diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fa58b6..b9f8b7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# (2026-06-28) Version 1.25.0 + +- (**Feature**) [♻️ Context management](./docs/configuration/text-generation.md#️-context-management) now works with every provider, not only [OpenAI](./docs/providers.md#openai). Token counting previously went through [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs), which is accurate only for OpenAI models and silently mis-counted everything else (worst of all for non-English text). OpenAI agents keep using tiktoken-rs; every other provider, including the recommended [Venice](./docs/providers.md#venice), now uses a provider-neutral approximation that needs no per-model tokenizer (ASCII counted at about four characters per token, other scripts such as Cyrillic and CJK at about two), landing within roughly 10-20% of the real count. See the [context management docs](./docs/configuration/text-generation.md#️-context-management). + +- (**Improvement**) Context management now trims a conversation on whole-turn boundaries for every provider, so an assistant reply is never kept without the user message it answered. This also adjusts how the OpenAI provider trims: a dangling assistant reply at the oldest edge of the kept history is now dropped along with its missing prompt, rather than left in place. + + # (2026-06-26) Version 1.24.0 - (**Feature**) Add an opt-in 💭 **thinking notice** for text generation. When enabled, a slow response (for example, from a reasoning model that runs for minutes) posts a "thinking…" placeholder after a short delay, refreshes it periodically with varying flavor text, and then edits that same message into the final answer, so a long wait no longer looks like a stuck bot. The notice is **disabled by default** and configurable per-room or globally via `text-generation set-thinking-notice-enabled true`. Fast responses (under the delay threshold) never show a placeholder. See the [text-generation configuration docs](./docs/configuration/text-generation.md#-thinking-notice). diff --git a/Cargo.lock b/Cargo.lock index 1a7bd6d..06c1b4b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -315,7 +315,7 @@ dependencies = [ [[package]] name = "baibot" -version = "1.24.0" +version = "1.25.0" dependencies = [ "anthropic", "anyhow", diff --git a/Cargo.toml b/Cargo.toml index 23bcd48..414e207 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,7 +7,7 @@ license = "AGPL-3.0-or-later" readme = "README.md" keywords = ["matrix", "chat", "bot", "AI", "LLM"] include = ["/etc/assets/baibot-torso-768.png", "/src", "/README.md", "/CHANGELOG.md", "/LICENSE"] -version = "1.24.0" +version = "1.25.0" edition = "2024" [lib] diff --git a/README.md b/README.md index f3f5374..880b91b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use - 🔒 Supports [encryption](./docs/features.md#-encryption) for Matrix communication and Account-Data-stored configuration -- ♻️ Supports [context-management](./docs/configuration/text-generation.md#️-context-management) handling on some models (automatically adjusting the message history length, etc.) +- ♻️ Supports [context-management](./docs/configuration/text-generation.md#️-context-management) for every [provider](./docs/providers.md) (automatically trimming older messages on whole-turn boundaries once a conversation outgrows the context window) - 🛠️ Allows **customizing much of the bot's [configuration](./docs/configuration/README.md)** at runtime (using commands sent via chat) diff --git a/docs/configuration/text-generation.md b/docs/configuration/text-generation.md index 0a2f0f3..3e2d937 100644 --- a/docs/configuration/text-generation.md +++ b/docs/configuration/text-generation.md @@ -50,9 +50,9 @@ Example: `!bai config room text-generation set-auto-usage only_for_voice` (this ### ♻️ Context Management -The bot also supports ♻️ **context management**, which automatically adjusts the message history length, etc. +The bot also supports ♻️ **context management**, which automatically trims the oldest messages once a conversation grows past the context window. It drops whole turns at a time, so a reply is never separated from the message it answered. -This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](../providers.md#openai) models. +Counting tokens precisely needs the model's own tokenizer. For [OpenAI](../providers.md#openai) models, the bot counts them with the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library. For every other provider, including the recommended [Venice](../providers.md#venice), the bot falls back to a provider-neutral **approximation** that needs no per-model tokenizer: it counts ASCII text at about four characters per token and other scripts (Cyrillic, CJK, and so on) at about two. Treat it as rough, within roughly 10-20% of the real count for typical text, which is plenty for keeping a long conversation inside the context window. This setting is **disabled by default**, but can be enabled via `!bai config room text-generation set-context-management-enabled true` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings)). diff --git a/docs/providers.md b/docs/providers.md index 48eaf52..5e33257 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -24,7 +24,7 @@ The list of supported providers is below. ### How to choose a provider -If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation) (incl. vision, incl. [🛠️ tools](./features.md#️-built-in-tools-openai-only)), [🖌️ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech). +If you're not sure which provider to start with, **we recommend [Venice](#venice)**: it's the most capable provider baibot supports (covering [💬 text-generation](./features.md#-text-generation) with vision, file inputs, prompt caching, and native web search, plus [🖌️ image-generation](./features.md#️-image-creation) incl. editing, [🦻 speech-to-text](./features.md#-speech-to-text), and [🗣️ text-to-speech](./features.md#️-text-to-speech)) and the only one that runs inference with no logging and no training on your data. If you'd rather start with the most widely-used option, [OpenAI](#openai) is a solid, well-supported choice too. You don't need to choose just one though. The bot supports [mixing & matching models](./features.md#-mixing--matching-models), so you can use multiple providers at the same time. @@ -176,7 +176,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo ### Venice -[Venice AI](https://venice.ai/chat?ref=kpXDe6) _(ref link with $10 bonus for you)_ runs inference on Venice-controlled GPUs or zero-data-retention partner infrastructure and stores no prompts or responses, so your conversations don't linger anywhere. It serves both frontier proprietary models and the latest open-source ones. +[Venice AI](https://venice.ai/chat?ref=kpXDe6) _(ref link with a $10 bonus for you)_ runs inference on Venice-controlled GPUs or zero-data-retention partner infrastructure and stores no prompts or responses, so your conversations don't linger anywhere. It serves both frontier proprietary models and the latest open-source ones. - 🆔 Identifier: `venice` - 🔗 Links: [🏠 Home page](https://venice.ai/chat?ref=kpXDe6), [👤 Sign up](https://venice.ai/chat?ref=kpXDe6), [📋 Models list](https://docs.venice.ai/models/overview) diff --git a/src/agent/provider/anthropic/controller.rs b/src/agent/provider/anthropic/controller.rs index 2d5f6c7..5bc8dcf 100644 --- a/src/agent/provider/anthropic/controller.rs +++ b/src/agent/provider/anthropic/controller.rs @@ -15,7 +15,7 @@ use crate::agent::provider::{ }; use crate::conversation::llm::{ Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage, - MessageContent as LLMMessageContent, shorten_messages_list_to_context_size, + MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size, }; use crate::strings; @@ -130,7 +130,7 @@ impl ControllerTrait for Controller { tracing::trace!("Shortening messages list to context size"); conversation_messages = shorten_messages_list_to_context_size( - &text_generation_config.model_id, + TokenEstimate::Approximate, &prompt_message, conversation_messages, Some(text_generation_config.max_response_tokens), diff --git a/src/agent/provider/openai/controller.rs b/src/agent/provider/openai/controller.rs index 2877780..6d601c4 100644 --- a/src/agent/provider/openai/controller.rs +++ b/src/agent/provider/openai/controller.rs @@ -24,7 +24,7 @@ use crate::{ }, conversation::llm::{ Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage, - MessageContent as LLMMessageContent, shorten_messages_list_to_context_size, + MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size, }, utils::base64::base64_decode, }; @@ -117,7 +117,7 @@ impl ControllerTrait for Controller { tracing::trace!("Shortening messages list to context size"); conversation_messages = shorten_messages_list_to_context_size( - &text_generation_config.model_id, + TokenEstimate::Tiktoken(&text_generation_config.model_id), &prompt_message, conversation_messages, text_generation_config.max_response_tokens, diff --git a/src/agent/provider/openai_compat/controller.rs b/src/agent/provider/openai_compat/controller.rs index 29b97df..ad2ca20 100644 --- a/src/agent/provider/openai_compat/controller.rs +++ b/src/agent/provider/openai_compat/controller.rs @@ -15,7 +15,7 @@ use crate::{ }, conversation::llm::{ Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage, - MessageContent as LLMMessageContent, shorten_messages_list_to_context_size, + MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size, }, }; use crate::{ @@ -114,7 +114,7 @@ impl ControllerTrait for Controller { tracing::trace!("Shortening messages list to context size"); conversation_messages = shorten_messages_list_to_context_size( - &text_generation_config.model_id, + TokenEstimate::Approximate, &prompt_message, conversation_messages, text_generation_config.max_response_tokens, diff --git a/src/agent/provider/venice/chat.rs b/src/agent/provider/venice/chat.rs index 4b67538..5230c1b 100644 --- a/src/agent/provider/venice/chat.rs +++ b/src/agent/provider/venice/chat.rs @@ -8,7 +8,7 @@ use crate::agent::AgentPurpose; use crate::agent::provider::entity::{TextGenerationParams, TextGenerationResult}; use crate::conversation::llm::{ Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage, - MessageContent as LLMMessageContent, shorten_messages_list_to_context_size, + MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size, }; use crate::strings; @@ -64,7 +64,7 @@ pub async fn generate_text( if params.context_management_enabled { conversation_messages = shorten_messages_list_to_context_size( - &text_generation_config.model_id, + TokenEstimate::Approximate, &prompt_message, conversation_messages, text_generation_config.max_response_tokens, diff --git a/src/conversation/llm/mod.rs b/src/conversation/llm/mod.rs index 8af5c53..b7f967f 100644 --- a/src/conversation/llm/mod.rs +++ b/src/conversation/llm/mod.rs @@ -6,5 +6,5 @@ mod utils; mod tests; pub use entity::*; -pub use tokenization::shorten_messages_list_to_context_size; +pub use tokenization::{TokenEstimate, shorten_messages_list_to_context_size}; pub use utils::*; diff --git a/src/conversation/llm/tokenization.rs b/src/conversation/llm/tokenization.rs index 9865765..b3ace6a 100644 --- a/src/conversation/llm/tokenization.rs +++ b/src/conversation/llm/tokenization.rs @@ -4,6 +4,22 @@ use tiktoken_rs::tokenizer; use super::{Author, Message, MessageContent}; +/// How to count the tokens in a conversation when trimming it to fit the context window. +pub enum TokenEstimate<'a> { + /// Count via the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library. + /// Accurate for OpenAI models; every other model falls back to the gpt-4 + /// tokenizer, which misreads it (badly so for non-English text). Use this only + /// for the OpenAI provider. + Tiktoken(&'a str), + + /// Provider-neutral approximation that needs no per-model tokenizer. Expect it + /// to land within roughly 10-20% of the real count for typical text, leaning + /// slightly high: over-counting trims a little extra history, while + /// under-counting would overflow the model's real context window. Use this for + /// every non-OpenAI provider. + Approximate, +} + fn get_bpe_for_model(model: &str) -> &'static CoreBPE { let tokenizer = tokenizer::get_tokenizer(model) .or_else(|| tokenizer::get_tokenizer("gpt-4")) @@ -13,21 +29,27 @@ fn get_bpe_for_model(model: &str) -> &'static CoreBPE { } pub fn shorten_messages_list_to_context_size( - model: &str, + estimate: TokenEstimate<'_>, prompt_message: &Option, mut messages: Vec, max_response_tokens: Option, max_context_tokens: u32, ) -> Vec { - // Loading the tokenization data is an expensive process, so - // se construct the BPE instance once and then use it for all messages. - let bpe = get_bpe_for_model(model); + // Loading the tiktoken data is expensive, so we resolve the counter once up + // front and reuse it for every message. + let tiktoken = match estimate { + TokenEstimate::Tiktoken(model) => Some((get_bpe_for_model(model), model)), + TokenEstimate::Approximate => None, + }; + let count = |message: &Message| match tiktoken { + Some((bpe, model)) => tiktoken_token_size_for_message(bpe, model, message), + None => approximate_token_size_for_message(message), + }; // We want to retain the prompt in all cases, so we always count it first. // We also always reserve enough tokens for the maximum response we expect. let mut current_context_length: u32 = if let Some(prompt_message) = prompt_message { - calculate_token_size_for_message(bpe, model, prompt_message) - + max_response_tokens.unwrap_or(0) + count(prompt_message) + max_response_tokens.unwrap_or(0) } else { 0 }; @@ -37,7 +59,7 @@ pub fn shorten_messages_list_to_context_size( let mut messages_to_keep: Vec = Vec::new(); for message in messages { - let tokens_for_message = calculate_token_size_for_message(bpe, model, &message); + let tokens_for_message = count(&message); if current_context_length + tokens_for_message > max_context_tokens { break; @@ -48,14 +70,26 @@ pub fn shorten_messages_list_to_context_size( messages_to_keep.push(message); } + // Cut on a turn boundary: the loop may stop right after an assistant reply + // whose triggering user message did not fit, which would leave the kept window + // starting on an orphaned reply. `messages_to_keep` is newest-first here, so + // the oldest kept messages are at the end; drop any trailing assistant messages + // until the window begins at the start of a turn (a user message). + while matches!( + messages_to_keep.last().map(|message| &message.author), + Some(Author::Assistant) + ) { + messages_to_keep.pop(); + } + messages_to_keep.reverse(); messages_to_keep } -/// Calculate the token size of a message for a given model, with a preloaded CoreBPE object. -/// Related to `calculate_token_size_for_model_message`. -fn calculate_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Message) -> u32 { +/// Token size of a message via tiktoken, for a preloaded CoreBPE object. +/// Accurate only for OpenAI models (see [`TokenEstimate::Tiktoken`]). +fn tiktoken_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Message) -> u32 { let (tokens_per_message, tokens_per_name) = if model.starts_with("gpt-3.5") { ( 4, // every message follows {role/name}\n{content}\n @@ -80,6 +114,52 @@ fn calculate_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Messag (text_length + role_length + tokens_per_message + tokens_per_name) as u32 } +/// ASCII text averages about four characters per token. +const ASCII_TOKENS_PER_CHAR: f32 = 0.25; + +/// Non-ASCII scripts (Cyrillic, CJK, and others) pack more information per +/// character: real tokenizers land around two characters per token for them, so +/// each counts as half a token. CJK runs a touch denser than that, so its estimate +/// can read slightly low, still within the tolerance this approximation targets. +const WIDE_TOKENS_PER_CHAR: f32 = 0.5; + +/// Structural per-message overhead (role marker plus message framing), mirroring +/// the small constant the tiktoken path adds. +const APPROX_TOKENS_PER_MESSAGE: u32 = 4; + +/// Provider-neutral, tokenizer-free token size of a message +/// (see [`TokenEstimate::Approximate`]). +fn approximate_token_size_for_message(message: &Message) -> u32 { + let text_tokens = match &message.content { + MessageContent::Text(text) => approximate_token_size_for_text(text), + // Images and files are not counted as text, matching the tiktoken path. + MessageContent::Image(..) | MessageContent::File(..) => 0, + }; + + text_tokens + APPROX_TOKENS_PER_MESSAGE +} + +/// Rough token estimate for a piece of text, with no tokenizer. +/// +/// ASCII characters count as a quarter-token each (~4 chars/token); characters +/// outside ASCII count as half a token each (~2 chars/token), matching how real +/// tokenizers treat Cyrillic and CJK. Weighting non-ASCII up keeps the estimate +/// from badly under-counting non-English text, the case the tiktoken fallback gets +/// most wrong. +fn approximate_token_size_for_text(text: &str) -> u32 { + let mut estimate = 0.0_f32; + + for character in text.chars() { + estimate += if character.is_ascii() { + ASCII_TOKENS_PER_CHAR + } else { + WIDE_TOKENS_PER_CHAR + }; + } + + estimate.ceil() as u32 +} + pub mod test { #[test] fn message_size_counting_works() { @@ -94,7 +174,7 @@ pub mod test { timestamp: chrono::Utc::now(), }; - let tokens = super::calculate_token_size_for_message(bpe, model, &message); + let tokens = super::tiktoken_token_size_for_message(bpe, model, &message); assert_eq!(8, tokens); } @@ -118,7 +198,7 @@ pub mod test { assert_eq!( prompt_length, - super::calculate_token_size_for_message(bpe, model, &prompt) + super::tiktoken_token_size_for_message(bpe, model, &prompt) ); let mut conversation_messages = Vec::new(); @@ -133,7 +213,7 @@ pub mod test { assert_eq!( first_length, - super::calculate_token_size_for_message(bpe, model, &first) + super::tiktoken_token_size_for_message(bpe, model, &first) ); conversation_messages.push(first); @@ -148,7 +228,7 @@ pub mod test { assert_eq!( second_length, - super::calculate_token_size_for_message(bpe, model, &second) + super::tiktoken_token_size_for_message(bpe, model, &second) ); conversation_messages.push(second); @@ -165,7 +245,7 @@ pub mod test { assert_eq!( third_length, - super::calculate_token_size_for_message(bpe, model, &third) + super::tiktoken_token_size_for_message(bpe, model, &third) ); conversation_messages.push(third.clone()); @@ -182,7 +262,7 @@ pub mod test { assert_eq!( forth_length, - super::calculate_token_size_for_message(bpe, model, &forth) + super::tiktoken_token_size_for_message(bpe, model, &forth) ); conversation_messages.push(forth.clone()); @@ -190,7 +270,7 @@ pub mod test { assert_eq!(4, conversation_messages.len()); let new_conversation_messages = super::shorten_messages_list_to_context_size( - model, + super::TokenEstimate::Tiktoken(model), &Some(prompt), conversation_messages, max_response_tokens, @@ -229,7 +309,7 @@ pub mod test { assert_eq!( prompt_length, - super::calculate_token_size_for_message(bpe, model, &prompt) + super::tiktoken_token_size_for_message(bpe, model, &prompt) ); let mut conversation_messages = Vec::new(); @@ -244,7 +324,7 @@ pub mod test { assert_eq!( first_length, - super::calculate_token_size_for_message(bpe, model, &first) + super::tiktoken_token_size_for_message(bpe, model, &first) ); conversation_messages.push(first); @@ -259,7 +339,7 @@ pub mod test { assert_eq!( second_length, - super::calculate_token_size_for_message(bpe, model, &second) + super::tiktoken_token_size_for_message(bpe, model, &second) ); conversation_messages.push(second); @@ -276,7 +356,7 @@ pub mod test { assert_eq!( third_length, - super::calculate_token_size_for_message(bpe, model, &third) + super::tiktoken_token_size_for_message(bpe, model, &third) ); conversation_messages.push(third.clone()); @@ -293,7 +373,7 @@ pub mod test { assert_eq!( forth_length, - super::calculate_token_size_for_message(bpe, model, &forth) + super::tiktoken_token_size_for_message(bpe, model, &forth) ); conversation_messages.push(forth.clone()); @@ -301,7 +381,7 @@ pub mod test { assert_eq!(4, conversation_messages.len()); let new_conversation_messages = super::shorten_messages_list_to_context_size( - model, + super::TokenEstimate::Tiktoken(model), &Some(prompt), conversation_messages, max_response_tokens, @@ -320,4 +400,126 @@ pub mod test { forth.content ); } + + #[test] + fn approximate_counting_weights_ascii_and_wide_scripts() { + // 12 ASCII characters at ~4 chars/token = 3 text tokens. + assert_eq!(3, super::approximate_token_size_for_text("Hello there!")); + + // 5 CJK characters at ~0.5 token/char = 3 text tokens. The ASCII rate would + // have under-counted these to 2, the failure mode this path avoids. + assert_eq!(3, super::approximate_token_size_for_text("こんにちは")); + + let message = super::Message { + author: super::Author::User, + sender_id: None, + content: super::MessageContent::Text("Hello there!".to_string()), + timestamp: chrono::Utc::now(), + }; + + // 3 text tokens plus the per-message overhead (4). + assert_eq!(7, super::approximate_token_size_for_message(&message)); + } + + #[test] + fn approximate_shortening_trims_to_budget() { + let prompt = super::Message { + author: super::Author::Prompt, + sender_id: None, + content: super::MessageContent::Text("You are a bot!".to_string()), + timestamp: chrono::Utc::now(), + }; + + let older = super::Message { + author: super::Author::User, + sender_id: None, + content: super::MessageContent::Text("This is the older message.".to_string()), + timestamp: chrono::Utc::now(), + }; + let newer = super::Message { + // A user message, so it is a valid window start: keeping a lone + // assistant reply would be an orphan and get trimmed (see + // `shortening_cuts_on_a_turn_boundary`). + author: super::Author::User, + sender_id: None, + content: super::MessageContent::Text("This is the newer message.".to_string()), + timestamp: chrono::Utc::now(), + }; + + // Budget room for the prompt and only the newest message. + let max_context_tokens = super::approximate_token_size_for_message(&prompt) + + super::approximate_token_size_for_message(&newer); + + let new_conversation_messages = super::shorten_messages_list_to_context_size( + super::TokenEstimate::Approximate, + &Some(prompt), + vec![older, newer.clone()], + None, + max_context_tokens, + ); + + assert_eq!(1, new_conversation_messages.len()); + assert_eq!( + new_conversation_messages.first().unwrap().content, + newer.content + ); + } + + #[test] + fn shortening_cuts_on_a_turn_boundary() { + // A four-message conversation of two full turns. All four messages are the + // same length, so they cost the same number of tokens. + let prompt = super::Message { + author: super::Author::Prompt, + sender_id: None, + content: super::MessageContent::Text("system".to_string()), + timestamp: chrono::Utc::now(), + }; + + let user_one = super::Message { + author: super::Author::User, + sender_id: None, + content: super::MessageContent::Text("user msg 1".to_string()), + timestamp: chrono::Utc::now(), + }; + let asst_one = super::Message { + author: super::Author::Assistant, + sender_id: None, + content: super::MessageContent::Text("asst msg 1".to_string()), + timestamp: chrono::Utc::now(), + }; + let user_two = super::Message { + author: super::Author::User, + sender_id: None, + content: super::MessageContent::Text("user msg 2".to_string()), + timestamp: chrono::Utc::now(), + }; + let asst_two = super::Message { + author: super::Author::Assistant, + sender_id: None, + content: super::MessageContent::Text("asst msg 2".to_string()), + timestamp: chrono::Utc::now(), + }; + + let per_message = super::approximate_token_size_for_message(&user_one); + // Budget fits the prompt plus three messages. By raw token budget the loop + // would keep asst_two, user_two, and asst_one, but asst_one's own user + // message (user_one) does not fit, so it must be dropped too rather than + // left as an orphaned reply. + let max_context_tokens = + super::approximate_token_size_for_message(&prompt) + (per_message * 3); + + let kept = super::shorten_messages_list_to_context_size( + super::TokenEstimate::Approximate, + &Some(prompt), + vec![user_one, asst_one, user_two.clone(), asst_two.clone()], + None, + max_context_tokens, + ); + + // Only the last whole turn survives; the orphaned asst_one is dropped. + assert_eq!(2, kept.len()); + assert_eq!(kept.first().unwrap().content, user_two.content); + assert_eq!(kept.last().unwrap().content, asst_two.content); + } } diff --git a/src/strings/help/cfg.rs b/src/strings/help/cfg.rs index 4d1cd0a..1218556 100644 --- a/src/strings/help/cfg.rs +++ b/src/strings/help/cfg.rs @@ -128,7 +128,7 @@ pub fn text_generation_context_management_intro() -> String { format!( "{}\n{}", "Controls the bot's ability to **intelligently drop old messages from the conversation context** when it gets too large.", - "This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](./providers.md#openai) models.", + "Counting tokens precisely needs the model's own tokenizer. For [OpenAI](./providers.md#openai) models the bot uses the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library; for every other provider (including the recommended [Venice](./providers.md#venice)) it falls back to a provider-neutral **approximation** (ASCII counted at ~4 characters per token, other scripts at ~2), within roughly 10-20% of the real count for typical text.", ) }