Compare commits

...

83 Commits

Author SHA1 Message Date
renovate[bot]
c65ae48a53 Update docker.io/rust Docker tag to v1.98.1 2026-09-09 17:09:27 +00:00
renovate[bot]
072fe3ec9f Update Rust crate reqwest to v0.13.5 2026-09-08 21:07:45 +00:00
renovate[bot]
0d7ae70d1d Update actions-rust-lang/setup-rust-toolchain action to v2 2026-09-08 07:07:30 +03:00
daru
f912162c4d Include Docker Compose example in installation guide (#262)
Adds a Docker Compose example alongside the existing `docker run` one, using a tagged release and the same hardening options.

Co-authored-by: daru <darupeter@pm.me>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nh4Z2JCU4aVC5STFN7usfW
2026-09-07 09:08:52 +03:00
renovate[bot]
264f71d885 Update Rust to v1.98.1 2026-09-04 01:55:41 +00:00
renovate[bot]
2861ea6141 Update docker.io/ollama/ollama Docker tag to v0.33.3 2026-09-03 19:57:33 +00:00
renovate[bot]
2014e25f77 Update ghcr.io/element-hq/synapse Docker tag to v1.160.0 2026-09-03 03:00:45 +00:00
renovate[bot]
b5c5fdb357 Update dependency prek to v0.5.2 2026-09-02 21:32:26 +00:00
renovate[bot]
5ee8d18439 Update ghcr.io/element-hq/element-web Docker tag to v1.12.27 2026-09-01 18:48:07 +00:00
renovate[bot]
194a47bfc6 Update dependency prek to v0.5.1 2026-09-01 10:01:11 +00:00
renovate[bot]
4f3a753f52 Update docker.io/ollama/ollama Docker tag to v0.33.2 2026-08-28 22:07:30 +00:00
renovate[bot]
1a7b194588 Update dependency prek to v0.5.0 2026-08-27 08:33:23 +00:00
renovate[bot]
616607433f Update docker.io/ollama/ollama Docker tag to v0.33.1 2026-08-27 02:54:32 +00:00
renovate[bot]
f1043e03de Update docker.io/ollama/ollama Docker tag to v0.33.0 2026-08-26 01:57:04 +00:00
Slavi Pantaleev
6f3132edfd justfile: git hooks resolve prek through the mise shim
prek bakes the full path of the currently installed version into the hooks it
generates (var/mise/installs/prek/<version>/...). That path stops existing as
soon as the pinned version changes or old versions are pruned, and the hook's
PATH fallback finds no prek either, so every commit fails until the hook is
regenerated by hand. It also means a hook keeps running the version it was
generated with, long after mise.toml has moved on.

Rewriting PREK to mise's shim makes the hook resolve whatever mise.toml pins at
the time it runs. The accompanying MISE_DATA_DIR / MISE_TRUSTED_CONFIG_PATHS
exports keep that resolution inside this project - without them mise falls back
to the global data directory and silently installs a second copy of the tool.

The recipe becomes a shebang recipe, because just runs each line of a plain
recipe in its own shell and the patch loop needs to span several lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 16:37:53 +03:00
renovate[bot]
4f86a321e2 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26.8.1 (#261)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-25 12:44:12 +00:00
Slavi Pantaleev
9d876a58a8 Drop Dockerfile.ci, which nothing has built since 2025
It was added in 941e5f0 (2024-09-21) as a cache-less variant, because
the BuildKit cache mounts in the main Dockerfile were causing trouble in
CI, and the workflow pointed at it with `file: Dockerfile.ci`. That line
was removed in 6719538 (2025-02-27, "Switch to using native ARM64
builders"), which returned the build to the default Dockerfile - but the
file itself stayed behind.

Nothing has referenced it since. It had meanwhile drifted from the real
Dockerfile (no RELEASE_BUILD argument, no cache mounts) while Renovate
kept bumping its base image pin, so every one of those bumps was a pull
request for a file that is never built.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 09:26:29 +03:00
Slavi Pantaleev
dfb2288fa3 Renovate: merge routine updates by pushing, without pull requests
ci.yml runs on `push: ["**"]`, so a Renovate branch is already compiled,
linted with `clippy -D warnings` and unit-tested before anything reaches
main; a red branch makes Renovate open a pull request instead of merging.
That gate now covers the Dockerfile base too, via the build job added in
the previous commit.

Cargo, the Rust toolchain, prek and the workflows' own actions therefore
merge by branch push. The local development images under etc/services/**
do too, on a different justification recorded in the rule itself: CI does
not run them and they reach no shipped artifact, so the worst case is a
broken `just services-start`.

Majors keep their pull request - the rule is deliberately last, so it
overrides the others for every manager.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 09:25:10 +03:00
Slavi Pantaleev
24d99627ae CI: build the container image when a Dockerfile changes
The prek job never builds an image, so a bump of the Dockerfile's base
image reached main unvalidated and only failed afterwards in Publish -
by which point ghcr.io/etkecc/baibot:latest had already been attempted.

Add a gate job that looks for Dockerfile changes against main, and a
build job that builds the image the way Publish does but with
`push: false`. The build is gated rather than unconditional because it
is a full Rust release build: running it on every push would turn a
~1 minute pipeline into a ~10 minute one for changes that cannot affect
the image. It is skipped on main, where Publish already builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 09:25:10 +03:00
renovate[bot]
f7c88fe532 Update docker.io/rust Docker tag to v1.98.0 (#260)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-21 01:58:56 +03:00
Slavi Pantaleev
f25d50c7f2 Update Rust to v1.98.0
Fixes a new clippy lint (needless_late_init) by turning the
text_to_speech_stage_params if/else into an expression.

Supersedes https://github.com/etkecc/baibot/pull/259

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 01:58:26 +03:00
renovate[bot]
f22ac1636f Update docker.io/ollama/ollama Docker tag to v0.32.15 2026-08-20 08:29:41 +03:00
renovate[bot]
43904e5032 Update ghcr.io/element-hq/synapse Docker tag to v1.159.0 2026-08-19 08:14:15 +03:00
renovate[bot]
4e29d2200f Update ghcr.io/element-hq/element-web Docker tag to v1.12.26 2026-08-18 19:37:12 +03:00
renovate[bot]
30fe2a8a8c Update dependency prek to v0.4.14 2026-08-17 10:07:29 +03:00
renovate[bot]
e59e88f12e Update docker.io/ollama/ollama Docker tag to v0.32.14 2026-08-17 00:41:52 +03:00
Slavi Pantaleev
f27d8cfdcf mise: drop the settings block - yes is ignored in project configs
Since mise 2026.6.4 (advisory GHSA-436v-8fw5-4mj8), trust-control settings
(`yes`, `ci`, `trusted_config_paths`, `paranoid`) in non-global configs are
ignored, and every mise invocation prints a warning about this one.
Removing it changes nothing on current mise - the setting was already dead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 09:14:50 +03:00
renovate[bot]
909c5215d0 Update docker.io/ollama/ollama Docker tag to v0.32.13 2026-08-15 05:47:45 +03:00
renovate[bot]
ca1147c10a Update docker.io/postgres Docker tag to v18.6 2026-08-14 07:08:00 +03:00
renovate[bot]
4a4e6c1090 Update docker.io/ollama/ollama Docker tag to v0.32.11 2026-08-14 07:07:35 +03:00
renovate[bot]
ea3af62afc Update docker.io/ollama/ollama Docker tag to v0.32.10 2026-08-14 06:59:04 +03:00
renovate[bot]
ee7ffc8970 Update docker.io/ollama/ollama Docker tag to v0.32.9 2026-08-12 07:55:38 +03:00
renovate[bot]
45f5b7a1b0 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26.7.3 2026-08-12 07:55:29 +03:00
renovate[bot]
6d0396270e Update dependency prek to v0.4.13 2026-08-10 16:02:29 +03:00
renovate[bot]
d52a5ea0ac Update docker.io/ollama/ollama Docker tag to v0.32.6 2026-08-06 07:03:58 +03:00
renovate[bot]
30985baf0c Update ghcr.io/element-hq/element-web Docker tag to v1.12.25 2026-08-05 14:02:42 +03:00
renovate[bot]
077a86e5c9 Update ghcr.io/element-hq/synapse Docker tag to v1.158.0 2026-08-05 05:08:48 +03:00
renovate[bot]
9b3d5400ff Update Rust crate base64 to v0.23.1 2026-08-04 21:54:00 +03:00
renovate[bot]
721abe8c75 Update dependency prek to v0.4.12 2026-08-03 17:10:23 +03:00
renovate[bot]
dae5fec4c3 Update Rust crate async-openai to v0.41.3 2026-08-01 00:58:42 +03:00
renovate[bot]
8b3644f3db Update Rust crate async-openai to v0.41.2 2026-07-31 06:30:40 +03:00
renovate[bot]
570a4be83c Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26.7.2 2026-07-31 06:30:09 +03:00
renovate[bot]
24266a4886 Update ghcr.io/element-hq/synapse Docker tag to v1.157.2 2026-07-28 23:15:01 +03:00
renovate[bot]
be94441ddb Update docker.io/ollama/ollama Docker tag to v0.32.5 2026-07-28 07:17:04 +03:00
renovate[bot]
a4280eb8c0 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26.7.1 2026-07-28 07:16:08 +03:00
renovate[bot]
9ad9a04040 Update docker.io/ollama/ollama Docker tag to v0.32.4 2026-07-26 00:56:04 +03:00
renovate[bot]
57ca095df0 Update dependency prek to v0.4.11 2026-07-25 07:30:18 +03:00
renovate[bot]
4b5cc24513 Update Rust crate base64 to 0.23.* 2026-07-23 20:59:41 +03:00
renovate[bot]
53daa06151 Update docker.io/ollama/ollama Docker tag to v0.32.3 2026-07-23 20:59:25 +03:00
renovate[bot]
77103e2523 Update ghcr.io/element-hq/synapse Docker tag to v1.157.1 2026-07-22 21:11:02 +03:00
renovate[bot]
0a755d287e Update docker.io/ollama/ollama Docker tag to v0.32.2 2026-07-22 09:57:30 +03:00
renovate[bot]
fe3296594b Update ghcr.io/element-hq/element-web Docker tag to v1.12.24 2026-07-22 06:46:43 +03:00
renovate[bot]
1e7f557334 Update ghcr.io/element-hq/synapse Docker tag to v1.157.0 2026-07-22 06:46:26 +03:00
renovate[bot]
be09ccf5da Update Rust crate tokio to v1.53.1 2026-07-21 07:18:42 +03:00
renovate[bot]
fa3646774a Update Rust crate serde_json to v1.0.151 2026-07-20 12:12:56 +03:00
renovate[bot]
1774bf29eb Update Rust crate serde to v1.0.229 2026-07-19 06:13:19 +03:00
renovate[bot]
2da4e6793a Update Rust crate anyhow to v1.0.104 2026-07-19 06:13:04 +03:00
renovate[bot]
166c1f7659 Update Rust crate tokio to 1.53.* 2026-07-17 15:32:13 +03:00
renovate[bot]
0f0868b795 Update docker.io/rust Docker tag to v1.97.1 2026-07-17 05:31:35 +03:00
renovate[bot]
65d24e4681 Update Rust to v1.97.1 2026-07-16 17:55:44 +03:00
renovate[bot]
ec83b38a1a Update Rust crate tokio to v1.52.4 2026-07-16 17:55:33 +03:00
renovate[bot]
ea13dd42d7 Update docker.io/ollama/ollama Docker tag to v0.32.1 2026-07-16 14:28:35 +03:00
renovate[bot]
b405859db7 Update dependency prek to v0.4.10 2026-07-16 14:28:19 +03:00
renovate[bot]
23a09cdf60 Update Rust crate regex to v1.13.1 2026-07-16 08:12:47 +03:00
renovate[bot]
dd13deff33 Update docker.io/ollama/ollama Docker tag to v0.32.0 2026-07-14 07:06:35 +03:00
renovate[bot]
0c109d4f92 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26.6.2 2026-07-13 07:17:51 +03:00
Slavi Pantaleev
e346951395 Switch continuwuity env vars to CONTINUWUITY_ prefix
The CONDUWUIT_ prefix still works in v26.6.1, but is a legacy
compatibility name from before the conduwuit -> continuwuity rebrand.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:59:26 +03:00
renovate[bot]
eee3c05e86 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v26 (#213)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-12 13:58:12 +03:00
renovate[bot]
4bf44665a5 Update dependency prek to v0.4.9 2026-07-11 21:34:08 +03:00
renovate[bot]
eae0851328 Update docker.io/rust Docker tag to v1.97.0 2026-07-10 07:17:52 +03:00
Slavi Pantaleev
5212d6643e Update Rust to v1.97.0
Fixes new clippy lint (redundant reference in format! argument) and
reflows the affected format! calls per rustfmt.

Supersedes https://github.com/etkecc/baibot/pull/211

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 20:49:43 +03:00
Slavi Pantaleev
b4a070f536 Update regex to 1.13.* and mxidwc to 1.0.3
Supersedes https://github.com/etkecc/baibot/pull/210

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 20:46:08 +03:00
renovate[bot]
de2b24f91b Update docker.io/ollama/ollama Docker tag to v0.31.2 2026-07-08 21:28:24 +03:00
renovate[bot]
fbe956c0b7 Update ghcr.io/element-hq/element-web Docker tag to v1.12.23 2026-07-07 22:00:31 +03:00
renovate[bot]
a64b7da7ae Update ghcr.io/element-hq/synapse Docker tag to v1.156.0 2026-07-07 22:00:22 +03:00
renovate[bot]
ac5e776345 Update dependency prek to v0.4.8 2026-07-04 17:17:33 +03:00
renovate[bot]
bea27df2e0 Update dependency prek to v0.4.6 (#203) 2026-07-01 08:05:03 +03:00
renovate[bot]
b7c1b6f48f Update docker.io/rust Docker tag to v1.96.1 2026-07-01 07:18:07 +03:00
renovate[bot]
8a5a38f241 Update Rust to v1.96.1 2026-07-01 07:05:55 +03:00
renovate[bot]
5306d3bb8c Update docker.io/ollama/ollama Docker tag to v0.31.1 2026-07-01 07:05:40 +03:00
renovate[bot]
75631982e7 Update docker.io/ollama/ollama Docker tag to v0.31.0 2026-06-30 08:57:43 +03:00
Slavi Pantaleev
6058bf733b Adjust changelog entry date 2026-06-29 17:10:30 +03:00
Aine
a445933c0b Provider-neutral context management 2026-06-29 17:10:04 +03:00
29 changed files with 531 additions and 181 deletions

View File

@@ -22,7 +22,7 @@ jobs:
# Toolchain version + components come from rust-toolchain.toml. rustflags is
# cleared so plain builds don't fail on warnings; the clippy hook still does.
- uses: actions-rust-lang/setup-rust-toolchain@v1
- uses: actions-rust-lang/setup-rust-toolchain@v2
with:
rustflags: ''
@@ -42,3 +42,63 @@ jobs:
- name: Unit tests
run: just test
# The prek job never builds a container image, so a bump of the Dockerfile's
# base image reaches main unvalidated and fails later, in Publish, after
# ghcr.io/etkecc/baibot:latest has already been attempted. These two jobs close
# that gap: decide whether a Dockerfile changed, and if so build the image the
# way Publish does - but without pushing anything.
#
# The build is gated rather than unconditional because it is a full Rust
# release build; running it on every push would turn a ~1 minute pipeline into
# a ~10 minute one for changes that cannot affect the image.
docker-gate:
name: Decide whether the image needs building
runs-on: ubuntu-latest
outputs:
build: ${{ steps.decide.outputs.build }}
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Look for Dockerfile changes against main
id: decide
run: |
if [ "${{ github.event_name }}" = 'workflow_dispatch' ]; then
echo 'Forced via workflow_dispatch.'
echo 'build=true' >> "$GITHUB_OUTPUT"
exit 0
fi
# Publish builds and pushes from main, so a main-side build here would
# be redundant. This gate exists for branches, before they merge.
if [ "${{ github.ref_name }}" = 'main' ]; then
echo 'On main; Publish covers this.'
echo 'build=false' >> "$GITHUB_OUTPUT"
exit 0
fi
git fetch --no-tags origin main
if git diff --name-only origin/main HEAD -- Dockerfile | grep -q .; then
echo 'A Dockerfile changed; the image will be built.'
echo 'build=true' >> "$GITHUB_OUTPUT"
else
echo 'No Dockerfile changed.'
echo 'build=false' >> "$GITHUB_OUTPUT"
fi
docker-build:
name: Build the container image (without publishing it)
needs: docker-gate
if: needs.docker-gate.outputs.build == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
# No build cache on purpose: a bump of the base image is exactly the case
# where a cold build is the honest test.
- name: Build
uses: docker/build-push-action@v7
with:
push: false

View File

@@ -5,6 +5,10 @@ repos:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
# This is the stock Synapse sample homeserver.yaml (mostly commented-out
# docs). prek's stricter YAML parser (serde-saphyr, since v0.4.6) rejects
# its long runs of consecutive comment lines in several places.
exclude: '^etc/services/synapse/config/homeserver\.yaml$'
- id: check-merge-conflict
- id: check-added-large-files
args: ['--maxkb=1024']

View File

@@ -1,3 +1,10 @@
# (2026-06-29) Version 1.25.0
- (**Feature**) [♻️ Context management](./docs/configuration/text-generation.md#️-context-management) now works with every provider, not only [OpenAI](./docs/providers.md#openai). Token counting previously went through [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs), which is accurate only for OpenAI models and silently mis-counted everything else (worst of all for non-English text). OpenAI agents keep using tiktoken-rs; every other provider, including the recommended [Venice](./docs/providers.md#venice), now uses a provider-neutral approximation that needs no per-model tokenizer (ASCII counted at about four characters per token, other scripts such as Cyrillic and CJK at about two), landing within roughly 10-20% of the real count. See the [context management docs](./docs/configuration/text-generation.md#️-context-management).
- (**Improvement**) Context management now trims a conversation on whole-turn boundaries for every provider, so an assistant reply is never kept without the user message it answered. This also adjusts how the OpenAI provider trims: a dangling assistant reply at the oldest edge of the kept history is now dropped along with its missing prompt, rather than left in place.
# (2026-06-26) Version 1.24.0
- (**Feature**) Add an opt-in 💭 **thinking notice** for text generation. When enabled, a slow response (for example, from a reasoning model that runs for minutes) posts a "thinking…" placeholder after a short delay, refreshes it periodically with varying flavor text, and then edits that same message into the final answer, so a long wait no longer looks like a stuck bot. The notice is **disabled by default** and configurable per-room or globally via `text-generation set-thinking-notice-enabled true`. Fast responses (under the delay threshold) never show a placeholder. See the [text-generation configuration docs](./docs/configuration/text-generation.md#-thinking-notice).

77
Cargo.lock generated
View File

@@ -95,9 +95,9 @@ dependencies = [
[[package]]
name = "anyhow"
version = "1.0.103"
version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]]
name = "anymap2"
@@ -184,9 +184,9 @@ checksum = "4288f83726785267c6f2ef073a3d83dc3f9b81464e9f99898240cced85fce35a"
[[package]]
name = "async-openai"
version = "0.41.1"
version = "0.41.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3007014661d5b98168b7b6f1014147bce8b1362a194783543eeb9f6117a20be9"
checksum = "d72db2750faea2ca5edbf6d0c50277a89dc8f75f5e6ddd695ef30f75e335019b"
dependencies = [
"async-openai-macros",
"base64 0.22.1",
@@ -196,7 +196,7 @@ dependencies = [
"futures",
"getrandom 0.3.4",
"rand 0.9.4",
"reqwest 0.13.4",
"reqwest 0.13.5",
"secrecy",
"serde",
"serde_json",
@@ -315,12 +315,12 @@ dependencies = [
[[package]]
name = "baibot"
version = "1.24.0"
version = "1.25.0"
dependencies = [
"anthropic",
"anyhow",
"async-openai",
"base64 0.22.1",
"base64 0.23.1",
"chrono",
"etke_openai_api_rust",
"matrix-sdk",
@@ -329,7 +329,7 @@ dependencies = [
"mxlink",
"quick_cache 0.7.0",
"regex",
"reqwest 0.13.4",
"reqwest 0.13.5",
"serde",
"serde_json",
"serde_yaml_ng",
@@ -353,6 +353,12 @@ version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]]
name = "base64"
version = "0.23.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]]
name = "base64ct"
version = "1.8.3"
@@ -2188,7 +2194,7 @@ dependencies = [
"oauth2-reqwest",
"percent-encoding",
"pin-project-lite",
"reqwest 0.13.4",
"reqwest 0.13.5",
"ruma",
"rustls",
"rustls-native-certs 0.8.3",
@@ -2480,9 +2486,9 @@ dependencies = [
[[package]]
name = "mxidwc"
version = "1.0.2"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e253f96a03d24d1c0006c5661b1e20a13dc3861f509d76c33b6b44349d4ff3b"
checksum = "45b5d51fcf414d2aa6bffc6cd9b037e62732734a944c5da4ace6b9895ec37b93"
dependencies = [
"regex",
]
@@ -2583,7 +2589,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "234fb5c965bbce983ee5de636a7a51d6a3223da8067ea02f9ab2d2d78ac08be2"
dependencies = [
"oauth2",
"reqwest 0.13.4",
"reqwest 0.13.5",
]
[[package]]
@@ -3064,9 +3070,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.12.4"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba"
checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
dependencies = [
"aho-corasick",
"memchr",
@@ -3076,9 +3082,9 @@ dependencies = [
[[package]]
name = "regex-automata"
version = "0.4.14"
version = "0.4.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad"
dependencies = [
"aho-corasick",
"memchr",
@@ -3134,11 +3140,11 @@ dependencies = [
[[package]]
name = "reqwest"
version = "0.13.4"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3"
checksum = "16a1cfa75cc186dd73d5818e510e042e40927bccc9c236b061cea97e1eb08029"
dependencies = [
"base64 0.22.1",
"base64 0.23.1",
"bytes",
"futures-core",
"futures-util",
@@ -3611,9 +3617,9 @@ checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
[[package]]
name = "serde"
version = "1.0.228"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
@@ -3642,22 +3648,22 @@ dependencies = [
[[package]]
name = "serde_core"
version = "1.0.228"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
"syn 3.0.0",
]
[[package]]
@@ -3675,9 +3681,9 @@ dependencies = [
[[package]]
name = "serde_json"
version = "1.0.150"
version = "1.0.151"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9"
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
dependencies = [
"itoa",
"memchr",
@@ -3899,6 +3905,17 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2fac314a64dc9a36e61a9eb4261a5e9bbfbc922b27e518af97bc32b926cf967"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "sync_wrapper"
version = "1.0.2"
@@ -4064,9 +4081,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]]
name = "tokio"
version = "1.52.3"
version = "1.53.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"
checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed"
dependencies = [
"bytes",
"libc",

View File

@@ -7,7 +7,7 @@ license = "AGPL-3.0-or-later"
readme = "README.md"
keywords = ["matrix", "chat", "bot", "AI", "LLM"]
include = ["/etc/assets/baibot-torso-768.png", "/src", "/README.md", "/CHANGELOG.md", "/LICENSE"]
version = "1.24.0"
version = "1.25.0"
edition = "2024"
[lib]
@@ -18,7 +18,7 @@ path = "src/lib.rs"
anthropic = { git = "https://github.com/etkecc/anthropic-rs.git", branch = "fix-content-block-image" }
anyhow = "1.0.*"
async-openai = { version = "0.41.0", features = ["audio", "chat-completion", "image", "responses"] }
base64 = "0.22.*"
base64 = "0.23.*"
chrono = { version = "0.4.*", default-features = false, features = ["std", "now"] }
# We'd rather not depend on this, but we cannot use the ruma-events EventContent macro without it.
matrix-sdk = { version = "0.18.0", default-features = false }
@@ -27,7 +27,7 @@ mxidwc = "1.0.*"
mxlink = ">=1.15.0"
etke_openai_api_rust = "0.1.*"
quick_cache = "0.7.*"
regex = "1.12.*"
regex = "1.13.*"
# HTTP client for the native `venice` provider. rustls only (no extra TLS stack), matching the
# reqwest copy async-openai/matrix-sdk/mxlink already use.
reqwest = { version = "0.13.*", default-features = false, features = ["json", "multipart", "rustls"] }
@@ -36,7 +36,7 @@ serde_json = "1.0.*"
serde_yaml_ng = "0.10.*"
tempfile = "3.27.*"
tiktoken-rs = { version = "0.12.*", default-features = false }
tokio = { version = "1.52.*", features = ["rt", "rt-multi-thread", "macros"] }
tokio = { version = "1.53.*", features = ["rt", "rt-multi-thread", "macros"] }
tracing = "0.1.*"
tracing-subscriber = { version = "0.3.*", features = ["env-filter"] }
url = "2.5.*"

View File

@@ -4,7 +4,7 @@
# #
#######################################
FROM docker.io/rust:1.96.0-slim-trixie AS build
FROM docker.io/rust:1.98.1-slim-trixie AS build
RUN apt-get update && apt-get install -y build-essential pkg-config libssl-dev libsqlite3-dev

View File

@@ -1,35 +0,0 @@
#######################################
# #
# Stage 1: building #
# #
#######################################
FROM docker.io/rust:1.96.0-slim-trixie AS build
RUN apt-get update && apt-get install -y build-essential pkg-config libssl-dev libsqlite3-dev
WORKDIR /app
COPY . /app
RUN cargo build --release
#######################################
# #
# Stage 2: packaging #
# #
#######################################
FROM docker.io/debian:trixie-slim
RUN apt-get update && apt-get install -y ca-certificates sqlite3 && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=build /app/target/release/baibot .
ENTRYPOINT ["/bin/sh", "-c"]
CMD ["/app/baibot"]

View File

@@ -30,7 +30,7 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use
- 🔒 Supports [encryption](./docs/features.md#-encryption) for Matrix communication and Account-Data-stored configuration
- ♻️ Supports [context-management](./docs/configuration/text-generation.md#️-context-management) handling on some models (automatically adjusting the message history length, etc.)
- ♻️ Supports [context-management](./docs/configuration/text-generation.md#️-context-management) for every [provider](./docs/providers.md) (automatically trimming older messages on whole-turn boundaries once a conversation outgrows the context window)
- 🛠️ Allows **customizing much of the bot's [configuration](./docs/configuration/README.md)** at runtime (using commands sent via chat)

View File

@@ -50,9 +50,9 @@ Example: `!bai config room text-generation set-auto-usage only_for_voice` (this
### ♻️ Context Management
The bot also supports ♻️ **context management**, which automatically adjusts the message history length, etc.
The bot also supports ♻️ **context management**, which automatically trims the oldest messages once a conversation grows past the context window. It drops whole turns at a time, so a reply is never separated from the message it answered.
This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](../providers.md#openai) models.
Counting tokens precisely needs the model's own tokenizer. For [OpenAI](../providers.md#openai) models, the bot counts them with the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library. For every other provider, including the recommended [Venice](../providers.md#venice), the bot falls back to a provider-neutral **approximation** that needs no per-model tokenizer: it counts ASCII text at about four characters per token and other scripts (Cyrillic, CJK, and so on) at about two. Treat it as rough, within roughly 10-20% of the real count for typical text, which is plenty for keeping a long conversation inside the context window.
This setting is **disabled by default**, but can be enabled via `!bai config room text-generation set-context-management-enabled true` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings)).

View File

@@ -57,6 +57,34 @@ CONTAINER_IMAGE_NAME=ghcr.io/etkecc/baibot:v1.0.0
$CONTAINER_IMAGE_NAME
```
Alternatively, you can use [Docker Compose](https://docs.docker.com/compose/) with a `compose.yml` file like this:
```yaml
services:
baibot:
container_name: baibot
# Adjust the version tag to point to the latest available tagged version.
# If building your own container image name, adjust to something like `localhost/baibot:latest`.
image: ghcr.io/etkecc/baibot:v1.0.0
# Set `UID` and `GID` in a `.env` file next to `compose.yml` (e.g. `UID=1000`, `GID=1000`)
# or export them in your shell (`export UID GID="$(id -g)"`).
# These should match the user that owns the data directory.
user: "${UID:-1000}:${GID:-1000}"
environment:
# Other settings can also be set via environment variables.
# See the 🛠️ Configuration documentation (docs/configuration/README.md) for details.
BAIBOT_PERSISTENCE_DATA_DIR_PATH: /data
volumes:
- /path/to/config.yml:/app/config.yml:ro
- /path/to/data:/data
cap_drop:
- ALL
read_only: true
tmpfs:
- /tmp:rw,noexec,nosuid,size=1024m
restart: unless-stopped
```
💡 If you've defined the `persistence.data_dir_path` setting in the `config.yml` file, you can skip the `BAIBOT_PERSISTENCE_DATA_DIR_PATH` environment variable.

View File

@@ -24,7 +24,7 @@ The list of supported providers is below.
### How to choose a provider
If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation) (incl. vision, incl. [🛠️ tools](./features.md#️-built-in-tools-openai-only)), [🖌️ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech).
If you're not sure which provider to start with, **we recommend [Venice](#venice)**: it's the most capable provider baibot supports (covering [💬 text-generation](./features.md#-text-generation) with vision, file inputs, prompt caching, and native web search, plus [🖌️ image-generation](./features.md#️-image-creation) incl. editing, [🦻 speech-to-text](./features.md#-speech-to-text), and [🗣️ text-to-speech](./features.md#️-text-to-speech)) and the only one that runs inference with no logging and no training on your data. If you'd rather start with the most widely-used option, [OpenAI](#openai) is a solid, well-supported choice too.
You don't need to choose just one though. The bot supports [mixing & matching models](./features.md#-mixing--matching-models), so you can use multiple providers at the same time.
@@ -176,7 +176,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
### Venice
[Venice AI](https://venice.ai/chat?ref=kpXDe6) _(ref link with $10 bonus for you)_ runs inference on Venice-controlled GPUs or zero-data-retention partner infrastructure and stores no prompts or responses, so your conversations don't linger anywhere. It serves both frontier proprietary models and the latest open-source ones.
[Venice AI](https://venice.ai/chat?ref=kpXDe6) _(ref link with a $10 bonus for you)_ runs inference on Venice-controlled GPUs or zero-data-retention partner infrastructure and stores no prompts or responses, so your conversations don't linger anywhere. It serves both frontier proprietary models and the latest open-source ones.
- 🆔 Identifier: `venice`
- 🔗 Links: [🏠 Home page](https://venice.ai/chat?ref=kpXDe6), [👤 Sign up](https://venice.ai/chat?ref=kpXDe6), [📋 Models list](https://docs.venice.ai/models/overview)

View File

@@ -1,14 +1,14 @@
services:
continuwuity:
image: forgejo.ellis.link/continuwuation/continuwuity:v0.5.10
image: forgejo.ellis.link/continuwuation/continuwuity:v26.8.1
user: "${UID}:${GID}"
restart: unless-stopped
cap_drop:
- ALL
read_only: true
environment:
CONDUWUIT_CONFIG: /etc/continuwuity/continuwuity.toml
CONDUWUIT_DATABASE_PATH: /var/lib/continuwuity
CONTINUWUITY_CONFIG: /etc/continuwuity/continuwuity.toml
CONTINUWUITY_DATABASE_PATH: /var/lib/continuwuity
ports:
- "${SERVICE_CONTINUWUITY_BIND_PORT_CLIENT_API}:6167"
volumes:

View File

@@ -1,6 +1,6 @@
services:
element-web:
image: ghcr.io/element-hq/element-web:v1.12.22
image: ghcr.io/element-hq/element-web:v1.12.27
user: "${UID}:${GID}"
restart: unless-stopped
environment:

View File

@@ -1,6 +1,6 @@
services:
ollama:
image: docker.io/ollama/ollama:0.30.11
image: docker.io/ollama/ollama:0.33.3
restart: unless-stopped
ports:
- "${SERVICE_OLLAMA_BIND_PORT_HTTP}:11434"

View File

@@ -1,6 +1,6 @@
services:
postgres:
image: docker.io/postgres:18.4-alpine
image: docker.io/postgres:18.6-alpine
user: ${UID}:${GID}
restart: unless-stopped
environment:
@@ -14,7 +14,7 @@ services:
- /etc/passwd:/etc/passwd:ro
synapse:
image: ghcr.io/element-hq/synapse:v1.155.0
image: ghcr.io/element-hq/synapse:v1.160.0
user: "${UID}:${GID}"
restart: unless-stopped
entrypoint: python

View File

@@ -271,7 +271,29 @@ prek-run-on-all *args: _ensure_mise_tools_installed
# Installs the git pre-commit hook (runs prek automatically before each commit)
prek-install-git-pre-commit-hook: _ensure_mise_tools_installed
@just --justfile {{ justfile() }} mise exec -- prek install
#!/usr/bin/env sh
set -eu
just --justfile {{ justfile() }} mise exec -- prek install
# The installed git hooks run later under Git, outside this just/mise environment,
# so they need to be told how to find their tooling:
#
# - MISE_DATA_DIR / MISE_TRUSTED_CONFIG_PATHS make mise resolve against this project's
# own data directory. Without them mise falls back to the global one and silently
# installs a second copy of the tool there.
# - prek bakes the full path of the currently installed version into the hook
# (var/mise/installs/prek/<version>/...), which stops working as soon as the pinned
# version changes or old versions are pruned. Pointing at mise's shim instead makes
# the hook resolve whatever mise.toml pins, at the time it runs.
#
# Which hook files prek installs depends on `default_install_hook_types` in
# .pre-commit-config.yaml, so patch every hook file that prek generated.
for hook in "{{ justfile_directory() }}"/.git/hooks/*; do
[ -f "$hook" ] || continue
grep -q 'generated by prek' "$hook" || continue
grep -q '^export MISE_DATA_DIR=' "$hook" || sed -i '2iexport MISE_DATA_DIR="{{ mise_data_dir }}"' "$hook"
grep -q '^export MISE_TRUSTED_CONFIG_PATHS=' "$hook" || sed -i '3iexport MISE_TRUSTED_CONFIG_PATHS="{{ mise_trusted_config_paths }}"' "$hook"
sed -i 's#^PREK=".*"$#PREK="{{ mise_data_dir }}/shims/prek"#' "$hook"
done
# Internal - ensures var/mise directory exists
_ensure_mise_data_directory:

View File

@@ -1,6 +1,2 @@
[tools]
prek = "0.4.5"
[settings]
# Disable automatic trust prompts - we trust this config
yes = true
prek = "0.5.2"

View File

@@ -5,5 +5,62 @@
],
"labels": [
"dependencies"
],
"packageRules": [
{
"description": "Cargo dependencies, the Rust toolchain pin, prek (via mise) and the workflows' own actions merge by pushing to main, without a pull request. ci.yml runs on `push: [\"**\"]`, so the Renovate branch itself is compiled, linted with `clippy -D warnings` and unit-tested first; a failure leaves the branch red and Renovate raises a pull request instead of merging.",
"matchManagers": [
"cargo",
"rust-toolchain",
"mise",
"github-actions"
],
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "branch",
"platformAutomerge": false
},
{
"description": "The release image's base (Dockerfile). Gated by ci.yml's docker-build job, which builds the image exactly as Publish does but with `push: false`, and which only runs when the Dockerfile actually changed.",
"matchManagers": [
"dockerfile"
],
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "branch",
"platformAutomerge": false
},
{
"description": "Local development service images under etc/services/** - the homeservers and LLM backends that `just services-start` brings up. They are never part of a shipped artifact and CI does not run them, so the justification here is blast radius rather than validation: the worst case is a broken local development stack, fixed by pinning back.",
"matchManagers": [
"docker-compose"
],
"matchFileNames": [
"etc/services/**"
],
"matchUpdateTypes": [
"minor",
"patch",
"digest"
],
"automerge": true,
"automergeType": "branch",
"platformAutomerge": false
},
{
"description": "Major updates always get a pull request and a human. This is deliberately the last rule so that it overrides the automerge rules above for every manager.",
"matchUpdateTypes": [
"major"
],
"automerge": false
}
]
}

View File

@@ -1,4 +1,4 @@
[toolchain]
channel = "1.96.0"
channel = "1.98.1"
components = ["rustfmt", "clippy"]
profile = "default"

View File

@@ -15,7 +15,7 @@ use crate::agent::provider::{
};
use crate::conversation::llm::{
Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage,
MessageContent as LLMMessageContent, shorten_messages_list_to_context_size,
MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size,
};
use crate::strings;
@@ -130,7 +130,7 @@ impl ControllerTrait for Controller {
tracing::trace!("Shortening messages list to context size");
conversation_messages = shorten_messages_list_to_context_size(
&text_generation_config.model_id,
TokenEstimate::Approximate,
&prompt_message,
conversation_messages,
Some(text_generation_config.max_response_tokens),

View File

@@ -24,7 +24,7 @@ use crate::{
},
conversation::llm::{
Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage,
MessageContent as LLMMessageContent, shorten_messages_list_to_context_size,
MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size,
},
utils::base64::base64_decode,
};
@@ -117,7 +117,7 @@ impl ControllerTrait for Controller {
tracing::trace!("Shortening messages list to context size");
conversation_messages = shorten_messages_list_to_context_size(
&text_generation_config.model_id,
TokenEstimate::Tiktoken(&text_generation_config.model_id),
&prompt_message,
conversation_messages,
text_generation_config.max_response_tokens,

View File

@@ -15,7 +15,7 @@ use crate::{
},
conversation::llm::{
Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage,
MessageContent as LLMMessageContent, shorten_messages_list_to_context_size,
MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size,
},
};
use crate::{
@@ -114,7 +114,7 @@ impl ControllerTrait for Controller {
tracing::trace!("Shortening messages list to context size");
conversation_messages = shorten_messages_list_to_context_size(
&text_generation_config.model_id,
TokenEstimate::Approximate,
&prompt_message,
conversation_messages,
text_generation_config.max_response_tokens,

View File

@@ -8,7 +8,7 @@ use crate::agent::AgentPurpose;
use crate::agent::provider::entity::{TextGenerationParams, TextGenerationResult};
use crate::conversation::llm::{
Author as LLMAuthor, Conversation as LLMConversation, Message as LLMMessage,
MessageContent as LLMMessageContent, shorten_messages_list_to_context_size,
MessageContent as LLMMessageContent, TokenEstimate, shorten_messages_list_to_context_size,
};
use crate::strings;
@@ -64,7 +64,7 @@ pub async fn generate_text(
if params.context_management_enabled {
conversation_messages = shorten_messages_list_to_context_size(
&text_generation_config.model_id,
TokenEstimate::Approximate,
&prompt_message,
conversation_messages,
text_generation_config.max_response_tokens,

View File

@@ -136,7 +136,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-generation prefix-requirement-type"
)
@@ -144,7 +144,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-prefix-requirement-type VALUE"
)
@@ -152,7 +152,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-prefix-requirement-type"
)
@@ -176,12 +176,12 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(command_prefix, "text-generation auto-usage")
strings::help::cfg::current_setting_show(command_prefix, "text-generation auto-usage")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-auto-usage VALUE"
)
@@ -189,10 +189,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-auto-usage"
)
strings::help::cfg::current_setting_unset(command_prefix, "text-generation set-auto-usage")
));
message.push_str("\n\n");
@@ -211,7 +208,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-generation context-management-enabled"
)
@@ -219,7 +216,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-context-management-enabled VALUE"
)
@@ -227,7 +224,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-context-management-enabled"
)
@@ -249,7 +246,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-generation thinking-notice-enabled"
)
@@ -257,7 +254,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-thinking-notice-enabled VALUE"
)
@@ -265,7 +262,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-thinking-notice-enabled"
)
@@ -289,7 +286,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-generation sender-context-mode"
)
@@ -297,7 +294,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-sender-context-mode VALUE"
)
@@ -305,7 +302,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-sender-context-mode"
)
@@ -323,15 +320,12 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
command_prefix,
"text-generation prompt-override"
)
strings::help::cfg::current_setting_show(command_prefix, "text-generation prompt-override")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-prompt-override VALUE"
)
@@ -339,7 +333,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-prompt-override"
)
@@ -357,7 +351,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-generation temperature-override"
)
@@ -365,7 +359,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-generation set-temperature-override VALUE"
)
@@ -373,7 +367,7 @@ fn build_section_text_generation(command_prefix: &str, bot_username: &str) -> St
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-generation set-temperature-override"
)
@@ -410,12 +404,12 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(command_prefix, "speech-to-text flow-type")
strings::help::cfg::current_setting_show(command_prefix, "speech-to-text flow-type")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"speech-to-text set-flow-type VALUE"
)
@@ -423,7 +417,7 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(command_prefix, "speech-to-text set-flow-type")
strings::help::cfg::current_setting_unset(command_prefix, "speech-to-text set-flow-type")
));
message.push_str("\n\n");
@@ -444,7 +438,7 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"speech-to-text msg-type-for-non-threaded-only-transcribed-messages"
)
@@ -452,7 +446,7 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"speech-to-text set-msg-type-for-non-threaded-only-transcribed-messages VALUE"
)
@@ -460,7 +454,7 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"speech-to-text set-msg-type-for-non-threaded-only-transcribed-messages"
)
@@ -478,12 +472,12 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(command_prefix, "speech-to-text language")
strings::help::cfg::current_setting_show(command_prefix, "speech-to-text language")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"speech-to-text set-language VALUE"
)
@@ -491,7 +485,7 @@ fn build_section_speech_to_text(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(command_prefix, "speech-to-text set-language")
strings::help::cfg::current_setting_unset(command_prefix, "speech-to-text set-language")
));
message.push_str("\n\n");
@@ -526,7 +520,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-to-speech bot-msgs-flow-type"
)
@@ -534,7 +528,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-to-speech set-bot-msgs-flow-type VALUE"
)
@@ -542,7 +536,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-to-speech set-bot-msgs-flow-type"
)
@@ -566,7 +560,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(
strings::help::cfg::current_setting_show(
command_prefix,
"text-to-speech user-msgs-flow-type"
)
@@ -574,7 +568,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-to-speech set-user-msgs-flow-type VALUE"
)
@@ -582,7 +576,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-to-speech set-user-msgs-flow-type"
)
@@ -600,12 +594,12 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(command_prefix, "text-to-speech speed-override")
strings::help::cfg::current_setting_show(command_prefix, "text-to-speech speed-override")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-to-speech set-speed-override VALUE"
)
@@ -613,7 +607,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-to-speech set-speed-override"
)
@@ -631,12 +625,12 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push_str("\n\n");
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_show(command_prefix, "text-to-speech voice-override")
strings::help::cfg::current_setting_show(command_prefix, "text-to-speech voice-override")
));
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_set(
strings::help::cfg::current_setting_set(
command_prefix,
"text-to-speech set-voice-override VALUE"
)
@@ -644,7 +638,7 @@ fn build_section_text_to_speech(command_prefix: &str) -> String {
message.push('\n');
message.push_str(&format!(
"- {}",
&strings::help::cfg::current_setting_unset(
strings::help::cfg::current_setting_unset(
command_prefix,
"text-to-speech set-voice-override"
)

View File

@@ -147,9 +147,7 @@ pub async fn handle(
// Let's proceed below where we potentially handle text-generation.
}
let text_to_speech_stage_params: Option<TextToSpeechParams>;
if message_context
let text_to_speech_stage_params: Option<TextToSpeechParams> = if message_context
.room_config_context()
.should_auto_text_generate(original_message_is_audio)
{
@@ -210,7 +208,7 @@ pub async fn handle(
return Ok(());
};
text_to_speech_stage_params = match message_context
match message_context
.room_config_context()
.text_to_speech_bot_messages_flow_type()
{
@@ -243,7 +241,7 @@ pub async fn handle(
text_to_speech_eligible_payload,
response_type,
)),
};
}
} else {
tracing::debug!("Not generating text due to auto-usage configuration");
@@ -262,7 +260,7 @@ pub async fn handle(
event_id: message_context.event_id().clone(),
};
text_to_speech_stage_params = match message_context
match message_context
.room_config_context()
.text_to_speech_user_messages_flow_type()
{
@@ -275,8 +273,8 @@ pub async fn handle(
text_to_speech_eligible_payload,
response_type,
)),
};
}
}
};
// We're potentially dealing with some text in text_to_speech_eligible_payload - either coming directly from the user or generated by an agent.

View File

@@ -6,5 +6,5 @@ mod utils;
mod tests;
pub use entity::*;
pub use tokenization::shorten_messages_list_to_context_size;
pub use tokenization::{TokenEstimate, shorten_messages_list_to_context_size};
pub use utils::*;

View File

@@ -4,6 +4,22 @@ use tiktoken_rs::tokenizer;
use super::{Author, Message, MessageContent};
/// How to count the tokens in a conversation when trimming it to fit the context window.
pub enum TokenEstimate<'a> {
/// Count via the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library.
/// Accurate for OpenAI models; every other model falls back to the gpt-4
/// tokenizer, which misreads it (badly so for non-English text). Use this only
/// for the OpenAI provider.
Tiktoken(&'a str),
/// Provider-neutral approximation that needs no per-model tokenizer. Expect it
/// to land within roughly 10-20% of the real count for typical text, leaning
/// slightly high: over-counting trims a little extra history, while
/// under-counting would overflow the model's real context window. Use this for
/// every non-OpenAI provider.
Approximate,
}
fn get_bpe_for_model(model: &str) -> &'static CoreBPE {
let tokenizer = tokenizer::get_tokenizer(model)
.or_else(|| tokenizer::get_tokenizer("gpt-4"))
@@ -13,21 +29,27 @@ fn get_bpe_for_model(model: &str) -> &'static CoreBPE {
}
pub fn shorten_messages_list_to_context_size(
model: &str,
estimate: TokenEstimate<'_>,
prompt_message: &Option<Message>,
mut messages: Vec<Message>,
max_response_tokens: Option<u32>,
max_context_tokens: u32,
) -> Vec<Message> {
// Loading the tokenization data is an expensive process, so
// se construct the BPE instance once and then use it for all messages.
let bpe = get_bpe_for_model(model);
// Loading the tiktoken data is expensive, so we resolve the counter once up
// front and reuse it for every message.
let tiktoken = match estimate {
TokenEstimate::Tiktoken(model) => Some((get_bpe_for_model(model), model)),
TokenEstimate::Approximate => None,
};
let count = |message: &Message| match tiktoken {
Some((bpe, model)) => tiktoken_token_size_for_message(bpe, model, message),
None => approximate_token_size_for_message(message),
};
// We want to retain the prompt in all cases, so we always count it first.
// We also always reserve enough tokens for the maximum response we expect.
let mut current_context_length: u32 = if let Some(prompt_message) = prompt_message {
calculate_token_size_for_message(bpe, model, prompt_message)
+ max_response_tokens.unwrap_or(0)
count(prompt_message) + max_response_tokens.unwrap_or(0)
} else {
0
};
@@ -37,7 +59,7 @@ pub fn shorten_messages_list_to_context_size(
let mut messages_to_keep: Vec<Message> = Vec::new();
for message in messages {
let tokens_for_message = calculate_token_size_for_message(bpe, model, &message);
let tokens_for_message = count(&message);
if current_context_length + tokens_for_message > max_context_tokens {
break;
@@ -48,14 +70,26 @@ pub fn shorten_messages_list_to_context_size(
messages_to_keep.push(message);
}
// Cut on a turn boundary: the loop may stop right after an assistant reply
// whose triggering user message did not fit, which would leave the kept window
// starting on an orphaned reply. `messages_to_keep` is newest-first here, so
// the oldest kept messages are at the end; drop any trailing assistant messages
// until the window begins at the start of a turn (a user message).
while matches!(
messages_to_keep.last().map(|message| &message.author),
Some(Author::Assistant)
) {
messages_to_keep.pop();
}
messages_to_keep.reverse();
messages_to_keep
}
/// Calculate the token size of a message for a given model, with a preloaded CoreBPE object.
/// Related to `calculate_token_size_for_model_message`.
fn calculate_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Message) -> u32 {
/// Token size of a message via tiktoken, for a preloaded CoreBPE object.
/// Accurate only for OpenAI models (see [`TokenEstimate::Tiktoken`]).
fn tiktoken_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Message) -> u32 {
let (tokens_per_message, tokens_per_name) = if model.starts_with("gpt-3.5") {
(
4, // every message follows <im_start>{role/name}\n{content}<im_end>\n
@@ -80,6 +114,52 @@ fn calculate_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Messag
(text_length + role_length + tokens_per_message + tokens_per_name) as u32
}
/// ASCII text averages about four characters per token.
const ASCII_TOKENS_PER_CHAR: f32 = 0.25;
/// Non-ASCII scripts (Cyrillic, CJK, and others) pack more information per
/// character: real tokenizers land around two characters per token for them, so
/// each counts as half a token. CJK runs a touch denser than that, so its estimate
/// can read slightly low, still within the tolerance this approximation targets.
const WIDE_TOKENS_PER_CHAR: f32 = 0.5;
/// Structural per-message overhead (role marker plus message framing), mirroring
/// the small constant the tiktoken path adds.
const APPROX_TOKENS_PER_MESSAGE: u32 = 4;
/// Provider-neutral, tokenizer-free token size of a message
/// (see [`TokenEstimate::Approximate`]).
fn approximate_token_size_for_message(message: &Message) -> u32 {
let text_tokens = match &message.content {
MessageContent::Text(text) => approximate_token_size_for_text(text),
// Images and files are not counted as text, matching the tiktoken path.
MessageContent::Image(..) | MessageContent::File(..) => 0,
};
text_tokens + APPROX_TOKENS_PER_MESSAGE
}
/// Rough token estimate for a piece of text, with no tokenizer.
///
/// ASCII characters count as a quarter-token each (~4 chars/token); characters
/// outside ASCII count as half a token each (~2 chars/token), matching how real
/// tokenizers treat Cyrillic and CJK. Weighting non-ASCII up keeps the estimate
/// from badly under-counting non-English text, the case the tiktoken fallback gets
/// most wrong.
fn approximate_token_size_for_text(text: &str) -> u32 {
let mut estimate = 0.0_f32;
for character in text.chars() {
estimate += if character.is_ascii() {
ASCII_TOKENS_PER_CHAR
} else {
WIDE_TOKENS_PER_CHAR
};
}
estimate.ceil() as u32
}
pub mod test {
#[test]
fn message_size_counting_works() {
@@ -94,7 +174,7 @@ pub mod test {
timestamp: chrono::Utc::now(),
};
let tokens = super::calculate_token_size_for_message(bpe, model, &message);
let tokens = super::tiktoken_token_size_for_message(bpe, model, &message);
assert_eq!(8, tokens);
}
@@ -118,7 +198,7 @@ pub mod test {
assert_eq!(
prompt_length,
super::calculate_token_size_for_message(bpe, model, &prompt)
super::tiktoken_token_size_for_message(bpe, model, &prompt)
);
let mut conversation_messages = Vec::new();
@@ -133,7 +213,7 @@ pub mod test {
assert_eq!(
first_length,
super::calculate_token_size_for_message(bpe, model, &first)
super::tiktoken_token_size_for_message(bpe, model, &first)
);
conversation_messages.push(first);
@@ -148,7 +228,7 @@ pub mod test {
assert_eq!(
second_length,
super::calculate_token_size_for_message(bpe, model, &second)
super::tiktoken_token_size_for_message(bpe, model, &second)
);
conversation_messages.push(second);
@@ -165,7 +245,7 @@ pub mod test {
assert_eq!(
third_length,
super::calculate_token_size_for_message(bpe, model, &third)
super::tiktoken_token_size_for_message(bpe, model, &third)
);
conversation_messages.push(third.clone());
@@ -182,7 +262,7 @@ pub mod test {
assert_eq!(
forth_length,
super::calculate_token_size_for_message(bpe, model, &forth)
super::tiktoken_token_size_for_message(bpe, model, &forth)
);
conversation_messages.push(forth.clone());
@@ -190,7 +270,7 @@ pub mod test {
assert_eq!(4, conversation_messages.len());
let new_conversation_messages = super::shorten_messages_list_to_context_size(
model,
super::TokenEstimate::Tiktoken(model),
&Some(prompt),
conversation_messages,
max_response_tokens,
@@ -229,7 +309,7 @@ pub mod test {
assert_eq!(
prompt_length,
super::calculate_token_size_for_message(bpe, model, &prompt)
super::tiktoken_token_size_for_message(bpe, model, &prompt)
);
let mut conversation_messages = Vec::new();
@@ -244,7 +324,7 @@ pub mod test {
assert_eq!(
first_length,
super::calculate_token_size_for_message(bpe, model, &first)
super::tiktoken_token_size_for_message(bpe, model, &first)
);
conversation_messages.push(first);
@@ -259,7 +339,7 @@ pub mod test {
assert_eq!(
second_length,
super::calculate_token_size_for_message(bpe, model, &second)
super::tiktoken_token_size_for_message(bpe, model, &second)
);
conversation_messages.push(second);
@@ -276,7 +356,7 @@ pub mod test {
assert_eq!(
third_length,
super::calculate_token_size_for_message(bpe, model, &third)
super::tiktoken_token_size_for_message(bpe, model, &third)
);
conversation_messages.push(third.clone());
@@ -293,7 +373,7 @@ pub mod test {
assert_eq!(
forth_length,
super::calculate_token_size_for_message(bpe, model, &forth)
super::tiktoken_token_size_for_message(bpe, model, &forth)
);
conversation_messages.push(forth.clone());
@@ -301,7 +381,7 @@ pub mod test {
assert_eq!(4, conversation_messages.len());
let new_conversation_messages = super::shorten_messages_list_to_context_size(
model,
super::TokenEstimate::Tiktoken(model),
&Some(prompt),
conversation_messages,
max_response_tokens,
@@ -320,4 +400,126 @@ pub mod test {
forth.content
);
}
#[test]
fn approximate_counting_weights_ascii_and_wide_scripts() {
// 12 ASCII characters at ~4 chars/token = 3 text tokens.
assert_eq!(3, super::approximate_token_size_for_text("Hello there!"));
// 5 CJK characters at ~0.5 token/char = 3 text tokens. The ASCII rate would
// have under-counted these to 2, the failure mode this path avoids.
assert_eq!(3, super::approximate_token_size_for_text("こんにちは"));
let message = super::Message {
author: super::Author::User,
sender_id: None,
content: super::MessageContent::Text("Hello there!".to_string()),
timestamp: chrono::Utc::now(),
};
// 3 text tokens plus the per-message overhead (4).
assert_eq!(7, super::approximate_token_size_for_message(&message));
}
#[test]
fn approximate_shortening_trims_to_budget() {
let prompt = super::Message {
author: super::Author::Prompt,
sender_id: None,
content: super::MessageContent::Text("You are a bot!".to_string()),
timestamp: chrono::Utc::now(),
};
let older = super::Message {
author: super::Author::User,
sender_id: None,
content: super::MessageContent::Text("This is the older message.".to_string()),
timestamp: chrono::Utc::now(),
};
let newer = super::Message {
// A user message, so it is a valid window start: keeping a lone
// assistant reply would be an orphan and get trimmed (see
// `shortening_cuts_on_a_turn_boundary`).
author: super::Author::User,
sender_id: None,
content: super::MessageContent::Text("This is the newer message.".to_string()),
timestamp: chrono::Utc::now(),
};
// Budget room for the prompt and only the newest message.
let max_context_tokens = super::approximate_token_size_for_message(&prompt)
+ super::approximate_token_size_for_message(&newer);
let new_conversation_messages = super::shorten_messages_list_to_context_size(
super::TokenEstimate::Approximate,
&Some(prompt),
vec![older, newer.clone()],
None,
max_context_tokens,
);
assert_eq!(1, new_conversation_messages.len());
assert_eq!(
new_conversation_messages.first().unwrap().content,
newer.content
);
}
#[test]
fn shortening_cuts_on_a_turn_boundary() {
// A four-message conversation of two full turns. All four messages are the
// same length, so they cost the same number of tokens.
let prompt = super::Message {
author: super::Author::Prompt,
sender_id: None,
content: super::MessageContent::Text("system".to_string()),
timestamp: chrono::Utc::now(),
};
let user_one = super::Message {
author: super::Author::User,
sender_id: None,
content: super::MessageContent::Text("user msg 1".to_string()),
timestamp: chrono::Utc::now(),
};
let asst_one = super::Message {
author: super::Author::Assistant,
sender_id: None,
content: super::MessageContent::Text("asst msg 1".to_string()),
timestamp: chrono::Utc::now(),
};
let user_two = super::Message {
author: super::Author::User,
sender_id: None,
content: super::MessageContent::Text("user msg 2".to_string()),
timestamp: chrono::Utc::now(),
};
let asst_two = super::Message {
author: super::Author::Assistant,
sender_id: None,
content: super::MessageContent::Text("asst msg 2".to_string()),
timestamp: chrono::Utc::now(),
};
let per_message = super::approximate_token_size_for_message(&user_one);
// Budget fits the prompt plus three messages. By raw token budget the loop
// would keep asst_two, user_two, and asst_one, but asst_one's own user
// message (user_one) does not fit, so it must be dropped too rather than
// left as an orphaned reply.
let max_context_tokens =
super::approximate_token_size_for_message(&prompt) + (per_message * 3);
let kept = super::shorten_messages_list_to_context_size(
super::TokenEstimate::Approximate,
&Some(prompt),
vec![user_one, asst_one, user_two.clone(), asst_two.clone()],
None,
max_context_tokens,
);
// Only the last whole turn survives; the orphaned asst_one is dropped.
assert_eq!(2, kept.len());
assert_eq!(kept.first().unwrap().content, user_two.content);
assert_eq!(kept.last().unwrap().content, asst_two.content);
}
}

View File

@@ -101,13 +101,13 @@ pub fn post_creation_helpful_commands(
for purpose in supported_purposes {
message.push_str(&format!(
"\n- {}",
&set_as_purpose_handler_in_room(agent_identifier, purpose, command_prefix,)
set_as_purpose_handler_in_room(agent_identifier, purpose, command_prefix,)
));
if !is_room_local {
message.push_str(&format!(
"\n- {}",
&set_as_purpose_handler_globally(agent_identifier, purpose, command_prefix,)
set_as_purpose_handler_globally(agent_identifier, purpose, command_prefix,)
));
}
}

View File

@@ -128,7 +128,7 @@ pub fn text_generation_context_management_intro() -> String {
format!(
"{}\n{}",
"Controls the bot's ability to **intelligently drop old messages from the conversation context** when it gets too large.",
"This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](./providers.md#openai) models.",
"Counting tokens precisely needs the model's own tokenizer. For [OpenAI](./providers.md#openai) models the bot uses the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library; for every other provider (including the recommended [Venice](./providers.md#venice)) it falls back to a provider-neutral **approximation** (ASCII counted at ~4 characters per token, other scripts at ~2), within roughly 10-20% of the real count for typical text.",
)
}