Compare commits

..

11 Commits

Author SHA1 Message Date
Slavi Pantaleev
407bb022d9 Release 1.14.1 2026-02-10 14:42:20 +02:00
Slavi Pantaleev
faf92cac09 Switch from deprecated serde_yaml to serde_yaml_ng
serde_yaml is deprecated and unmaintained. serde_yaml_ng is the community
fork with a compatible API, so this is a straightforward rename across the
codebase.
2026-02-10 14:34:57 +02:00
Slavi Pantaleev
a82e9a1d1f Add prek pre-commit hooks via mise, fix formatting and clippy warnings
- Add mise.toml (prek 0.3.2) and .pre-commit-config.yaml with hooks for
  trailing whitespace, end-of-file, YAML check, merge conflicts, large files,
  cargo fmt, cargo clippy (-D warnings), and unit tests
- Add prek/mise recipes to justfile
- Run cargo fmt to fix formatting issues
- Fix all clippy warnings: collapse nested if statements, derive Default for Avatar
2026-02-10 14:33:19 +02:00
Slavi Pantaleev
8f87f05a08 Update dependencies to fix security vulnerabilities
- Bump mxlink (>=1.11.0 -> >=1.12.0): pulls in fixes for time and bytes CVEs
- Bump async-openai (0.32.3 -> 0.32.4)
- Bump tempfile (3.24.* -> 3.25.*)
- Run cargo update to bump transitive dependencies, notably:
  - time (0.3.46 -> 0.3.47): fix stack exhaustion DoS
2026-02-10 14:28:38 +02:00
Slavi Pantaleev
61d18b2e13 Release 1.14.0 2026-02-04 03:19:30 +02:00
Slavi Pantaleev
d831c08306 Add support for OpenAI built-in tools (web_search, code_interpreter)
Document the new tools feature in README, features.md, and providers.md.
Update the `!bai providers` command output to show vision/tools support
consistently for all providers.

Based on #62 by @yeslayla which migrated the OpenAI provider to the
Responses API.

See: https://github.com/etkecc/baibot/pull/62
2026-02-04 03:17:19 +02:00
Slavi Pantaleev
c70387b0c3 Fix sticker generation for newer GPT image models
Sticker generation was failing when using newer GPT image models
(gpt-image-1, gpt-image-1-mini, gpt-image-1.5). The issue occurred
because stickers requested 256x256 size, but these models only support
1024x1024, 1536x1024, 1024x1536, and auto.

To reproduce, send `!bai sticker Something` to an agent configured
with a GPT image model. The error was:

  invalid_request_error: Invalid value: '256x256'. Supported values
  are: '1024x1024', '1024x1536', '1536x1024', and 'auto'. (param: size)
  (code: invalid_value)

The fix replaces the hardcoded 256x256 size override with a
`smallest_size_possible` flag, letting each provider determine the
appropriate sticker size based on the model being used.

The `openai_compat` provider still defaults to requesting 256x256 in all cases
(regardless of model name).
2026-02-04 02:39:51 +02:00
Slavi Pantaleev
38516f2e17 Update services 2026-02-04 02:28:12 +02:00
Slavi Pantaleev
5481b5a763 Update dependencies 2026-02-04 01:54:53 +02:00
Slavi Pantaleev
26bc437678 Add tools config to OpenAI example in config.yml.dist 2026-02-04 01:41:27 +02:00
Layla
ec93f1ee2a Implement OpenAI's response API and add support for built-in tools (web search & code interpreter). 2026-02-04 01:41:27 +02:00
40 changed files with 709 additions and 379 deletions

36
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,36 @@
repos:
# Fast built-in hooks (Rust-native, no dependencies)
- repo: builtin
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-merge-conflict
- id: check-added-large-files
args: ['--maxkb=1024']
# Local hooks that run project-specific tools
- repo: local
hooks:
- id: cargo-fmt-check
name: Cargo Format Check
entry: cargo fmt --all -- --check
language: system
files: '\.rs$'
pass_filenames: false
- id: cargo-clippy
name: Cargo Clippy
entry: cargo clippy -- -D warnings
language: system
files: '\.rs$'
pass_filenames: false
priority: 100
- id: test-unit
name: Unit Tests
entry: just test
language: system
files: '\.rs$'
pass_filenames: false
priority: 100

View File

@@ -1,3 +1,23 @@
# (2026-02-10) Version 1.14.1
- (**Security**) Dependency updates to fix security vulnerabilities ([time](https://crates.io/crates/time) stack exhaustion DoS, [bytes](https://crates.io/crates/bytes) integer overflow), via [mxlink](https://crates.io/crates/mxlink) 1.12.0
- (**Internal Improvement**) Switch from deprecated [serde_yaml](https://crates.io/crates/serde_yaml) to its maintained fork [serde_yaml_ng](https://crates.io/crates/serde_yaml_ng)
- (**Internal Improvement**) Add [prek](https://github.com/nicholasgasior/prek) pre-commit hooks via [mise](https://mise.jdx.dev/) for automated code quality checks (formatting, clippy, tests)
- (**Internal Improvement**) Fix clippy warnings and formatting issues
# (2026-02-04) Version 1.14.0
- (**Feature**) The `openai` provider now uses OpenAI's [Responses API](https://platform.openai.com/docs/api-reference/responses) (instead of the older Chat Completions API), adding support for [🛠️ built-in tools](./docs/features.md#️-built-in-tools-openai-only) (`web_search` and `code_interpreter`). These tools are **disabled by default** and can be enabled via the `text_generation.tools` configuration (see the [sample configuration](https://github.com/etkecc/baibot/blob/c70387b0c38d8d0f30bba2179a2a21a3710dbeaf/docs/sample-provider-configs/openai.yml#L12-L15)). To enable tools on an existing agent, you need to [update the agent](./docs/agents.md#updating-agents) to re-create it with the `text_generation.tools` section added and enable the tools you need. Thanks to [Layla Manley](https://github.com/yeslayla) for the contribution in [#62](https://github.com/etkecc/baibot/pull/62)!
- (**Bugfix**) Fix sticker generation for newer GPT image models (`gpt-image-1`, `gpt-image-1-mini`, `gpt-image-1.5`) which don't support the previously hardcoded `256x256` size (minimum is `1024x1024`)
- (**Internal Improvement**) Dependency updates
# (2026-01-23) Version 1.13.0
- (**Improvement**) Extend auto-switching to support cheaper models (`gpt-image-1-mini`) for `gpt-image-1` and `gpt-image-1.5` when generating stickers ([e0b4a40](https://github.com/etkecc/baibot/commit/e0b4a40))

379
Cargo.lock generated
View File

@@ -38,7 +38,7 @@ checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures",
"cpufeatures 0.2.17",
]
[[package]]
@@ -95,9 +95,9 @@ dependencies = [
[[package]]
name = "anyhow"
version = "1.0.100"
version = "1.0.101"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
checksum = "5f0e0fee31ef5ed1ba1316088939cea399010ed7731dba877ed44aeb407a75ea"
[[package]]
name = "anymap2"
@@ -166,9 +166,9 @@ dependencies = [
[[package]]
name = "async-compression"
version = "0.4.37"
version = "0.4.39"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d10e4f991a553474232bc0a31799f6d24b034a84c0971d80d2e2f78b2e576e40"
checksum = "68650b7df54f0293fd061972a0fb05aaf4fc0879d3b3d21a638a182c5c543b9f"
dependencies = [
"compression-codecs",
"compression-core",
@@ -178,9 +178,9 @@ dependencies = [
[[package]]
name = "async-openai"
version = "0.32.3"
version = "0.32.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "16aa4b9090c12270cd0925f735958ab6f0296af7ac08ce0f53fe320bcaf1e23d"
checksum = "f4d91c2005832450ad9ac92411b9d4cec39e30be21adc76553efc47f812d59c7"
dependencies = [
"async-openai-macros",
"backoff",
@@ -288,7 +288,7 @@ dependencies = [
[[package]]
name = "baibot"
version = "1.13.0"
version = "1.14.1"
dependencies = [
"anthropic",
"anyhow",
@@ -304,7 +304,7 @@ dependencies = [
"regex",
"serde",
"serde_json",
"serde_yaml",
"serde_yaml_ng",
"tempfile",
"tiktoken-rs",
"tokio",
@@ -384,7 +384,7 @@ dependencies = [
"cc",
"cfg-if",
"constant_time_eq",
"cpufeatures",
"cpufeatures 0.2.17",
]
[[package]]
@@ -439,9 +439,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
[[package]]
name = "bytes"
version = "1.11.0"
version = "1.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b35204fbdc0b3f4446b89fc1ac2cf84a8a68971995d0bf2e925ec7cd960f9cb3"
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
[[package]]
name = "bytesize"
@@ -460,9 +460,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.2.53"
version = "1.2.55"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "755d2fce177175ffca841e9a06afdb2c4ab0f593d53b4dee48147dfaade85932"
checksum = "47b26a0954ae34af09b50f0de26458fa95369a0d478d8236d3f93082b219bd29"
dependencies = [
"find-msvc-tools",
"shlex",
@@ -488,7 +488,18 @@ checksum = "c3613f74bd2eac03dad61bd53dbe620703d4371614fe0bc3b9f04dd36fe4e818"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures",
"cpufeatures 0.2.17",
]
[[package]]
name = "chacha20"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"rand_core 0.10.0",
]
[[package]]
@@ -498,7 +509,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10cd79432192d1c0f4e1a0fef9527696cc039165d729fb41b3f4f4f354c2dc35"
dependencies = [
"aead",
"chacha20",
"chacha20 0.9.1",
"cipher",
"poly1305",
"zeroize",
@@ -625,6 +636,15 @@ dependencies = [
"libc",
]
[[package]]
name = "cpufeatures"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
dependencies = [
"libc",
]
[[package]]
name = "crc32fast"
version = "1.5.0"
@@ -667,7 +687,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be"
dependencies = [
"cfg-if",
"cpufeatures",
"cpufeatures 0.2.17",
"curve25519-dalek-derive",
"digest",
"fiat-crypto",
@@ -1152,15 +1172,15 @@ checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d"
[[package]]
name = "find-msvc-tools"
version = "0.1.8"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8591b0bcc8a98a64310a2fae1bb3e9b8564dd10e381e6e28010fde8e8e8568db"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "flate2"
version = "1.1.8"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b375d6465b98090a5f25b1c7703f3859783755aa9a80433b36e0379a3ec2f369"
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -1344,6 +1364,20 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "getrandom"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec"
dependencies = [
"cfg-if",
"libc",
"r-efi",
"rand_core 0.10.0",
"wasip2",
"wasip3",
]
[[package]]
name = "gloo-timers"
version = "0.3.0"
@@ -1467,6 +1501,12 @@ dependencies = [
"http 1.4.0",
]
[[package]]
name = "heck"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hermit-abi"
version = "0.5.2"
@@ -1679,14 +1719,13 @@ dependencies = [
[[package]]
name = "hyper-util"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "727805d60e7938b76b826a6ef209eb70eaa1812794f9424d4a4e2d740662df5f"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [
"base64 0.22.1",
"bytes",
"futures-channel",
"futures-core",
"futures-util",
"http 1.4.0",
"http-body 1.0.1",
@@ -1695,7 +1734,7 @@ dependencies = [
"libc",
"percent-encoding",
"pin-project-lite",
"socket2 0.6.1",
"socket2 0.6.2",
"tokio",
"tower-service",
"tracing",
@@ -1703,9 +1742,9 @@ dependencies = [
[[package]]
name = "iana-time-zone"
version = "0.1.64"
version = "0.1.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb"
checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
dependencies = [
"android_system_properties",
"core-foundation-sys",
@@ -1806,6 +1845,12 @@ dependencies = [
"zerovec",
]
[[package]]
name = "id-arena"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
[[package]]
name = "ident_case"
version = "1.0.1"
@@ -2019,10 +2064,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
[[package]]
name = "libc"
version = "0.2.180"
name = "leb128fmt"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bcc35a38544a891a5f7c865aca548a982ccb3b8650a5b06d0fd33a10283c56fc"
checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
[[package]]
name = "libc"
version = "0.2.181"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "459427e2af2b9c839b132acb702a1c654d95e10f8c326bfc2ad11310e458b1c5"
[[package]]
name = "libsqlite3-sys"
@@ -2452,9 +2503,9 @@ dependencies = [
[[package]]
name = "memchr"
version = "2.7.6"
version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f52b00d39961fc5b2736ea853c9cc86238e165017a493d1d5c8eac6bdc4cc273"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "mime"
@@ -2516,9 +2567,9 @@ dependencies = [
[[package]]
name = "mxlink"
version = "1.11.0"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f5f917fdf8eed1fe7085255be92bdd045df070bb69886190808e8ce2d74f4c0a"
checksum = "e5bf83148b4188a9d62a265642c36360af262006413634222c2706bc52e38ccb"
dependencies = [
"base64 0.22.1",
"chacha20poly1305",
@@ -2526,7 +2577,7 @@ dependencies = [
"matrix-sdk",
"mime",
"quick_cache",
"rand 0.9.2",
"rand 0.10.0",
"serde",
"serde_json",
"thiserror 2.0.18",
@@ -2578,9 +2629,9 @@ dependencies = [
[[package]]
name = "num-conv"
version = "0.1.0"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9"
checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050"
[[package]]
name = "num-traits"
@@ -2806,7 +2857,7 @@ version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8159bd90725d2df49889a078b54f4f79e87f1f8a8444194cdca81d38f5393abf"
dependencies = [
"cpufeatures",
"cpufeatures 0.2.17",
"opaque-debug",
"universal-hash",
]
@@ -2841,6 +2892,16 @@ version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c"
[[package]]
name = "prettyplease"
version = "0.2.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
dependencies = [
"proc-macro2",
"syn 2.0.114",
]
[[package]]
name = "proc-macro-crate"
version = "3.4.0"
@@ -2946,7 +3007,7 @@ dependencies = [
"quinn-udp",
"rustc-hash 2.1.1",
"rustls 0.23.36",
"socket2 0.6.1",
"socket2 0.6.2",
"thiserror 2.0.18",
"tokio",
"tracing",
@@ -2983,16 +3044,16 @@ dependencies = [
"cfg_aliases",
"libc",
"once_cell",
"socket2 0.6.1",
"socket2 0.6.2",
"tracing",
"windows-sys 0.60.2",
]
[[package]]
name = "quote"
version = "1.0.43"
version = "1.0.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc74d9a594b72ae6656596548f56f667211f8a97b3d4c3d467150794690dc40a"
checksum = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4"
dependencies = [
"proc-macro2",
]
@@ -3024,6 +3085,17 @@ dependencies = [
"rand_core 0.9.5",
]
[[package]]
name = "rand"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8"
dependencies = [
"chacha20 0.10.0",
"getrandom 0.4.1",
"rand_core 0.10.0",
]
[[package]]
name = "rand_chacha"
version = "0.3.1"
@@ -3062,6 +3134,12 @@ dependencies = [
"getrandom 0.3.4",
]
[[package]]
name = "rand_core"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba"
[[package]]
name = "rand_xoshiro"
version = "0.7.0"
@@ -3073,15 +3151,15 @@ dependencies = [
[[package]]
name = "readlock"
version = "0.1.10"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6961798ceb493ecff2e50d3d9a6a9108124ee8298f81484d0ff623b6c32469e9"
checksum = "6da6f291b23556edd9edaf655a0be2ad8ef8002ff5f1bca62b264f3f58b53f34"
[[package]]
name = "readlock-tokio"
version = "0.1.5"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5f78bdff9b59a104cb171ceecc310c25e60ca02d7a95895afff559edd711fa1b"
checksum = "fc7e264f9ec4f3d112e8e2f214e8e7cb5cf3b83278f3570b7e00bfe13d3bd8ff"
dependencies = [
"tokio",
]
@@ -3097,9 +3175,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.12.2"
version = "1.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843bc0191f75f3e22651ae5f1e72939ab2f72a4bc30fa80a066bd66edefc24d4"
checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276"
dependencies = [
"aho-corasick",
"memchr",
@@ -3109,9 +3187,9 @@ dependencies = [
[[package]]
name = "regex-automata"
version = "0.4.13"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5276caf25ac86c8d810222b3dbb938e512c55c6831a10f3e6ed1c93b84041f1c"
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
dependencies = [
"aho-corasick",
"memchr",
@@ -3120,9 +3198,9 @@ dependencies = [
[[package]]
name = "regex-syntax"
version = "0.8.8"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a2d987857b319362043e95f5353c0535c1f58eec5336fdfcf626430af7def58"
checksum = "a96887878f22d7bad8a3b6dc5b7440e0ada9a245242924394987b21cf2210a4c"
[[package]]
name = "reqwest"
@@ -3612,9 +3690,9 @@ checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "ryu"
version = "1.0.22"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a50f4cf475b65d88e057964e0e9bb1f0aa9bbb2036dc65c64596b42932536984"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "schannel"
@@ -3814,10 +3892,10 @@ dependencies = [
]
[[package]]
name = "serde_yaml"
version = "0.9.34+deprecated"
name = "serde_yaml_ng"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
checksum = "7b4db627b98b36d4203a7b458cf3573730f2bb591b28871d916dfa9efabfd41f"
dependencies = [
"indexmap",
"itoa",
@@ -3833,7 +3911,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
dependencies = [
"cfg-if",
"cpufeatures",
"cpufeatures 0.2.17",
"digest",
]
@@ -3844,7 +3922,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"cpufeatures 0.2.17",
"digest",
]
@@ -3880,15 +3958,15 @@ checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2"
[[package]]
name = "siphasher"
version = "1.0.1"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d"
checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e"
[[package]]
name = "slab"
version = "0.4.11"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a2ae44ef20feb57a68b23d846850f861394c2e02dc425a50098ae8c90267589"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "smallvec"
@@ -3911,9 +3989,9 @@ dependencies = [
[[package]]
name = "socket2"
version = "0.6.1"
version = "0.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "17129e116933cf371d018bb80ae557e889637989d8638274fb25622827b03881"
checksum = "86f4aa3ad99f2088c990dfa82d367e19cb29268ed67c574d10d0a4bfe71f07e0"
dependencies = [
"libc",
"windows-sys 0.60.2",
@@ -4049,12 +4127,12 @@ dependencies = [
[[package]]
name = "tempfile"
version = "3.24.0"
version = "3.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "655da9c7eb6305c55742045d5a8d2037996d61d8de95806335c7c86ce0f82e9c"
checksum = "0136791f7c95b1f6dd99f9cc786b91bb81c3800b639b3478e561ddb7be95e5f1"
dependencies = [
"fastrand",
"getrandom 0.3.4",
"getrandom 0.4.1",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -4137,9 +4215,9 @@ dependencies = [
[[package]]
name = "time"
version = "0.3.45"
version = "0.3.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9e442fc33d7fdb45aa9bfeb312c095964abdf596f7567261062b2a7107aaabd"
checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c"
dependencies = [
"deranged",
"itoa",
@@ -4152,15 +4230,15 @@ dependencies = [
[[package]]
name = "time-core"
version = "0.1.7"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b36ee98fd31ec7426d599183e8fe26932a8dc1fb76ddb6214d05493377d34ca"
checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca"
[[package]]
name = "time-macros"
version = "0.2.25"
version = "0.2.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71e552d1249bf61ac2a52db88179fd0673def1e1ad8243a00d9ec9ed71fee3dd"
checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215"
dependencies = [
"num-conv",
"time-core",
@@ -4201,7 +4279,7 @@ dependencies = [
"libc",
"mio",
"pin-project-lite",
"socket2 0.6.1",
"socket2 0.6.2",
"tokio-macros",
"windows-sys 0.61.2",
]
@@ -4471,9 +4549,9 @@ checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142"
[[package]]
name = "unicode-ident"
version = "1.0.22"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5"
checksum = "537dd038a89878be9b64dd4bd1b260315c1bb94f4d784956b81e27a088d9a09e"
[[package]]
name = "unicode-normalization"
@@ -4563,9 +4641,9 @@ checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]]
name = "uuid"
version = "1.19.0"
version = "1.20.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2e054861b4bd027cd373e18e8d8d8e6548085000e41290d95ce0c373a654b4a"
checksum = "ee48d38b119b0cd71fe4141b30f5ba9c7c5d9f4e7a3a8b4a674e4b6ef789976f"
dependencies = [
"getrandom 0.3.4",
"js-sys",
@@ -4645,6 +4723,15 @@ dependencies = [
"wit-bindgen",
]
[[package]]
name = "wasip3"
version = "0.4.0+wasi-0.3.0-rc-2026-01-06"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5"
dependencies = [
"wit-bindgen",
]
[[package]]
name = "wasm-bindgen"
version = "0.2.108"
@@ -4704,6 +4791,28 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "wasm-encoder"
version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319"
dependencies = [
"leb128fmt",
"wasmparser",
]
[[package]]
name = "wasm-metadata"
version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909"
dependencies = [
"anyhow",
"indexmap",
"wasm-encoder",
"wasmparser",
]
[[package]]
name = "wasm-streams"
version = "0.4.2"
@@ -4735,6 +4844,18 @@ dependencies = [
"web-sys",
]
[[package]]
name = "wasmparser"
version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe"
dependencies = [
"bitflags 2.10.0",
"hashbrown 0.15.5",
"indexmap",
"semver",
]
[[package]]
name = "web-sys"
version = "0.3.85"
@@ -4774,14 +4895,14 @@ version = "0.26.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
dependencies = [
"webpki-roots 1.0.5",
"webpki-roots 1.0.6",
]
[[package]]
name = "webpki-roots"
version = "1.0.5"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12bed680863276c63889429bfd6cab3b99943659923822de1c8a39c49e4d722c"
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
dependencies = [
"rustls-pki-types",
]
@@ -5097,6 +5218,88 @@ name = "wit-bindgen"
version = "0.51.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5"
dependencies = [
"wit-bindgen-rust-macro",
]
[[package]]
name = "wit-bindgen-core"
version = "0.51.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc"
dependencies = [
"anyhow",
"heck",
"wit-parser",
]
[[package]]
name = "wit-bindgen-rust"
version = "0.51.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21"
dependencies = [
"anyhow",
"heck",
"indexmap",
"prettyplease",
"syn 2.0.114",
"wasm-metadata",
"wit-bindgen-core",
"wit-component",
]
[[package]]
name = "wit-bindgen-rust-macro"
version = "0.51.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a"
dependencies = [
"anyhow",
"prettyplease",
"proc-macro2",
"quote",
"syn 2.0.114",
"wit-bindgen-core",
"wit-bindgen-rust",
]
[[package]]
name = "wit-component"
version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2"
dependencies = [
"anyhow",
"bitflags 2.10.0",
"indexmap",
"log",
"serde",
"serde_derive",
"serde_json",
"wasm-encoder",
"wasm-metadata",
"wasmparser",
"wit-parser",
]
[[package]]
name = "wit-parser"
version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736"
dependencies = [
"anyhow",
"id-arena",
"indexmap",
"log",
"semver",
"serde",
"serde_derive",
"serde_json",
"unicode-xid",
"wasmparser",
]
[[package]]
name = "writeable"
@@ -5147,18 +5350,18 @@ dependencies = [
[[package]]
name = "zerocopy"
version = "0.8.33"
version = "0.8.39"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "668f5168d10b9ee831de31933dc111a459c97ec93225beb307aed970d1372dfd"
checksum = "db6d35d663eadb6c932438e763b262fe1a70987f9ae936e60158176d710cae4a"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.33"
version = "0.8.39"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c7962b26b0a8685668b671ee4b54d007a67d4eaf05fda79ac0ecf41e32270f1"
checksum = "4122cd3169e94605190e77839c9a40d40ed048d305bfdc146e7df40ab0f3e517"
dependencies = [
"proc-macro2",
"quote",
@@ -5241,6 +5444,6 @@ dependencies = [
[[package]]
name = "zmij"
version = "1.0.16"
version = "1.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dfcd145825aace48cff44a8844de64bf75feec3080e0aa5cdbde72961ae51a65"
checksum = "4de98dfa5d5b7fef4ee834d0073d560c9ca7b6c46a71d058c48db7960f8cfaf7"

View File

@@ -7,7 +7,7 @@ license = "AGPL-3.0-or-later"
readme = "README.md"
keywords = ["matrix", "chat", "bot", "AI", "LLM"]
include = ["/etc/assets/baibot-torso-768.png", "/src", "/README.md", "/CHANGELOG.md", "/LICENSE"]
version = "1.13.0"
version = "1.14.1"
edition = "2024"
[lib]
@@ -17,7 +17,7 @@ path = "src/lib.rs"
[dependencies]
anthropic = { git = "https://github.com/etkecc/anthropic-rs.git", branch = "fix-content-block-image" }
anyhow = "1.0.*"
async-openai = { version = "0.32.3", features = ["audio", "chat-completion", "image"] }
async-openai = { version = "0.32.4", features = ["audio", "chat-completion", "image", "responses"] }
base64 = "0.22.*"
chrono = { version = "0.4.*", default-features = false, features = ["std", "now"] }
# We'd rather not depend on this, but we cannot use the ruma-events EventContent macro without it.
@@ -25,14 +25,14 @@ chrono = { version = "0.4.*", default-features = false, features = ["std", "now"
matrix-sdk = { version = "0.16.0", default-features = false, features = ["native-tls"] }
mime_guess = "2.0.*"
mxidwc = "1.0.*"
mxlink = ">=1.11.0"
mxlink = ">=1.12.0"
etke_openai_api_rust = "0.1.*"
quick_cache = "0.6.*"
regex = "1.12.*"
serde = { version = "1.0.*", features = ["derive"], default-features = false }
serde_json = "1.0.*"
serde_yaml = "0.9.*"
tempfile = "3.24.*"
serde_yaml_ng = "0.10.*"
tempfile = "3.25.*"
tiktoken-rs = { version = "0.9.*", default-features = false }
tokio = { version = "1.49.*", features = ["rt", "rt-multi-thread", "macros"] }
tracing = "0.1.*"

View File

@@ -17,7 +17,7 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use
- Supports **different use purposes** (depending on the [☁️ provider](./docs/providers.md) & model):
- [💬 text-generation](./docs/features.md#-text-generation): communicating with you via text (though certain models may "see" images as well)
- [💬 text-generation](./docs/features.md#-text-generation): communicating with you via text (though certain models may "see" images as well). The [OpenAI provider](./docs/providers.md#openai) also supports [🛠️ built-in tools](./docs/features.md#️-built-in-tools-openai-only) (web search, code interpreter)
- [🦻 speech-to-text](./docs/features.md#-speech-to-text): turning your voice messages into text
- [🗣️ text-to-speech](./docs/features.md#%EF%B8%8F-text-to-speech): turning bot or users text messages into voice messages
- [🖌️ image-generation](./docs/features.md#image-generation): creating and editing images based on instructions

View File

@@ -40,6 +40,23 @@ You may also wish to see:
- [📖 Usage / 💬 Text Generation](./usage.md#-text-generation) section for more details on how to use the bot for Text Generation in a room
#### 🛠️ Built-in Tools (OpenAI only)
The [OpenAI provider](./providers.md#openai) supports built-in tools that extend the model's capabilities:
- [🔍 Web Search](https://platform.openai.com/docs/guides/tools-web-search) (`web_search`): allows the model to search the web for up-to-date information. [🖼️ Screenshot](./screenshots/text-generation-tools-web-search.webp)
- [💻 Code Interpreter](https://platform.openai.com/docs/guides/tools-code-interpreter) (`code_interpreter`): allows the model to write and execute Python code in a sandbox
These tools are **disabled by default** and need to be explicitly enabled in the agent's `text_generation.tools` configuration. See the [OpenAI sample configuration](https://github.com/etkecc/baibot/blob/c70387b0c38d8d0f30bba2179a2a21a3710dbeaf/docs/sample-provider-configs/openai.yml#L12-L15) for reference.
To enable tools on an existing dynamically-created agent, you need to [update the agent](./agents.md#updating-agents) to re-create it with the `text_generation.tools` section added and enable the tools you need
💡 **Note**: These tools run on OpenAI's infrastructure and may incur additional costs. Web search results include citations that are incorporated into the response.
#### On-demand involvement
In the following 2 cases, it's useful to involve the bot in conversations on-demand:

View File

@@ -23,7 +23,7 @@ The list of supported providers is below.
### How to choose a provider
If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation) (no vision), [🖌️ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech).
If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation) (incl. vision, incl. [🛠️ tools](./features.md#️-built-in-tools-openai-only)), [🖌️ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech).
You don't need to choose just one though. The bot supports [mixing & matching models](./features.md#-mixing--matching-models), so you can use multiple providers at the same time.
@@ -47,7 +47,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
- 🆔 Identifier: `anthropic`
- 🔗 Links: [🏠 Home page](https://www.anthropic.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/Anthropic), [👤 Sign up](https://console.anthropic.com/), [📋 Models list](https://docs.anthropic.com/en/docs/about-claude/models)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (incl. vision)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (incl. vision, no tools)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local anthropic my-anthropic-agent`
- create a global agent: `!bai agent create-global anthropic my-anthropic-agent`
@@ -61,7 +61,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
- 🆔 Identifier: `groq`
- 🔗 Links: [🏠 Home page](https://groq.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/Groq), [👤 Sign up](https://console.groq.com/login), [📋 Models list](https://console.groq.com/docs/models)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local groq my-groq-agent`
- create a global agent: `!bai agent create-global groq my-groq-agent`
@@ -75,7 +75,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
- 🆔 Identifier: `localai`
- 🔗 Links: [🏠 Home page](https://localai.io/), [📋 Models list](https://localai.io/gallery.html)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local localai my-localai-agent`
- create a global agent: `!bai agent create-global localai my-localai-agent`
@@ -89,7 +89,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
- 🆔 Identifier: `mistral`
- 🔗 Links: [🏠 Home page](https://mistral.ai/), [🌐 Wiki](https://en.wikipedia.org/wiki/Mistral_AI), [👤 Sign up](https://auth.mistral.ai/ui/registration), [📋 Models list](https://docs.mistral.ai/getting-started/models/)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local mistral my-mistral-agent`
- create a global agent: `!bai agent create-global mistral my-mistral-agent`
@@ -103,7 +103,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
- 🆔 Identifier: `ollama`
- 🔗 Links: [🏠 Home page](https://ollama.com/), [📋 Models list](https://ollama.com/library)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local ollama my-ollama-agent`
- create a global agent: `!bai agent create-global ollama my-ollama-agent`
@@ -120,7 +120,7 @@ For services which are not fully compatible with the OpenAI API, consider using
- 🆔 Identifier: `openai`
- 🔗 Links: [🏠 Home page](https://openai.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/OpenAI), [👤 Sign up](https://platform.openai.com/signup), [📋 Models list](https://platform.openai.com/docs/models)
- 🌟 Capabilities: [🖌️ image-generation](./features.md#️-image-creation), [💬 text-generation](./features.md#-text-generation) (incl. vision), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🌟 Capabilities: [🖌️ image-generation](./features.md#️-image-creation), [💬 text-generation](./features.md#-text-generation) (incl. vision, incl. [🛠️ tools](./features.md#️-built-in-tools-openai-only)), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local openai my-openai-agent`
- create a global agent: `!bai agent create-global openai my-openai-agent`
@@ -137,7 +137,7 @@ Some of these popular services already have **shortcut** providers (leading to t
This provider is just as featureful as the [OpenAI](#openai) provider, but is more compatible with services which do not fully adhere to the [OpenAI API spec](https://github.com/openai/openai-openapi/).
- 🆔 Identifier: `openai-compatible`
- 🌟 Capabilities: [🖌️ image-generation](./features.md#️-image-creation), [💬 text-generation](./features.md#-text-generation) (no vision), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🌟 Capabilities: [🖌️ image-generation](./features.md#️-image-creation), [💬 text-generation](./features.md#-text-generation) (no vision, no tools), [🗣️ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local openai-compatible my-openai-compatible-agent`
- create a global agent: `!bai agent create-global openai-compatible my-openai-compatible-agent`
@@ -151,7 +151,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
- 🆔 Identifier: `openrouter`
- 🔗 Links: [🏠 Home page](https://openrouter.ai/), [👤 Sign up](https://openrouter.ai/), [📋 Models list](https://openrouter.ai/models)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local openrouter my-openrouter-agent`
- create a global agent: `!bai agent create-global openrouter my-openrouter-agent`
@@ -165,7 +165,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
- 🆔 Identifier: `together-ai`
- 🔗 Links: [🏠 Home page](https://www.together.ai/), [👤 Sign up](https://api.together.ai/signup), [📋 Models list](https://api.together.xyz/models)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision)
- 🌟 Capabilities: [💬 text-generation](./features.md#-text-generation) (no vision, no tools)
- 🗲 Quick start:
- create a room-local agent: `!bai agent create-room-local together-ai my-together-ai-agent`
- create a global agent: `!bai agent create-global together-ai my-together-ai-agent`

View File

@@ -9,6 +9,10 @@ text_generation:
max_response_tokens: null
max_completion_tokens: 128000
max_context_tokens: 400000
# Built-in tools
tools:
web_search: false
code_interpreter: false
speech_to_text:
model_id: whisper-1
text_to_speech:

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

View File

@@ -90,6 +90,10 @@ agents:
# max_response_tokens: null
# max_completion_tokens: 128000
# max_context_tokens: 400000
# # Built-in tools
# tools:
# web_search: false
# code_interpreter: false
# speech_to_text:
# model_id: whisper-1
# text_to_speech:

View File

@@ -14,7 +14,7 @@ services:
- /etc/passwd:/etc/passwd:ro
synapse:
image: ghcr.io/element-hq/synapse:v1.144.0
image: ghcr.io/element-hq/synapse:v1.146.0
user: "${UID}:${GID}"
restart: unless-stopped
entrypoint: python
@@ -27,7 +27,7 @@ services:
- ./synapse/media-store:/media-store
element-web:
image: ghcr.io/element-hq/element-web:v1.12.7
image: ghcr.io/element-hq/element-web:v1.12.9
user: "${UID}:${GID}"
restart: unless-stopped
environment:

View File

@@ -1,6 +1,6 @@
services:
ollama:
image: docker.io/ollama/ollama:0.13.5
image: docker.io/ollama/ollama:0.15.4
restart: unless-stopped
ports:
- "${SERVICE_OLLAMA_BIND_PORT_HTTP}:11434"

View File

@@ -2,6 +2,9 @@ project_name := "baibot"
container_image_name := "localhost/baibot"
project_container_network := "baibot"
mise_data_dir := env("MISE_DATA_DIR", justfile_directory() / "var/mise")
mise_trusted_config_paths := justfile_directory() / "mise.toml"
# Show help by default
default:
@just --list --justfile {{ justfile() }}
@@ -159,6 +162,44 @@ synapse-register-regular-user username password: services-prepare
clippy *extra_args:
cargo clippy {{ extra_args }}
# Checks that the code compiles without building
check:
cargo check
# Invokes mise with the project-local data directory
mise *args: _ensure_mise_data_directory
#!/bin/sh
export MISE_DATA_DIR="{{ mise_data_dir }}"
export MISE_TRUSTED_CONFIG_PATHS="{{ mise_trusted_config_paths }}"
mise {{ args }}
# Runs prek (pre-commit hooks manager) with the given arguments
prek *args: _ensure_mise_tools_installed
@just --justfile {{ justfile() }} mise exec -- prek {{ args }}
# Runs pre-commit hooks on staged files
prek-run-on-staged *args: _ensure_mise_tools_installed
@just --justfile {{ justfile() }} mise exec -- prek run {{ args }}
# Runs pre-commit hooks on all files
prek-run-on-all *args: _ensure_mise_tools_installed
@just --justfile {{ justfile() }} mise exec -- prek run --all-files {{ args }}
# Installs the git pre-commit hook (runs prek automatically before each commit)
prek-install-git-pre-commit-hook: _ensure_mise_tools_installed
@just --justfile {{ justfile() }} mise exec -- prek install
# Internal - ensures var/mise directory exists
_ensure_mise_data_directory:
#!/bin/sh
if [ ! -d "{{ mise_data_dir }}" ]; then
mkdir -p "{{ mise_data_dir }}"
fi
# Internal - ensures mise tools are installed
_ensure_mise_tools_installed: _ensure_mise_data_directory
@just --justfile {{ justfile() }} mise install --quiet
_prepare-var-services-env:
#!/bin/sh
cd {{ justfile_directory() }};
@@ -248,4 +289,3 @@ _prepare-var-app-container-data:
if [ ! -f var/app/container/data ]; then
mkdir -p var/app/container/data
fi

6
mise.toml Normal file
View File

@@ -0,0 +1,6 @@
[tools]
prek = "0.3.2"
[settings]
# Disable automatic trust prompts - we trust this config
yes = true

View File

@@ -33,11 +33,11 @@ pub struct AgentDefinition {
)]
pub provider: AgentProvider,
pub config: serde_yaml::Value,
pub config: serde_yaml_ng::Value,
}
impl AgentDefinition {
pub fn new(id: String, provider: AgentProvider, config: serde_yaml::Value) -> Self {
pub fn new(id: String, provider: AgentProvider, config: serde_yaml_ng::Value) -> Self {
Self {
id,
provider,

View File

@@ -15,7 +15,7 @@ pub enum Error {
// Contains the error from the constructor function
ConstructionFailed(anyhow::Error),
// Contains the error from the YAML deserialization function
Yaml(serde_yaml::Error),
Yaml(serde_yaml_ng::Error),
}
pub type Result<T> = std::result::Result<T, Error>;
@@ -69,7 +69,7 @@ pub(super) fn create(
pub fn create_from_provider_and_yaml_value_config(
provider: &AgentProvider,
identifier: &PublicIdentifier,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
) -> Result<AgentInstance> {
let definition = AgentDefinition::new(identifier.prefixless(), provider.to_owned(), config);
@@ -79,7 +79,7 @@ pub fn create_from_provider_and_yaml_value_config(
fn create_controller_from_provider_and_json_value_config(
agent_id: &str,
provider: &AgentProvider,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
) -> Result<ControllerType> {
match provider {
AgentProvider::Anthropic => {
@@ -112,43 +112,43 @@ fn create_controller_from_provider_and_json_value_config(
}
}
pub fn default_config_for_provider(provider: &AgentProvider) -> serde_yaml::Value {
pub fn default_config_for_provider(provider: &AgentProvider) -> serde_yaml_ng::Value {
match provider {
AgentProvider::Anthropic => {
let config = super::provider::anthropic::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::Groq => {
let config = super::provider::groq::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::LocalAI => {
let config = super::provider::localai::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::Mistral => {
let config = super::provider::mistral::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::Ollama => {
let config = super::provider::ollama::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::OpenAI => {
let config = super::provider::openai::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::OpenAICompat => {
let config = super::provider::openai_compat::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::OpenRouter => {
let config = super::provider::openrouter::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
AgentProvider::TogetherAI => {
let config = super::provider::togetherai::default_config();
serde_yaml::to_value(config).expect("Failed to serialize config")
serde_yaml_ng::to_value(config).expect("Failed to serialize config")
}
}
}

View File

@@ -146,10 +146,10 @@ impl ControllerTrait for Controller {
.temperature_override
.unwrap_or(text_generation_config.temperature);
if let Some(prompt_message) = prompt_message {
if let LLMMessageContent::Text(text) = &prompt_message.content {
request.system = text.clone();
}
if let Some(prompt_message) = prompt_message
&& let LLMMessageContent::Text(text) = &prompt_message.content
{
request.system = text.clone();
}
request.model = text_generation_config.model_id.clone();

View File

@@ -12,12 +12,12 @@ use super::controller::ControllerType;
pub fn create_controller_from_yaml_value_config(
agent_id: &str,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
) -> AgentInstantiationResult<ControllerType> {
let config = match &config {
serde_yaml::Value::Mapping(_) => {
serde_yaml_ng::Value::Mapping(_) => {
let config: Config =
serde_yaml::from_value(config).map_err(AgentInstantiationError::Yaml)?;
serde_yaml_ng::from_value(config).map_err(AgentInstantiationError::Yaml)?;
config
.validate()

View File

@@ -69,6 +69,7 @@ impl AgentProvider {
models_list_url: Some("https://docs.anthropic.com/en/docs/about-claude/models"),
supported_purposes: vec![AgentPurpose::TextGeneration],
text_generation_supports_vision: true,
text_generation_supports_tools: false,
},
Self::Groq => AgentProviderInfo {
id: Self::Groq.to_static_str(),
@@ -80,11 +81,12 @@ impl AgentProvider {
models_list_url: Some("https://console.groq.com/docs/models"),
supported_purposes: vec![AgentPurpose::TextGeneration, AgentPurpose::SpeechToText],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::LocalAI => AgentProviderInfo {
id: Self::LocalAI.to_static_str(),
name: "LocalAI",
description: "LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.",
description: "LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.",
homepage_url: Some("https://localai.io/"),
wiki_url: None,
sign_up_url: None,
@@ -95,6 +97,7 @@ impl AgentProvider {
AgentPurpose::SpeechToText,
],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::Mistral => AgentProviderInfo {
id: Self::Mistral.to_static_str(),
@@ -106,6 +109,7 @@ impl AgentProvider {
models_list_url: Some("https://docs.mistral.ai/getting-started/models/"),
supported_purposes: vec![AgentPurpose::TextGeneration],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::Ollama => AgentProviderInfo {
id: Self::Ollama.to_static_str(),
@@ -117,6 +121,7 @@ impl AgentProvider {
models_list_url: Some("https://ollama.com/library"),
supported_purposes: vec![AgentPurpose::TextGeneration],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::OpenAI => AgentProviderInfo {
id: Self::OpenAI.to_static_str(),
@@ -133,6 +138,7 @@ impl AgentProvider {
AgentPurpose::SpeechToText,
],
text_generation_supports_vision: true,
text_generation_supports_tools: true,
},
Self::OpenAICompat => AgentProviderInfo {
id: Self::OpenAICompat.to_static_str(),
@@ -149,6 +155,7 @@ impl AgentProvider {
AgentPurpose::SpeechToText,
],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::OpenRouter => AgentProviderInfo {
id: Self::OpenRouter.to_static_str(),
@@ -160,6 +167,7 @@ impl AgentProvider {
models_list_url: Some("https://openrouter.ai/models"),
supported_purposes: vec![AgentPurpose::TextGeneration],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
Self::TogetherAI => AgentProviderInfo {
id: Self::TogetherAI.to_static_str(),
@@ -171,6 +179,7 @@ impl AgentProvider {
models_list_url: Some("https://api.together.xyz/models"),
supported_purposes: vec![AgentPurpose::TextGeneration],
text_generation_supports_vision: false,
text_generation_supports_tools: false,
},
}
}
@@ -192,4 +201,5 @@ pub struct AgentProviderInfo {
pub models_list_url: Option<&'static str>,
pub supported_purposes: Vec<AgentPurpose>,
pub text_generation_supports_vision: bool,
pub text_generation_supports_tools: bool,
}

View File

@@ -2,7 +2,7 @@ use mxlink::mime;
#[derive(Default)]
pub struct ImageGenerationParams {
pub size_override: Option<String>,
pub smallest_size_possible: bool,
pub cheaper_model_switching_allowed: bool,
@@ -10,8 +10,8 @@ pub struct ImageGenerationParams {
}
impl ImageGenerationParams {
pub fn with_size_override(mut self, value: Option<String>) -> Self {
self.size_override = value;
pub fn with_smallest_size_possible(mut self, value: bool) -> Self {
self.smallest_size_possible = value;
self
}

View File

@@ -64,6 +64,9 @@ pub struct TextGenerationConfig {
#[serde(default)]
pub max_context_tokens: u32,
#[serde(default)]
pub tools: ToolsConfig,
}
impl Default for TextGenerationConfig {
@@ -75,6 +78,7 @@ impl Default for TextGenerationConfig {
max_response_tokens: None,
max_completion_tokens: Some(128_000),
max_context_tokens: 400_000,
tools: ToolsConfig::default(),
}
}
}
@@ -83,6 +87,15 @@ fn default_text_model_id() -> String {
"gpt-5.2".to_owned()
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ToolsConfig {
#[serde(default)]
pub web_search: bool,
#[serde(default)]
pub code_interpreter: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SpeechToTextConfig {
#[serde(default = "default_speech_to_text_model_id")]

View File

@@ -5,10 +5,13 @@ use async_openai::{
config::OpenAIConfig,
types::{
audio::{AudioInput, CreateSpeechRequestArgs, CreateTranscriptionRequestArgs},
chat::{ChatCompletionRequestMessage, CreateChatCompletionRequestArgs},
images::{
CreateImageEditRequestArgs, CreateImageRequestArgs,
Image, ImageInput, ImageModel, ImageResponseFormat,
CreateImageEditRequestArgs, CreateImageRequestArgs, Image, ImageInput, ImageModel,
ImageResponseFormat,
},
responses::{
CodeInterpreterContainerAuto, CodeInterpreterTool, CodeInterpreterToolContainer,
CreateResponseArgs, OutputItem, OutputMessageContent, Tool, WebSearchTool,
},
},
};
@@ -28,12 +31,9 @@ use crate::{
use crate::{
agent::{
AgentPurpose,
provider::{
entity::{
ImageEditResult, ImageGenerationResult, ImageSource, PingResult,
TextToSpeechParams, TextToSpeechResult,
},
openai::utils::convert_string_to_enum,
provider::entity::{
ImageEditResult, ImageGenerationResult, ImageSource, PingResult, TextToSpeechParams,
TextToSpeechResult,
},
},
strings,
@@ -129,28 +129,45 @@ impl ControllerTrait for Controller {
conversation_messages.insert(0, prompt_message);
}
let openai_conversation_messages: Vec<ChatCompletionRequestMessage> =
super::utils::convert_llm_messages_to_openai_messages(conversation_messages);
let input =
super::utils::convert_llm_messages_to_openai_response_input(conversation_messages);
let messages_count = openai_conversation_messages.len();
let messages_count = match &input {
async_openai::types::responses::InputParam::Items(items) => items.len(),
_ => 1,
};
let temperature = params
.temperature_override
.unwrap_or(text_generation_config.temperature);
let mut request_builder = CreateChatCompletionRequestArgs::default();
let mut request_builder = CreateResponseArgs::default();
request_builder
.model(&text_generation_config.model_id)
.temperature(temperature)
.messages(openai_conversation_messages);
.input(input);
if let Some(max_response_tokens) = text_generation_config.max_response_tokens {
request_builder.max_tokens(max_response_tokens);
let mut tools = Vec::new();
if text_generation_config.tools.web_search {
tools.push(Tool::WebSearch(WebSearchTool::default()));
}
if text_generation_config.tools.code_interpreter {
tools.push(Tool::CodeInterpreter(CodeInterpreterTool {
container: CodeInterpreterToolContainer::Auto(
CodeInterpreterContainerAuto::default(),
),
}));
}
if let Some(max_completion_tokens) = text_generation_config.max_completion_tokens {
request_builder.max_completion_tokens(max_completion_tokens);
if !tools.is_empty() {
request_builder.tools(tools);
}
if let Some(max_response_tokens) = text_generation_config.max_response_tokens {
request_builder.max_output_tokens(max_response_tokens);
} else if let Some(max_completion_tokens) = text_generation_config.max_completion_tokens {
request_builder.max_output_tokens(max_completion_tokens);
}
let request = request_builder.build()?;
@@ -160,33 +177,28 @@ impl ControllerTrait for Controller {
model = format!("{:?}", request.model),
?messages_count,
request = request_as_json,
"Sending OpenAI chat completion API request"
"Sending OpenAI response API request"
);
}
let response = self.client.chat().create(request).await?;
let response = self.client.responses().create(request).await?;
tracing::trace!(
?response,
"Got response from the OpenAI chat completion API"
);
tracing::trace!(?response, "Got response from the OpenAI response API");
// We only request 1 result, so there should only be 1 choice.
if let Some(choice) = response.choices.into_iter().next() {
match choice.message.content {
Some(text) => {
return Ok(TextGenerationResult { text });
}
None => {
return Err(anyhow::anyhow!(
"No content was found in the response choice from the OpenAI chat completion API"
));
for item in response.output {
if let OutputItem::Message(message) = item {
for content in message.content {
if let OutputMessageContent::OutputText(text_content) = content {
return Ok(TextGenerationResult {
text: text_content.text,
});
}
}
}
}
Err(anyhow::anyhow!(
"No response messages choices were returned from the OpenAI chat completion API"
"No response messages choices were returned from the OpenAI response API"
))
}
@@ -257,9 +269,7 @@ impl ControllerTrait for Controller {
ImageModel::GptImage1 => ImageModel::GptImage1Mini,
ImageModel::GptImage1dot5 => ImageModel::GptImage1Mini,
ImageModel::GptImage1Mini => ImageModel::GptImage1Mini,
ImageModel::Other(_) => {
ImageModel::DallE2
}
ImageModel::Other(_) => ImageModel::DallE2,
}
} else {
original_model
@@ -295,10 +305,11 @@ impl ControllerTrait for Controller {
image_generation_config.quality.clone()
};
let size = params
.size_override
.map(|s| convert_string_to_enum::<async_openai::types::images::ImageSize>(&s).unwrap())
.or(image_generation_config.size);
let size = if params.smallest_size_possible {
Some(get_sticker_size(&model))
} else {
image_generation_config.size
};
let response_format = match model.clone() {
ImageModel::DallE2 => Some(ImageResponseFormat::B64Json),
@@ -393,9 +404,15 @@ impl ControllerTrait for Controller {
}
let dalle2_size = match image_generation_config.size {
Some(async_openai::types::images::ImageSize::S256x256) => Some(async_openai::types::images::ImageSize::S256x256),
Some(async_openai::types::images::ImageSize::S512x512) => Some(async_openai::types::images::ImageSize::S512x512),
Some(async_openai::types::images::ImageSize::S1024x1024) => Some(async_openai::types::images::ImageSize::S1024x1024),
Some(async_openai::types::images::ImageSize::S256x256) => {
Some(async_openai::types::images::ImageSize::S256x256)
}
Some(async_openai::types::images::ImageSize::S512x512) => {
Some(async_openai::types::images::ImageSize::S512x512)
}
Some(async_openai::types::images::ImageSize::S1024x1024) => {
Some(async_openai::types::images::ImageSize::S1024x1024)
}
_ => None,
};
@@ -404,12 +421,8 @@ impl ControllerTrait for Controller {
.map_err(|err| anyhow::anyhow!(err))?;
let response_format = match model.clone() {
ImageModel::DallE2 => {
Some(ImageResponseFormat::B64Json)
}
ImageModel::DallE3 => {
Some(ImageResponseFormat::B64Json)
}
ImageModel::DallE2 => Some(ImageResponseFormat::B64Json),
ImageModel::DallE3 => Some(ImageResponseFormat::B64Json),
// gpt-image-1 only outputs base64 and we don't need to specify the response format.
// In fact, specifying the response format results in an error.
ImageModel::GptImage1 => None,
@@ -621,3 +634,17 @@ fn audio_mime_type_to_file_name(mime_type: &mxlink::mime::Mime) -> Option<String
Some(format!("audio.{}", file_extension))
}
/// Returns the smallest supported size for stickers based on what the image model supports.
fn get_sticker_size(model: &ImageModel) -> async_openai::types::images::ImageSize {
use async_openai::types::images::ImageSize;
match model {
ImageModel::DallE2 => ImageSize::S256x256,
ImageModel::DallE3 => ImageSize::S1024x1024,
ImageModel::GptImage1 => ImageSize::S1024x1024,
ImageModel::GptImage1Mini => ImageSize::S1024x1024,
ImageModel::GptImage1dot5 => ImageSize::S1024x1024,
ImageModel::Other(_) => ImageSize::S1024x1024,
}
}

View File

@@ -20,12 +20,12 @@ pub const OPENAI_IMAGE_MODEL_GPT_IMAGE_1_DOT_5: &str = "gpt-image-1.5";
pub fn create_controller_from_yaml_value_config(
agent_id: &str,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
) -> AgentInstantiationResult<ControllerType> {
let config = match &config {
serde_yaml::Value::Mapping(_) => {
serde_yaml_ng::Value::Mapping(_) => {
let config: Config =
serde_yaml::from_value(config).map_err(AgentInstantiationError::Yaml)?;
serde_yaml_ng::from_value(config).map_err(AgentInstantiationError::Yaml)?;
config
.validate()

View File

@@ -1,12 +1,6 @@
use async_openai::types::{
chat::{
ChatCompletionRequestAssistantMessageArgs, ChatCompletionRequestMessage,
ChatCompletionRequestMessageContentPartImage,
ChatCompletionRequestSystemMessageArgs,
ChatCompletionRequestUserMessageArgs, ChatCompletionRequestUserMessageContent,
ChatCompletionRequestUserMessageContentPart,
ImageUrlArgs,
},
use async_openai::types::responses::{
EasyInputContent, EasyInputMessage, ImageDetail, InputContent, InputImageContent, InputItem,
InputParam, MessageType, Role,
};
use crate::conversation::llm::{
@@ -14,93 +8,41 @@ use crate::conversation::llm::{
};
use crate::utils::base64::base64_encode;
pub fn convert_llm_messages_to_openai_messages(
pub fn convert_llm_messages_to_openai_response_input(
conversation_messages: Vec<LLMMessage>,
) -> Vec<ChatCompletionRequestMessage> {
let mut openai_conversation_messages: Vec<ChatCompletionRequestMessage> =
Vec::with_capacity(conversation_messages.len());
) -> InputParam {
let mut items = Vec::with_capacity(conversation_messages.len());
for message in conversation_messages {
let openai_message = convert_llm_message_to_openai_message(message);
if let Some(openai_message) = openai_message {
openai_conversation_messages.push(openai_message);
}
}
let role = match message.author {
LLMAuthor::Prompt => Role::System,
LLMAuthor::Assistant => Role::Assistant,
LLMAuthor::User => Role::User,
};
openai_conversation_messages
}
let content = match message.content {
LLMMessageContent::Text(text) => EasyInputContent::Text(text),
LLMMessageContent::Image(image_details) => {
let image_url = format!(
"data:{};base64,{}",
image_details.mime,
base64_encode(&image_details.data)
);
fn convert_llm_message_to_openai_message(
llm_message: LLMMessage,
) -> Option<ChatCompletionRequestMessage> {
match &llm_message.content {
LLMMessageContent::Text(text) => Some(match llm_message.author {
LLMAuthor::Prompt => ChatCompletionRequestSystemMessageArgs::default()
.content(text.clone())
.build()
.expect("Failed building OpenAI system message")
.into(),
LLMAuthor::Assistant => ChatCompletionRequestAssistantMessageArgs::default()
.content(text.clone())
.build()
.expect("Failed building OpenAI assistant message")
.into(),
LLMAuthor::User => ChatCompletionRequestUserMessageArgs::default()
.content(text.clone())
.build()
.expect("Failed building OpenAI user message")
.into(),
}),
LLMMessageContent::Image(image_details) => {
let image_url = format!(
"data:{};base64,{}",
image_details.mime,
base64_encode(&image_details.data)
);
let part = ChatCompletionRequestUserMessageContentPart::ImageUrl(
ChatCompletionRequestMessageContentPartImage {
image_url: ImageUrlArgs::default()
.url(image_url)
.build()
.expect("Failed building OpenAI image url"),
},
);
let message_content = ChatCompletionRequestUserMessageContent::Array(vec![part]);
match llm_message.author {
LLMAuthor::User => Some(
ChatCompletionRequestUserMessageArgs::default()
.content(message_content)
.build()
.expect("Failed building OpenAI user message")
.into(),
),
_ => {
tracing::warn!(
"OpenAI API does not support image content for messages authored by {:?}. This message part will be skipped.",
llm_message.author
);
None
}
EasyInputContent::ContentList(vec![InputContent::InputImage(InputImageContent {
image_url: Some(image_url),
detail: ImageDetail::Auto,
file_id: None,
})])
}
}
}
}
};
pub(super) fn convert_string_to_enum<T>(value: &str) -> Result<T, String>
where
T: serde::de::DeserializeOwned,
{
// This is a hacky way to construct an enum from the string we have.
let enum_result: serde_json::Result<T> = serde_json::from_str(&format!("\"{}\"", value));
match enum_result {
Ok(enum_result) => Ok(enum_result),
Err(err) => {
tracing::debug!(?err, "Failed to parse into enum");
Err(format!("The value ({}) is not supported.", value))
}
items.push(InputItem::EasyMessage(EasyInputMessage {
r#type: MessageType::Message,
role,
content,
}));
}
InputParam::Items(items)
}

View File

@@ -95,6 +95,7 @@ impl TryInto<OpenAITextGenerationConfig> for TextGenerationConfig {
max_response_tokens: self.max_response_tokens,
max_completion_tokens: None,
max_context_tokens: self.max_context_tokens,
tools: Default::default(),
})
}
}
@@ -161,13 +162,14 @@ impl TryInto<OpenAITextToSpeechConfig> for TextToSpeechConfig {
type Error = String;
fn try_into(self) -> Result<OpenAITextToSpeechConfig, Self::Error> {
let model_id = convert_string_to_enum::<async_openai::types::audio::SpeechModel>(&self.model_id)?;
let model_id =
convert_string_to_enum::<async_openai::types::audio::SpeechModel>(&self.model_id)?;
let voice = convert_string_to_enum::<async_openai::types::audio::Voice>(&self.voice)?;
let response_format = convert_string_to_enum::<async_openai::types::audio::SpeechResponseFormat>(
&self.response_format,
)?;
let response_format = convert_string_to_enum::<
async_openai::types::audio::SpeechResponseFormat,
>(&self.response_format)?;
Ok(OpenAITextToSpeechConfig {
model_id,
@@ -224,25 +226,25 @@ impl TryInto<OpenAIImageGenerationConfig> for ImageGenerationConfig {
fn try_into(self) -> Result<OpenAIImageGenerationConfig, Self::Error> {
let size = if let Some(size) = &self.size {
Some(convert_string_to_enum::<async_openai::types::images::ImageSize>(
size,
)?)
Some(convert_string_to_enum::<
async_openai::types::images::ImageSize,
>(size)?)
} else {
None
};
let style = if let Some(style) = &self.style {
Some(convert_string_to_enum::<async_openai::types::images::ImageStyle>(
style,
)?)
Some(convert_string_to_enum::<
async_openai::types::images::ImageStyle,
>(style)?)
} else {
None
};
let quality = if let Some(quality) = &self.quality {
Some(convert_string_to_enum::<async_openai::types::images::ImageQuality>(
quality,
)?)
Some(convert_string_to_enum::<
async_openai::types::images::ImageQuality,
>(quality)?)
} else {
None
};

View File

@@ -3,6 +3,8 @@ use etke_openai_api_rust::chat::{ChatApi, ChatBody};
use etke_openai_api_rust::images::{ImagesApi, ImagesBody};
use etke_openai_api_rust::{Auth, Message, OpenAI};
const SMALLEST_IMAGE_SIZE: &str = "256x256";
use super::super::ControllerTrait;
use crate::utils::base64::base64_decode;
use crate::{
@@ -303,9 +305,11 @@ impl ControllerTrait for Controller {
// when they span multiple lines.
let prompt = prompt.replace("\n", " ");
let size: Option<String> = params
.size_override
.or_else(|| image_generation_config.size.clone());
let size: Option<String> = if params.smallest_size_possible {
Some(SMALLEST_IMAGE_SIZE.to_owned())
} else {
image_generation_config.size.clone()
};
let request = ImagesBody {
model: Some(image_generation_config.model_id.to_owned()),

View File

@@ -26,12 +26,12 @@ use super::controller::ControllerType;
pub fn create_controller_from_yaml_value_config(
agent_id: &str,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
) -> AgentInstantiationResult<ControllerType> {
let config = match &config {
serde_yaml::Value::Mapping(_) => {
serde_yaml_ng::Value::Mapping(_) => {
let config: Config =
serde_yaml::from_value(config).map_err(AgentInstantiationError::Yaml)?;
serde_yaml_ng::from_value(config).map_err(AgentInstantiationError::Yaml)?;
config
.validate()

View File

@@ -4,10 +4,10 @@ use std::{future::Future, pin::Pin};
use mxlink::matrix_sdk::Room;
use mxlink::matrix_sdk::media::{MediaFormat, MediaRequestParameters};
use mxlink::matrix_sdk::ruma::api::client::profile::{AvatarUrl, DisplayName};
use mxlink::matrix_sdk::ruma::{
MilliSecondsSinceUnixEpoch, OwnedUserId, events::room::MediaSource,
};
use mxlink::matrix_sdk::ruma::api::client::profile::{AvatarUrl, DisplayName};
use mxlink::{
InitConfig, LoginConfig, LoginCredentials, LoginEncryption, MatrixLink, PersistenceConfig,

View File

@@ -21,7 +21,7 @@ pub fn load() -> anyhow::Result<Config> {
}
let config_str = std::fs::read_to_string(config_file_path)?;
let mut config: Config = serde_yaml::from_str(&config_str)?;
let mut config: Config = serde_yaml_ng::from_str(&config_str)?;
// Allow environment variables to override some configuration keys
for (key, value) in env::vars() {

View File

@@ -28,18 +28,18 @@ pub async fn handle_set(
message_context: &MessageContext,
patterns: &Option<Vec<String>>,
) -> anyhow::Result<()> {
if let Some(patterns) = patterns {
if let Err(err) = mxidwc::parse_patterns_vector(patterns) {
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
&strings::access::failed_to_parse_patterns(&err.to_string()),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
if let Some(patterns) = patterns
&& let Err(err) = mxidwc::parse_patterns_vector(patterns)
{
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
&strings::access::failed_to_parse_patterns(&err.to_string()),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
return Ok(());
}
return Ok(());
}
let mut global_config_manager_guard = bot.global_config_manager().lock().await;

View File

@@ -24,18 +24,18 @@ pub async fn handle_set(
message_context: &MessageContext,
patterns: &Option<Vec<String>>,
) -> anyhow::Result<()> {
if let Some(patterns) = patterns {
if let Err(err) = mxidwc::parse_patterns_vector(patterns) {
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
&strings::access::failed_to_parse_patterns(&err.to_string()),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
if let Some(patterns) = patterns
&& let Err(err) = mxidwc::parse_patterns_vector(patterns)
{
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
&strings::access::failed_to_parse_patterns(&err.to_string()),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
return Ok(());
}
return Ok(());
}
let mut global_config_manager_guard = bot.global_config_manager().lock().await;

View File

@@ -15,7 +15,7 @@ use crate::{Bot, entity::MessageContext};
struct ParsedAgentConfig {
agent: AgentInstance,
config: serde_yaml::Value,
config: serde_yaml_ng::Value,
}
pub async fn handle_room_local(
@@ -250,7 +250,7 @@ async fn send_guide(
provider: &AgentProvider,
) -> anyhow::Result<()> {
let sample_config = crate::agent::default_config_for_provider(provider);
let sample_config_pretty_yaml = serde_yaml::to_string(&sample_config)?;
let sample_config_pretty_yaml = serde_yaml_ng::to_string(&sample_config)?;
bot.messaging()
.send_text_markdown_no_fail(
@@ -263,7 +263,7 @@ async fn send_guide(
Ok(())
}
fn parse_from_message_to_yaml_value(text: &str) -> Result<serde_yaml::Value, String> {
fn parse_from_message_to_yaml_value(text: &str) -> Result<serde_yaml_ng::Value, String> {
let mut text = text.trim();
if text.starts_with("```") {
@@ -274,10 +274,10 @@ fn parse_from_message_to_yaml_value(text: &str) -> Result<serde_yaml::Value, Str
text = text.trim_end_matches("```");
}
let config: serde_yaml::Value = serde_yaml::from_str(text).map_err(|e| e.to_string())?;
let config: serde_yaml_ng::Value = serde_yaml_ng::from_str(text).map_err(|e| e.to_string())?;
match config {
serde_yaml::Value::Mapping(_) => {}
serde_yaml_ng::Value::Mapping(_) => {}
_ => {
return Err("Not a valid YAML hashmap".to_owned());
}

View File

@@ -2,12 +2,12 @@
fn agent_config_parsing_works() {
struct TestCase {
input: String,
expected: Option<serde_yaml::Value>,
expected: Option<serde_yaml_ng::Value>,
}
let provider = crate::agent::AgentProvider::OpenAI;
let sample_config = crate::agent::default_config_for_provider(&provider);
let sample_config_pretty_yaml = serde_yaml::to_string(&sample_config).unwrap();
let sample_config_pretty_yaml = serde_yaml_ng::to_string(&sample_config).unwrap();
let test_cases = vec![
// Invalid input

View File

@@ -64,7 +64,7 @@ pub async fn handle(
PublicIdentifier::Static(_) => {}
};
let config_yaml_pretty = serde_yaml::to_string(&agent.definition().config)?;
let config_yaml_pretty = serde_yaml_ng::to_string(&agent.definition().config)?;
bot.messaging()
.send_text_markdown_no_fail(

View File

@@ -39,18 +39,18 @@ async fn dispatch_config_related_handler(
message_context: &MessageContext,
bot: &Bot,
) -> anyhow::Result<()> {
if let SettingsStorageSource::Global = config_type {
if !message_context.sender_can_manage_global_config() {
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
strings::global_config::no_permissions_to_administrate(),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
return Ok(());
}
};
if let SettingsStorageSource::Global = config_type
&& !message_context.sender_can_manage_global_config()
{
bot.messaging()
.send_error_markdown_no_fail(
message_context.room(),
strings::global_config::no_permissions_to_administrate(),
MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
)
.await;
return Ok(());
}
let room_settings = match config_type {
SettingsStorageSource::Room => &message_context.room_config().settings,

View File

@@ -12,9 +12,6 @@ use crate::strings;
use crate::utils::mime::get_file_extension;
use crate::{Bot, entity::MessageContext};
// We may make this configurable (per room, etc.) in the future, but for now it's hardcoded.
const STICKER_SIZE: &str = "256x256";
pub async fn handle_image(
bot: &Bot,
matrix_link: MatrixLink,
@@ -177,7 +174,7 @@ pub async fn handle_sticker(
);
let params = ImageGenerationParams::default()
.with_size_override(Some(STICKER_SIZE.to_owned()))
.with_smallest_size_possible(true)
.with_cheaper_model_switching_allowed(true)
.with_cheaper_quality_switching_allowed(true);

View File

@@ -154,35 +154,35 @@ pub async fn process_matrix_messages(
let mut message = message.clone();
if i == 0 && !params.first_message_prefixes_to_strip.is_empty() {
if let MatrixMessageContent::Text(message_text) = &message.content {
let mut message_text = message_text.clone();
if i == 0
&& !params.first_message_prefixes_to_strip.is_empty()
&& let MatrixMessageContent::Text(message_text) = &message.content
{
let mut message_text = message_text.clone();
for prefix in &params.first_message_prefixes_to_strip {
if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
message_text = message_text_stripped.to_owned();
}
for prefix in &params.first_message_prefixes_to_strip {
if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
message_text = message_text_stripped.to_owned();
}
message.content = MatrixMessageContent::Text(message_text.trim().to_owned());
}
message.content = MatrixMessageContent::Text(message_text.trim().to_owned());
}
// We only strip `bot_user_prefixes_to_strip`-defined prefixes from messages that mention the bot user.
if !params.bot_user_prefixes_to_strip.is_empty()
&& message.mentioned_users.contains(&params.bot_user_id)
&& let MatrixMessageContent::Text(message_text) = &message.content
{
if let MatrixMessageContent::Text(message_text) = &message.content {
let mut message_text = message_text.clone();
let mut message_text = message_text.clone();
for prefix in &params.bot_user_prefixes_to_strip {
if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
message_text = message_text_stripped.to_owned();
}
for prefix in &params.bot_user_prefixes_to_strip {
if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
message_text = message_text_stripped.to_owned();
}
message.content = MatrixMessageContent::Text(message_text.trim().to_owned());
}
message.content = MatrixMessageContent::Text(message_text.trim().to_owned());
}
messages_filtered.push(message);

View File

@@ -88,9 +88,10 @@ impl ConfigHomeserver {
/// - `Default`: Use the built-in default avatar (null, empty string, or missing in config)
/// - `Keep`: Don't touch the avatar, keep whatever is already set ("keep" in config)
/// - `Custom(String)`: Use a custom avatar from the specified file path
#[derive(Debug, Clone, PartialEq, Serialize)]
#[derive(Debug, Clone, Default, PartialEq, Serialize)]
pub enum Avatar {
/// Use the built-in default avatar
#[default]
Default,
/// Keep the current avatar, don't change it
Keep,
@@ -98,12 +99,6 @@ pub enum Avatar {
Custom(String),
}
impl Default for Avatar {
fn default() -> Self {
Avatar::Default
}
}
impl<'de> Deserialize<'de> for Avatar {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
@@ -181,13 +176,13 @@ pub struct ConfigUserEncryption {
impl ConfigUserEncryption {
pub fn validate(&self) -> anyhow::Result<()> {
if let Some(passphrase) = &self.recovery_passphrase {
if passphrase.is_empty() {
return Err(anyhow::anyhow!(
"The user.encryption.recovery_passphrase ({}) configuration must either be null or set to a non-empty passphrase",
super::env::BAIBOT_USER_ENCRYPTION_RECOVERY_PASSPHRASE
));
}
if let Some(passphrase) = &self.recovery_passphrase
&& passphrase.is_empty()
{
return Err(anyhow::anyhow!(
"The user.encryption.recovery_passphrase ({}) configuration must either be null or set to a non-empty passphrase",
super::env::BAIBOT_USER_ENCRYPTION_RECOVERY_PASSPHRASE
));
}
Ok(())

View File

@@ -109,11 +109,21 @@ pub fn help_provider_details(id: &str, info: &AgentProviderInfo) -> String {
let mut purpose_line = format!("{} {}", purpose.emoji(), purpose.as_str());
if let AgentPurpose::TextGeneration = purpose {
let mut extras = vec![];
if info.text_generation_supports_vision {
purpose_line = format!("{} ({})", purpose_line, "incl. vision");
extras.push("incl. vision");
} else {
purpose_line = format!("{} ({})", purpose_line, "no vision");
extras.push("no vision");
}
if info.text_generation_supports_tools {
extras.push("incl. tools");
} else {
extras.push("no tools");
}
purpose_line = format!("{} ({})", purpose_line, extras.join(", "));
}
capabilities.push(purpose_line);

View File

@@ -64,7 +64,7 @@ To create a sticker, send a command like `%command_prefix% sticker A huge bowl o
The difference from **creating images** is that the bot will:
- create a smaller-resolution image (`256x256`) - smaller/quicker, but still good enough for a sticker
- create a smaller-resolution image (as small as the model allows) - smaller/quicker, but still good enough for a sticker
- potentially switch to a different (cheaper or otherwise more suitable) model, if available
- post the image directly to the room (as a reply to your message), without starting a threaded conversation
"#;