Compare commits

...

46 Commits

Author SHA1 Message Date
Slavi Pantaleev
2455117e41 Prepare 1.16.1 release 2026-03-24 14:05:35 +02:00
Slavi Pantaleev
3e9c110afc Update dependencies 2026-03-24 14:04:40 +02:00
Slavi Pantaleev
d9b5524c97 Fix OpenAI response input for async-openai 0.34
async-openai 0.34 adds a required phase field to EasyInputMessage,
which broke our Responses API request construction.

Set phase to None because baibot does not currently model assistant commentary vs final-answer turns,
so omitting phase preserves the previous behavior while keeping the request compatible with the new crate.
2026-03-24 12:51:56 +02:00
renovate[bot]
9b169a7d28 Update Rust crate async-openai to 0.34.0 2026-03-24 12:40:25 +02:00
Slavi Pantaleev
cb29419d75 Prepare 1.16.0 release
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:25:52 +02:00
Slavi Pantaleev
51ca8c9948 Update dependencies 2026-03-20 12:23:51 +02:00
Slavi Pantaleev
12b938d2d1 Skip files with application/octet-stream MIME type
Files with unrecognized MIME types (application/octet-stream) are
not supported by any LLM provider and would cause errors that
permanently break the conversation thread. Instead, represent them
as a text message describing the attachment so the LLM is still
aware a file was sent without the thread becoming unusable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:22:38 +02:00
Slavi Pantaleev
f3d1b32ad7 Use mime_guess for file extension MIME type detection
Replace the hand-maintained extension-to-MIME mapping with the
mime_guess crate, which was already in the dependency tree.
This covers hundreds of file extensions out of the box.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:22:38 +02:00
Slavi Pantaleev
3d3bd3c9f9 Add support for file attachments (m.file) in conversations
Files sent as m.file Matrix messages are now downloaded, MIME-detected,
and forwarded to LLM providers alongside the conversation context,
similar to how m.image is already handled.

- OpenAI provider: sends files inline as base64 data URLs
- Anthropic provider: skips files with a warning (library limitation)
- OpenAI-compat provider: skips files with a warning (library limitation)

Controller routing respects the existing prefix requirement setting.
MIME detection expanded to cover PDF, text, code, and document formats.
Docs updated to reflect file support and known limitations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:22:38 +02:00
renovate[bot]
a25845e89e Update Rust crate quick_cache to v0.6.21 2026-03-19 19:22:00 +02:00
renovate[bot]
479f54b93d Update docker.io/ollama/ollama Docker tag to v0.18.2 2026-03-19 08:15:14 +02:00
renovate[bot]
90ab6807ac Update Rust crate quick_cache to v0.6.20 2026-03-18 09:15:53 +02:00
renovate[bot]
5022f79bf5 Update docker.io/ollama/ollama Docker tag to v0.18.1 2026-03-17 07:25:56 +02:00
renovate[bot]
2ba3b5a437 Update docker.io/ollama/ollama Docker tag to v0.18.0 2026-03-14 22:34:33 +02:00
renovate[bot]
2819011b9d Update Rust crate tracing-subscriber to v0.3.23 2026-03-13 20:44:43 +02:00
renovate[bot]
ede9065f77 Update Rust crate async-openai to v0.33.1 2026-03-13 11:54:12 +02:00
renovate[bot]
5e61f5b3a3 Update ghcr.io/element-hq/synapse Docker tag to v1.149.1 2026-03-11 20:05:48 +02:00
renovate[bot]
8633e82f62 Update Rust crate quick_cache to v0.6.19 2026-03-11 20:05:32 +02:00
Slavi Pantaleev
6aa3d70d57 Bump default OpenAI text-generation model (gpt-5.2 -> gpt-5.4) 2026-03-11 09:53:02 +02:00
renovate[bot]
75e29ee3fb Update Rust crate tempfile to 3.27.* 2026-03-11 08:50:06 +02:00
renovate[bot]
eab7978b9f Update ghcr.io/element-hq/element-web Docker tag to v1.12.12 2026-03-10 22:06:39 +02:00
renovate[bot]
765e7c17b2 Update ghcr.io/element-hq/synapse Docker tag to v1.149.0 2026-03-10 20:19:06 +02:00
Slavi Pantaleev
748d2b7fd4 Prepare 1.15.0 release
Add the 1.15.0 changelog entry covering access-token authentication support, Rust 1.93.0 toolchain pinning, docs updates, and dependency updates. Bump crate version to 1.15.0 in Cargo.toml and Cargo.lock.
2026-03-07 11:13:42 +02:00
Slavi Pantaleev
527759dd02 Document Matrix authentication modes
Add a dedicated configuration/authentication doc covering password and access-token setup,
including environment-variable mappings and token-generation example.

Link to it from configuration docs and align the sample config wording with Matrix Authentication Service/OIDC terminology.
2026-03-07 11:08:50 +02:00
Slavi Pantaleev
3290255bad Pin local Rust toolchain to 1.93.0
Add rust-toolchain.toml so local development and ad-hoc cargo commands use the same known-good compiler version as CI.
This avoids the matrix-sdk query-depth overflow seen on newer stable toolchains.

Related to 9b987395b3

Ref: https://github.com/etkecc/baibot/pull/83#issuecomment-4008463792
2026-03-07 10:40:56 +02:00
Slavi Pantaleev
9b987395b3 Pin GitHub CI Rust toolchain to 1.93.0
Using the floating stable toolchain currently breaks this project. Repro via just build-debug:

```
Compiling matrix-sdk v0.16.0
error: queries overflow the depth limit!
help: consider increasing the recursion limit by adding #![recursion_limit = "256"] to your crate (matrix_sdk)
note: query depth increased by 130 when computing layout of matrix-sdk Client::sync async body
error: could not compile matrix-sdk (lib) due to 1 previous error
```

Pinning CI to 1.93.0 keeps CI on the known-good toolchain until upstream/toolchain compatibility is addressed.
We should have had that to begin with (we're pinning as much as we can anyway), but..

Ref: https://github.com/etkecc/baibot/pull/83#issuecomment-4008463792
2026-03-07 10:38:35 +02:00
Taylor Southwick
4852d1fe92 Add support for access tokens using MAS (#83)
* Add support for access tokens using MAS

* use 1.13.0

* Update dependencies

* Harden auth credential selection in matrix link init

Use the same non-empty access-token criterion for auth mode selection and bind the token directly from the branch condition.
Return explicit configuration errors for missing or empty `device_id`/`password` instead of panicking, so invalid auth config fails gracefully.

* Centralize and harden user auth config handling

Move authentication-mode resolution into typed config parsing with ConfigUserAuth,
so downstream login setup consumes validated credentials instead of re-checking raw optional fields.

Enforce explicit password-vs-token selection, validate token/device/user-id requirements in one place,
and normalize empty auth env overrides to unset values for consistent behavior across YAML and environment input.

* Add auth config unit tests

Move auth_config tests into a dedicated cfg test module file to keep production config code compact while preserving behavior coverage. The tests cover password/token mode selection, missing/both auth method rejection, missing device_id, and empty-value handling.

* Use conventional mxlink version requirement

Replace the unconventional wildcard lower-bound expression with a standard semver lower bound for readability and tooling consistency.

---------

Co-authored-by: Slavi Pantaleev <slavi@devture.com>
2026-03-07 10:26:40 +02:00
renovate[bot]
8bd313f0d4 Update docker/build-push-action action to v7 2026-03-06 10:15:37 +02:00
renovate[bot]
711e1099d6 Update docker.io/ollama/ollama Docker tag to v0.17.7 2026-03-06 08:16:12 +02:00
renovate[bot]
91c8dd8f7d Update docker/metadata-action action to v6 2026-03-05 22:22:11 +02:00
renovate[bot]
afc5572d6a Update docker/login-action action to v4 2026-03-04 17:13:28 +02:00
renovate[bot]
73e13dcf2f Update docker.io/ollama/ollama Docker tag to v0.17.6 2026-03-04 07:36:04 +02:00
renovate[bot]
2bebd109b1 Update forgejo.ellis.link/continuwuation/continuwuity Docker tag to v0.5.6 2026-03-04 07:35:22 +02:00
renovate[bot]
7f7c58be1f Update Rust crate tokio to 1.50.* 2026-03-03 16:27:53 +02:00
renovate[bot]
47e5a464a0 Update docker.io/ollama/ollama Docker tag to v0.17.5 2026-03-01 08:09:47 +02:00
renovate[bot]
85f751e514 Update docker.io/ollama/ollama Docker tag to v0.17.4 2026-02-27 07:08:48 +02:00
renovate[bot]
95acad3558 Update docker.io/postgres Docker tag to v18.3 2026-02-27 06:36:26 +02:00
renovate[bot]
304056c59a Update docker.io/ollama/ollama Docker tag to v0.17.2 2026-02-27 06:36:19 +02:00
renovate[bot]
bedc0335f1 Update docker.io/ollama/ollama Docker tag to v0.17.1 2026-02-26 13:33:16 +02:00
renovate[bot]
a8be8c3c1e Update ghcr.io/element-hq/element-web Docker tag to v1.12.11 2026-02-24 16:54:27 +02:00
renovate[bot]
826fa728a9 Update ghcr.io/element-hq/synapse Docker tag to v1.148.0 2026-02-24 16:53:10 +02:00
renovate[bot]
5aef8e8b2f Update Rust crate tempfile to 3.26.* 2026-02-24 08:21:45 +02:00
renovate[bot]
f70f20181e Update Rust crate chrono to v0.4.44 2026-02-24 08:16:54 +02:00
renovate[bot]
fcdd4f39ee Update docker.io/ollama/ollama Docker tag to v0.17.0 2026-02-24 08:16:31 +02:00
renovate[bot]
891adfec49 Update Rust crate anyhow to v1.0.102 2026-02-20 08:49:48 +02:00
renovate[bot]
35ab79844b Update docker.io/ollama/ollama Docker tag to v0.16.3 2026-02-20 08:49:37 +02:00
35 changed files with 684 additions and 236 deletions

View File

@@ -17,7 +17,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- uses: dtolnay/rust-toolchain@1.93.0
- name: Install SQLite3
run: sudo apt-get update && sudo apt-get install -y libsqlite3-dev
- run: cargo test --all-features
@@ -30,7 +30,7 @@ jobs:
steps:
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
with:
images: |
ghcr.io/${{ github.repository }}
@@ -58,14 +58,14 @@ jobs:
- name: Checkout
uses: actions/checkout@v6
- name: Log in to the GitHub Container registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
with:
tags: |
type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
@@ -77,7 +77,7 @@ jobs:
ghcr.io/${{ github.repository }}
- name: Build and push Docker images
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
@@ -95,7 +95,7 @@ jobs:
steps:
- name: Log in to the GitHub Container registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}

View File

@@ -1,3 +1,29 @@
# (2026-03-24) Version 1.16.1
- (**Bugfix**) Fix compatibility with [async-openai](https://crates.io/crates/async-openai) 0.34.0 by populating the new `phase` field required for OpenAI Responses API message inputs. baibot does not currently distinguish between assistant `commentary` and `final_answer` turns, so using `None` preserves the previous behavior while remaining compatible with the updated crate.
- (**Internal Improvement**) Dependency updates.
# (2026-03-20) Version 1.16.0
- (**Feature**) Add support for file attachments (`m.file` Matrix messages) in conversations. Files like PDFs, text documents, spreadsheets, code files, etc. are now downloaded and forwarded to the LLM alongside the conversation context, similar to how images (`m.image`) are already handled. See the [💬 Text Generation](./docs/features.md#-text-generation) documentation for details and known limitations.
- (**Improvement**) Use the [mime_guess](https://crates.io/crates/mime_guess) crate for MIME type detection from file extensions, replacing a hand-maintained mapping. This covers hundreds of file extensions out of the box.
# (2026-03-07) Version 1.15.0
- (**Feature**) Add support for authentication via access tokens (for [Matrix Authentication Service](https://github.com/element-hq/matrix-authentication-service)/OIDC-enabled homeservers) as an alternative to password authentication. See [🔐 Authentication](./docs/configuration/authentication.md) for setup details. Thanks to [Taylor Southwick](https://github.com/twsouthwick) for the contribution in [#83](https://github.com/etkecc/baibot/pull/83)!
- (**Internal Improvement**) Pin the Rust toolchain to `1.93.0` in both CI and local development to avoid `matrix-sdk` build failures on newer stable toolchains.
- (**Internal Improvement**) Documentation updates.
- (**Internal Improvement**) Dependency updates.
# (2026-02-18) Version 1.14.3
- (**Internal Improvement**) Add [Renovate](https://docs.renovatebot.com/) configuration for automated dependency updates

373
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -7,7 +7,7 @@ license = "AGPL-3.0-or-later"
readme = "README.md"
keywords = ["matrix", "chat", "bot", "AI", "LLM"]
include = ["/etc/assets/baibot-torso-768.png", "/src", "/README.md", "/CHANGELOG.md", "/LICENSE"]
version = "1.14.3"
version = "1.16.1"
edition = "2024"
[lib]
@@ -17,7 +17,7 @@ path = "src/lib.rs"
[dependencies]
anthropic = { git = "https://github.com/etkecc/anthropic-rs.git", branch = "fix-content-block-image" }
anyhow = "1.0.*"
async-openai = { version = "0.33.0", features = ["audio", "chat-completion", "image", "responses"] }
async-openai = { version = "0.34.0", features = ["audio", "chat-completion", "image", "responses"] }
base64 = "0.22.*"
chrono = { version = "0.4.*", default-features = false, features = ["std", "now"] }
# We'd rather not depend on this, but we cannot use the ruma-events EventContent macro without it.
@@ -25,16 +25,16 @@ chrono = { version = "0.4.*", default-features = false, features = ["std", "now"
matrix-sdk = { version = "0.16.0", default-features = false, features = ["native-tls"] }
mime_guess = "2.0.*"
mxidwc = "1.0.*"
mxlink = ">=1.12.0"
mxlink = ">=1.13.0"
etke_openai_api_rust = "0.1.*"
quick_cache = "0.6.*"
regex = "1.12.*"
serde = { version = "1.0.*", features = ["derive"], default-features = false }
serde_json = "1.0.*"
serde_yaml_ng = "0.10.*"
tempfile = "3.25.*"
tempfile = "3.27.*"
tiktoken-rs = { version = "0.9.*", default-features = false }
tokio = { version = "1.49.*", features = ["rt", "rt-multi-thread", "macros"] }
tokio = { version = "1.50.*", features = ["rt", "rt-multi-thread", "macros"] }
tracing = "0.1.*"
tracing-subscriber = { version = "0.3.*", features = ["env-filter"] }
url = "2.5.*"

View File

@@ -22,6 +22,8 @@ You can see the list of supported environment variables in the [🦀 src/entity/
> [!WARNING]
> The static configuration contains an `initial_global_config` key, which is used to populate the bot's global configuration (stored as [dynamic configuration](#dynamic-configuration)) the first time the bot starts. Modifying this subsequently will not have any effect. After initial global configuration creation, it's expected to be managed dynamically via chat commands.
For Matrix-account authentication setup, see [🔐 Authentication](./authentication.md).
### Dynamic configuration

View File

@@ -0,0 +1,23 @@
## 🔐 Authentication
baibot supports 2 authentication modes for the Matrix account (`user.*` keys in config).
Set **exactly one** mode. If both are set (or neither is set), startup validation fails.
### Password authentication
- Config key: `user.password`
- Environment variable: `BAIBOT_USER_PASSWORD`
### Access token authentication
- Config keys: `user.access_token` + `user.device_id`
- Environment variables: `BAIBOT_USER_ACCESS_TOKEN` + `BAIBOT_USER_DEVICE_ID`
Access-token authentication is useful for OIDC-enabled homeservers (e.g. those using [Matrix Authentication Service](https://github.com/element-hq/matrix-authentication-service)).
Example token-generation command:
```sh
mas-cli manage issue-compatibility-token <username> [device_id]
```

View File

@@ -8,7 +8,7 @@ You can also use **different models within the same room** (e.g. [💬 text-gene
The bot supports the following use-purposes:
- [💬 text-generation](../features.md#-text-generation): communicating with you via text (though certain models may "see" images as well)
- [💬 text-generation](../features.md#-text-generation): communicating with you via text (though certain models may also process images and files)
- [🦻 speech-to-text](../features.md#-speech-to-text): turning your voice messages into text
- [🗣️ text-to-speech](../features.md#️-text-to-speech): turning bot or users text messages into voice messages
- [🖌️ image-generation](../features.md#image-generation): generating images based on instructions

View File

@@ -8,7 +8,7 @@ You can also use **different models within the same room** (e.g. [💬 text-gene
The bot supports the following use-purposes:
- [💬 text-generation](#-text-generation): communicating with you via text (though certain models may "see" images as well)
- [💬 text-generation](#-text-generation): communicating with you via text (though certain models may also process images and files)
- [🦻 speech-to-text](#-speech-to-text): turning your voice messages into text
- [🗣️ text-to-speech](#%EF%B8%8F-text-to-speech): turning bot or users text messages into voice messages
- [🖌️ image-generation](#%EF%B8%8F-image-generation): generating images based on instructions
@@ -26,7 +26,7 @@ Text Generation is the bot's ability to **respond to users' messages with text**
![Screenshot of Text Generation - a user sends a message and the bot replies in a new conversation thread](./screenshots/text-generation.webp)
Some models also support vision, so you may be able to mix text and images in the same conversation.
Some models also support vision and document understanding, so you may be able to mix text, images, and files (PDFs, text documents, etc.) in the same conversation. Note that certain providers may not support all file types or may have issues with specific files (e.g. scanned/image-based PDFs). If a file is rejected by the provider, the conversation thread may become unusable — start a new thread to work around this.
In multi-user (group) rooms, to avoid disturbing the normal conversation between people, the bot is auto-configured to only respond to messages starting with the command prefix (`!bai`) or direct mentions via the [💬 Text Generation / 🗟 Prefix Requirement Type](./configuration/text-generation.md#-prefix-requirement-type) setting.

View File

@@ -1,7 +1,7 @@
base_url: https://api.openai.com/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: gpt-5.2
model_id: gpt-5.4
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
# Reasoning models need to use `max_completion_tokens` instead of `max_response_tokens`.

View File

@@ -11,7 +11,7 @@ This is related to the [💬 Text Generation](./features.md#-text-generation) fe
If there's a text-generation handler agent configured, the bot **may** respond to messages sent in the room.
Some models also support vision, so you may be able to mix text and images in the same conversation.
Some models also support vision and document understanding, so you may be able to mix text, images, and files (PDFs, text documents, etc.) in the same conversation.
See screenshots of:

View File

@@ -5,8 +5,17 @@ homeserver:
user:
mxid_localpart: baibot
# Authentication: set EITHER password OR access_token + device_id.
#
# Password-based login (traditional homeservers):
password: baibot
# Access token login (for Matrix Authentication Service/OIDC-enabled homeservers):
# Generate a token via: mas-cli manage issue-compatibility-token <username> [device_id]
# access_token: null
# device_id: null
# The name the bot uses as a display name and when it refers to itself.
# Leave empty to use the default (baibot).
name: baibot
@@ -82,7 +91,7 @@ agents:
# base_url: https://api.openai.com/v1
# api_key: ""
# text_generation:
# model_id: gpt-5.2
# model_id: gpt-5.4
# prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
# temperature: 1.0
# # Reasoning models need to use `max_completion_tokens` instead of `max_response_tokens`.

View File

@@ -1,6 +1,6 @@
services:
continuwuity:
image: forgejo.ellis.link/continuwuation/continuwuity:v0.5.5
image: forgejo.ellis.link/continuwuation/continuwuity:v0.5.6
user: "${UID}:${GID}"
restart: unless-stopped
cap_drop:

View File

@@ -1,6 +1,6 @@
services:
element-web:
image: ghcr.io/element-hq/element-web:v1.12.10
image: ghcr.io/element-hq/element-web:v1.12.12
user: "${UID}:${GID}"
restart: unless-stopped
environment:

View File

@@ -1,6 +1,6 @@
services:
ollama:
image: docker.io/ollama/ollama:0.16.2
image: docker.io/ollama/ollama:0.18.2
restart: unless-stopped
ports:
- "${SERVICE_OLLAMA_BIND_PORT_HTTP}:11434"

View File

@@ -1,6 +1,6 @@
services:
postgres:
image: docker.io/postgres:18.2-alpine
image: docker.io/postgres:18.3-alpine
user: ${UID}:${GID}
restart: unless-stopped
environment:
@@ -14,7 +14,7 @@ services:
- /etc/passwd:/etc/passwd:ro
synapse:
image: ghcr.io/element-hq/synapse:v1.147.1
image: ghcr.io/element-hq/synapse:v1.149.1
user: "${UID}:${GID}"
restart: unless-stopped
entrypoint: python

4
rust-toolchain.toml Normal file
View File

@@ -0,0 +1,4 @@
[toolchain]
channel = "1.93.0"
components = ["rustfmt", "clippy"]
profile = "default"

View File

@@ -28,6 +28,13 @@ pub(super) fn create_anthropic_message_request(llm_messages: Vec<LLMMessage>) ->
},
}]
}
LLMMessageContent::File(file_details) => {
tracing::warn!(
"The Anthropic provider's library does not support file/document content. This file message ({}) will be skipped.",
file_details.filename(),
);
continue;
}
};
let message = Message { role, content };

View File

@@ -84,7 +84,7 @@ impl Default for TextGenerationConfig {
}
fn default_text_model_id() -> String {
"gpt-5.2".to_owned()
"gpt-5.4".to_owned()
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]

View File

@@ -1,6 +1,6 @@
use async_openai::types::responses::{
EasyInputContent, EasyInputMessage, ImageDetail, InputContent, InputImageContent, InputItem,
InputParam, MessageType, Role,
EasyInputContent, EasyInputMessage, ImageDetail, InputContent, InputFileArgs,
InputImageContent, InputItem, InputParam, MessageType, Role,
};
use crate::conversation::llm::{
@@ -35,12 +35,28 @@ pub fn convert_llm_messages_to_openai_response_input(
file_id: None,
})])
}
LLMMessageContent::File(file_details) => {
let file_data = format!(
"data:{};base64,{}",
file_details.mime,
base64_encode(&file_details.data)
);
let file_content = InputFileArgs::default()
.file_data(file_data)
.filename(file_details.filename())
.build()
.expect("Failed to build InputFileContent");
EasyInputContent::ContentList(vec![InputContent::InputFile(file_content)])
}
};
items.push(InputItem::EasyMessage(EasyInputMessage {
r#type: MessageType::Message,
role,
content,
phase: None,
}));
}

View File

@@ -40,6 +40,12 @@ fn convert_llm_message_to_openai_message(llm_message: LLMMessage) -> Option<Mess
);
None
}
LLMMessageContent::File(_file_details) => {
tracing::warn!(
"The OpenAI-compat provider's library does not support file content. This file message will be skipped."
);
None
}
}
}

View File

@@ -25,7 +25,7 @@ use crate::agent::Manager as AgentManager;
use crate::entity::catch_up_marker::{
CatchUpMarker, CatchUpMarkerManager, DelayedCatchUpMarkerManager,
};
use crate::entity::cfg::{Avatar, Config};
use crate::entity::cfg::{Avatar, Config, ConfigUserAuth};
use crate::entity::globalconfig::{GlobalConfig, GlobalConfigurationManager};
use crate::entity::roomconfig::{RoomConfig, RoomConfigurationManager};
@@ -395,10 +395,22 @@ async fn create_matrix_link(config: &Config) -> anyhow::Result<MatrixLink> {
let session_encryption_key = config.persistence.session_encryption_key()?;
let db_dir_path: std::path::PathBuf = config.persistence.db_dir_path()?;
let login_creds = LoginCredentials::UserPassword(
config.user.mxid_localpart.to_owned(),
config.user.password.to_owned(),
);
let user_auth = config.user.auth_config(&config.homeserver.server_name)?;
let login_creds = match user_auth {
ConfigUserAuth::UserPassword { username, password } => {
LoginCredentials::UserPassword(username, password)
}
ConfigUserAuth::AccessToken {
user_id,
device_id,
access_token,
} => LoginCredentials::AccessToken {
user_id,
device_id,
access_token,
},
};
let login_encryption = LoginEncryption::new(
config.user.encryption.recovery_passphrase.clone(),

View File

@@ -29,7 +29,15 @@ pub fn load() -> anyhow::Result<Config> {
cfg_env::BAIBOT_HOMESERVER_SERVER_NAME => config.homeserver.server_name = value,
cfg_env::BAIBOT_HOMESERVER_URL => config.homeserver.url = value,
cfg_env::BAIBOT_USER_MXID_LOCALPART => config.user.mxid_localpart = value,
cfg_env::BAIBOT_USER_PASSWORD => config.user.password = value,
cfg_env::BAIBOT_USER_PASSWORD => {
config.user.password = optional_non_empty(value);
}
cfg_env::BAIBOT_USER_ACCESS_TOKEN => {
config.user.access_token = optional_non_empty(value);
}
cfg_env::BAIBOT_USER_DEVICE_ID => {
config.user.device_id = optional_non_empty(value);
}
cfg_env::BAIBOT_USER_ENCRYPTION_RECOVERY_PASSPHRASE => {
config.user.encryption.recovery_passphrase = Some(value);
}
@@ -120,3 +128,7 @@ pub fn load() -> anyhow::Result<Config> {
Ok(config)
}
fn optional_non_empty(value: String) -> Option<String> {
if value.is_empty() { None } else { Some(value) }
}

View File

@@ -41,6 +41,8 @@ pub enum ChatCompletionControllerType {
Image,
File,
ThreadMention,
ReplyMention,
}
@@ -419,7 +421,8 @@ async fn handle_stage_text_generation(
| ChatCompletionControllerType::TextMention
| ChatCompletionControllerType::TextDirect
| ChatCompletionControllerType::Audio
| ChatCompletionControllerType::Image => {
| ChatCompletionControllerType::Image
| ChatCompletionControllerType::File => {
Some(message_context.combined_admin_and_user_regexes())
}

View File

@@ -58,6 +58,18 @@ pub fn determine_controller(
)
}
}
MessagePayload::File(_file_message_content) => {
let prefix_requirement_type = message_context
.room_config_context()
.text_generation_prefix_requirement_type();
match prefix_requirement_type {
TextGenerationPrefixRequirementType::CommandPrefix => ControllerType::Ignore,
TextGenerationPrefixRequirementType::No => {
ControllerType::ChatCompletion(ChatCompletionControllerType::File)
}
}
}
MessagePayload::Audio(_) => {
ControllerType::ChatCompletion(ChatCompletionControllerType::Audio)
}

View File

@@ -1,5 +1,7 @@
use chrono::{DateTime, Utc};
use mxlink::matrix_sdk::ruma::events::room::message::ImageMessageEventContent;
use mxlink::matrix_sdk::ruma::events::room::message::{
FileMessageEventContent, ImageMessageEventContent,
};
use mxlink::mime::Mime;
use crate::agent::provider::ImageSource;
@@ -48,10 +50,35 @@ impl From<ImageDetails> for ImageSource {
}
}
#[derive(Debug, Clone)]
pub struct FileDetails {
pub event_content: FileMessageEventContent,
pub mime: Mime,
pub data: Vec<u8>,
}
impl FileDetails {
pub fn new(event_content: FileMessageEventContent, mime: Mime, data: Vec<u8>) -> Self {
Self {
event_content,
mime,
data,
}
}
pub fn filename(&self) -> String {
self.event_content
.filename
.clone()
.unwrap_or(self.event_content.body.clone())
}
}
#[derive(Debug, Clone)]
pub enum MessageContent {
Text(String),
Image(ImageDetails),
File(FileDetails),
}
impl PartialEq for MessageContent {
@@ -62,6 +89,7 @@ impl PartialEq for MessageContent {
// We can probably do better than this by inspecting `.event_conten1t.source`, but for now this is good enough.
a.filename() == b.filename()
}
(MessageContent::File(a), MessageContent::File(b)) => a.filename() == b.filename(),
_ => false,
}
}

View File

@@ -74,6 +74,7 @@ fn calculate_token_size_for_message(bpe: &CoreBPE, model: &str, message: &Messag
let text_length = match &message.content {
MessageContent::Text(text) => bpe.encode_with_special_tokens(text).len() as i32,
MessageContent::Image(..) => 0,
MessageContent::File(..) => 0,
};
(text_length + role_length + tokens_per_message + tokens_per_name) as u32

View File

@@ -1,6 +1,6 @@
use mxlink::matrix_sdk::ruma::OwnedUserId;
use super::entity::{Author, ImageDetails, Message, MessageContent};
use super::entity::{Author, FileDetails, ImageDetails, Message, MessageContent};
use crate::conversation::matrix::{MatrixMessage, MatrixMessageContent};
use crate::utils::text_to_speech as text_to_speech_utils;
@@ -32,6 +32,15 @@ fn convert_bot_message(matrix_message: &MatrixMessage) -> Option<Message> {
)),
timestamp: matrix_message.timestamp.to_owned(),
}),
MatrixMessageContent::File(file_content, mime_type, media_bytes) => Some(Message {
author: Author::Assistant,
content: MessageContent::File(FileDetails::new(
file_content.clone(),
mime_type.clone(),
media_bytes.clone(),
)),
timestamp: matrix_message.timestamp.to_owned(),
}),
}
}
@@ -90,5 +99,14 @@ fn convert_user_message(matrix_message: &MatrixMessage) -> Option<Message> {
)),
timestamp: matrix_message.timestamp.to_owned(),
}),
MatrixMessageContent::File(file_content, mime_type, media_bytes) => Some(Message {
author: Author::User,
content: MessageContent::File(FileDetails::new(
file_content.clone(),
mime_type.clone(),
media_bytes.clone(),
)),
timestamp: matrix_message.timestamp.to_owned(),
}),
}
}

View File

@@ -2,7 +2,9 @@ use chrono::{DateTime, Utc};
use regex::Regex;
use mxlink::matrix_sdk::ruma::OwnedUserId;
use mxlink::matrix_sdk::ruma::events::room::message::ImageMessageEventContent;
use mxlink::matrix_sdk::ruma::events::room::message::{
FileMessageEventContent, ImageMessageEventContent,
};
use mxlink::mime::Mime;
#[derive(Clone)]
@@ -18,6 +20,7 @@ pub enum MatrixMessageContent {
Text(String),
Notice(String),
Image(ImageMessageEventContent, Mime, Vec<u8>),
File(FileMessageEventContent, Mime, Vec<u8>),
}
#[derive(Clone)]

View File

@@ -234,6 +234,7 @@ pub async fn convert_matrix_native_event_to_matrix_message(
MessageType::Text(text_content) => (text_content.body.clone(), false),
MessageType::Notice(notice_content) => (notice_content.body.clone(), true),
MessageType::Image(image_content) => (image_content.body.clone(), false),
MessageType::File(file_content) => (file_content.body.clone(), false),
_ => return Ok(None),
};
@@ -291,6 +292,67 @@ pub async fn convert_matrix_native_event_to_matrix_message(
}));
}
if let MessageType::File(file_content) = &room_message.msgtype {
let media_request = mxlink::matrix_sdk::media::MediaRequestParameters {
source: file_content.source.to_owned(),
format: mxlink::matrix_sdk::media::MediaFormat::File,
};
let file_name = file_content
.filename
.clone()
.unwrap_or(file_content.body.clone());
let mime_type = file_content
.info
.as_ref()
.and_then(|info| info.mimetype.clone())
.and_then(|mimetype| mimetype.parse::<mxlink::mime::Mime>().ok())
.unwrap_or_else(|| get_mime_type_from_file_name(&file_name));
tracing::debug!("Determined mime type {} for file {}", mime_type, file_name);
if mime_type == mxlink::mime::APPLICATION_OCTET_STREAM {
tracing::debug!(
"Skipping file {} with unsupported MIME type {}. It will be represented as a text message.",
file_name,
mime_type,
);
return Ok(Some(MatrixMessage {
sender_id: matrix_native_event.sender().to_owned(),
content: MatrixMessageContent::Text(format!(
"[A file ({}) was attached but skipped because its content type ({}) is not supported. Let the user know.]",
file_name, mime_type,
)),
mentioned_users,
timestamp,
}));
}
let span = tracing::debug_span!("get_media_content", file_name = %file_name, mime_type = %mime_type);
let media_bytes = matrix_link
.client()
.media()
.get_media_content(&media_request, true)
.instrument(span)
.await?;
tracing::debug!(
"Downloaded {} bytes for file {}",
media_bytes.len(),
file_name
);
return Ok(Some(MatrixMessage {
sender_id: matrix_native_event.sender().to_owned(),
content: MatrixMessageContent::File(file_content.clone(), mime_type, media_bytes),
mentioned_users,
timestamp,
}));
}
Ok(Some(MatrixMessage {
sender_id: matrix_native_event.sender().to_owned(),
content: if is_notice {

View File

@@ -1,6 +1,7 @@
use std::path::PathBuf;
use mxlink::helpers::encryption::EncryptionKey;
use mxlink::matrix_sdk::ruma::{OwnedDeviceId, OwnedUserId};
use serde::{Deserialize, Deserializer, Serialize};
use crate::{
@@ -38,7 +39,7 @@ pub struct Config {
impl Config {
pub fn validate(&self) -> anyhow::Result<()> {
self.homeserver.validate()?;
self.user.validate()?;
self.user.validate(&self.homeserver.server_name)?;
self.persistence.validate()?;
self.room.validate()?;
self.access.validate()?;
@@ -57,6 +58,19 @@ impl Config {
}
}
#[derive(Debug)]
pub enum ConfigUserAuth {
UserPassword {
username: String,
password: String,
},
AccessToken {
user_id: OwnedUserId,
device_id: OwnedDeviceId,
access_token: String,
},
}
#[derive(Debug, Serialize, Deserialize)]
pub struct ConfigHomeserver {
pub server_name: String,
@@ -127,7 +141,15 @@ impl Avatar {
#[derive(Debug, Serialize, Deserialize)]
pub struct ConfigUser {
pub mxid_localpart: String,
pub password: String,
#[serde(default)]
pub password: Option<String>,
#[serde(default)]
pub access_token: Option<String>,
#[serde(default)]
pub device_id: Option<String>,
#[serde(default = "super::defaults::name")]
pub name: String,
@@ -140,7 +162,7 @@ pub struct ConfigUser {
}
impl ConfigUser {
pub fn validate(&self) -> anyhow::Result<()> {
pub fn validate(&self, homeserver_server_name: &str) -> anyhow::Result<()> {
if self.mxid_localpart.is_empty() {
return Err(anyhow::anyhow!(
"The user.mxid_localpart ({}) configuration must be set",
@@ -148,12 +170,7 @@ impl ConfigUser {
));
}
if self.password.is_empty() {
return Err(anyhow::anyhow!(
"The user.password ({}) configuration must be set",
super::env::BAIBOT_USER_PASSWORD
));
}
self.auth_config(homeserver_server_name)?;
if self.name.is_empty() {
return Err(anyhow::anyhow!(
@@ -166,6 +183,57 @@ impl ConfigUser {
Ok(())
}
pub fn auth_config(&self, homeserver_server_name: &str) -> anyhow::Result<ConfigUserAuth> {
let password = self.password.as_deref().filter(|value| !value.is_empty());
let access_token = self
.access_token
.as_deref()
.filter(|value| !value.is_empty());
match (password, access_token) {
(Some(_), Some(_)) => Err(anyhow::anyhow!(
"Set exactly one authentication method: either user.password ({}) OR user.access_token ({}) + user.device_id ({})",
super::env::BAIBOT_USER_PASSWORD,
super::env::BAIBOT_USER_ACCESS_TOKEN,
super::env::BAIBOT_USER_DEVICE_ID
)),
(None, None) => Err(anyhow::anyhow!(
"Set one authentication method: either user.password ({}) OR user.access_token ({}) + user.device_id ({})",
super::env::BAIBOT_USER_PASSWORD,
super::env::BAIBOT_USER_ACCESS_TOKEN,
super::env::BAIBOT_USER_DEVICE_ID
)),
(Some(password), None) => Ok(ConfigUserAuth::UserPassword {
username: self.mxid_localpart.to_owned(),
password: password.to_owned(),
}),
(None, Some(access_token)) => {
let device_id = self
.device_id
.as_deref()
.filter(|value| !value.is_empty())
.ok_or_else(|| {
anyhow::anyhow!(
"user.device_id ({}) must be set when using access token authentication",
super::env::BAIBOT_USER_DEVICE_ID
)
})?;
let user_id = OwnedUserId::try_from(format!(
"@{}:{}",
self.mxid_localpart, homeserver_server_name
))
.map_err(|e| anyhow::anyhow!("Invalid user ID: {e}"))?;
Ok(ConfigUserAuth::AccessToken {
user_id,
device_id: OwnedDeviceId::from(device_id),
access_token: access_token.to_owned(),
})
}
}
}
}
#[derive(Debug, Default, Serialize, Deserialize)]
@@ -468,3 +536,7 @@ impl TryInto<GlobalConfig> for ConfigInitialGlobalConfig {
Ok(entity)
}
}
#[cfg(test)]
#[path = "config_tests.rs"]
mod config_tests;

View File

@@ -0,0 +1,117 @@
use super::{Avatar, ConfigUser, ConfigUserAuth, ConfigUserEncryption};
use crate::entity::cfg::env;
fn base_user() -> ConfigUser {
ConfigUser {
mxid_localpart: "baibot".to_owned(),
password: None,
access_token: None,
device_id: None,
name: "baibot".to_owned(),
encryption: ConfigUserEncryption {
recovery_passphrase: None,
recovery_reset_allowed: false,
},
avatar: Avatar::Default,
}
}
#[test]
fn auth_config_uses_password_mode() {
let mut user = base_user();
user.password = Some("secret".to_owned());
let auth = user
.auth_config("example.com")
.expect("password auth should be valid");
match auth {
ConfigUserAuth::UserPassword { username, password } => {
assert_eq!(username, "baibot");
assert_eq!(password, "secret");
}
ConfigUserAuth::AccessToken { .. } => {
panic!("expected password auth mode");
}
}
}
#[test]
fn auth_config_uses_access_token_mode() {
let mut user = base_user();
user.access_token = Some("token123".to_owned());
user.device_id = Some("DEVICE1".to_owned());
let auth = user
.auth_config("example.com")
.expect("access token auth should be valid");
match auth {
ConfigUserAuth::AccessToken {
user_id,
device_id,
access_token,
} => {
assert_eq!(user_id.as_str(), "@baibot:example.com");
assert_eq!(device_id.as_str(), "DEVICE1");
assert_eq!(access_token, "token123");
}
ConfigUserAuth::UserPassword { .. } => {
panic!("expected access token auth mode");
}
}
}
#[test]
fn auth_config_rejects_both_auth_methods() {
let mut user = base_user();
user.password = Some("secret".to_owned());
user.access_token = Some("token123".to_owned());
user.device_id = Some("DEVICE1".to_owned());
let err = user
.auth_config("example.com")
.expect_err("both auth methods should be rejected");
assert!(
err.to_string()
.contains("exactly one authentication method")
);
}
#[test]
fn auth_config_rejects_missing_auth() {
let user = base_user();
let err = user
.auth_config("example.com")
.expect_err("missing auth should be rejected");
assert!(err.to_string().contains("Set one authentication method"));
}
#[test]
fn auth_config_rejects_access_token_without_device_id() {
let mut user = base_user();
user.access_token = Some("token123".to_owned());
let err = user
.auth_config("example.com")
.expect_err("access token mode without device_id should be rejected");
assert!(err.to_string().contains(env::BAIBOT_USER_DEVICE_ID));
}
#[test]
fn auth_config_treats_empty_strings_as_unset() {
let mut user = base_user();
user.password = Some(String::new());
user.access_token = Some(String::new());
user.device_id = Some(String::new());
let err = user
.auth_config("example.com")
.expect_err("empty auth values should be treated as unset");
assert!(err.to_string().contains("Set one authentication method"));
}

View File

@@ -5,6 +5,8 @@ pub const BAIBOT_HOMESERVER_URL: &str = "BAIBOT_HOMESERVER_URL";
pub const BAIBOT_USER_MXID_LOCALPART: &str = "BAIBOT_USER_MXID_LOCALPART";
pub const BAIBOT_USER_PASSWORD: &str = "BAIBOT_USER_PASSWORD";
pub const BAIBOT_USER_ACCESS_TOKEN: &str = "BAIBOT_USER_ACCESS_TOKEN";
pub const BAIBOT_USER_DEVICE_ID: &str = "BAIBOT_USER_DEVICE_ID";
pub const BAIBOT_USER_NAME: &str = "BAIBOT_USER_NAME";
pub const BAIBOT_USER_AVATAR: &str = "BAIBOT_USER_AVATAR";
pub const BAIBOT_USER_ENCRYPTION_RECOVERY_PASSPHRASE: &str =

View File

@@ -2,4 +2,4 @@ mod config;
pub mod defaults;
pub mod env;
pub use config::{Avatar, Config};
pub use config::{Avatar, Config, ConfigUserAuth};

View File

@@ -1,5 +1,6 @@
use mxlink::matrix_sdk::ruma::events::room::message::{
AudioMessageEventContent, ImageMessageEventContent, MessageType, TextMessageEventContent,
AudioMessageEventContent, FileMessageEventContent, ImageMessageEventContent, MessageType,
TextMessageEventContent,
};
use mxlink::matrix_sdk::ruma::{OwnedEventId, OwnedUserId};
@@ -30,6 +31,7 @@ pub enum MessagePayload {
Text(TextMessageEventContent),
Audio(AudioMessageEventContent),
Image(ImageMessageEventContent),
File(FileMessageEventContent),
Reaction {
key: String,
@@ -57,6 +59,7 @@ impl TryInto<MessagePayload> for MessageType {
MessagePayload::Audio(audio_content)
}
MessageType::Image(image_content) => MessagePayload::Image(image_content),
MessageType::File(file_content) => MessagePayload::File(file_content),
other => {
return Err(format!("Unsupported message type: {:?}", other));
}

View File

@@ -17,18 +17,7 @@ pub fn get_file_extension(mime_type: &mime::Mime) -> String {
}
pub fn get_mime_type_from_file_name(file_name: &str) -> mime::Mime {
let extension = file_name.rsplit('.').next().unwrap_or("");
match extension.to_lowercase().as_str() {
"jpg" | "jpeg" => mime::IMAGE_JPEG,
"png" => mime::IMAGE_PNG,
"gif" => mime::IMAGE_GIF,
"webp" => "image/webp".parse().unwrap(),
"svg" => mime::IMAGE_SVG,
"tiff" | "tif" => "image/tiff".parse().unwrap(),
"bmp" => "image/bmp".parse().unwrap(),
"heic" | "heif" => "image/heic".parse().unwrap(),
"avif" => "image/avif".parse().unwrap(),
_ => mime::APPLICATION_OCTET_STREAM,
}
mime_guess::from_path(file_name)
.first()
.unwrap_or(mime::APPLICATION_OCTET_STREAM)
}