Release 1.4.0

It seems like Element iOS benefits from this.

.github/workflows/workflow.yml

@@ -24,6 +24,10 @@ jobs:
     name: Build and Publish
     runs-on: self-hosted
     steps:
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: arm64
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v1
       - name: Login to ghcr.io
@@ -49,3 +53,4 @@ jobs:
           push: true
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
+          file: Dockerfile.ci

CHANGELOG.md

@@ -1 +1,83 @@
-There's nothing here yet.
+# (2024-11-19) Version 1.4.0
+
+- (**Improvement**) Dependency updates. This version is based on [mxlink](https://crates.io/crates/mxlink)@1.4.0 (which is based on the newly released [matrix-sdk](https://crates.io/crates/matrix-sdk)@[0.8.0](https://github.com/matrix-org/matrix-rust-sdk/releases/tag/matrix-sdk-0.8.0)). Once you run this version at least once and your matrix-sdk datastore gets upgraded to the new schema, **you will not be able to downgrade to older baibot versions** (based on the older matrix-sdk), unless you start with an empty datastore.
+
+- (**Bugfix**) Add missing typing notices sending functionality while generating images ([9d166e35ba](https://github.com/etkecc/baibot/commit/9d166e35ba6fc0daaf69318870e92436f3302056))
+
+
+# (2024-11-12) Version 1.3.2
+
+Dependency updates.
+
+
+# (2024-10-03) Version 1.3.1
+
+- (**Improvement**) Improves fallback user mentions support for old clients (like Element iOS) which use the bot's display name (not its full Matrix User ID). ([d9a045a5e4](https://github.com/etkecc/baibot/commit/d9a045a5e41d2b99694f92ec9e90f47529546d89))
+
+
+# (2024-10-03) Version 1.3.0
+
+**TLDR**: you can now use OpenAI's [o1](https://platform.openai.com/docs/models/o1) models, benefit from [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) and mention the bot again from old clients lacking proper [user mentions support](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) (like Element iOS).
+
+- (**Feature**) Introduces a new `baibot_conversation_start_time_utc` [prompt variable](./docs/configuration/text-generation.md#️-prompt-override) which is not a moving target (like the `baibot_now_utc` variable) and allows [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) to work. All default/sample configs have been adjusted to make use of this new variable, but users need to adjust your existing dynamically-created agents to start using it. ([85e66406dc](https://github.com/etkecc/baibot/commit/85e66406dc6f430741c7819f420e2df4ae6e8d3b))
+
+- (**Improvement**) Allows for the `max_response_tokens` configuration value for the [OpenAI provider](./docs/providers.md#openai) to be set to `null` to allow [o1](https://platform.openai.com/docs/models/o1) models (which do not support `max_response_tokens`) to be used. See the new o1 sample config [here](./docs/sample-provider-configs/openai-o1.yml). ([db9422740c](https://github.com/etkecc/baibot/commit/db9422740ceca32956d9628b6326b8be206344e2))
+
+- (**Improvement**) Switches the sample configs for the [OpenAI provider](./docs/providers.md#openai) to point to the `gpt-4o` model, which since 2024-10-02 is the same as the `gpt-4o-2024-08-06` model. We previously explicitly pointed the bot to the `gpt-4o-2024-08-06` model, because it was much better (longer context window). Now that `gpt-4o` points to the same powerful model, we don't need to pin its version anymore. Existing users may wish to adjust their configuration to match. ([90fbad5b64](https://github.com/etkecc/baibot/commit/90fbad5b643cd06c23179f055a309ec6a7cba161))
+
+- (**Bugfix**) Restores fallback user mentions support (via regular text, not via the [user mentions spec](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions)) to allow certain old clients (like Element iOS) to be able to mention the bot again. Support for this was intentionally removed recently (in [v1.2.0](#2024-10-01-version-120)), but it turned out to be too early to do this. ([b40226826f](https://github.com/etkecc/baibot/commit/b40226826fe914d0d5d265230ebc5bac8058b6f7))
+
+
+# (2024-10-01) Version 1.2.0
+
+- (**Feature**) Adds support for [on-demand involvement](./docs/features.md#on-demand-involvement) of the bot (via mention) in arbitrary threads and reply chains ([9908512968](https://github.com/etkecc/baibot/commit/990851296828168c2106eb3f4668833e9e5a7463)) - fixes [issue #15](https://github.com/etkecc/baibot/issues/15)
+
+- (**Improvement**) Simplifies [Transcribe-only mode](./docs/features.md#transcribe-only-mode) reply format (removing `> 🦻` prefixing) to allow easier forwarding, etc. ([e6aa956423](https://github.com/etkecc/baibot/commit/e6aa95642376ee7d87932d0e66dcfedf261b188b)) - fixes [issue #14](https://github.com/etkecc/baibot/issues/14)
+
+- (**Bugfix**) Fixes speech-to-text replies rendering incorrectly in certain clients, due to them confusing our old reply format with [fallback for rich replies](https://spec.matrix.org/v1.11/client-server-api/#fallbacks-for-rich-replies) ([e6aa956423](https://github.com/etkecc/baibot/commit/e6aa95642376ee7d87932d0e66dcfedf261b188b)) - fixes [issue #17](https://github.com/etkecc/baibot/issues/17)
+
+
+# (2024-09-22) Version 1.1.1
+
+- (**Bugfix**) Fix thread messages being lost due to lack of pagination support ([d4ddd29660](https://github.com/etkecc/baibot/commit/d4ddd29660d9f51d248119dd6032e68ab29e7d35)) - fixes [issue #13](https://github.com/etkecc/baibot/issues/13)
+
+- (**Bugfix**) Fix Anthropic conversations getting stuck when being impatient and sending multiple consecutive messages ([8b12bdf2b3](https://github.com/etkecc/baibot/commit/8b12bdf2b3196abea0e8db33d7c50fff48341cb9)) - fixes [issue #13](https://github.com/etkecc/baibot/issues/13)
+
+
+# (2024-09-21) Version 1.1.0
+
+- (**Feature**) Adds support for [prompt variables](./docs/configuration/text-generation.md#️-prompt-override) (date/time, bot name, model id) ([2a5a2d6a4d](https://github.com/etkecc/baibot/commit/2a5a2d6a4dbf5fd7cb504ac07d4187fdc32ae395)) - fixes [issue #10](https://github.com/etkecc/baibot/issues/10)
+
+- (**Improvement**) [Dockerfile](./Dockerfile) changes to produce ~20MB smaller container images ([354063abb7](https://github.com/etkecc/baibot/commit/354063abb79035069bd3b26c53214874e9cdd95d))
+
+- (**Improvement**) [Dockerfile](./Dockerfile) changes to optimize local (debug) runs in a container ([c8c5e0e540](https://github.com/etkecc/baibot/commit/c8c5e0e540ab981e849452eb3ddb0378105e1fc6))
+
+- (**Improvement**) CI changes to try and work around multi-arch image issues like [this one](https://github.com/etkecc/baibot/issues/2) ([5de7559ed6](https://github.com/etkecc/baibot/commit/5de7559ed685a41c22dfc12283681f02f4c2ee00))
+
+
+# (2024-09-19) Version 1.0.6
+
+Improvements to:
+
+- messages sent by the bot - better onboarding flow, especially when no agents have been created yet
+- documentation pages
+
+
+# (2024-09-14) Version 1.0.5
+
+Further [improves](https://github.com/etkecc/baibot/commit/3b25b92a81a05ebaf1c6dbabf675fbfbe6c9f418) the typing notification logic, so that it tolerates edge cases better.
+
+
+# (2024-09-14) Version 1.0.4
+
+[Improves](https://github.com/etkecc/baibot/commit/dd1dd78312e3db7f92b37fb3b4750fbe35de7115) the typing notification logic.
+
+
+# (2024-09-13) Version 1.0.3
+
+Contains [fixes](https://github.com/etkecc/rust-mxlink/commit/f339fc85e69aa7f614394ad303d1614cd307319c) for [some](https://github.com/etkecc/baibot/issues/1) startup failures caused by partial initialization (errors during startup).
+
+
+# (2024-09-12) Version 1.0.0
+
+Initial release. 🎉

Cargo.toml

@@ -5,9 +5,9 @@ authors = ["Slavi Pantaleev <slavi@devture.com>"]
 repository = "https://github.com/etkecc/baibot"
 license = "AGPL-3.0-or-later"
 readme = "README.md"
-keywords = ["matrix", "chat", "LLM", "openai", "anthropic", "localai"]
+keywords = ["matrix", "chat", "bot", "AI", "LLM"]
-exclude = [".editorconfig", "justfile", "docs/", "etc/", "Dockerfile", ".dockerignore", ".gitignore"]
+include = ["/etc/assets/baibot-torso-768.png", "/src", "/README.md", "/CHANGELOG.md", "/LICENSE"]
-version = "1.0.0"
+version = "1.4.0"
 edition = "2021"
 
 [lib]
@@ -17,21 +17,22 @@ path = "src/lib.rs"
 [dependencies]
 anthropic-rs = "0.1.*"
 anyhow = "1.0.*"
-async-openai = "0.24.*"
+async-openai = "0.26.*"
 base64 = "0.22.*"
+chrono = { version = "0.4.*", default-features = false, features = ["std", "now"] }
 # We'd rather not depend on this, but we cannot use the ruma-events EventContent macro without it.
-matrix-sdk = { version = "0.7.1", default-features = false }
+matrix-sdk = { version = "0.8.0", default-features = false }
 mxidwc = "1.0.*"
-mxlink = "1.0.*"
+mxlink = ">=1.4.0"
-openai_api_rust = { git = "https://github.com/etkecc/openai_api_rust.git", branch = "next" }
+etke_openai_api_rust = "0.1.*"
 quick_cache = "0.6.*"
-regex = "1.10.*"
+regex = "1.11.*"
 serde = { version = "1.0.*", features = ["derive"], default-features = false }
 serde_json = "1.0.*"
 serde_yaml = "0.9.*"
-tempfile = "3.12.*"
+tempfile = "3.14.*"
-tiktoken-rs = { version = "0.5.*", features = ["async-openai"] }
+tiktoken-rs = { version = "0.6.*", features = ["async-openai"] }
-tokio = { version = "1.40.*", features = ["rt", "rt-multi-thread", "macros"] }
+tokio = { version = "1.41.*", features = ["rt", "rt-multi-thread", "macros"] }
 tracing = "0.1.*"
 tracing-subscriber = { version = "0.3.*", features = ["env-filter"] }
 url = "2.5.*"

Dockerfile

@@ -4,7 +4,7 @@
 #                                     #
 #######################################
 
-FROM docker.io/rust:1.80.1-slim-bookworm AS build
+FROM docker.io/rust:1.82.0-slim-bookworm AS build
 
 RUN apt-get update && apt-get install -y build-essential pkg-config libssl-dev libsqlite3-dev
 
@@ -15,14 +15,23 @@ WORKDIR /app
 
 COPY . /app
 
+ARG RELEASE_BUILD=true
+
 RUN --mount=type=cache,target=/cargo,sharing=locked \
 	--mount=type=cache,target=/target,sharing=locked \
-	cargo build --release
+	if [ "$RELEASE_BUILD" = "true" ]; then \
+		cargo build --release; \
+	else \
+		cargo build; \
+	fi
 
 # Move it out of the mounted cache, so we can copy it in the next stage.
 RUN --mount=type=cache,target=/target,sharing=locked \
-	cp /target/release/baibot /baibot
+	if [ "$RELEASE_BUILD" = "true" ]; then \
-
+		cp /target/release/baibot /baibot; \
+	else \
+		cp /target/debug/baibot /baibot; \
+	fi
 
 #######################################
 #                                     #
@@ -32,7 +41,9 @@ RUN --mount=type=cache,target=/target,sharing=locked \
 
 FROM docker.io/debian:bookworm-slim
 
-RUN apt-get update && apt-get install -y ca-certificates sqlite3
+RUN apt-get update && apt-get install -y ca-certificates sqlite3 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 

Dockerfile.ci

@@ -0,0 +1,35 @@
+#######################################
+#                                     #
+# Stage 1: building                   #
+#                                     #
+#######################################
+
+FROM docker.io/rust:1.82.0-slim-bookworm AS build
+
+RUN apt-get update && apt-get install -y build-essential pkg-config libssl-dev libsqlite3-dev
+
+WORKDIR /app
+
+COPY . /app
+
+RUN cargo build --release
+
+#######################################
+#                                     #
+# Stage 2: packaging                  #
+#                                     #
+#######################################
+
+FROM docker.io/debian:bookworm-slim
+
+RUN apt-get update && apt-get install -y ca-certificates sqlite3 && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY --from=build /app/target/release/baibot .
+
+ENTRYPOINT ["/bin/sh", "-c"]
+
+CMD ["/app/baibot"]

README.md

@@ -5,7 +5,7 @@
 
 🤖 baibot is an [AI](https://en.wikipedia.org/wiki/Artificial_intelligence) ([Large Language Model](https://en.wikipedia.org/wiki/Large_language_model)) bot for [Matrix](https://matrix.org/) built by [etke.cc](https://etke.cc/) (managed Matrix servers).
 
-The name is pronounced 'bye' and is a play on [AI](https://en.wikipedia.org/wiki/Artificial_intelligence), referencing the fictional character [🇧🇬 Bai Ganyo](https://en.wikipedia.org/wiki/Bay_Ganyo).
+The name is pronounced 'bye'-bot and is a play on [AI](https://en.wikipedia.org/wiki/Artificial_intelligence), referencing the fictional character [🇧🇬 Bai Ganyo](https://en.wikipedia.org/wiki/Bay_Ganyo).
 
 It's designed as a more private and [featureful](#-features) alternative to [matrix-chatgpt-bot](https://github.com/matrixgpt/matrix-chatgpt-bot).
 It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use the [AIChat](https://github.com/sigoden/aichat) CLI tool and instead does everything in-process, without forking.
@@ -19,8 +19,8 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use
 
   - [💬 text-generation](./docs/features.md#-text-generation): communicating with you via text
   - [🦻 speech-to-text](./docs/features.md#-speech-to-text): turning your voice messages into text
-  - [🗣️ text-to-speech](./docs/features.md#️-text-to-speech): turning bot or users text messages into voice messages
+  - [🗣️ text-to-speech](./docs/features.md#%EF%B8%8F-text-to-speech): turning bot or users text messages into voice messages
-  - [🖌️ image-generation](./docs/features.md#-image-generation): generating images based on instructions
+  - [🖌️ image-generation](./docs/features.md#%EF%B8%8F-image-generation): generating images based on instructions
 
 - 🪄 Supports [seamless voice interaction](./docs/features.md#seamless-voice-interaction) (turning user voice messages into text, answering in text, then turning that text back into voice)
 
@@ -41,7 +41,7 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use
 
 ![Introduction and general usage](./docs/screenshots/introduction-and-general-usage.webp)
 
-You can find more screenshots on the the [🌟 Features](./docs/features.md) and other [📚 Documentation](./docs/README.md) pages, as well as in the [docs/screenshots](./docs/screenshots) directory.
+You can find more screenshots on the [🌟 Features](./docs/features.md) and other [📚 Documentation](./docs/README.md) pages, as well as in the [docs/screenshots](./docs/screenshots) directory.
 
 
 ## 🚀 Getting Started

docs/README.md

@@ -1,9 +1,11 @@
 # Table of Contents
 
 - [🔒 Access](./access.md)
-- [🤖 Agents](./agents.md) and [☁️ Providers](./providers.md)
+- [🤖 Agents](./agents.md)
 - [🛠️ Configuration](./configuration/README.md)
 - [🌟 Features](./features.md)
+- [🤝 Handlers](./configuration/handlers.md)
+- [☁️ Providers](./providers.md)
 - [📖 Usage](./usage.md)
 - [🚀 Installation](./installation.md)
 - [💻 Development](./development.md)

docs/access.md

@@ -5,17 +5,22 @@ This bot employs access control to decide who can use its services and manage it
 
 ### 👋 Joining rooms
 
-The bot automatically joins rooms when invited by someone considered a bot [user](#-users).
+The bot automatically joins rooms only when invited by someone considered a bot [👥 user](#-users).
 
 
 ### 👥 Users
 
-The bot will ignore messages (and room invitations) from unallowed users.
-
-Users can **use all the bot's [features](./features.md)** ([💬 Text Generation](./features.md#-text-generation), [🦻 Speech-to-Text](./features.md#-speech-to-text), etc.), but **cannot manage the bot's configuration**.
-
 The bot can be used by users that match some [dynamically](./configuration/README.md#dynamic-configuration) configured [Matrix user id](https://spec.matrix.org/v1.11/#users) patterns.
 
+Users:
+
+- ✅ can **invite the bot to rooms**
+- ✅ can **use all the bot's [features](./features.md)** ([💬 Text Generation](./features.md#-text-generation), [🦻 Speech-to-Text](./features.md#-speech-to-text), etc.) by sending room messages
+- ✅ can **mention the bot** in threads and reply chains to provoke it to respond to non-user messages (see [🌟 Features / 💬 Text Generation / On-demand involvement](./features.md#on-demand-involvement))
+- ✅ can **change the bot's configuration in a room** (e.g. `!bai config room ...` commands)
+- ❌ cannot **change the bot's global configuration** (e.g. `!bai config global ...` commands)
+- ❌ cannot **create new [🤖 Agents](./agents.md)** (neither in rooms, nor globally). See [💼 Room-local agent managers](#-room-local-agent-managers) for controlling which users can create agents.
+
 The following commands are available:
 - **Show** the currently allowed users: `!bai access users`
 - **Set** the list of allowed users: `!bai access set-users SPACE_SEPARATED_PATTERNS`
@@ -27,6 +32,8 @@ Example patterns: `@*:example.com @*:another.com @someone:company.org`
 
 Administrators can **manage the bot's configuration and access control**.
 
+Administrators are [👥 Users](#-users) and [💼 Room-local agent managers](#-room-local-agent-managers) implicitly, so they inherit all their permissions.
+
 The bot can be administrated by users that match some [statically](./configuration/README.md#static-configuration) configured [Matrix user id](https://spec.matrix.org/v1.11/#users) patterns.
 
 Administrators cannot be changed without adjusting the bot's configuration on the server.
@@ -35,12 +42,11 @@ Administrators cannot be changed without adjusting the bot's configuration on th
 ### 💼 Room-local agent managers
 
 Room-local agent managers are users privileged to **create their own [agents](./agents.md)** (see `!bai agent`) in rooms.
-Letting regular users create agents which contact arbitrary network services **may be a security issue**.
 
-No room-local agent manager patterns are configured, so new agents can only be created by administrators.
+**⚠️ WARNING**: Letting regular users create agents which contact arbitrary network services **may be a security issue**.
 
 The following commands are available:
 - **Show** the currently allowed users: `!bai access room-local-agent-managers`
 - **Set** the list of allowed users: `!bai access set-room-local-agent-managers SPACE_SEPARATED_PATTERNS`
 
-Example patterns: `@*:synapse.127.0.0.1.nip.io @*:another.com @someone:company.org`
+Example patterns: `@*:example.com @*:another.com @someone:company.org`

docs/configuration/text-generation.md

@@ -13,7 +13,7 @@ You may also wish to see:
 
 In Direct Message rooms with the bot (1:1 rooms), it most usually makes sense for the bot to respond to **all** of your messages, as shown on this [🖼️ screenshot](../screenshots/text-generation.webp).
 
-In group rooms (with multiple users), it may be more appropriate for the bot to only respond to messages that are **prefixed** with the command prefix (e.g. `!bai`), so that other chat exchange in the room will not trigger it. Such a setup is shown on this [🖼️ screenshot](../screenshots/text-generation-prefix-requirement.webp).
+In group rooms (with multiple users), it may be more appropriate for the bot to only respond to messages that are **prefixed** with the command prefix (e.g. `!bai`) or which are [mentioning](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) the bot (e.g. `@baibot`), so that other chat exchange in the room will not trigger it. Such a setup is shown on the [🖼️ On-demand involvement in the room](../screenshots/text-generation-prefix-requirement.webp) screenshot.
 
 There are exceptions to these rules, and you can configure the bot to respond only to prefixed messages in a 1:1 room, or to respond to all messages even in a multi-user group room.
 
@@ -23,11 +23,14 @@ To support such use-cases, the bot has a `text-generation prefix-requirement-typ
 
 - `command_prefix`: indicates that the bot would require that messages be prefixed with the command prefix (e.g. `!bai`) and would ignore all messages that are not prefixed
 
-By default, the bot is **auto-configured (upon joining a new room)** to use the `no` setting in rooms that only include 2 users (you and the bot), and `command_prefix` in rooms with more than 2 users. To prevent surprises, the bot will **not** adjust this setting subsequently. You can manually adjust it via `!bai config room set-prefix-requirement-type VALUE`.
+By default, the bot is **auto-configured (upon joining a new room)** to use the `no` setting in rooms that only include 2 users (you and the bot), and `command_prefix` in rooms with more than 2 users. To prevent surprises, the bot will **not** adjust this setting subsequently. You can manually adjust it via `!bai config room text-generation set-prefix-requirement-type VALUE`.
 
 Example: `!bai config room text-generation set-prefix-requirement-type command_prefix` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
 
-Regardless of this configuration, **the bot will also respond to messages which directly [mention](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) the bot** (e.g. `@baibot`), even if they are not prefixed. An example of this can be seen on this [🖼️ screenshot](../screenshots/text-generation-prefix-requirement.webp).
+Regardless of this configuration, **the bot will also respond to messages by allowed [👥 Users](../access.md#-users) which directly [mention](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) the bot** (e.g. `@baibot`), even if they are not prefixed. An example of this can be seen on these screenshots:
+
+- [🖼️ On-demand involvement in a thread](../screenshots/text-generation-on-demand-thread-involvement.webp)
+- [🖼️ On-demand involvement in a reply chain](../screenshots/text-generation-on-demand-reply-involvement.webp)
 
 
 ### 🪄 Auto Usage
@@ -51,7 +54,7 @@ The bot also supports ♻️ **context management**, which automatically adjusts
 
 This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](../providers.md#openai) models.
 
-This setting is **disabled by default**, but can be enabled via `!bai config room context-management-enabled true` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings)).
+This setting is **disabled by default**, but can be enabled via `!bai config room text-generation set-context-management-enabled true` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings)).
 
 
 ### ⌨️ Prompt Override
@@ -68,6 +71,22 @@ Where appropriate, you'll mention best practices and common pitfalls.
 
 A prompt override can also be set globally, see [🛠️ Room Settings](./README.md#room-settings).
 
+Prompts may contain the following **placeholder variables** which will be replaced *every time* the bot is interacted with:
+
+| Placeholder               | Description | Example |
+|---------------------------|-------------|---------|
+| `{{ baibot_name }}`       | Name of the bot as configured in the `user.name` field in the [Static configuration](./README.md#static-configuration) | `Baibot` |
+| `{{ baibot_model_id }}`   | Text-Generation model ID as configured in the [🤖 agent](../agents.md)'s configuration | `gpt-4o` |
+| `{{ baibot_now_utc }}`    | Current date and time in UTC (⚠️ usage may break prompt caching - see below) | `2024-09-20 (Friday), 14:26:42 UTC` |
+| `{{ baibot_conversation_start_time_utc }}` | The date and time in UTC that the conversation started | `2024-09-20 (Friday), 14:26:42 UTC` |
+
+💡 `{{ baibot_now_utc }}` changes as time goes on, which prevents [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) from working. It's better to use `{{ baibot_conversation_start_time_utc }}` in prompts, as its value doesn't change yet still orients the bot to the current date/time.
+
+Here's a prompt that combines some of the above variables:
+
+> You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
+
+
 ### 🌡️ Temperature Override
 
 You can override the [temperature](https://blogs.novita.ai/what-are-large-language-model-settings-temperature-top-p-and-max-tokens/#what-is-llm-temperature) (randomness / creativity) parameter configured at the [🤖 agent](../agents.md) level.

docs/features.md

@@ -10,8 +10,8 @@ The bot supports the following use-purposes:
 
 - [💬 text-generation](#-text-generation): communicating with you via text
 - [🦻 speech-to-text](#-speech-to-text): turning your voice messages into text
-- [🗣️ text-to-speech](#️-text-to-speech): turning bot or users text messages into voice messages
+- [🗣️ text-to-speech](#%EF%B8%8F-text-to-speech): turning bot or users text messages into voice messages
-- [🖌️ image-generation](#-image-generation): generating images based on instructions
+- [🖌️ image-generation](#%EF%B8%8F-image-generation): generating images based on instructions
 
 In a given room, each different purpose can be served by a different [☁️ provider](./providers.md) and model. This combination of provider and model configuration is called an [🤖 agent](./agents.md). Each purpose can be served by a different **handler** agent.
 
@@ -28,14 +28,32 @@ Text Generation is the bot's ability to **respond to users' text messages with t
 
 In multi-user (group) rooms, to avoid disturbing the normal conversation between people, the bot is auto-configured to only respond to messages starting with the command prefix (`!bai`) or direct mentions via the [💬 Text Generation / 🗟 Prefix Requirement Type](./configuration/text-generation.md#-prefix-requirement-type) setting.
 
+Normally, the bot only responds to allowed [👥 Users](./access.md#-users). In certain cases, it's useful for an allowed user to provoke the bot to respond even in foreign threads or reply chains. You can learn more about this feature in the [On-demand involvement](./features.md#on-demand-involvement) section below.
+
 A few other features (like [🗣️ Text-to-Speech](#️-text-to-speech) and [🦻 Speech-to-Text](#-speech-to-text)) combine well with Text Generation, so you **don't necessarily need to communicate with the bot via text** (with [Seamless voice interaction](#seamless-voice-interaction), you can communicate only with voice).
 
 You may also wish to see:
 
-- [🛠️ Configuration / 💬 Text Generation](./configuration/README.md#-text-generation) for configuration options related to Text Generation
+- [🛠️ Configuration / 💬 Text Generation](./configuration/text-generation.md) for configuration options related to Text Generation
 - [📖 Usage / 💬 Text Generation](./usage.md#-text-generation) section for more details on how to use the bot for Text Generation in a room
 
 
+#### On-demand involvement
+
+In the following 2 cases, it's useful to involve the bot in conversations on-demand:
+
+1. In multi-user rooms (with the [🗟 Prefix Requirement](./configuration/text-generation.md#-prefix-requirement-type) setting set to "required")
+2. In rooms with foreign users (users that are not authorized bot [👥 users](./access.md#-users))
+
+In these instances, an allowed [👥 user](./access.md#-users) can also provoke the bot to respond to **any** thread or reply chain by [mentioning](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) the bot (e.g. `@baibot Hello!`). The following screenshots demonstrate this behavior:
+
+- [🖼️ On-demand involvement in the room](./screenshots/text-generation-prefix-requirement.webp)
+- [🖼️ On-demand involvement in a thread](./screenshots/text-generation-on-demand-thread-involvement.webp) (the Alice user in this example is not an allowed user, yet her messages are still considered as part of the conversation context)
+- [🖼️ On-demand involvement in a reply chain](./screenshots/text-generation-on-demand-reply-involvement.webp) (the Alice user in this example is not an allowed user, yet her messages are still considered as part of the conversation context)
+
+💡 **NOTE**: Normally, the bot **only considers messages from allowed [👥 Users](./access.md#-users)** and ignores all other messages when responding. However, **when the bot is explicitly invoked (via mention)** in a thread or reply chain, **it will consider all messages** in the thread and reply chain (even those from foreign users) as part of the conversation context.
+
+
 ### 🗣️ Text-to-Speech
 
 Text-to-Speech is the bot's ability to **turn text messages into voice messages**.
@@ -48,7 +66,9 @@ By default, the bot **doesn't** perform text-to-speech. It can be configured for
 
 Another use-case is to have the bot operate in [Text-to-Speech-only mode](#text-to-speech-only-mode).
 
-- [🛠️ Configuration / 🗣️ Text-to-Speech](./configuration/README.md#-text-to-speech) for configuration options related to Text-to-Speech
+You may also wish to see:
+
+- [🛠️ Configuration / 🗣️ Text-to-Speech](./configuration/text-to-speech.md) for configuration options related to Text-to-Speech
 - [📖 Usage / 🗣️ Text-to-Speech](./usage.md#-text-to-speech) section for more details on how to use the bot for Text-to-Speech in a room
 
 
@@ -77,6 +97,10 @@ The default flow is shown in the screenshot above: your voice messages are trans
 
 You may also configure the bot for [Seamless voice interaction](#seamless-voice-interaction) or [Transcribe-only mode](#transcribe-only-mode), etc.
 
+You may also wish to see:
+
+- [🛠️ Configuration / 🦻 Speech-to-Text](./configuration/speech-to-text.md) for configuration options related to Speech-to-Text
+- [📖 Usage / 🦻 Speech-to-Text](./usage.md#-speech-to-text) section for more details on how to use the bot for Speech-to-Text in a room
 
 #### Seamless voice interaction
 
@@ -121,7 +145,7 @@ See a [🖼️ Screenshot of the Image Generation feature](./screenshots/image-g
 
 You may also wish to see:
 
-- [🛠️ Configuration / 🖌️ Image Generation](./configuration/README.md#-image-generation) for configuration options related to Image Generation
+- [🛠️ Configuration / 🖌️ Image Generation](./configuration/image-generation.md) for configuration options related to Image Generation
 - [📖 Usage / 🖌️ Image Generation](./usage.md#-image-generation) section for more details on how to use the bot for Image Generation in a room
 - [🫵 Sticker Generation](#-sticker-generation) - a special case of Image Generation
 

docs/installation.md

@@ -15,8 +15,16 @@
 
 We provide prebuilt container images for the `amd64` and `arm64` architectures, so **you don't necessarily need to build images yourself** and can jump to [Running in a container](#-running-in-a-container).
 
-If you nevertheless wish to build a container image yourself, you can do so by running `just build-container-image`.
+If you nevertheless wish to build a container image yourself, you can do so by running:
-This will build and tag your container image as `localhost/baibot:latest`.
+
+- (recommended) `just build-container-image-release` to build a release version of the container image
+
+- or `just build-container-image-debug` to build a debug version of the container image
+
+Debug images are faster to build but are larger in size.
+Release images are ~5x smaller in size, but are slower to build.
+
+Both of these commands will build and tag your container image as `localhost/baibot:latest`.
 
 
 ### 🐋 Running in a container

docs/providers.md

@@ -23,17 +23,20 @@ The list of supported providers is below.
 
 ### How to choose a provider
 
-If you're not sure which provider to start with, we **recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation), [🖌️ image-generation](./features.md#️-image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech).
+If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [💬 text-generation](./features.md#-text-generation), [🖌️ image-generation](./features.md#️-image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [🗣️ text-to-speech](./features.md#️-text-to-speech).
 
 You don't need to choose just one though. The bot supports [mixing & matching models](./features.md#-mixing--matching-models), so you can use multiple providers at the same time.
 
 
 ### How to use a provider
 
-- sign up for it
+1. 📝 **Sign up for it**
-- obtain an API key
+
-- [create a new agent](./agents.md#creating-agents)
+2. 🔑 **Obtain an API key**
-- set it as a handler for some types of messages (see [Mixing & matching models](./features.md#-mixing--matching-models)) for a specific room or globally
+
+3. 🤖 **Create one or more agents** in a given room or globally. Next to each provider in the [list below](#supported-providers) you'll see **🗲 Quick start** commands, but you may also refer to the [agent creation guide](./agents.md#creating-agents).
+
+4. 🤝 **Set the new agent as a handler** for a given use-purpose like text-generation, image-generation, etc. The agent creation wizard will tell you how, but you may also refer to the [🤝 Handlers](./configuration/handlers.md) guide.
 
 
 ### Supported providers
@@ -49,7 +52,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
   - create a room-local agent: `!bai agent create-room-local anthropic my-anthropic-agent`
   - create a global agent: `!bai agent create-global anthropic my-anthropic-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/anthropic.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/anthropic.yml).
 
 
 ### Groq
@@ -63,7 +66,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
   - create a room-local agent: `!bai agent create-room-local groq my-groq-agent`
   - create a global agent: `!bai agent create-global groq my-groq-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/groq.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/groq.yml).
 
 
 ### LocalAI
@@ -77,7 +80,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
   - create a room-local agent: `!bai agent create-room-local localai my-localai-agent`
   - create a global agent: `!bai agent create-global localai my-localai-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/localai.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/localai.yml).
 
 
 ### Mistral
@@ -91,7 +94,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
   - create a room-local agent: `!bai agent create-room-local mistral my-mistral-agent`
   - create a global agent: `!bai agent create-global mistral my-mistral-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/mistral.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/mistral.yml).
 
 
 ### Ollama
@@ -105,7 +108,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo
   - create a room-local agent: `!bai agent create-room-local ollama my-ollama-agent`
   - create a global agent: `!bai agent create-global ollama my-ollama-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/ollama.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/ollama.yml).
 
 
 ### OpenAI
@@ -122,7 +125,10 @@ For services which are not fully compatible with the OpenAI API, consider using
   - create a room-local agent: `!bai agent create-room-local openai my-openai-agent`
   - create a global agent: `!bai agent create-global openai my-openai-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/openai.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which:
+
+- in the general case looks [like this](./sample-provider-configs/openai.yml)
+- for the [o1](https://platform.openai.com/docs/models/o1) models needs to look [like this](./sample-provider-configs/openai-o1.yml)
 
 
 ### OpenAI Compatible
@@ -139,7 +145,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
   - create a room-local agent: `!bai agent create-room-local openai-compatible my-openai-compatible-agent`
   - create a global agent: `!bai agent create-global openai-compatible my-openai-compatible-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/openai-compatible.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/openai-compatible.yml).
 
 
 ### OpenRouter
@@ -153,7 +159,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
   - create a room-local agent: `!bai agent create-room-local openrouter my-openrouter-agent`
   - create a global agent: `!bai agent create-global openrouter my-openrouter-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/openrouter.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/openrouter.yml).
 
 
 ### Together AI
@@ -167,4 +173,4 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo
   - create a room-local agent: `!bai agent create-room-local together-ai my-together-ai-agent`
   - create a global agent: `!bai agent create-global together-ai my-together-ai-agent`
 
-💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](../sample-provider-configs/together-ai.yml).
+💡 When creating an agent, the bot will show you an up-to-date sample configuration for this provider which looks [like this](./sample-provider-configs/together-ai.yml).

docs/sample-provider-configs/anthropic.yml

@@ -2,7 +2,7 @@ base_url: https://api.anthropic.com/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: claude-3-5-sonnet-20240620
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 8192
   max_context_tokens: 204800

docs/sample-provider-configs/groq.yml

@@ -2,7 +2,7 @@ base_url: https://api.groq.com/openai/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: llama3-70b-8192
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 131072

docs/sample-provider-configs/localai.yml

@@ -2,7 +2,7 @@ base_url: http://my-localai-self-hosted-service:8080/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: gpt-4
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/mistral.yml

@@ -2,7 +2,7 @@ base_url: https://api.mistral.ai/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: mistral-large-latest
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/ollama.yml

@@ -2,7 +2,7 @@ base_url: http://my-ollama-self-hosted-service:11434/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: gemma2:2b
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/openai-compatible.yml

@@ -2,7 +2,7 @@ base_url: ''
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: some-model
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/openai-o1.yml

@@ -0,0 +1,24 @@
+base_url: https://api.openai.com/v1
+api_key: YOUR_API_KEY_HERE
+text_generation:
+  model_id: o1-mini
+  # o1 models do not support a system prompt
+  prompt: null
+  temperature: 1.0
+  # o1 models do not support max_response_tokens.
+  # They use `max_completion_tokens` as an alternative,
+  # but we don't support it yet (see https://github.com/64bit/async-openai/issues/272).
+  max_response_tokens: null
+  max_context_tokens: 128000
+speech_to_text:
+  model_id: whisper-1
+text_to_speech:
+  model_id: tts-1-hd
+  voice: onyx
+  speed: 1.0
+  response_format: opus
+image_generation:
+  model_id: dall-e-3
+  style: vivid
+  size: 1024x1024
+  quality: standard

docs/sample-provider-configs/openai.yml

@@ -1,8 +1,8 @@
 base_url: https://api.openai.com/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
-  model_id: gpt-4o-2024-08-06
+  model_id: gpt-4o
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 16384
   max_context_tokens: 128000

docs/sample-provider-configs/openrouter.yml

@@ -2,7 +2,7 @@ base_url: https://openrouter.ai/api/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: mattshumer/reflection-70b:free
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 2048
   max_context_tokens: 8192

docs/sample-provider-configs/together-ai.yml

@@ -2,7 +2,7 @@ base_url: https://api.together.xyz/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
-  prompt: You are a brief, but helpful bot.
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 2048
   max_context_tokens: 8192

docs/usage.md

@@ -11,10 +11,11 @@ This is related to the [💬 Text Generation](./features.md#-text-generation) fe
 
 If there's a text-generation handler agent configured, the bot **may** respond to messages sent in the room.
 
-🖼️ See screenshots of:
+See screenshots of:
 
-- the [default Text Generation flow](./screenshots/text-generation.webp) for 1:1 rooms
+- 🖼️ [the default Text Generation flow](./screenshots/text-generation.webp) in 1:1 rooms
-- the [Text Generation flow in multi-user rooms](./screenshots/text-generation-prefix-requirement.webp) (where the [🗟 Prefix Requirement](./configuration/text-generation.md#-prefix-requirement-type) setting is auto-configured to "required")
+- 🖼️ [the Text Generation flow in multi-user rooms](./screenshots/text-generation-prefix-requirement.webp) (where the [🗟 Prefix Requirement](./configuration/text-generation.md#-prefix-requirement-type) setting is auto-configured to "required")
+- the [on-demand involvement](./features.md#on-demand-involvement) feature
 
 Whether the bot responds depends on:
 
@@ -24,9 +25,9 @@ Whether the bot responds depends on:
 
 - (🎨 agent capabilities) whether the configured `text-generation` (or `catch-all`) handler agent actually supports text-generation. The provider may lack support for this feature or it may be disabled in the [🤖 agents](./agents.md) configuration
 
-- (the [🗟 Prefix Requirement](./configuration/text-generation.md#-prefix-requirement-type) setting) whether a prefix (e.g. `!bai`) is required in front of messages sent to the room. For multi-user rooms, this setting defaults to "required"
+- (the [🗟 Prefix Requirement](./configuration/text-generation.md#-prefix-requirement-type) setting) whether a prefix (e.g. `!bai`) or user mention (e.g. `@baibot`) is required for messages sent to the room. For multi-user rooms, this setting defaults to "required". See [🌟 Features / 💬 Text Generation / On-demand involvement](./features.md#on-demand-involvement) for details.
 
-Room messages start a threaded conversation where you can continue back-and-forth communication with the bot.
+Room messages start a threaded conversation where you can continue back-and-forth communication with the bot. Using [on-demand involvement](./features.md#on-demand-involvement), you can can also mention the bot to provoke it to get involved in any conversation thread or reply chain.
 
 Unless you've enabled the [♻️ Context Management](./features.md#️-context-management) feature, all messages will be sent to the agent's API each time. If the context management feature is enabled, older messages may be dropped.
 

etc/app/config.yml.dist

@@ -72,8 +72,8 @@ agents:
     #     base_url: https://api.openai.com/v1
     #     api_key: ""
     #     text_generation:
-    #       model_id: gpt-4o-2024-08-06
+    #       model_id: gpt-4o
-    #       prompt: You are a brief, but helpful bot.
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 16384
     #       max_context_tokens: 128000
@@ -97,7 +97,7 @@ agents:
     #     api_key: null
     #     text_generation:
     #       model_id: gpt-4
-    #       prompt: You are a brief, but helpful bot.
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 16384
     #       max_context_tokens: 128000
@@ -122,7 +122,7 @@ agents:
     #     api_key: null
     #     text_generation:
     #       model_id: "gemma2:2b"
-    #       prompt: "You are an assistant based on the gemma2:2b model. Be brief in your responses."
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 4096
     #       max_context_tokens: 128000

etc/services/core/compose.yml

@@ -1,6 +1,6 @@
 services:
   postgres:
-    image: docker.io/postgres:16.3-alpine
+    image: docker.io/postgres:16.4-alpine
     user: ${UID}:${GID}
     restart: unless-stopped
     environment:
@@ -13,7 +13,7 @@ services:
     - /etc/passwd:/etc/passwd:ro
 
   synapse:
-    image: ghcr.io/element-hq/synapse:v1.114.0
+    image: ghcr.io/element-hq/synapse:v1.118.0
     user: "${UID}:${GID}"
     restart: unless-stopped
     entrypoint: python
@@ -26,7 +26,7 @@ services:
     - ./synapse/media-store:/media-store
 
   element-web:
-    image: docker.io/vectorim/element-web:v1.11.77
+    image: docker.io/vectorim/element-web:v1.11.84
     user: "${UID}:${GID}"
     restart: unless-stopped
     ports:

etc/services/core/synapse/config/homeserver.yaml

@@ -579,7 +579,9 @@ rc_login:
 #
 #federation_rr_transactions_per_room_per_second: 50
 
-
+# Authenticated media is not supported yet.
+# See: https://github.com/etkecc/baibot/issues/12
+enable_authenticated_media: false
 
 # Directory where uploaded images and attachments are stored.
 #

etc/services/ollama/compose.yml

@@ -1,6 +1,6 @@
 services:
   ollama:
-    image: docker.io/ollama/ollama:0.3.9
+    image: docker.io/ollama/ollama:0.4.1
     restart: unless-stopped
     ports:
       - "${SERVICE_OLLAMA_BIND_PORT_HTTP}:11434"

justfile

@@ -13,7 +13,8 @@ run-locally *extra_args: app-local-prepare
 	BAIBOT_PERSISTENCE_DATA_DIR_PATH={{ justfile_directory() }}/var/app/local/data \
 	cargo run -- {{ extra_args }}
 
-run-in-container *extra_args: app-container-prepare build-container-image
+# Builds and runs the bot in a container
+run-in-container *extra_args: app-container-prepare build-container-image-debug
 	/usr/bin/env docker run \
 	-it \
 	--rm \
@@ -25,7 +26,7 @@ run-in-container *extra_args: app-container-prepare build-container-image
 	--env BAIBOT_PERSISTENCE_DATA_DIR_PATH=/data \
 	--mount type=bind,src={{ justfile_directory() }}/var/app/container/config.yml,dst=/app/config.yml,ro \
 	--mount type=bind,src={{ justfile_directory() }}/var/app/container/data,dst=/data \
-	{{ container_image_name }} {{ extra_args }}
+	{{ container_image_name }}:latest {{ extra_args }}
 
 # Runs tests
 test *extra_args:
@@ -38,9 +39,16 @@ build-debug *extra_args:
 # Builds an optimized release binary (target/release/*)
 build-release *extra_args: (build-debug "--release")
 
-# Builds a container image
+# Builds a container image (debug mode)
-build-container-image tag='latest':
+build-container-image-debug tag='latest': (_build-container-image "false" tag)
+
+# Builds a container image (release mode)
+build-container-image-release tag='latest': (_build-container-image "true" tag)
+
+_build-container-image release_build tag:
 	/usr/bin/env docker build \
+	--build-arg RELEASE_BUILD={{ release_build }} \
+	-f {{ justfile_directory() }}/Dockerfile \
 	-t {{ container_image_name }}:{{ tag }} \
 	.
 

src/agent/mod.rs

@@ -19,3 +19,7 @@ pub use instantiation::Result as AgentInstantiationResult;
 
 pub use provider::{AgentProvider, AgentProviderInfo, ControllerTrait};
 pub use purpose::AgentPurpose;
+
+pub(super) fn default_prompt() -> &'static str {
+    "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
+}

src/agent/provider/anthropic/config.rs

@@ -2,7 +2,7 @@ use serde::{Deserialize, Serialize};
 
 use anthropic_rs::models::claude::ClaudeModel;
 
-use crate::agent::provider::ConfigTrait;
+use crate::agent::{default_prompt, provider::ConfigTrait};
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct Config {
@@ -58,7 +58,7 @@ impl Default for TextGenerationConfig {
     fn default() -> Self {
         Self {
             model_id: default_text_model_id(),
-            prompt: Some("You are a brief, but helpful bot.".to_owned()),
+            prompt: Some(default_prompt().to_owned()),
             temperature: super::super::default_temperature(),
             max_response_tokens: 8192,
             max_context_tokens: 204_800,

src/agent/provider/anthropic/controller.rs

@@ -2,7 +2,7 @@ use std::fmt::Debug;
 use std::str::FromStr;
 use std::sync::Arc;
 
-use anthropic_rs::completion::message::ContentType;
+use anthropic_rs::completion::message::{ContentType, System};
 use anthropic_rs::{
     client::Client as AnthropicClient, config::Config as AnthropicConfig,
     models::claude::ClaudeModel,
@@ -72,6 +72,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -95,11 +96,12 @@ impl ControllerTrait for Controller {
             ));
         };
 
-        let prompt_text = params
+        let prompt_text = params.prompt_variables.format(
-            .prompt_override
+            params
-            .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
+                .prompt_override
-            .trim()
+                .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
-            .to_owned();
+                .trim(),
+        );
 
         let prompt_message = if prompt_text.is_empty() {
             None
@@ -107,9 +109,19 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 
+        // Avoid the situation where multiple user or assistant messages are sent consecutively,
+        // to avoid errors like:
+        // > API error: Error response: error Api error: invalid_request_error messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
+        // as reported here: https://github.com/etkecc/baibot/issues/13
+        //
+        // As https://docs.anthropic.com/en/api/messages says:
+        // > Our models are trained to operate on alternating user and assistant conversational turns.
+        let conversation = conversation.combine_consecutive_messages();
+
         let mut conversation_messages = conversation.messages;
 
         if params.context_management_enabled {
@@ -119,7 +131,7 @@ impl ControllerTrait for Controller {
                 &text_generation_config.model_id,
                 &prompt_message,
                 conversation_messages,
-                text_generation_config.max_response_tokens,
+                Some(text_generation_config.max_response_tokens),
                 text_generation_config.max_context_tokens,
             );
 
@@ -147,7 +159,7 @@ impl ControllerTrait for Controller {
             .unwrap_or(text_generation_config.temperature);
 
         if let Some(prompt_message) = prompt_message {
-            request.system = Some(prompt_message.message_text);
+            request.system = Some(System::Text(prompt_message.message_text));
         }
 
         request.model = model;
@@ -225,20 +237,25 @@ impl ControllerTrait for Controller {
         }
     }
 
-    fn text_generation_prompt(&self) -> Option<String> {
+    fn text_generation_model_id(&self) -> Option<String> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
+            .map(|config| config.model_id.to_owned())
+    }
 
-        text_generation_config.prompt.clone()
+    fn text_generation_prompt(&self) -> Option<String> {
+        self.config
+            .text_generation
+            .as_ref()
+            .and_then(|config| config.prompt.clone())
     }
 
     fn text_generation_temperature(&self) -> Option<f32> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
-
+            .map(|config| config.temperature)
-        Some(text_generation_config.temperature)
     }
 
     fn text_to_speech_voice(&self) -> Option<String> {

src/agent/provider/controller.rs

@@ -13,6 +13,8 @@ pub trait ControllerTrait {
 
     fn ping(&self) -> impl std::future::Future<Output = anyhow::Result<PingResult>> + Send;
 
+    fn text_generation_model_id(&self) -> Option<String>;
+
     fn text_generation_prompt(&self) -> Option<String>;
 
     fn text_generation_temperature(&self) -> Option<f32>;
@@ -63,6 +65,14 @@ impl ControllerTrait for ControllerType {
         }
     }
 
+    fn text_generation_model_id(&self) -> Option<String> {
+        match &self {
+            ControllerType::OpenAI(controller) => controller.text_generation_model_id(),
+            ControllerType::OpenAICompat(controller) => controller.text_generation_model_id(),
+            ControllerType::Anthropic(controller) => controller.text_generation_model_id(),
+        }
+    }
+
     fn text_generation_prompt(&self) -> Option<String> {
         match &self {
             ControllerType::OpenAI(controller) => controller.text_generation_prompt(),

src/agent/provider/entity/mod.rs

@@ -9,5 +9,7 @@ pub use agent_provider::{AgentProvider, AgentProviderInfo};
 pub use image_generation::{ImageGenerationParams, ImageGenerationResult};
 pub use ping::PingResult;
 pub use speech_to_text::{SpeechToTextParams, SpeechToTextResult};
-pub use text_generation::{TextGenerationParams, TextGenerationResult};
+pub use text_generation::{
+    TextGenerationParams, TextGenerationPromptVariables, TextGenerationResult,
+};
 pub use text_to_speech::{TextToSpeechParams, TextToSpeechResult};

src/agent/provider/entity/text_generation/mod.rs

@@ -1,8 +1,13 @@
+mod prompt_variables;
+
+pub use prompt_variables::TextGenerationPromptVariables;
+
 #[derive(Default)]
 pub struct TextGenerationParams {
     pub context_management_enabled: bool,
     pub prompt_override: Option<String>,
     pub temperature_override: Option<f32>,
+    pub prompt_variables: TextGenerationPromptVariables,
 }
 
 pub struct TextGenerationResult {

src/agent/provider/entity/text_generation/prompt_variables.rs

@@ -0,0 +1,106 @@
+use chrono::{DateTime, Utc};
+use std::collections::HashMap;
+
+pub struct TextGenerationPromptVariables {
+    map: HashMap<String, String>,
+}
+
+impl Default for TextGenerationPromptVariables {
+    fn default() -> Self {
+        let now = Utc::now();
+        Self::new("unnamed", "unknown-model", now, Some(now))
+    }
+}
+
+impl TextGenerationPromptVariables {
+    pub fn new(
+        bot_name: &str,
+        model_id: &str,
+        now_time: DateTime<Utc>,
+        conversation_start_time: Option<DateTime<Utc>>,
+    ) -> Self {
+        let mut map = HashMap::new();
+
+        map.insert("baibot_name".to_string(), bot_name.to_string());
+        map.insert("baibot_model_id".to_string(), model_id.to_string());
+        map.insert("baibot_now_utc".to_string(), format_utc_time(now_time));
+
+        let baibot_conversation_start_time_utc = match conversation_start_time {
+            Some(conversation_start_time) => format_utc_time(conversation_start_time),
+            None => "unknown".to_string(),
+        };
+
+        map.insert(
+            "baibot_conversation_start_time_utc".to_string(),
+            baibot_conversation_start_time_utc,
+        );
+
+        Self { map }
+    }
+
+    pub fn format(&self, text: &str) -> String {
+        let mut formatted_text = text.to_string();
+
+        for (key, value) in &self.map {
+            let placeholder = format!("{{{{ {} }}}}", key);
+            formatted_text = formatted_text.replace(&placeholder, value);
+        }
+
+        formatted_text
+    }
+}
+
+fn format_utc_time(time: DateTime<Utc>) -> String {
+    time.format("%Y-%m-%d (%A), %H:%M:%S UTC").to_string()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use chrono::{TimeZone, Timelike};
+
+    #[test]
+    fn test_new() {
+        // Intentionally injecting some sub-seconds to ensure formatting would ignore them.
+        let now_utc = Utc
+            .with_ymd_and_hms(2024, 9, 20, 18, 34, 15)
+            .unwrap()
+            .with_nanosecond(250000000)
+            .unwrap();
+
+        let conversation_start_time_utc = Utc
+            .with_ymd_and_hms(2024, 9, 19, 18, 34, 15)
+            .unwrap()
+            .with_nanosecond(250000000)
+            .unwrap();
+
+        let variables = TextGenerationPromptVariables::new(
+            "baibot",
+            "gpt-4o",
+            now_utc,
+            Some(conversation_start_time_utc),
+        );
+
+        assert_eq!(
+            variables.map.get("baibot_name"),
+            Some(&"baibot".to_string())
+        );
+        assert_eq!(
+            variables.map.get("baibot_model_id"),
+            Some(&"gpt-4o".to_string())
+        );
+        assert_eq!(
+            variables.map.get("baibot_now_utc"),
+            Some(&format_utc_time(now_utc))
+        );
+        assert_eq!(
+            variables.map.get("baibot_conversation_start_time_utc"),
+            Some(&format_utc_time(conversation_start_time_utc))
+        );
+
+        let prompt = "Hello, I'm {{ baibot_name }} using {{ baibot_model_id }}. The date/time now is {{ baibot_now_utc }} and this conversation started at {{ baibot_conversation_start_time_utc }}.";
+        let expected = "Hello, I'm baibot using gpt-4o. The date/time now is 2024-09-20 (Friday), 18:34:15 UTC and this conversation started at 2024-09-19 (Thursday), 18:34:15 UTC.";
+
+        assert_eq!(variables.format(prompt), expected);
+    }
+}

src/agent/provider/groq/mod.rs

@@ -15,7 +15,7 @@ pub fn default_config() -> Config {
     if let Some(ref mut config) = config.text_generation.as_mut() {
         config.model_id = "llama3-70b-8192".to_owned();
         config.max_context_tokens = 131_072;
-        config.max_response_tokens = 4096;
+        config.max_response_tokens = Some(4096);
     }
 
     if let Some(ref mut config) = config.speech_to_text.as_mut() {

src/agent/provider/localai/mod.rs

@@ -1,6 +1,5 @@
-// LocalAI is based on OpenAI (async-openai), because it seems to be fully compatible.
+// At the time of testing, LocalAI can be powered by `openai`, but we use `openai_compat` for better reliability
-// Moreover, openai_api_rust does not support speech-to-text, so if we wish to use this feature
+// in the event of future updates to `async-openai`.
-// we need to stick to async-openai.
 
 use super::openai_compat::Config;
 
@@ -14,7 +13,7 @@ pub fn default_config() -> Config {
     if let Some(ref mut config) = config.text_generation.as_mut() {
         config.model_id = "gpt-4".to_owned();
         config.max_context_tokens = 128_000;
-        config.max_response_tokens = 4096;
+        config.max_response_tokens = Some(4096);
     }
 
     if let Some(ref mut config) = config.text_to_speech.as_mut() {

src/agent/provider/mod.rs

@@ -21,5 +21,5 @@ pub use config::ConfigTrait;
 
 pub use entity::{
     AgentProvider, AgentProviderInfo, ImageGenerationParams, PingResult, SpeechToTextParams,
-    SpeechToTextResult, TextGenerationParams, TextToSpeechParams,
+    SpeechToTextResult, TextGenerationParams, TextGenerationPromptVariables, TextToSpeechParams,
 };

src/agent/provider/ollama/mod.rs

@@ -17,7 +17,7 @@ pub fn default_config() -> Config {
     if let Some(ref mut config) = config.text_generation.as_mut() {
         config.model_id = "gemma2:2b".to_owned();
         config.max_context_tokens = 128_000;
-        config.max_response_tokens = 4096;
+        config.max_response_tokens = Some(4096);
     }
 
     config

src/agent/provider/openai/config.rs

@@ -1,6 +1,6 @@
 use serde::{Deserialize, Serialize};
 
-use crate::agent::provider::ConfigTrait;
+use crate::agent::{default_prompt, provider::ConfigTrait};
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct Config {
@@ -56,7 +56,7 @@ pub struct TextGenerationConfig {
     pub temperature: f32,
 
     #[serde(default)]
-    pub max_response_tokens: u32,
+    pub max_response_tokens: Option<u32>,
 
     #[serde(default)]
     pub max_context_tokens: u32,
@@ -66,16 +66,16 @@ impl Default for TextGenerationConfig {
     fn default() -> Self {
         Self {
             model_id: default_text_model_id(),
-            prompt: Some("You are a brief, but helpful bot.".to_owned()),
+            prompt: Some(default_prompt().to_owned()),
             temperature: super::super::default_temperature(),
-            max_response_tokens: 16_384,
+            max_response_tokens: Some(16_384),
             max_context_tokens: 128_000,
         }
     }
 }
 
 fn default_text_model_id() -> String {
-    "gpt-4o-2024-08-06".to_owned()
+    "gpt-4o".to_owned()
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]

src/agent/provider/openai/controller.rs

@@ -63,6 +63,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -86,11 +87,12 @@ impl ControllerTrait for Controller {
             ));
         };
 
-        let prompt_text = params
+        let prompt_text = params.prompt_variables.format(
-            .prompt_override
+            params
-            .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
+                .prompt_override
-            .trim()
+                .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
-            .to_owned();
+                .trim(),
+        );
 
         let prompt_message = if prompt_text.is_empty() {
             None
@@ -98,6 +100,7 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 
@@ -130,12 +133,18 @@ impl ControllerTrait for Controller {
             .temperature_override
             .unwrap_or(text_generation_config.temperature);
 
-        let request = CreateChatCompletionRequestArgs::default()
+        let mut request_builder = CreateChatCompletionRequestArgs::default();
-            .max_tokens(text_generation_config.max_response_tokens)
+
+        request_builder
             .model(&text_generation_config.model_id)
             .temperature(temperature)
-            .messages(openai_conversation_messages)
+            .messages(openai_conversation_messages);
-            .build()?;
+
+        if let Some(max_response_tokens) = text_generation_config.max_response_tokens {
+            request_builder.max_tokens(max_response_tokens);
+        }
+
+        let request = request_builder.build()?;
 
         if let Ok(request_as_json) = serde_json::to_string(&request) {
             tracing::trace!(
@@ -391,20 +400,25 @@ impl ControllerTrait for Controller {
         }
     }
 
-    fn text_generation_prompt(&self) -> Option<String> {
+    fn text_generation_model_id(&self) -> Option<String> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
+            .map(|config| config.model_id.to_owned())
+    }
 
-        text_generation_config.prompt.clone()
+    fn text_generation_prompt(&self) -> Option<String> {
+        self.config
+            .text_generation
+            .as_ref()
+            .and_then(|config| config.prompt.clone())
     }
 
     fn text_generation_temperature(&self) -> Option<f32> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
-
+            .map(|config| config.temperature)
-        Some(text_generation_config.temperature)
     }
 
     fn text_to_speech_voice(&self) -> Option<String> {

src/agent/provider/openai_compat/config.rs

@@ -1,5 +1,6 @@
 use serde::{Deserialize, Serialize};
 
+use crate::agent::default_prompt;
 use crate::agent::provider::openai::{
     ImageGenerationConfig as OpenAIImageGenerationConfig,
     SpeechToTextConfig as OpenAISpeechToTextConfig,
@@ -65,7 +66,7 @@ pub struct TextGenerationConfig {
     pub temperature: f32,
 
     #[serde(default)]
-    pub max_response_tokens: u32,
+    pub max_response_tokens: Option<u32>,
 
     #[serde(default)]
     pub max_context_tokens: u32,
@@ -75,9 +76,9 @@ impl Default for TextGenerationConfig {
     fn default() -> Self {
         Self {
             model_id: default_text_model_id(),
-            prompt: Some("You are a brief, but helpful bot.".to_owned()),
+            prompt: Some(default_prompt().to_owned()),
             temperature: super::super::default_temperature(),
-            max_response_tokens: 4096,
+            max_response_tokens: Some(4096),
             max_context_tokens: 128_000,
         }
     }

src/agent/provider/openai_compat/controller.rs

@@ -1,7 +1,7 @@
-use openai_api_rust::audio::{AudioApi, AudioBody};
+use etke_openai_api_rust::audio::{AudioApi, AudioBody};
-use openai_api_rust::chat::{ChatApi, ChatBody};
+use etke_openai_api_rust::chat::{ChatApi, ChatBody};
-use openai_api_rust::images::{ImagesApi, ImagesBody};
+use etke_openai_api_rust::images::{ImagesApi, ImagesBody};
-use openai_api_rust::{Auth, Message, OpenAI};
+use etke_openai_api_rust::{Auth, Message, OpenAI};
 
 use super::super::ControllerTrait;
 use crate::agent::utils::base64_decode;
@@ -61,6 +61,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -84,11 +85,12 @@ impl ControllerTrait for Controller {
             ));
         };
 
-        let prompt_text = params
+        let prompt_text = params.prompt_variables.format(
-            .prompt_override
+            params
-            .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
+                .prompt_override
-            .trim()
+                .unwrap_or(self.text_generation_prompt().unwrap_or("".to_owned()))
-            .to_owned();
+                .trim(),
+        );
 
         let prompt_message = if prompt_text.is_empty() {
             None
@@ -96,6 +98,7 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 
@@ -130,12 +133,15 @@ impl ControllerTrait for Controller {
 
         let max_tokens = text_generation_config
             .max_response_tokens
-            .try_into()
+            .map(|max_response_tokens| {
-            .expect("Failed converting max_response_tokens from u32 to i32");
+                max_response_tokens
+                    .try_into()
+                    .expect("Failed converting max_response_tokens from u32 to i32")
+            });
 
         let request = ChatBody {
             model: text_generation_config.model_id.clone(),
-            max_tokens: Some(max_tokens),
+            max_tokens,
             temperature: Some(temperature),
             top_p: None,
             n: Some(1),
@@ -409,20 +415,25 @@ impl ControllerTrait for Controller {
         }
     }
 
-    fn text_generation_prompt(&self) -> Option<String> {
+    fn text_generation_model_id(&self) -> Option<String> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
+            .map(|config| config.model_id.to_owned())
+    }
 
-        text_generation_config.prompt.clone()
+    fn text_generation_prompt(&self) -> Option<String> {
+        self.config
+            .text_generation
+            .as_ref()
+            .and_then(|config| config.prompt.clone())
     }
 
     fn text_generation_temperature(&self) -> Option<f32> {
-        let Some(text_generation_config) = &self.config.text_generation else {
+        self.config
-            return None;
+            .text_generation
-        };
+            .as_ref()
-
+            .map(|config| config.temperature)
-        Some(text_generation_config.temperature)
     }
 
     fn text_to_speech_voice(&self) -> Option<String> {

src/agent/provider/openai_compat/mod.rs

@@ -56,7 +56,7 @@ pub fn default_config() -> Config {
 
     if let Some(text_generation) = &mut config.text_generation {
         text_generation.model_id = "some-model".to_string();
-        text_generation.max_response_tokens = 4096;
+        text_generation.max_response_tokens = Some(4096);
         text_generation.max_context_tokens = 128_000;
     }
 

src/agent/provider/openai_compat/utils.rs

@@ -1,4 +1,4 @@
-use openai_api_rust::{Message, Role};
+use etke_openai_api_rust::{Message, Role};
 
 use crate::agent::provider::openai::Config as OpenAIConfig;
 

src/agent/provider/openrouter/mod.rs

@@ -14,7 +14,7 @@ pub fn default_config() -> Config {
     if let Some(ref mut config) = config.text_generation.as_mut() {
         config.model_id = "mattshumer/reflection-70b:free".to_owned();
         config.max_context_tokens = 8192;
-        config.max_response_tokens = 2048;
+        config.max_response_tokens = Some(2048);
     }
 
     config

src/agent/provider/togetherai/mod.rs

@@ -14,7 +14,7 @@ pub fn default_config() -> Config {
     if let Some(ref mut config) = config.text_generation.as_mut() {
         config.model_id = "meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo".to_owned();
         config.max_context_tokens = 8192;
-        config.max_response_tokens = 2048;
+        config.max_response_tokens = Some(2048);
     }
 
     config

src/bot/implementation.rs

@@ -1,7 +1,7 @@
 use std::sync::Arc;
 use std::{future::Future, pin::Pin};
 
-use mxlink::matrix_sdk::media::{MediaFormat, MediaRequest};
+use mxlink::matrix_sdk::media::{MediaFormat, MediaRequestParameters};
 use mxlink::matrix_sdk::ruma::{
     events::room::MediaSource, MilliSecondsSinceUnixEpoch, OwnedUserId,
 };
@@ -9,6 +9,7 @@ use mxlink::matrix_sdk::Room;
 
 use mxlink::{
     InitConfig, LoginConfig, LoginCredentials, LoginEncryption, MatrixLink, PersistenceConfig,
+    TypingNoticeGuard,
 };
 
 use mxlink::helpers::account_data_config::{
@@ -171,6 +172,24 @@ impl Bot {
         self.matrix_link().user_id()
     }
 
+    pub(crate) async fn user_display_name_in_room(&self, room: &Room) -> Option<String> {
+        let bot_display_name = self
+            .room_display_name_fetcher()
+            .own_display_name_in_room(room)
+            .await;
+
+        match bot_display_name {
+            Ok(value) => value,
+            Err(err) => {
+                tracing::warn!(
+                    ?err,
+                    "Failed to fetch bot display name. Proceeding without it"
+                );
+                None
+            }
+        }
+    }
+
     pub(crate) fn reacting(&self) -> super::reacting::Reacting {
         super::reacting::Reacting::new(self.clone())
     }
@@ -210,6 +229,14 @@ impl Bot {
             .await
     }
 
+    pub(crate) async fn start_typing_notice(&self, room: &Room) -> TypingNoticeGuard {
+        self.inner
+            .matrix_link
+            .rooms()
+            .start_typing_notice(room)
+            .await
+    }
+
     pub async fn start(&self) -> anyhow::Result<()> {
         self.rooms().attach_event_handlers().await;
         self.messaging().attach_event_handlers().await;
@@ -260,7 +287,7 @@ impl Bot {
         let desired_display_name = self.inner.config.user.name.clone();
 
         let profile = account
-            .get_profile()
+            .fetch_user_profile()
             .await
             .map_err(|e| anyhow::anyhow!("Failed fetching profile: {:?}", e))?;
 
@@ -283,7 +310,7 @@ impl Bot {
 
         let should_update_avatar = match &profile.avatar_url {
             Some(avatar_url) => {
-                let request = MediaRequest {
+                let request = MediaRequestParameters {
                     source: MediaSource::Plain(avatar_url.to_owned()),
                     format: MediaFormat::File,
                 };

src/bot/messaging.rs

@@ -11,7 +11,7 @@ use mxlink::{CallbackError, MessageResponseType};
 use tracing::Instrument;
 
 use crate::{
-    conversation::matrix::determine_thread_context_for_room_event,
+    conversation::matrix::determine_interaction_context_for_room_event,
     entity::{MessageContext, MessagePayload, RoomConfigContext, TriggerEventInfo},
 };
 
@@ -239,8 +239,11 @@ impl Messaging {
             }
         };
 
-        let thread_context = determine_thread_context_for_room_event(
+        let bot_display_name = self.bot.user_display_name_in_room(&room).await;
+
+        let interaction_context = determine_interaction_context_for_room_event(
             self.bot.user_id(),
+            &bot_display_name,
             &room,
             &event,
             &payload,
@@ -248,16 +251,18 @@ impl Messaging {
         )
         .await;
 
-        let thread_context = match thread_context {
+        let interaction_context = match interaction_context {
             Ok(value) => value,
             Err(err) => {
-                tracing::error!(?err, "Failed to determine thread context for event");
+                tracing::error!(?err, "Failed to determine interaction context for event");
                 return Ok(());
             }
         };
 
-        let Some(thread_context) = thread_context else {
+        let Some(interaction_context) = interaction_context else {
-            tracing::debug!("Ignoring message with unknown thread context (likely not a threaded message or a top-level message)");
+            tracing::debug!(
+                "Ignoring message with unknown interaction context (likely not a message for us)"
+            );
             return Ok(());
         };
 
@@ -276,33 +281,14 @@ impl Messaging {
             room_config_context,
             self.bot.admin_pattern_regexes().clone(),
             trigger_event_info,
-            thread_context.info.clone(),
+            interaction_context.thread_info.clone(),
-        );
+        )
+        .with_bot_display_name(bot_display_name);
 
-        let bot_display_name = self
-            .bot
-            .room_display_name_fetcher()
-            .own_display_name_in_room(message_context.room())
-            .await;
-
-        let bot_display_name = match bot_display_name {
-            Ok(value) => value,
-            Err(err) => {
-                tracing::warn!(
-                    ?err,
-                    "Failed to fetch bot display name. Proceeding without it"
-                );
-                None
-            }
-        };
-
-        // The first event in the thread determines which handler processes the current event.
         let controller_type = crate::controller::determine_controller(
             self.bot.command_prefix(),
-            &thread_context.first_message,
+            &interaction_context.trigger,
             &message_context,
-            self.bot.user_id(),
-            &bot_display_name,
         );
 
         tracing::info!(?controller_type, "Determined controller");
@@ -310,7 +296,7 @@ impl Messaging {
         let _ = room
             .send_single_receipt(
                 ReceiptType::Read,
-                thread_context.info.clone().into(),
+                interaction_context.thread_info.clone().into(),
                 event.event_id.clone(),
             )
             .await;

src/bot/reacting.rs

@@ -1,8 +1,8 @@
 use mxlink::matrix_sdk::{
     ruma::{
         events::{
-            room::message::Relation, AnyMessageLikeEvent, AnySyncTimelineEvent, AnyTimelineEvent,
+            room::message::Relation, AnySyncMessageLikeEvent, AnySyncTimelineEvent,
-            MessageLikeEvent,
+            SyncMessageLikeEvent,
         },
         OwnedEventId, OwnedUserId,
     },
@@ -139,7 +139,7 @@ impl Reacting {
             }
         };
 
-        let reacted_to_event_any_timeline_event = match reacted_to_event.event.deserialize() {
+        let reacted_to_event_any_timeline_event = match reacted_to_event.raw().deserialize() {
             Ok(value) => value,
             Err(err) => {
                 tracing::error!(
@@ -154,7 +154,7 @@ impl Reacting {
         let reacted_to_event_sender_id: OwnedUserId =
             reacted_to_event_any_timeline_event.sender().to_owned();
 
-        let AnyTimelineEvent::MessageLike(reacted_to_event_message_like) =
+        let AnySyncTimelineEvent::MessageLike(reacted_to_event_message_like) =
             reacted_to_event_any_timeline_event
         else {
             tracing::debug!(
@@ -164,7 +164,7 @@ impl Reacting {
             return Ok(());
         };
 
-        let AnyMessageLikeEvent::RoomMessage(reacted_to_event_room_message) =
+        let AnySyncMessageLikeEvent::RoomMessage(reacted_to_event_room_message) =
             reacted_to_event_message_like
         else {
             tracing::debug!(
@@ -174,7 +174,7 @@ impl Reacting {
             return Ok(());
         };
 
-        let MessageLikeEvent::Original(reacted_to_event_room_message_original) =
+        let SyncMessageLikeEvent::Original(reacted_to_event_room_message_original) =
             reacted_to_event_room_message
         else {
             tracing::debug!(?reacted_to_event_id, "Ignoring redacted reacted-to event",);

src/controller/access/dispatching.rs

@@ -13,7 +13,7 @@ pub async fn dispatch_controller(
     match handler {
         AccessControllerType::Help => {}
         _ => {
-            if !message_context.sender_can_manage_global_config()? {
+            if !message_context.sender_can_manage_global_config() {
                 bot.messaging()
                     .send_error_markdown_no_fail(
                         message_context.room(),

src/controller/access/help.rs

@@ -80,24 +80,21 @@ fn build_section_users(
         message.push_str(&strings::access::users_no_patterns());
     }
 
-    let can_manage_global_config = message_context.sender_can_manage_global_config();
+    if message_context.sender_can_manage_global_config() {
-    if let Ok(can_manage_global_config) = can_manage_global_config {
+        message.push_str("\n\n");
-        if can_manage_global_config {
-            message.push_str("\n\n");
 
-            message.push_str(strings::the_following_commands_are_available());
+        message.push_str(strings::the_following_commands_are_available());
-            message.push('\n');
+        message.push('\n');
 
-            message.push_str(&strings::help::access::users_command_get(command_prefix));
+        message.push_str(&strings::help::access::users_command_get(command_prefix));
-            message.push('\n');
+        message.push('\n');
 
-            message.push_str(&strings::help::access::users_command_set(command_prefix));
+        message.push_str(&strings::help::access::users_command_set(command_prefix));
-            message.push_str("\n\n");
+        message.push_str("\n\n");
 
-            message.push_str(&strings::help::access::example_user_patterns(
+        message.push_str(&strings::help::access::example_user_patterns(
-                homeserver_name,
+            homeserver_name,
-            ));
+        ));
-        }
     }
 
     message
@@ -156,27 +153,24 @@ fn build_section_room_local_agent_managers(
         message.push_str(&strings::access::room_local_agent_managers_no_patterns());
     }
 
-    let can_manage_global_config = message_context.sender_can_manage_global_config();
+    if message_context.sender_can_manage_global_config() {
-    if let Ok(can_manage_global_config) = can_manage_global_config {
+        message.push_str("\n\n");
-        if can_manage_global_config {
+        message.push_str(strings::the_following_commands_are_available());
-            message.push_str("\n\n");
+        message.push('\n');
-            message.push_str(strings::the_following_commands_are_available());
-            message.push('\n');
 
-            message.push_str(
+        message.push_str(
-                &strings::help::access::room_local_agent_managers_command_get(command_prefix),
+            &strings::help::access::room_local_agent_managers_command_get(command_prefix),
-            );
+        );
-            message.push('\n');
+        message.push('\n');
 
-            message.push_str(
+        message.push_str(
-                &strings::help::access::room_local_agent_managers_command_set(command_prefix),
+            &strings::help::access::room_local_agent_managers_command_set(command_prefix),
-            );
+        );
-            message.push_str("\n\n");
+        message.push_str("\n\n");
 
-            message.push_str(&strings::help::access::example_user_patterns(
+        message.push_str(&strings::help::access::example_user_patterns(
-                homeserver_name,
+            homeserver_name,
-            ));
+        ));
-        }
     }
 
     message

src/controller/agent/create/mod.rs

@@ -100,8 +100,6 @@ pub async fn handle_room_local(
         return Ok(());
     };
 
-    message_context.room().typing_notice(true).await?;
-
     if !try_to_ping_agent_or_complain(bot, message_context, &parsed_config.agent).await {
         return Ok(());
     }
@@ -140,7 +138,7 @@ pub async fn handle_global(
     provider: &str,
     agent_id_prefixless: &str,
 ) -> anyhow::Result<()> {
-    if !message_context.sender_can_manage_global_config()? {
+    if !message_context.sender_can_manage_global_config() {
         bot.messaging()
             .send_error_markdown_no_fail(
                 message_context.room(),
@@ -215,8 +213,6 @@ pub async fn handle_global(
         return Ok(());
     };
 
-    message_context.room().typing_notice(true).await?;
-
     if !try_to_ping_agent_or_complain(bot, message_context, &parsed_config.agent).await {
         return Ok(());
     }

src/controller/agent/delete/mod.rs

@@ -50,7 +50,7 @@ pub async fn handle(
                 .await
         }
         PublicIdentifier::DynamicGlobal(_) => {
-            if !message_context.sender_can_manage_global_config()? {
+            if !message_context.sender_can_manage_global_config() {
                 bot.messaging()
                     .send_error_markdown_no_fail(
                         message_context.room(),

src/controller/agent/details/mod.rs

@@ -47,7 +47,7 @@ pub async fn handle(
             }
         }
         PublicIdentifier::DynamicGlobal(_) => {
-            if !message_context.sender_can_manage_global_config()? {
+            if !message_context.sender_can_manage_global_config() {
                 bot.messaging()
                     .send_error_markdown_no_fail(
                         message_context.room(),

src/controller/agent/help/mod.rs

@@ -12,10 +12,7 @@ pub async fn handle(bot: &Bot, message_context: &MessageContext) -> anyhow::Resu
 
     message.push_str(&format!("## {}", strings::help::agent::heading()));
     message.push_str("\n\n");
-    message.push_str(&strings::help::agent::intro(
+    message.push_str(&strings::help::agent::intro(bot.command_prefix()));
-        bot.command_prefix(),
-        can_manage_agents,
-    ));
     message.push('\n');
     message.push_str(&strings::help::agent::intro_capabilities());
     message.push_str("\n\n");
@@ -39,7 +36,7 @@ pub async fn handle(bot: &Bot, message_context: &MessageContext) -> anyhow::Resu
         ));
         message.push('\n');
 
-        if message_context.sender_can_manage_global_config()? {
+        if message_context.sender_can_manage_global_config() {
             message.push_str(&strings::help::agent::create_agent_global(
                 bot.command_prefix(),
             ));

src/controller/cfg/dispatching/mod.rs

@@ -40,7 +40,7 @@ async fn dispatch_config_related_handler(
     bot: &Bot,
 ) -> anyhow::Result<()> {
     if let SettingsStorageSource::Global = config_type {
-        if !message_context.sender_can_manage_global_config()? {
+        if !message_context.sender_can_manage_global_config() {
             bot.messaging()
                 .send_error_markdown_no_fail(
                     message_context.room(),

src/controller/chat_completion/mod.rs

@@ -4,7 +4,9 @@ use mxlink::{MatrixLink, MessageResponseType};
 
 use tracing::Instrument;
 
-use crate::agent::provider::{SpeechToTextParams, TextGenerationParams};
+use crate::agent::provider::{
+    SpeechToTextParams, TextGenerationParams, TextGenerationPromptVariables,
+};
 use crate::agent::AgentInstance;
 use crate::agent::AgentPurpose;
 use crate::agent::ControllerTrait;
@@ -16,13 +18,28 @@ use crate::entity::roomconfig::{
 use crate::entity::MessagePayload;
 use crate::strings;
 use crate::utils::text_to_speech::create_transcribed_message_text;
-use crate::{conversation::create_llm_conversation_for_matrix_thread, entity::MessageContext, Bot};
+use crate::{
+    conversation::{
+        create_llm_conversation_for_matrix_reply_chain, create_llm_conversation_for_matrix_thread,
+        matrix::create_list_of_bot_user_prefixes_to_strip,
+    },
+    entity::MessageContext,
+    Bot,
+};
 
 #[derive(Debug, PartialEq)]
 pub enum ChatCompletionControllerType {
-    ViaText { prefixes_to_strip: Vec<String> },
+    // Invoked via a command prefix (e.g. `!bai Hello!`)
+    TextCommand,
+    // Invoked via a mention (e.g. `@baibot Hello!`)
+    TextMention,
+    // Invoked via a direct message (e.g. `Hello!`)
+    TextDirect,
 
-    ViaAudio,
+    Audio,
+
+    ThreadMention,
+    ReplyMention,
 }
 
 struct TextToSpeechEligiblePayload {
@@ -43,6 +60,8 @@ pub async fn handle(
 ) -> anyhow::Result<()> {
     let mut original_message_is_audio = false;
 
+    let mut _typing_notice_guard: Option<mxlink::TypingNoticeGuard> = None;
+
     let speech_to_text_flow_type = message_context
         .room_config_context()
         .speech_to_text_flow_type();
@@ -58,11 +77,11 @@ pub async fn handle(
                 return Ok(());
             }
             SpeechToTextFlowType::TranscribeAndGenerateText => {
-                tracing::debug!("Will be trascribing and possibly generating text..");
+                tracing::debug!("Will be transcribing and possibly generating text..");
                 MessageResponseType::InThread(message_context.thread_info().clone())
             }
             SpeechToTextFlowType::OnlyTranscribe => {
-                tracing::debug!("Will only be trascribing audio to text..");
+                tracing::debug!("Will only be transcribing audio to text..");
                 if message_context.thread_info().is_thread_root_only() {
                     MessageResponseType::Reply(message_context.thread_info().root_event_id.clone())
                 } else {
@@ -71,6 +90,10 @@ pub async fn handle(
             }
         };
 
+        if _typing_notice_guard.is_none() {
+            _typing_notice_guard = Some(bot.start_typing_notice(message_context.room()).await);
+        }
+
         let Some(speech_to_text_created_event_id_result) =
             handle_stage_speech_to_text(bot, message_context, audio_content, response_type).await
         else {
@@ -96,6 +119,10 @@ pub async fn handle(
         .room_config_context()
         .should_auto_text_generate(original_message_is_audio)
     {
+        if _typing_notice_guard.is_none() {
+            _typing_notice_guard = Some(bot.start_typing_notice(message_context.room()).await);
+        }
+
         let speech_to_text_created_event_id_reaction_event_id =
             if let Some(speech_to_text_created_event_id) = speech_to_text_created_event_id {
                 let reaction_event_response = bot
@@ -113,7 +140,15 @@ pub async fn handle(
                 None
             };
 
-        let response_type = MessageResponseType::InThread(message_context.thread_info().clone());
+        let response_type = match controller_type {
+            // When we're triggered via a reply mention, we reply to the message that triggered us.
+            ChatCompletionControllerType::ReplyMention => {
+                MessageResponseType::Reply(message_context.thread_info().last_event_id.clone())
+            }
+
+            // In all other cases, we're dealing with a threaded conversation, so we reply in the thread.
+            _ => MessageResponseType::InThread(message_context.thread_info().clone()),
+        };
 
         let text_to_speech_eligible_payload = handle_stage_text_generation(
             bot,
@@ -213,6 +248,10 @@ pub async fn handle(
 
     match text_to_speech_stage_params {
         Some(TextToSpeechParams::Perform(text_to_speech_eligible_payload, response_type)) => {
+            if _typing_notice_guard.is_none() {
+                _typing_notice_guard = Some(bot.start_typing_notice(message_context.room()).await);
+            }
+
             let _tts_result = generate_and_send_tts_for_message(
                 bot,
                 matrix_link.clone(),
@@ -337,26 +376,64 @@ async fn handle_stage_text_generation(
     )
     .await?;
 
-    _ = message_context.room().typing_notice(true).await;
+    // We only strip text from the first message if we're invoked via a command prefix.
-
+    // Otherwise, we do bot-user mentions stripping on all messages below.
-    let prefixes_to_strip = match controller_type {
+    let first_message_prefixes_to_strip = match controller_type {
-        ChatCompletionControllerType::ViaText { prefixes_to_strip } => prefixes_to_strip.clone(),
+        ChatCompletionControllerType::TextCommand => vec![bot.command_prefix().to_owned()],
-        ChatCompletionControllerType::ViaAudio => vec![],
+        _ => vec![],
     };
 
-    let params = MatrixMessageProcessingParams::new(
+    let bot_user_prefixes_to_strip = create_list_of_bot_user_prefixes_to_strip(
-        bot.user_id().as_str().to_owned(),
+        bot.user_id(),
-        message_context.combined_admin_and_user_regexes(),
+        message_context.bot_display_name(),
-    )
+    );
-    .with_first_message_stripped_prefixes(prefixes_to_strip);
 
-    let conversation = create_llm_conversation_for_matrix_thread(
+    let allowed_users = match controller_type {
-        matrix_link.clone(),
+        // Regular chat completion only operates on messages from allowed users.
-        message_context.room(),
+        ChatCompletionControllerType::TextCommand
-        message_context.thread_info().root_event_id.clone(),
+        | ChatCompletionControllerType::TextMention
-        &params,
+        | ChatCompletionControllerType::TextDirect
-    )
+        | ChatCompletionControllerType::Audio => {
-    .await;
+            Some(message_context.combined_admin_and_user_regexes())
+        }
+
+        // When we're triggered via an explicit mention (thread or reply), we wish to operate against the mention's whole context
+        // (the whole thread or the whole reply chain upward of the message that triggered us).
+        //
+        // This is to allow admins and users to trigger text-generation for other users' messages.
+        // When we're dragged into a conversation by a known (to us) user, we'd like to process all messages in the conversation,
+        // not just those from allowed users.
+        ChatCompletionControllerType::ThreadMention
+        | ChatCompletionControllerType::ReplyMention => None,
+    };
+
+    let params = MatrixMessageProcessingParams::new(bot.user_id().to_owned(), allowed_users)
+        .with_first_message_prefixes_to_strip(first_message_prefixes_to_strip)
+        .with_bot_user_prefixes_to_strip(bot_user_prefixes_to_strip);
+
+    let conversation = match controller_type {
+        // When we're triggered via a reply mention, the context is the whole reply chain upward of the message that triggered us.
+        ChatCompletionControllerType::ReplyMention => {
+            create_llm_conversation_for_matrix_reply_chain(
+                &bot.room_event_fetcher().clone(),
+                message_context.room(),
+                message_context.thread_info().last_event_id.clone(),
+                &params,
+            )
+            .await
+        }
+
+        // Everything else is happening in a thread, so the context is the whole thread.
+        _ => {
+            create_llm_conversation_for_matrix_thread(
+                matrix_link.clone(),
+                message_context.room(),
+                message_context.thread_info().root_event_id.clone(),
+                &params,
+            )
+            .await
+        }
+    };
 
     let conversation = match conversation {
         Ok(conversation) => conversation,
@@ -393,6 +470,17 @@ async fn handle_stage_text_generation(
 
     let start_time = std::time::Instant::now();
 
+    let controller = agent.controller();
+
+    let prompt_variables = TextGenerationPromptVariables::new(
+        bot.name(),
+        &controller
+            .text_generation_model_id()
+            .unwrap_or("unknown-model".to_owned()),
+        chrono::Utc::now(),
+        conversation.start_time(),
+    );
+
     let params = TextGenerationParams {
         context_management_enabled: message_context
             .room_config_context()
@@ -405,10 +493,11 @@ async fn handle_stage_text_generation(
         temperature_override: message_context
             .room_config_context()
             .text_generation_temperature_override(),
+
+        prompt_variables,
     };
 
-    let result = agent
+    let result = controller
-        .controller()
         .generate_text(conversation, params)
         .instrument(span)
         .await;
@@ -486,7 +575,7 @@ async fn handle_stage_speech_to_text_actual_transcribing(
 ) -> anyhow::Result<OwnedEventId> {
     let src = &audio_content.source;
 
-    let media_request = mxlink::matrix_sdk::media::MediaRequest {
+    let media_request = mxlink::matrix_sdk::media::MediaRequestParameters {
         source: src.to_owned(),
         format: mxlink::matrix_sdk::media::MediaFormat::File,
     };
@@ -498,8 +587,6 @@ async fn handle_stage_speech_to_text_actual_transcribing(
         .get_media_content(&media_request, true)
         .await?;
 
-    _ = message_context.room().typing_notice(true).await;
-
     let span = tracing::debug_span!(
         "speech_to_text_generation",
         agent_id = agent.identifier().as_string()
@@ -525,16 +612,53 @@ async fn handle_stage_speech_to_text_actual_transcribing(
         .instrument(span)
         .await?;
 
-    let transcribed_text = create_transcribed_message_text(&speech_to_text_result.text);
+    // Only use the `> 🦻 Transcribed text` format if we're posting in a thread.
+    //
+    // If we're dealing with a regular reply (which would be the case in "Transcribe-only mode" = speech-to-text/flow-type=only_transcribe),
+    // we don't want to use the `> 🦻 Transcribed text` format for 2 reasons:
+    //
+    // 1. This kind of blockquote-formatting can be confused by clients for a fallback-for-rich-replies
+    //    (see https://spec.matrix.org/v1.11/client-server-api/#fallbacks-for-rich-replies).
+    //    It makes certain clients render our messages incorrectly.
+    //
+    // 2. Transcribe-only mode is typically used for memos. Sticking to a plain-text format
+    //    allows people to copy-paste the text or forward it to another room more easily (without having to strip formatting, etc.)
+    //
+    // When sending a bare reply, we'd better annotate the message with a 🦻 reaction instead,
+    // to make it clear to users that it's a transcription.
+    //
+    // Regardless of how we post this message, it will be posted as a notice,
+    // which can indicate to the bot (for potential future text-generation purposes) that this message is not a bot message.
+    let (transcribed_text, annotate_message_with_reaction) =
+        if let MessageResponseType::InThread(_) = response_type {
+            (
+                create_transcribed_message_text(&speech_to_text_result.text),
+                false,
+            )
+        } else {
+            (speech_to_text_result.text, true)
+        };
 
     let result = bot
         .messaging()
         .send_notice_markdown_no_fail(message_context.room(), transcribed_text, response_type)
         .await;
 
-    result
+    let event_id = result
         .map(|result| result.event_id)
-        .ok_or_else(|| anyhow::anyhow!("Failed to send transcribed text"))
+        .ok_or_else(|| anyhow::anyhow!("Failed to send transcribed text"))?;
+
+    if annotate_message_with_reaction {
+        bot.reacting()
+            .react_no_fail(
+                message_context.room(),
+                event_id.clone(),
+                AgentPurpose::SpeechToText.emoji().to_owned(),
+            )
+            .await;
+    }
+
+    Ok(event_id)
 }
 
 async fn send_tts_offer_for_message(

src/controller/determination/mod.rs

@@ -1,13 +1,11 @@
 #[cfg(test)]
 mod tests;
 
-use mxlink::matrix_sdk::ruma::OwnedUserId;
-
 use super::chat_completion::ChatCompletionControllerType;
 use crate::{
     entity::{
-        roomconfig::TextGenerationPrefixRequirementType, MessageContext, MessagePayload,
+        roomconfig::TextGenerationPrefixRequirementType, InteractionTrigger, MessageContext,
-        ThreadContextFirstMessage,
+        MessagePayload,
     },
     strings,
 };
@@ -16,12 +14,16 @@ use super::ControllerType;
 
 pub fn determine_controller(
     command_prefix: &str,
-    first_thread_message: &ThreadContextFirstMessage,
+    first_thread_message: &InteractionTrigger,
     message_context: &MessageContext,
-    bot_user_id: &OwnedUserId,
-    bot_display_name: &Option<String>,
 ) -> ControllerType {
     match &first_thread_message.payload {
+        MessagePayload::SynthethicChatCompletionTriggerInThread => {
+            ControllerType::ChatCompletion(ChatCompletionControllerType::ThreadMention)
+        }
+        MessagePayload::SynthethicChatCompletionTriggerForReply => {
+            ControllerType::ChatCompletion(ChatCompletionControllerType::ReplyMention)
+        }
         MessagePayload::Text(text_message_content) => {
             let prefix_requirement_type = message_context
                 .room_config_context()
@@ -32,8 +34,6 @@ pub fn determine_controller(
                 &text_message_content.body,
                 prefix_requirement_type,
                 first_thread_message.is_mentioning_bot,
-                bot_user_id,
-                bot_display_name,
             )
         }
         MessagePayload::Encrypted(thread_info) => {
@@ -47,7 +47,7 @@ pub fn determine_controller(
             }
         }
         MessagePayload::Audio(_) => {
-            ControllerType::ChatCompletion(ChatCompletionControllerType::ViaAudio)
+            ControllerType::ChatCompletion(ChatCompletionControllerType::Audio)
         }
         MessagePayload::Reaction { .. } => {
             panic!("Handling reaction as first message in thread does not make sense")
@@ -60,8 +60,6 @@ fn determine_text_controller(
     text: &str,
     room_text_generation_prefix_requirement_type: TextGenerationPrefixRequirementType,
     is_mentioning_bot: bool,
-    bot_user_id: &OwnedUserId,
-    bot_display_name: &Option<String>,
 ) -> ControllerType {
     let text = text.trim();
 
@@ -102,53 +100,26 @@ fn determine_text_controller(
     // Otherwise, it depends on the prefix requirement for text generation - it may be routed for chat completion or ignored.
 
     if is_mentioning_bot {
-        // Different clients do mentions differently.
+        return ControllerType::ChatCompletion(ChatCompletionControllerType::TextMention);
-        // The body text containing the mention usually contains one of:
-        // - the full user ID (includes a @ prefix by default)
-        // - the localpart (with a @ prefix)
-        // - the localpart (without a @ prefix)
-        // - the display name (with a @ prefix)
-        // - the display name (without a @ prefix)
-        //
-        // Some add a `: ` suffix after the mention.
-        //
-        // There's no guarantee that the mention is at the start even.
-        // It being there is most common and we try to strip it from there
-        // as best as we can.
-        let bot_user_id_localpart = bot_user_id.localpart();
-
-        let mut prefixes_to_strip = vec![
-            bot_user_id.as_str().to_owned(),
-            format!("@{}", bot_user_id_localpart),
-            bot_user_id_localpart.to_owned(),
-        ];
-
-        if let Some(bot_display_name) = bot_display_name {
-            prefixes_to_strip.push(format!("@{}", bot_display_name));
-            prefixes_to_strip.push(bot_display_name.to_owned());
-        }
-
-        prefixes_to_strip.push(":".to_owned());
-
-        return ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
-            prefixes_to_strip,
-        });
     }
 
+    // Regardless of what the prefix requirement is, if we encounter a command prefix, we'll consider it a chat completion via command prefix invokation.
+    // This is to correctly indicate to the chat completion controller that a command prefix was used,
+    // so that it can be stripped from the beginning of the message.
+    if text.starts_with(command_prefix) {
+        return ControllerType::ChatCompletion(ChatCompletionControllerType::TextCommand);
+    }
+
+    // We're dealing with a regular message that does not start with a command prefix.
+
     match room_text_generation_prefix_requirement_type {
         TextGenerationPrefixRequirementType::CommandPrefix => {
-            if text.starts_with(command_prefix) {
+            // A prefix is required, but we've already checked (above) that the message does not start with a command prefix.
-                ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            // It's to be ignored.
-                    prefixes_to_strip: vec![command_prefix.to_owned()],
+            ControllerType::Ignore
-                })
-            } else {
-                ControllerType::Ignore
-            }
         }
         TextGenerationPrefixRequirementType::No => {
-            ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            ControllerType::ChatCompletion(ChatCompletionControllerType::TextDirect)
-                prefixes_to_strip: vec![],
-            })
         }
     }
 }

src/controller/determination/tests.rs

@@ -4,9 +4,6 @@ fn determine_text_controller() {
     use super::ControllerType;
     use crate::controller;
 
-    let bot_user_id = mxlink::matrix_sdk::ruma::owned_user_id!("@bot:example.com");
-    let bot_display_name = "Bot";
-
     let command_prefix = "!bai";
 
     struct TestCase {
@@ -44,9 +41,7 @@ fn determine_text_controller() {
             is_mentioning_bot: false,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::No,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextCommand),
-                prefixes_to_strip: vec![],
-            }),
         },
         TestCase {
             name: "Access top-level",
@@ -110,9 +105,7 @@ fn determine_text_controller() {
             is_mentioning_bot: false,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::No,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextDirect),
-                prefixes_to_strip: vec![],
-            }),
         },
         TestCase {
             name: "Regular text is ignored when prefix is required",
@@ -128,9 +121,7 @@ fn determine_text_controller() {
             is_mentioning_bot: false,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::CommandPrefix,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextCommand),
-                prefixes_to_strip: vec!["!bai".to_owned()],
-            }),
         },
         TestCase {
             name: "Command-prefixed text triggers completion even when prefix is not required",
@@ -138,58 +129,35 @@ fn determine_text_controller() {
             is_mentioning_bot: false,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::No,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextCommand),
-                prefixes_to_strip: vec![],
-            }),
         },
         TestCase {
-            name: "Regular message with bot mention triggers completion stripping bot id and display name (no prefix requirement)",
+            name: "Regular message with bot mention triggers completion (no prefix requirement)",
             input: "Regular text goes here",
             is_mentioning_bot: true,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::No,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextMention),
-                prefixes_to_strip: vec![
-                    "@bot:example.com".to_owned(),
-                    "@bot".to_owned(),
-                    "bot".to_owned(),
-                    "@Bot".to_owned(),
-                    "Bot".to_owned(),
-                    ":".to_owned(),
-                ],
-            }),
         },
-        // This test case is the same as the one above, just with a different prefix requirement.
+        // This test case is the same as the one above, just with a different prefix requirement setting.
         // We expect the same result.
         TestCase {
-            name: "Regular message with bot mention triggers completion stripping bot id and display name (command_prefix requirement)",
+            name:
+                "Regular message with bot mention triggers completion (command prefix requirement)",
             input: "Regular text goes here",
             is_mentioning_bot: true,
             room_text_generation_prefix_requirement_type:
                 super::TextGenerationPrefixRequirementType::CommandPrefix,
-            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::ViaText {
+            expected: ControllerType::ChatCompletion(ChatCompletionControllerType::TextMention),
-                prefixes_to_strip: vec![
-                    "@bot:example.com".to_owned(),
-                    "@bot".to_owned(),
-                    "bot".to_owned(),
-                    "@Bot".to_owned(),
-                    "Bot".to_owned(),
-                    ":".to_owned(),
-                ],
-            }),
         },
     ];
 
     for test_case in test_cases {
-        let bot_display_name = Some(bot_display_name.to_owned());
-
         let result = super::determine_text_controller(
             command_prefix,
             test_case.input,
             test_case.room_text_generation_prefix_requirement_type,
             test_case.is_mentioning_bot,
-            &bot_user_id,
-            &bot_display_name,
         );
         assert_eq!(result, test_case.expected, "Test case: {}", test_case.name);
     }

src/controller/help/mod.rs

@@ -3,7 +3,7 @@ use mxlink::MessageResponseType;
 use crate::{entity::MessageContext, strings, Bot};
 
 pub async fn handle(bot: &Bot, message_context: &MessageContext) -> anyhow::Result<()> {
-    let sender_can_manage_global_config = message_context.sender_can_manage_global_config()?;
+    let sender_can_manage_global_config = message_context.sender_can_manage_global_config();
     let sender_can_manage_room_local_agents =
         message_context.sender_can_manage_room_local_agents()?;
 
@@ -18,10 +18,7 @@ pub async fn handle(bot: &Bot, message_context: &MessageContext) -> anyhow::Resu
     // Agents
     message.push_str(&format!("## {}", strings::help::agent::heading()));
     message.push_str("\n\n");
-    message.push_str(&strings::help::agent::intro(
+    message.push_str(&strings::help::agent::intro(bot.command_prefix()));
-        bot.command_prefix(),
-        sender_can_manage_room_local_agents,
-    ));
     message.push_str("\n\n");
     message.push_str(&strings::help::agent::intro_handler_relation(
         bot.command_prefix(),

src/controller/image/generation.rs

@@ -34,9 +34,11 @@ pub async fn handle_image(
         return Ok(());
     };
 
+    let _typing_notice_guard = bot.start_typing_notice(message_context.room()).await;
+
     let params = MatrixMessageProcessingParams::new(
-        bot.user_id().as_str().to_owned(),
+        bot.user_id().to_owned(),
-        message_context.combined_admin_and_user_regexes(),
+        Some(message_context.combined_admin_and_user_regexes()),
     );
 
     let conversation = create_llm_conversation_for_matrix_thread(
@@ -56,8 +58,6 @@ pub async fn handle_image(
         original_prompt.to_owned()
     };
 
-    message_context.room().typing_notice(true).await?;
-
     let span = tracing::debug_span!(
         "image_generation",
         agent_id = agent.identifier().as_string()
@@ -139,7 +139,7 @@ pub async fn handle_sticker(
         return Ok(());
     };
 
-    message_context.room().typing_notice(true).await?;
+    let _typing_notice_guard = bot.start_typing_notice(message_context.room()).await;
 
     let span = tracing::debug_span!(
         "sticker_generation",

src/controller/image/prompt.rs

@@ -46,6 +46,8 @@ mod tests {
 
     #[test]
     fn test_build_prompt() {
+        let timestamp = chrono::Utc::now();
+
         let test_cases = vec![
             // Simple case
             TestCase {
@@ -59,6 +61,7 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of a dog\nOther criteria:\n- Must be blue",
             },
@@ -68,14 +71,17 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::Assistant,
                     message_text: "Whatever".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "Must be 3-legged.\nMust be flying.".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of an elephant\nOther criteria:\n- Must be blue\n- Must be 3-legged.. Must be flying.",
             },
@@ -85,18 +91,22 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::Assistant,
                     message_text: "Whatever".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "Again".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "again".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of a grizzly bear\nOther criteria:\n- Must be blue",
             },

src/controller/provider/mod.rs

@@ -9,17 +9,8 @@ pub fn determine_controller(_text: &str) -> ControllerType {
 }
 
 pub async fn handle_help(message_context: &MessageContext, bot: &Bot) -> anyhow::Result<()> {
-    if !message_context.sender_can_manage_room_local_agents()? {
+    let can_create_global_agents = message_context.sender_can_manage_global_config();
-        bot.messaging()
+    let can_create_room_local_agents = message_context.sender_can_manage_room_local_agents()?;
-            .send_error_markdown_no_fail(
-                message_context.room(),
-                &strings::provider::not_allowed(),
-                MessageResponseType::Reply(message_context.thread_info().root_event_id.clone()),
-            )
-            .await;
-
-        return Ok(());
-    }
 
     let mut message = String::new();
     message.push_str(&format!("## {}", strings::help::provider::heading()));
@@ -69,18 +60,26 @@ pub async fn handle_help(message_context: &MessageContext, bot: &Bot) -> anyhow:
             &provider_info,
         ));
 
-        message.push_str("- 🗲 Quick start:\n");
+        // We always show a "Quick start" section (even to unprivileged users),
-        message.push_str(&format!(
+        // because we're talking about it in a previous message.
-            "\t- create a room-local agent: `{command_prefix} agent create-room-local {provider_id} my-{provider_id}-agent`",
+        message.push_str("- 🗲 Quick start:");
-            command_prefix = bot.command_prefix(),
+        if can_create_room_local_agents {
-            provider_id = provider.to_static_str(),
+            message.push_str(&format!(
-        ));
+                "\n\t- create a room-local agent: `{command_prefix} agent create-room-local {provider_id} my-{provider_id}-agent`",
-        message.push('\n');
+                command_prefix = bot.command_prefix(),
-        message.push_str(&format!(
+                provider_id = provider.to_static_str(),
-            "\t- create a global agent: `{command_prefix} agent create-global {provider_id} my-{provider_id}-agent`",
+            ));
-            command_prefix = bot.command_prefix(),
+        }
-            provider_id = provider.to_static_str(),
+        if can_create_global_agents {
-        ));
+            message.push_str(&format!(
+                "\n\t- create a global agent: `{command_prefix} agent create-global {provider_id} my-{provider_id}-agent`",
+                command_prefix = bot.command_prefix(),
+                provider_id = provider.to_static_str(),
+            ));
+        }
+        if !can_create_room_local_agents && !can_create_global_agents {
+            message.push_str(" ask an administrator to create an agent for you (you lack permissions to do so yourself)");
+        }
 
         message.push_str("\n\n");
     }

src/controller/reaction/text_to_speech.rs

@@ -52,6 +52,8 @@ pub(super) async fn handle(
         return Ok(());
     };
 
+    let _typing_notice_guard = bot.start_typing_notice(message_context.room()).await;
+
     crate::controller::utils::text_to_speech::generate_and_send_tts_for_message(
         bot,
         matrix_link,

src/controller/utils/text_to_speech.rs

@@ -18,8 +18,6 @@ pub async fn generate_and_send_tts_for_message(
     text_message_event_id: &OwnedEventId,
     text_content: &str,
 ) -> bool {
-    _ = message_context.room().typing_notice(true).await;
-
     let reaction_event_response = bot
         .reacting()
         .react_no_fail(

src/conversation/llm/entity.rs

@@ -1,3 +1,5 @@
+use chrono::{DateTime, Utc};
+
 #[derive(Debug, Clone, PartialEq)]
 pub enum Author {
     Prompt,
@@ -9,9 +11,126 @@ pub enum Author {
 pub struct Message {
     pub author: Author,
     pub message_text: String,
+    pub timestamp: DateTime<Utc>,
 }
 
 #[derive(Debug)]
 pub struct Conversation {
     pub messages: Vec<Message>,
 }
+
+impl Conversation {
+    /// Combine consecutive messages by the same author into a single message.
+    ///
+    /// Certain models (like Anthropic) cannot tolerate consecutive messages by the same author,
+    /// so combining them helps avoid issues.
+    /// See: https://github.com/etkecc/baibot/issues/13
+    pub fn combine_consecutive_messages(&self) -> Conversation {
+        // We'll likely get fewer messages, but let's reserve the maximum we expect.
+        let mut new_messages = Vec::with_capacity(self.messages.len());
+        let mut last_seen_author: Option<Author> = None;
+
+        for message in &self.messages {
+            let Some(last_seen_author_clone) = last_seen_author.clone() else {
+                last_seen_author = Some(message.author.clone());
+                new_messages.push(message.clone());
+                continue;
+            };
+
+            if message.author != last_seen_author_clone {
+                last_seen_author = Some(message.author.clone());
+                new_messages.push(message.clone());
+                continue;
+            }
+
+            new_messages.last_mut().unwrap().message_text.push('\n');
+            new_messages
+                .last_mut()
+                .unwrap()
+                .message_text
+                .push_str(&message.message_text);
+        }
+
+        Conversation {
+            messages: new_messages,
+        }
+    }
+
+    pub fn start_time(&self) -> Option<DateTime<Utc>> {
+        self.messages.first().map(|message| message.timestamp)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use chrono::{TimeZone, Utc};
+
+    #[test]
+    fn combine_consecutive_messages() {
+        let timestamp_1 = Utc.with_ymd_and_hms(2024, 9, 20, 18, 34, 15).unwrap();
+
+        let timestamp_2 = Utc.with_ymd_and_hms(2024, 9, 21, 18, 34, 15).unwrap();
+
+        let timestamp_3 = Utc.with_ymd_and_hms(2024, 9, 22, 18, 34, 15).unwrap();
+
+        let conversation = Conversation {
+            messages: vec![
+                // User's turn
+                Message {
+                    author: Author::User,
+                    message_text: "Hello".to_string(),
+                    timestamp: timestamp_1,
+                },
+                Message {
+                    author: Author::User,
+                    message_text: "How are you?".to_string(),
+                    timestamp: timestamp_2,
+                },
+                Message {
+                    author: Author::User,
+                    message_text: "I'm OK, btw.".to_string(),
+                    timestamp: timestamp_3,
+                },
+                // Assistant's turn
+                Message {
+                    author: Author::Assistant,
+                    message_text: "Hi there!".to_string(),
+                    timestamp: timestamp_2,
+                },
+                Message {
+                    author: Author::Assistant,
+                    message_text: "I'm doing well, thank you.".to_string(),
+                    timestamp: timestamp_3,
+                },
+                // User's turn
+                Message {
+                    author: Author::User,
+                    message_text: "That's great!".to_string(),
+                    timestamp: timestamp_3,
+                },
+            ],
+        };
+
+        let conversation = conversation.combine_consecutive_messages();
+
+        assert_eq!(conversation.messages.len(), 3);
+        assert_eq!(conversation.messages[0].author, Author::User);
+        assert_eq!(
+            conversation.messages[0].message_text,
+            "Hello\nHow are you?\nI'm OK, btw."
+        );
+        assert_eq!(conversation.messages[0].timestamp, timestamp_1);
+
+        assert_eq!(conversation.messages[1].author, Author::Assistant);
+        assert_eq!(
+            conversation.messages[1].message_text,
+            "Hi there!\nI'm doing well, thank you."
+        );
+        assert_eq!(conversation.messages[1].timestamp, timestamp_2);
+
+        assert_eq!(conversation.messages[2].author, Author::User);
+        assert_eq!(conversation.messages[2].message_text, "That's great!");
+        assert_eq!(conversation.messages[2].timestamp, timestamp_3);
+    }
+}

src/conversation/llm/tests.rs

@@ -1,3 +1,5 @@
+use mxlink::matrix_sdk::ruma::OwnedUserId;
+
 use crate::utils::status::create_error_message_text;
 use crate::utils::text_to_speech::create_transcribed_message_text;
 
@@ -5,15 +7,18 @@ use super::*;
 
 #[test]
 fn test_messages_by_the_bot_are_identified_correctly() {
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
 
     let matrix_message = super::super::matrix::MatrixMessage {
         sender_id: bot_user_id.to_owned(),
         message_type: super::super::matrix::MatrixMessageType::Text,
         message_text: "Hello!".to_owned(),
+        mentioned_users: vec![],
+        timestamp: chrono::Utc::now(),
     };
 
-    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, bot_user_id).unwrap();
+    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, &bot_user_id).unwrap();
 
     assert_eq!(llm_message.author, Author::Assistant);
     assert_eq!(llm_message.message_text, "Hello!");
@@ -22,7 +27,8 @@ fn test_messages_by_the_bot_are_identified_correctly() {
 #[test]
 fn test_notice_messages_by_bot_with_speech_to_text_prefix_are_cleaned_up_and_considered_sent_by_user(
 ) {
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
 
     let source_message_text = "Hello!";
     let message_text = create_transcribed_message_text(source_message_text);
@@ -33,9 +39,11 @@ fn test_notice_messages_by_bot_with_speech_to_text_prefix_are_cleaned_up_and_con
         sender_id: bot_user_id.to_owned(),
         message_type: super::super::matrix::MatrixMessageType::Notice,
         message_text,
+        mentioned_users: vec![],
+        timestamp: chrono::Utc::now(),
     };
 
-    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, bot_user_id).unwrap();
+    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, &bot_user_id).unwrap();
 
     assert_eq!(llm_message.author, Author::User);
     assert_eq!(llm_message.message_text, source_message_text);
@@ -43,7 +51,8 @@ fn test_notice_messages_by_bot_with_speech_to_text_prefix_are_cleaned_up_and_con
 
 #[test]
 fn test_notice_error_messages_by_bot_are_ignored() {
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
 
     let source_message_text = "Some error happened";
     let message_text = create_error_message_text(source_message_text);
@@ -54,9 +63,11 @@ fn test_notice_error_messages_by_bot_are_ignored() {
         sender_id: bot_user_id.to_owned(),
         message_type: super::super::matrix::MatrixMessageType::Notice,
         message_text,
+        mentioned_users: vec![],
+        timestamp: chrono::Utc::now(),
     };
 
-    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, bot_user_id);
+    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, &bot_user_id);
 
     assert!(llm_message.is_none());
 }
@@ -68,7 +79,8 @@ fn test_other_notice_messages_by_the_bot_are_ignored() {
     // (except for speech-to-text-created transcriptions - see `test_notice_messages_by_bot_with_speech_to_text_prefix_are_cleaned_up_and_considered_sent_by_user()`).
     // This test is to make sure that we don't accidentally start accepting other notice messages.
 
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
 
     let message_text = "Something something";
 
@@ -76,9 +88,11 @@ fn test_other_notice_messages_by_the_bot_are_ignored() {
         sender_id: bot_user_id.to_owned(),
         message_type: super::super::matrix::MatrixMessageType::Notice,
         message_text: message_text.to_owned(),
+        mentioned_users: vec![],
+        timestamp: chrono::Utc::now(),
     };
 
-    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, bot_user_id);
+    let llm_message = convert_matrix_message_to_llm_message(&matrix_message, &bot_user_id);
 
     assert!(llm_message.is_none());
 }

src/conversation/llm/tokenization.rs

@@ -16,7 +16,7 @@ pub fn shorten_messages_list_to_context_size(
     model: &str,
     prompt_message: &Option<Message>,
     mut messages: Vec<Message>,
-    max_response_tokens: u32,
+    max_response_tokens: Option<u32>,
     max_context_tokens: u32,
 ) -> Vec<Message> {
     // Loading the tokenization data is an expensive process, so
@@ -26,7 +26,8 @@ pub fn shorten_messages_list_to_context_size(
     // We want to retain the prompt in all cases, so we always count it first.
     // We also always reserve enough tokens for the maximum response we expect.
     let mut current_context_length: u32 = if let Some(prompt_message) = prompt_message {
-        calculate_token_size_for_message(&bpe, model, prompt_message) + max_response_tokens
+        calculate_token_size_for_message(&bpe, model, prompt_message)
+            + max_response_tokens.unwrap_or(0)
     } else {
         0
     };
@@ -85,6 +86,7 @@ pub mod test {
         let message = super::Message {
             author: super::Author::User,
             message_text: "Hello there!".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
 
         let tokens = super::calculate_token_size_for_message(&bpe, model, &message);
@@ -98,11 +100,12 @@ pub mod test {
 
         let bpe = super::get_bpe_for_model(model);
 
-        let max_response_tokens: u32 = 5;
+        let max_response_tokens: Option<u32> = Some(5);
 
         let prompt = super::Message {
             author: super::Author::Prompt,
             message_text: "You are a bot!".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let prompt_length = 10;
 
@@ -116,6 +119,7 @@ pub mod test {
         let first = super::Message {
             author: super::Author::User,
             message_text: "Hello there!".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let first_length = 8;
 
@@ -129,6 +133,7 @@ pub mod test {
         let second = super::Message {
             author: super::Author::Assistant,
             message_text: "Hello!".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let second_length = 7;
 
@@ -143,6 +148,7 @@ pub mod test {
             author: super::Author::User,
             message_text: "This is the 3rd message in this conversation. It shall be preserved."
                 .to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let third_length = 21;
 
@@ -156,6 +162,7 @@ pub mod test {
         let forth = super::Message {
             author: super::Author::Assistant,
             message_text: "This is yet another message that shall be preserved.".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let forth_length = 15;
 
@@ -173,7 +180,7 @@ pub mod test {
             &Some(prompt),
             conversation_messages,
             max_response_tokens,
-            prompt_length + max_response_tokens + forth_length + third_length,
+            prompt_length + max_response_tokens.unwrap_or(0) + forth_length + third_length,
         );
 
         assert_eq!(2, new_conversation_messages.len());
@@ -195,11 +202,12 @@ pub mod test {
 
         let bpe = super::get_bpe_for_model(model);
 
-        let max_response_tokens: u32 = 5;
+        let max_response_tokens: Option<u32> = Some(5);
 
         let prompt = super::Message {
             author: super::Author::User,
             message_text: "あなたはボットです。".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let prompt_length = 14;
 
@@ -213,6 +221,7 @@ pub mod test {
         let first = super::Message {
             author: super::Author::User,
             message_text: "こんにちは!".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let first_length = 7;
 
@@ -226,6 +235,7 @@ pub mod test {
         let second = super::Message {
             author: super::Author::Assistant,
             message_text: "こんにちは。今日は元気ですか。".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let second_length = 15;
 
@@ -239,6 +249,7 @@ pub mod test {
         let third = super::Message {
             author: super::Author::User,
             message_text: "これは第3のメッセージなので、保存されます。".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let third_length = 22;
 
@@ -252,6 +263,7 @@ pub mod test {
         let forth = super::Message {
             author: super::Author::Assistant,
             message_text: "これはもう一つの保存されますメッセージです。".to_owned(),
+            timestamp: chrono::Utc::now(),
         };
         let forth_length = 21;
 
@@ -269,7 +281,7 @@ pub mod test {
             &Some(prompt),
             conversation_messages,
             max_response_tokens,
-            prompt_length + max_response_tokens + forth_length + third_length,
+            prompt_length + max_response_tokens.unwrap_or(0) + forth_length + third_length,
         );
 
         assert_eq!(2, new_conversation_messages.len());

src/conversation/llm/utils.rs

@@ -1,12 +1,14 @@
+use matrix_sdk::ruma::OwnedUserId;
+
 use super::{Author, Message};
 use crate::conversation::matrix::{MatrixMessage, MatrixMessageType};
 use crate::utils::text_to_speech as text_to_speech_utils;
 
 pub fn convert_matrix_message_to_llm_message(
     matrix_message: &MatrixMessage,
-    bot_user_id: &str,
+    bot_user_id: &OwnedUserId,
 ) -> Option<Message> {
-    if matrix_message.sender_id == bot_user_id {
+    if matrix_message.sender_id == bot_user_id.as_str() {
         return convert_bot_message(matrix_message);
     }
 
@@ -15,28 +17,43 @@ pub fn convert_matrix_message_to_llm_message(
 
 fn convert_bot_message(matrix_message: &MatrixMessage) -> Option<Message> {
     match matrix_message.message_type {
-        MatrixMessageType::Text => convert_bot_text_message(&matrix_message.message_text),
+        MatrixMessageType::Text => {
-        MatrixMessageType::Notice => convert_bot_notice_message(&matrix_message.message_text),
+            convert_bot_text_message(&matrix_message.message_text, &matrix_message.timestamp)
+        }
+        MatrixMessageType::Notice => {
+            convert_bot_notice_message(&matrix_message.message_text, &matrix_message.timestamp)
+        }
     }
 }
 
-fn convert_bot_text_message(text: &str) -> Option<Message> {
+fn convert_bot_text_message(
+    text: &str,
+    timestamp: &chrono::DateTime<chrono::Utc>,
+) -> Option<Message> {
     Some(Message {
         author: Author::Assistant,
         message_text: text.to_owned(),
+        timestamp: timestamp.to_owned(),
     })
 }
 
-fn convert_bot_notice_message(text: &str) -> Option<Message> {
+fn convert_bot_notice_message(
+    text: &str,
+    timestamp: &chrono::DateTime<chrono::Utc>,
+) -> Option<Message> {
     // Notice messages sent by the bot are usually transcriptions of previous messages sent by the user.
     // Such transcriptions are prefixed with an emoji and blockquoted.
     // If we find a notice that doesn't match this pattern, we skip it.
+    //
+    // It should be noted that transcriptions are sometimes posted as regular notice messages which do not include
+    // the `> 🦻` formatting. This function will not handle these properly.
 
     if let Some(text) = text_to_speech_utils::parse_transcribed_message_text(text) {
         // This is a transcription message. We remove the prefix and consider it as a message sent by the user.
         return Some(Message {
             author: Author::User,
             message_text: text.to_owned(),
+            timestamp: timestamp.to_owned(),
         });
     }
 
@@ -47,5 +64,6 @@ fn convert_user_message(matrix_message: &MatrixMessage) -> Option<Message> {
     Some(Message {
         author: Author::User,
         message_text: matrix_message.message_text.clone(),
+        timestamp: matrix_message.timestamp.to_owned(),
     })
 }

src/conversation/matrix/entity.rs

@@ -1,10 +1,15 @@
+use chrono::{DateTime, Utc};
 use regex::Regex;
 
+use mxlink::matrix_sdk::ruma::OwnedUserId;
+
 #[derive(Clone)]
 pub struct MatrixMessage {
-    pub sender_id: String,
+    pub sender_id: OwnedUserId,
     pub message_type: MatrixMessageType,
     pub message_text: String,
+    pub mentioned_users: Vec<OwnedUserId>,
+    pub timestamp: DateTime<Utc>,
 }
 
 #[derive(Clone)]
@@ -13,26 +18,42 @@ pub enum MatrixMessageType {
     Notice,
 }
 
-#[derive(Default, Clone)]
+#[derive(Clone)]
 pub struct MatrixMessageProcessingParams {
-    pub(crate) bot_user_id: String,
+    pub(crate) bot_user_id: OwnedUserId,
-    pub(crate) allowed_users: Vec<Regex>,
 
-    // If non-empty, these prefixes will be stripped when processing the message
+    /// The prefixes that will be stripped when processing the messages in the context (thread or reply chain),
-    pub(crate) first_message_stripped_prefixes: Vec<String>,
+    /// which are found to be mentioning the bot user (`bot_user_id`).
+    pub(crate) bot_user_prefixes_to_strip: Vec<String>,
+
+    /// The prefixes that will be stripped when processing the 1st message in the context (thread or reply chain).
+    pub(crate) first_message_prefixes_to_strip: Vec<String>,
+
+    /// A list of users whose messages are allowed.
+    /// If None, all messages are allowed.
+    /// If Some, only messages from the allowed users (and the bot itself, `bot_user_id`) are allowed.
+    pub(crate) allowed_users: Option<Vec<Regex>>,
 }
 
 impl MatrixMessageProcessingParams {
-    pub fn new(bot_user_id: String, allowed_users: Vec<Regex>) -> Self {
+    pub fn new(bot_user_id: OwnedUserId, allowed_users: Option<Vec<Regex>>) -> Self {
         Self {
             bot_user_id,
+            bot_user_prefixes_to_strip: vec![],
+
+            first_message_prefixes_to_strip: vec![],
+
             allowed_users,
-            ..Default::default()
         }
     }
 
-    pub fn with_first_message_stripped_prefixes(mut self, value: Vec<String>) -> Self {
+    pub fn with_bot_user_prefixes_to_strip(mut self, value: Vec<String>) -> Self {
-        self.first_message_stripped_prefixes = value;
+        self.bot_user_prefixes_to_strip = value;
+        self
+    }
+
+    pub fn with_first_message_prefixes_to_strip(mut self, value: Vec<String>) -> Self {
+        self.first_message_prefixes_to_strip = value;
         self
     }
 }

src/conversation/matrix/room_event_fetcher.rs

@@ -22,7 +22,7 @@ impl RoomEventFetcher {
         room: &Room,
     ) -> mxlink::matrix_sdk::Result<TimelineEvent> {
         let Some(lru_cache) = &self.lru_cache else {
-            return room.event(event_id).await;
+            return room.event(event_id, None).await;
         };
 
         let guard = lru_cache.get_value_or_guard_async(event_id).await;
@@ -33,7 +33,7 @@ impl RoomEventFetcher {
                 return Ok(config);
             }
             Err(guard) => {
-                let event = room.event(event_id).await?;
+                let event = room.event(event_id, None).await?;
 
                 let _ = guard.insert(event.clone());
 

src/conversation/matrix/utils/mod.rs

@@ -5,25 +5,37 @@ use std::sync::Arc;
 
 use mxlink::matrix_sdk::ruma::{OwnedEventId, OwnedUserId};
 use mxlink::matrix_sdk::{
+    deserialized_responses::TimelineEvent,
     ruma::events::{
+        relation::Thread,
         room::message::{
-            MessageType, OriginalSyncRoomMessageEvent, Relation, RoomMessageEventContent,
+            sanitize::remove_plain_reply_fallback, MessageType, OriginalSyncRoomMessageEvent,
+            Relation, RoomMessageEventContent,
         },
-        AnyMessageLikeEvent, AnyMessageLikeEventContent, AnyTimelineEvent, MessageLikeEvent,
+        AnyMessageLikeEventContent, AnySyncMessageLikeEvent, AnySyncTimelineEvent,
+        SyncMessageLikeEvent,
     },
     Room,
 };
-use mxlink::{MatrixLink, ThreadInfo};
+use mxlink::{MatrixLink, ThreadGetMessagesParams, ThreadInfo};
 
 use super::{MatrixMessage, MatrixMessageProcessingParams, MatrixMessageType, RoomEventFetcher};
-use crate::entity::{MessagePayload, ThreadContext, ThreadContextFirstMessage};
+use crate::entity::{InteractionContext, InteractionTrigger, MessagePayload};
+
+struct DetailedMessagePayload {
+    is_mentioning_bot: bool,
+    message_payload: MessagePayload,
+}
 
 pub async fn get_matrix_messages_in_thread(
     matrix_link: MatrixLink,
     room: &Room,
     thread_id: OwnedEventId,
 ) -> Result<Vec<MatrixMessage>, mxlink::matrix_sdk::Error> {
-    let messages_native = matrix_link.threads().get_messages(room, thread_id).await?;
+    let messages_native = matrix_link
+        .threads()
+        .get_messages(room, thread_id, ThreadGetMessagesParams::default())
+        .await?;
 
     let mut messages: Vec<MatrixMessage> = Vec::new();
 
@@ -39,23 +51,123 @@ pub async fn get_matrix_messages_in_thread(
     Ok(messages)
 }
 
-pub async fn process_matrix_messages_in_thread(
+pub async fn get_matrix_messages_in_reply_chain(
+    event_fetcher: &Arc<RoomEventFetcher>,
+    room: &Room,
+    event_id: OwnedEventId,
+) -> Result<Vec<MatrixMessage>, mxlink::matrix_sdk::Error> {
+    let messages_native =
+        get_matrix_messages_in_reply_chain_native(event_fetcher, room, event_id).await?;
+
+    let mut messages: Vec<MatrixMessage> = Vec::new();
+
+    for matrix_native_message in messages_native {
+        let Some(message) = convert_matrix_native_event_to_matrix_message(&matrix_native_message)
+        else {
+            continue;
+        };
+
+        messages.push(message);
+    }
+
+    Ok(messages)
+}
+
+async fn get_matrix_messages_in_reply_chain_native(
+    event_fetcher: &Arc<RoomEventFetcher>,
+    room: &Room,
+    event_id: OwnedEventId,
+) -> Result<Vec<AnySyncMessageLikeEvent>, mxlink::matrix_sdk::Error> {
+    let mut next_event_id = Some(event_id.clone());
+
+    let mut messages: Vec<AnySyncMessageLikeEvent> = Vec::new();
+    let mut handled_event_ids: Vec<OwnedEventId> = Vec::new();
+
+    while let Some(next_event_id_in_loop) = next_event_id {
+        let event = event_fetcher
+            .fetch_event_in_room(&next_event_id_in_loop, room)
+            .await
+            .unwrap();
+
+        if handled_event_ids.contains(&next_event_id_in_loop) {
+            tracing::warn!(
+                "Not following loop-causing event: {}",
+                next_event_id_in_loop
+            );
+            break;
+        }
+
+        handled_event_ids.push(next_event_id_in_loop.clone());
+
+        let event_deserialized = event.raw().deserialize()?;
+
+        let AnySyncTimelineEvent::MessageLike(message_like_event) = event_deserialized else {
+            tracing::warn!(
+                "Not proceeding past non-MessageLike event: {:?}",
+                event_deserialized
+            );
+            break;
+        };
+
+        next_event_id = match message_like_event.clone() {
+            AnySyncMessageLikeEvent::RoomEncrypted(_) => None,
+            AnySyncMessageLikeEvent::RoomMessage(room_message) => {
+                if let SyncMessageLikeEvent::Original(room_message_original) = room_message {
+                    match room_message_original.content.relates_to {
+                        Some(Relation::Reply { in_reply_to }) => Some(in_reply_to.event_id.clone()),
+                        _ => None,
+                    }
+                } else {
+                    None
+                }
+            }
+            _ => None,
+        };
+
+        messages.push(message_like_event);
+    }
+
+    messages.reverse();
+
+    Ok(messages)
+}
+
+pub async fn process_matrix_messages(
     messages: &[MatrixMessage],
     params: &MatrixMessageProcessingParams,
 ) -> Vec<MatrixMessage> {
     let mut messages_filtered: Vec<MatrixMessage> = Vec::new();
 
     for (i, message) in messages.iter().enumerate() {
-        if !is_message_from_allowed_sender(message, &params.bot_user_id, &params.allowed_users) {
+        if !is_message_from_allowed_sender(
+            message,
+            &params.bot_user_id,
+            params.allowed_users.as_deref(),
+        ) {
             continue;
         }
 
         let mut message = message.clone();
 
-        if i == 0 && !params.first_message_stripped_prefixes.is_empty() {
+        if i == 0 && !params.first_message_prefixes_to_strip.is_empty() {
             let mut message_text = message.message_text.clone();
 
-            for prefix in &params.first_message_stripped_prefixes {
+            for prefix in &params.first_message_prefixes_to_strip {
+                if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
+                    message_text = message_text_stripped.to_owned();
+                }
+            }
+
+            message.message_text = message_text.trim().to_owned();
+        }
+
+        // We only strip `bot_user_prefixes_to_strip`-defined prefixes from messages that mention the bot user.
+        if !params.bot_user_prefixes_to_strip.is_empty()
+            && message.mentioned_users.contains(&params.bot_user_id)
+        {
+            let mut message_text = message.message_text.clone();
+
+            for prefix in &params.bot_user_prefixes_to_strip {
                 if let Some(message_text_stripped) = message_text.strip_prefix(prefix) {
                     message_text = message_text_stripped.to_owned();
                 }
@@ -70,16 +182,25 @@ pub async fn process_matrix_messages_in_thread(
     messages_filtered
 }
 
+/// Tells if the given message is from an allowed sender.
+///
+/// If allowed_users is None, all messages are allowed.
+/// If allowed_users is Some, only messages from the allowed users (and the `bot_user_id`) are allowed.
 fn is_message_from_allowed_sender(
     matrix_message: &MatrixMessage,
-    bot_user_id: &str,
+    bot_user_id: &OwnedUserId,
-    allowed_users: &[regex::Regex],
+    allowed_users: Option<&[regex::Regex]>,
 ) -> bool {
-    if matrix_message.sender_id == bot_user_id {
+    if matrix_message.sender_id == *bot_user_id {
         return true;
     }
 
-    if mxidwc::match_user_id(&matrix_message.sender_id, allowed_users) {
+    if let Some(allowed_users) = allowed_users {
+        if mxidwc::match_user_id(matrix_message.sender_id.as_str(), allowed_users) {
+            return true;
+        }
+    } else {
+        // No allowed users configured, so all messages are allowed
         return true;
     }
 
@@ -87,7 +208,7 @@ fn is_message_from_allowed_sender(
 }
 
 pub fn convert_matrix_native_event_to_matrix_message(
-    matrix_native_event: &AnyMessageLikeEvent,
+    matrix_native_event: &AnySyncMessageLikeEvent,
 ) -> Option<MatrixMessage> {
     let Some(content) = matrix_native_event.original_content() else {
         // Redacted message
@@ -105,30 +226,67 @@ pub fn convert_matrix_native_event_to_matrix_message(
         _ => return None,
     };
 
+    let is_reply = matches!(room_message.relates_to, Some(Relation::Reply { .. }));
+
+    let text = if is_reply {
+        // For regular replies, we need to strip the fallback-for-rich replies part.
+        // See: https://spec.matrix.org/v1.11/client-server-api/#fallbacks-for-rich-replies
+        remove_plain_reply_fallback(&text).to_owned()
+    } else {
+        text
+    };
+
+    let timestamp = chrono::DateTime::<chrono::Utc>::from(
+        matrix_native_event
+            .origin_server_ts()
+            .to_system_time()
+            .unwrap_or_else(std::time::SystemTime::now),
+    );
+
+    let mentioned_users = room_message
+        .mentions
+        .map(|m| m.user_ids.iter().map(|u| u.to_owned()).collect())
+        .unwrap_or(vec![]);
+
     Some(MatrixMessage {
-        sender_id: matrix_native_event.sender().to_string(),
+        sender_id: matrix_native_event.sender().to_owned(),
         message_type: if is_notice {
             MatrixMessageType::Notice
         } else {
             MatrixMessageType::Text
         },
         message_text: text,
+        mentioned_users,
+        timestamp,
     })
 }
 
-/// Determines the thread context (relationship within the thread + first thread message payload) for an incoming (new) room event.
+/// Determines the interaction context for an incoming (new) room event.
-/// This room event is assumed to be the "newest message" in the thread (or a top-level message).
+///
-/// If the given event is a regular reply (not a thread reply), this function will return `None`.
+/// This context is created based on the "newest message" (`current_event`), which is:
-/// If the given event is a top-level message, this function will consider this event as the start of the thread.
+/// - either a top-level message, which may or may not be mentioning the bot
-/// If the given event is a thread reply, this function will inspect the thread root event and will return the thread context.
+///     - this function will inspect the event and will likely start a new threaded conversation
-/// If the thread root event is not found, is redacted, or is of some unsupported MessagePayload type, this function will return `None`.
+///
-pub async fn determine_thread_context_for_room_event(
+/// - or a thread reply
+///     - this function will inspect the thread root event and will return the interaction context
+///     - if the bot only reacts to prefixed messsages (or mentions), this function may ignore the given thread reply, unless it mentions the bot (which causes a synthetic "first message" to be produced)
+///     - if the thread root event is not found, is redacted, or is of some unsupported MessagePayload type, this function will return `None`
+///
+/// - or an in-room (non-threaded) reply to a room message, which may or may not be mentioning the bot
+///     - replies that do not mention the bot cause this function to return `None`
+///     - other replies create a interaction context which points to a "first message" which is synthetic
+#[tracing::instrument(name = "determine_interaction_context_for_room_event", skip_all, fields(room_id = room.room_id().as_str(), event_id = current_event.event_id.as_str()))]
+pub async fn determine_interaction_context_for_room_event(
     bot_user_id: &OwnedUserId,
+    bot_display_name: &Option<String>,
     room: &Room,
     current_event: &OriginalSyncRoomMessageEvent,
     current_event_payload: &MessagePayload,
     event_fetcher: &Arc<RoomEventFetcher>,
-) -> anyhow::Result<Option<ThreadContext>> {
+) -> anyhow::Result<Option<InteractionContext>> {
+    let current_event_is_mentioning_bot =
+        is_event_mentioning_bot(&current_event.content, bot_user_id, bot_display_name);
+
     let Some(relation) = &current_event.content.relates_to else {
         // This is a top-level message. We consider it the start of the thread.
         let thread_info = ThreadInfo::new(
@@ -136,25 +294,75 @@ pub async fn determine_thread_context_for_room_event(
             current_event.event_id.clone(),
         );
 
-        let is_mentioning_bot = is_event_mentioning_bot(&current_event.content, bot_user_id);
+        return Ok(Some(InteractionContext {
-
+            thread_info,
-        return Ok(Some(ThreadContext {
+            trigger: InteractionTrigger {
-            info: thread_info,
+                is_mentioning_bot: current_event_is_mentioning_bot,
-            first_message: ThreadContextFirstMessage {
-                is_mentioning_bot,
                 payload: current_event_payload.clone(),
             },
         }));
     };
 
-    let Relation::Thread(thread) = relation else {
+    match relation {
-        // This is a reply or a replacement, etc. It's not a thread.
+        Relation::Thread(thread) => {
-        // We don't care about this.
+            determine_interaction_context_for_room_event_related_to_thread(
-        return Ok(None);
+                bot_user_id,
-    };
+                bot_display_name,
+                room,
+                current_event,
+                event_fetcher,
+                current_event_is_mentioning_bot,
+                thread,
+            )
+            .await
+        }
+        Relation::Reply { in_reply_to } => {
+            determine_interaction_context_for_room_event_related_to_reply(
+                current_event,
+                current_event_is_mentioning_bot,
+                in_reply_to.event_id.clone(),
+            )
+            .await
+        }
 
+        // This is a replacement or something else. It's not something we support.
+        _ => return Ok(None),
+    }
+}
+
+async fn determine_interaction_context_for_room_event_related_to_thread(
+    bot_user_id: &OwnedUserId,
+    bot_display_name: &Option<String>,
+    room: &Room,
+    current_event: &OriginalSyncRoomMessageEvent,
+    event_fetcher: &Arc<RoomEventFetcher>,
+    current_event_is_mentioning_bot: bool,
+    thread: &Thread,
+) -> anyhow::Result<Option<InteractionContext>> {
     let thread_info = ThreadInfo::new(thread.event_id.clone(), current_event.event_id.clone());
 
+    tracing::trace!(
+        ?current_event_is_mentioning_bot,
+        is_thread_root_only = thread_info.is_thread_root_only(),
+        "Dealing with a thread reply",
+    );
+
+    if current_event_is_mentioning_bot && !thread_info.is_thread_root_only() {
+        // If the current event is a thread reply and is mentioning the bot,
+        // it's probably someone trying to involve us in the threaded conversation.
+        // See: https://github.com/etkecc/baibot/issues/15
+        //
+        // In such cases, we don't care what the thread root event is like or what the current event is like,
+        // we want text-generation to be triggered for this whole thread regardless.
+        return Ok(Some(InteractionContext {
+            thread_info,
+            trigger: InteractionTrigger {
+                is_mentioning_bot: true,
+                payload: MessagePayload::SynthethicChatCompletionTriggerInThread,
+            },
+        }));
+    }
+
     let start_time = std::time::Instant::now();
 
     let thread_start_timeline_event = event_fetcher
@@ -180,82 +388,47 @@ pub async fn determine_thread_context_for_room_event(
         "Fetched thread start event"
     );
 
-    let thread_start_timeline_event_deserialized =
+    let thread_start_detailed_message_payload = timeline_event_to_detailed_message_payload(
-        match thread_start_timeline_event.event.deserialize() {
+        &thread.event_id,
-            Ok(value) => value,
+        thread_start_timeline_event,
-            Err(err) => {
+        thread_info.clone(),
-                return Err(anyhow::format_err!(
+        bot_user_id,
-                    "Failed to deserialize thread start event {}: {:?}",
+        bot_display_name,
-                    thread.event_id,
+    )?;
-                    err
-                ));
-            }
-        };
 
-    let AnyTimelineEvent::MessageLike(thread_start_message_like_event) =
+    let Some(detailed_message_payload) = thread_start_detailed_message_payload else {
-        thread_start_timeline_event_deserialized
-    else {
-        tracing::trace!(
-            "Ignoring non-MessageLike thread start event: {:?}",
-            thread_start_timeline_event_deserialized
-        );
         return Ok(None);
     };
 
-    let (thread_start_message_is_mentioning_bot, thread_start_message_payload) =
+    Ok(Some(InteractionContext {
-        match thread_start_message_like_event {
+        thread_info,
-            AnyMessageLikeEvent::RoomEncrypted(room_message) => {
+        trigger: InteractionTrigger {
-                tracing::warn!(
+            is_mentioning_bot: detailed_message_payload.is_mentioning_bot,
-                    "Could not inspect thread start event {} because it failed to decrypt: {:?}",
+            payload: detailed_message_payload.message_payload,
-                    thread.event_id.clone(),
+        },
-                    room_message
+    }))
-                );
+}
 
-                // There's no way to know and it doesn't matter anyway.
+async fn determine_interaction_context_for_room_event_related_to_reply(
-                let is_mentioning_bot = false;
+    current_event: &OriginalSyncRoomMessageEvent,
+    current_event_is_mentioning_bot: bool,
+    reply_to_event_id: OwnedEventId,
+) -> anyhow::Result<Option<InteractionContext>> {
+    tracing::trace!(?current_event_is_mentioning_bot, "Dealing with a reply");
 
-                (
+    if !current_event_is_mentioning_bot {
-                    is_mentioning_bot,
+        // If the current event is not mentioning the bot, we don't care about it.
-                    MessagePayload::Encrypted(thread_info.clone()),
+        tracing::trace!("Ignoring reply event which does not mention the bot");
-                )
+        return Ok(None);
-            }
+    }
-            AnyMessageLikeEvent::RoomMessage(room_message) => {
-                if let MessageLikeEvent::Original(room_message_original) = room_message {
-                    let room_message_payload: Result<MessagePayload, String> =
-                        room_message_original.content.msgtype.clone().try_into();
 
-                    let Ok(room_message_payload) = room_message_payload else {
+    let thread_info = ThreadInfo::new(reply_to_event_id.clone(), current_event.event_id.clone());
-                        tracing::debug!(
-                            msg_type = room_message_original.content.msgtype(),
-                            "Ignoring thread start message of unknown type",
-                        );
-                        return Ok(None);
-                    };
 
-                    let is_mentioning_bot =
+    Ok(Some(InteractionContext {
-                        is_event_mentioning_bot(&room_message_original.content, bot_user_id);
+        thread_info,
-
+        trigger: InteractionTrigger {
-                    (is_mentioning_bot, room_message_payload)
+            is_mentioning_bot: true,
-                } else {
+            payload: MessagePayload::SynthethicChatCompletionTriggerForReply,
-                    tracing::error!("Ignoring thread start message which appears to be redacted");
-
-                    return Ok(None);
-                }
-            }
-            other => {
-                tracing::trace!(
-                    "Ignoring unknown MessageLike thread start event: {:?}",
-                    other
-                );
-                return Ok(None);
-            }
-        };
-
-    Ok(Some(ThreadContext {
-        info: thread_info,
-        first_message: ThreadContextFirstMessage {
-            is_mentioning_bot: thread_start_message_is_mentioning_bot,
-            payload: thread_start_message_payload,
         },
     }))
 }
@@ -263,6 +436,7 @@ pub async fn determine_thread_context_for_room_event(
 fn is_event_mentioning_bot(
     event_content: &RoomMessageEventContent,
     bot_user_id: &OwnedUserId,
+    bot_display_name: &Option<String>,
 ) -> bool {
     if let Some(mentions) = &event_content.mentions {
         mentions
@@ -274,11 +448,138 @@ fn is_event_mentioning_bot(
         // (see https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions),
         // we also do string matching here.
         //
-        // It may be even better to match not only against the MXID, but also against the bot's
+        // As of 2024-10-03, at least Element iOS does not support the new Mentions specification
-        // room-specific display name.
+        // and is still quite widespread.
         //
         // We may consider dropping this string-matching behavior altogether in the future,
         // so improving this compatibility block is not a high priority.
-        event_content.body().contains(bot_user_id.as_str())
+        if event_content.body().contains(bot_user_id.as_str()) {
+            return true;
+        }
+
+        if let Some(bot_display_name) = bot_display_name {
+            return event_content.body().contains(bot_display_name);
+        }
+
+        false
     }
 }
+
+fn timeline_event_to_detailed_message_payload(
+    timeline_event_id: &OwnedEventId,
+    timeline_event: TimelineEvent,
+    thread_info: ThreadInfo,
+    bot_user_id: &OwnedUserId,
+    bot_display_name: &Option<String>,
+) -> anyhow::Result<Option<DetailedMessagePayload>> {
+    let timeline_event_deserialized = match timeline_event.raw().deserialize() {
+        Ok(value) => value,
+        Err(err) => {
+            return Err(anyhow::format_err!(
+                "Failed to deserialize timeline event {}: {:?}",
+                timeline_event_id,
+                err
+            ));
+        }
+    };
+
+    let AnySyncTimelineEvent::MessageLike(thread_start_message_like_event) =
+        timeline_event_deserialized
+    else {
+        tracing::trace!(
+            "Ignoring non-MessageLike timeline event: {:?}",
+            timeline_event_deserialized
+        );
+        return Ok(None);
+    };
+
+    let (is_mentioning_bot, message_payload) = match thread_start_message_like_event {
+        AnySyncMessageLikeEvent::RoomEncrypted(room_message) => {
+            tracing::warn!(
+                "Could not inspect event {} because it failed to decrypt: {:?}",
+                timeline_event_id.clone(),
+                room_message
+            );
+
+            // There's no way to know and it doesn't matter anyway.
+            let is_mentioning_bot = false;
+
+            (
+                is_mentioning_bot,
+                MessagePayload::Encrypted(thread_info.clone()),
+            )
+        }
+        AnySyncMessageLikeEvent::RoomMessage(room_message) => {
+            if let SyncMessageLikeEvent::Original(room_message_original) = room_message {
+                let room_message_payload: Result<MessagePayload, String> =
+                    room_message_original.content.msgtype.clone().try_into();
+
+                let Ok(room_message_payload) = room_message_payload else {
+                    tracing::debug!(
+                        msg_type = room_message_original.content.msgtype(),
+                        "Ignoring event message of unknown type",
+                    );
+                    return Ok(None);
+                };
+
+                let is_mentioning_bot = is_event_mentioning_bot(
+                    &room_message_original.content,
+                    bot_user_id,
+                    bot_display_name,
+                );
+
+                (is_mentioning_bot, room_message_payload)
+            } else {
+                tracing::error!("Ignoring event message which appears to be redacted");
+
+                return Ok(None);
+            }
+        }
+        other => {
+            tracing::trace!("Ignoring unknown MessageLike event: {:?}", other);
+            return Ok(None);
+        }
+    };
+
+    Ok(Some(DetailedMessagePayload {
+        is_mentioning_bot,
+        message_payload,
+    }))
+}
+
+/// Creates a list of prefixes to strip from the beginning of message texts that mention the bot user.
+///
+/// Different clients do mentions differently.
+/// The body text containing the mention usually contains one of:
+/// - the full user ID (includes a @ prefix by default)
+/// - the localpart (with a @ prefix)
+/// - the localpart (without a @ prefix)
+/// - the display name (with a @ prefix)
+/// - the display name (without a @ prefix)
+///
+/// Some add a `: ` suffix after the mention.
+///
+/// There's no guarantee that the mention is at the start even.
+/// It being there is most common and we try to strip it from there
+/// as best as we can.
+pub fn create_list_of_bot_user_prefixes_to_strip(
+    bot_user_id: &OwnedUserId,
+    bot_display_name: &Option<String>,
+) -> Vec<String> {
+    let bot_user_id_localpart = bot_user_id.localpart();
+
+    let mut prefixes_to_strip = vec![
+        bot_user_id.as_str().to_owned(),
+        format!("@{}", bot_user_id_localpart),
+        bot_user_id_localpart.to_owned(),
+    ];
+
+    if let Some(bot_display_name) = bot_display_name {
+        prefixes_to_strip.push(format!("@{}", bot_display_name));
+        prefixes_to_strip.push(bot_display_name.to_owned());
+    }
+
+    prefixes_to_strip.push(":".to_owned());
+
+    prefixes_to_strip
+}

src/conversation/matrix/utils/tests.rs

@@ -1,29 +1,42 @@
+use chrono::{TimeZone, Utc};
+
+use mxlink::matrix_sdk::ruma::OwnedUserId;
+
 use crate::conversation::matrix::{
     MatrixMessage, MatrixMessageProcessingParams, MatrixMessageType,
 };
 
 #[test]
 fn is_message_from_allowed_sender() {
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
-    let allowed_user_id = "@user.someone:example.com";
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
-    let unallowed_user_id = "@another:example.com";
+    let allowed_user_id = OwnedUserId::try_from("@user.someone:example.com").unwrap();
+    let unallowed_user_id = OwnedUserId::try_from("@another:example.com").unwrap();
+
+    let timestamp = Utc.with_ymd_and_hms(2024, 9, 20, 18, 34, 15).unwrap();
 
     let bot_message = MatrixMessage {
         sender_id: bot_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let allowed_user_message = MatrixMessage {
         sender_id: allowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let unallowed_user_message = MatrixMessage {
         sender_id: unallowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let parsed_regex = match mxidwc::parse_pattern("@user.*:example.com") {
@@ -36,65 +49,106 @@ fn is_message_from_allowed_sender() {
     let allowed_users = vec![parsed_regex];
 
     assert!(
-        super::is_message_from_allowed_sender(&bot_message, bot_user_id, &vec![]),
+        super::is_message_from_allowed_sender(&bot_message, &bot_user_id, Some(&allowed_users)),
         "Bot message should be allowed"
     );
 
     assert!(
-        super::is_message_from_allowed_sender(&allowed_user_message, bot_user_id, &allowed_users),
+        super::is_message_from_allowed_sender(
+            &allowed_user_message,
+            &bot_user_id,
+            Some(&allowed_users)
+        ),
         "Allowed user message should be allowed"
     );
 
     assert!(
         !super::is_message_from_allowed_sender(
             &unallowed_user_message,
-            bot_user_id,
+            &bot_user_id,
-            &allowed_users
+            Some(&allowed_users),
         ),
         "Unallowed user message should be ignored"
     );
+
+    assert!(
+        super::is_message_from_allowed_sender(&unallowed_user_message, &bot_user_id, None,),
+        "An empty list of allowed users lets everyone through"
+    );
 }
 
 #[tokio::test]
-async fn process_matrix_messages_in_thread() {
+async fn process_matrix_messages() {
-    let bot_user_id = "@bot:example.com";
+    let bot_user_id =
-    let allowed_user_id = "@user.someone:example.com";
+        OwnedUserId::try_from("@bot:example.com").expect("Failed to parse bot user ID");
-    let unallowed_user_id = "@another:example.com";
+    let allowed_user_id = OwnedUserId::try_from("@user.someone:example.com").unwrap();
+    let unallowed_user_id = OwnedUserId::try_from("@another:example.com").unwrap();
+
+    let timestamp = Utc.with_ymd_and_hms(2024, 9, 20, 18, 34, 15).unwrap();
 
     let allowed_user_message = MatrixMessage {
         sender_id: allowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello from the user!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let allowed_user_message_with_prefix = MatrixMessage {
         sender_id: allowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "!bai Hello from the user!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let allowed_user_message_with_prefix_no_space = MatrixMessage {
         sender_id: allowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "!baiHello from the user!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let allowed_user_message_with_prefix_full_width_space = MatrixMessage {
         sender_id: allowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "!bai　Hello from the user!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let bot_message = MatrixMessage {
         sender_id: bot_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello from the bot!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
+    };
+
+    let allowed_user_message_with_bot_mention = MatrixMessage {
+        sender_id: allowed_user_id.to_owned(),
+        message_type: MatrixMessageType::Text,
+        message_text: "@baibot: Hello from the user!".to_owned(),
+        mentioned_users: vec![bot_user_id.to_owned()],
+        timestamp,
+    };
+
+    // The message text is the same as above - it mentions the bot, but the actually-mentioned user is another user.
+    let allowed_user_message_with_another_user_mention = MatrixMessage {
+        sender_id: allowed_user_id.to_owned(),
+        message_type: MatrixMessageType::Text,
+        message_text: allowed_user_message_with_bot_mention.message_text.clone(),
+        mentioned_users: vec![allowed_user_id.to_owned()],
+        timestamp,
     };
 
     let unallowed_user_message = MatrixMessage {
         sender_id: unallowed_user_id.to_owned(),
         message_type: MatrixMessageType::Text,
         message_text: "Hello from an unallowed user!".to_owned(),
+        mentioned_users: vec![],
+        timestamp,
     };
 
     let parsed_regex = match mxidwc::parse_pattern("@user.*:example.com") {
@@ -106,12 +160,24 @@ async fn process_matrix_messages_in_thread() {
 
     let allowed_users = vec![parsed_regex];
 
-    let message_processing_params_basic =
+    let message_processing_params_basic = super::MatrixMessageProcessingParams::new(
-        super::MatrixMessageProcessingParams::new(bot_user_id.to_owned(), allowed_users.clone());
+        bot_user_id.to_owned(),
+        Some(allowed_users.clone()),
+    );
 
     let message_processing_params_with_prefix_stripping =
-        super::MatrixMessageProcessingParams::new(bot_user_id.to_owned(), allowed_users.clone())
+        super::MatrixMessageProcessingParams::new(
-            .with_first_message_stripped_prefixes(vec!["!bai".to_owned()]);
+            bot_user_id.to_owned(),
+            Some(allowed_users.clone()),
+        )
+        .with_first_message_prefixes_to_strip(vec!["!bai".to_owned()]);
+
+    let message_processing_params_with_bot_user_prefix_stripping =
+        super::MatrixMessageProcessingParams::new(
+            bot_user_id.to_owned(),
+            Some(allowed_users.clone()),
+        )
+        .with_bot_user_prefixes_to_strip(vec!["@baibot: ".to_owned(), "@baibot".to_owned()]);
 
     struct TestCase {
         name: String,
@@ -195,10 +261,23 @@ async fn process_matrix_messages_in_thread() {
                 "!bai Hello from the user!".to_owned(),
             ],
         },
+        TestCase {
+            name: "Messages that mention the bot user get the bot user prefix stripped"
+                .to_owned(),
+            messages: vec![
+                allowed_user_message_with_bot_mention.clone(),
+                allowed_user_message_with_another_user_mention.clone(),
+            ],
+            message_processing_params: message_processing_params_with_bot_user_prefix_stripping.clone(),
+            expected_message_texts: vec![
+                "Hello from the user!".to_owned(),
+                "@baibot: Hello from the user!".to_owned(),
+            ],
+        },
     ];
 
     for test_case in test_cases {
-        let processed_messages = super::process_matrix_messages_in_thread(
+        let processed_messages = super::process_matrix_messages(
             &test_case.messages,
             &test_case.message_processing_params,
         )
@@ -216,3 +295,41 @@ async fn process_matrix_messages_in_thread() {
         );
     }
 }
+
+#[test]
+fn create_list_of_bot_user_prefixes_to_strip() {
+    let bot_user_id =
+        OwnedUserId::try_from("@baibot:example.com").expect("Failed to parse bot user ID");
+
+    // Test case 1: Bot user with no display name
+    let bot_display_name = None;
+    let prefixes =
+        super::create_list_of_bot_user_prefixes_to_strip(&bot_user_id, &bot_display_name);
+
+    assert_eq!(
+        prefixes,
+        vec![
+            "@baibot:example.com".to_string(),
+            "@baibot".to_string(),
+            "baibot".to_string(),
+            ":".to_string()
+        ]
+    );
+
+    // Test case 2: Bot user with display name
+    let bot_display_name = Some("Assistant".to_string());
+    let prefixes =
+        super::create_list_of_bot_user_prefixes_to_strip(&bot_user_id, &bot_display_name);
+
+    assert_eq!(
+        prefixes,
+        vec![
+            "@baibot:example.com".to_string(),
+            "@baibot".to_string(),
+            "baibot".to_string(),
+            "@Assistant".to_string(),
+            "Assistant".to_string(),
+            ":".to_string()
+        ]
+    );
+}

src/conversation/matrix_llm_bridge.rs

@@ -1,9 +1,14 @@
+use std::sync::Arc;
+
 use mxlink::matrix_sdk::ruma::OwnedEventId;
 use mxlink::MatrixLink;
 
+use crate::conversation::matrix::MatrixMessage;
+
 use super::llm::{convert_matrix_message_to_llm_message, Conversation, Message};
 use super::matrix::{
-    get_matrix_messages_in_thread, process_matrix_messages_in_thread, MatrixMessageProcessingParams,
+    get_matrix_messages_in_reply_chain, get_matrix_messages_in_thread, process_matrix_messages,
+    MatrixMessageProcessingParams, RoomEventFetcher,
 };
 
 pub async fn create_llm_conversation_for_matrix_thread(
@@ -14,7 +19,33 @@ pub async fn create_llm_conversation_for_matrix_thread(
 ) -> Result<Conversation, mxlink::matrix_sdk::Error> {
     let messages = get_matrix_messages_in_thread(matrix_link, room, thread_id).await?;
 
-    let messages_filtered = process_matrix_messages_in_thread(&messages, params).await;
+    let llm_messages = filter_messages_and_convert_to_llm_messages(messages, params).await;
+
+    Ok(Conversation {
+        messages: llm_messages,
+    })
+}
+
+pub async fn create_llm_conversation_for_matrix_reply_chain(
+    event_fetcher: &Arc<RoomEventFetcher>,
+    room: &mxlink::matrix_sdk::Room,
+    event_id: OwnedEventId,
+    params: &MatrixMessageProcessingParams,
+) -> Result<Conversation, mxlink::matrix_sdk::Error> {
+    let messages = get_matrix_messages_in_reply_chain(event_fetcher, room, event_id).await?;
+
+    let llm_messages = filter_messages_and_convert_to_llm_messages(messages, params).await;
+
+    Ok(Conversation {
+        messages: llm_messages,
+    })
+}
+
+async fn filter_messages_and_convert_to_llm_messages(
+    messages: Vec<MatrixMessage>,
+    params: &MatrixMessageProcessingParams,
+) -> Vec<Message> {
+    let messages_filtered = process_matrix_messages(&messages, params).await;
 
     let mut llm_messages: Vec<Message> = Vec::new();
 
@@ -28,7 +59,5 @@ pub async fn create_llm_conversation_for_matrix_thread(
         llm_messages.push(llm_message);
     }
 
-    Ok(Conversation {
+    llm_messages
-        messages: llm_messages,
-    })
 }

src/conversation/mod.rs

@@ -2,4 +2,6 @@ pub(crate) mod llm;
 pub(crate) mod matrix;
 mod matrix_llm_bridge;
 
-pub(crate) use matrix_llm_bridge::create_llm_conversation_for_matrix_thread;
+pub(crate) use matrix_llm_bridge::{
+    create_llm_conversation_for_matrix_reply_chain, create_llm_conversation_for_matrix_thread,
+};

src/entity/interaction_context.rs

@@ -0,0 +1,13 @@
+use mxlink::ThreadInfo;
+
+use super::MessagePayload;
+
+pub struct InteractionContext {
+    pub thread_info: ThreadInfo,
+    pub trigger: InteractionTrigger,
+}
+
+pub struct InteractionTrigger {
+    pub is_mentioning_bot: bool,
+    pub payload: MessagePayload,
+}

src/entity/message_context.rs

@@ -15,6 +15,8 @@ pub struct MessageContext {
     admin_whitelist_regexes: Vec<regex::Regex>,
     trigger_event_info: TriggerEventInfo,
     thread_info: ThreadInfo,
+
+    bot_display_name: Option<String>,
 }
 
 impl MessageContext {
@@ -31,9 +33,20 @@ impl MessageContext {
             admin_whitelist_regexes,
             trigger_event_info,
             thread_info,
+
+            bot_display_name: None,
         }
     }
 
+    pub fn with_bot_display_name(mut self, value: Option<String>) -> Self {
+        self.bot_display_name = value;
+        self
+    }
+
+    pub fn bot_display_name(&self) -> &Option<String> {
+        &self.bot_display_name
+    }
+
     pub fn room(&self) -> &Room {
         &self.room
     }
@@ -70,12 +83,12 @@ impl MessageContext {
         &self.thread_info
     }
 
-    pub fn sender_can_manage_global_config(&self) -> anyhow::Result<bool> {
+    pub fn sender_can_manage_global_config(&self) -> bool {
-        Ok(self.trigger_event_info.sender_is_admin)
+        self.trigger_event_info.sender_is_admin
     }
 
-    pub fn sender_can_manage_room_local_agents(&self) -> anyhow::Result<bool> {
+    pub fn sender_can_manage_room_local_agents(&self) -> mxidwc::Result<bool> {
-        Ok(self.sender_can_manage_global_config()?
+        Ok(self.sender_can_manage_global_config()
             || self.sender_is_allowed_room_local_agent_manager()?)
     }
 
@@ -102,21 +115,8 @@ impl MessageContext {
         combined
     }
 
-    fn sender_is_allowed_room_local_agent_manager(&self) -> anyhow::Result<bool> {
+    fn sender_is_allowed_room_local_agent_manager(&self) -> mxidwc::Result<bool> {
-        match &self
+        self.room_config_context()
-            .global_config()
+            .is_user_allowed_room_local_agent_manager(self.sender_id().clone())
-            .access
-            .room_local_agent_manager_patterns
-        {
-            None => Ok(false),
-            Some(patterns) => {
-                let allowed_regexes = mxidwc::parse_patterns_vector(patterns)?;
-
-                Ok(mxidwc::match_user_id(
-                    self.sender_id().as_str(),
-                    &allowed_regexes,
-                ))
-            }
-        }
     }
 }

src/entity/message_payload.rs

@@ -6,10 +6,28 @@ use mxlink::matrix_sdk::ruma::{OwnedEventId, OwnedUserId};
 use mxlink::ThreadInfo;
 
 /// MessagePayload is like matrix-sdk's MessageType, but represents only message types that the bot deals with and payloads are massaged a bit.
+///
+/// This also includes a few synthetic events.
 #[derive(Debug, Clone)]
 pub enum MessagePayload {
-    Text(TextMessageEventContent),
+    /// A synthetic message payload that indicates that the bot should produce a reply inside a thread.
+    /// This does not represent an actual message event, it's just a way to trigger a chat completion.
+    ///
+    /// When this is invoked, the ThreadInfo contains the full thread details (which represents our context).
+    ///
+    /// See: https://github.com/etkecc/baibot/issues/15
+    SynthethicChatCompletionTriggerInThread,
 
+    /// A synthetic message payload that indicates that the bot should produce a reply to a specific message.
+    /// This does not represent an actual message event, it's just a way to trigger a chat completion.
+    ///
+    /// When this is invoked, the ThreadInfo would refer to the reply-message that triggered us.
+    /// We can follow the chain upward from it to get the full context.
+    ///
+    /// See: https://github.com/etkecc/baibot/issues/15
+    SynthethicChatCompletionTriggerForReply,
+
+    Text(TextMessageEventContent),
     Audio(AudioMessageEventContent),
 
     Reaction {

src/entity/mod.rs

@@ -1,15 +1,15 @@
 pub mod catch_up_marker;
 pub mod cfg;
 pub mod globalconfig;
+mod interaction_context;
 mod message_context;
 mod message_payload;
 mod room_config_context;
 pub mod roomconfig;
-mod thread_context;
 mod trigger_event_info;
 
+pub use interaction_context::{InteractionContext, InteractionTrigger};
 pub use message_context::MessageContext;
 pub use message_payload::MessagePayload;
 pub use room_config_context::RoomConfigContext;
-pub use thread_context::{ThreadContext, ThreadContextFirstMessage};
 pub use trigger_event_info::TriggerEventInfo;

src/entity/room_config_context.rs

@@ -1,3 +1,5 @@
+use mxlink::matrix_sdk::ruma::OwnedUserId;
+
 use super::globalconfig::GlobalConfig;
 use super::roomconfig::RoomConfig;
 
@@ -180,4 +182,18 @@ impl RoomConfigContext {
                     .clone()
             })
     }
+
+    pub fn is_user_allowed_room_local_agent_manager(
+        &self,
+        user_id: OwnedUserId,
+    ) -> mxidwc::Result<bool> {
+        match &self.global_config.access.room_local_agent_manager_patterns {
+            None => Ok(false),
+            Some(patterns) => {
+                let allowed_regexes = mxidwc::parse_patterns_vector(patterns)?;
+
+                Ok(mxidwc::match_user_id(user_id.as_str(), &allowed_regexes))
+            }
+        }
+    }
 }

src/entity/thread_context.rs

@@ -1,13 +0,0 @@
-use mxlink::ThreadInfo;
-
-use super::MessagePayload;
-
-pub struct ThreadContext {
-    pub info: ThreadInfo,
-    pub first_message: ThreadContextFirstMessage,
-}
-
-pub struct ThreadContextFirstMessage {
-    pub is_mentioning_bot: bool,
-    pub payload: MessagePayload,
-}

src/strings/agent.rs

@@ -44,7 +44,7 @@ pub fn not_allowed_to_manage_static_agents() -> String {
 
 pub fn configuration_does_not_result_in_a_working_agent(err: anyhow::Error) -> String {
     format!(
-        "The provided configuration does not result in a working agent. The following error was encountered when trying to talk to the agent API:\n```\n{}```",
+        "The provided configuration does not result in a working agent. The following error was encountered when trying to talk to the agent API:\n```\n{}\n```",
         err,
     )
 }

src/strings/help/agent.rs

@@ -2,15 +2,8 @@ pub fn heading() -> String {
     "🤖 Agents".to_owned()
 }
 
-pub fn intro(command_prefix: &str, can_see_providers: bool) -> String {
+pub fn intro(command_prefix: &str) -> String {
-    format!(
+    format!("An agent is an instantiation and configuration of some **☁️ provider** (see `{command_prefix} provider`).")
-        "An agent is an instantiation and configuration of some **☁️ provider**{}.",
-        if can_see_providers {
-            format!(" (see `{command_prefix} provider`)")
-        } else {
-            "".to_owned()
-        }
-    )
 }
 
 pub fn intro_handler_relation(command_prefix: &str) -> String {
@@ -24,7 +17,7 @@ pub fn intro_capabilities() -> String {
 }
 
 pub fn no_permission_to_create_agents() -> &'static str {
-    "You are neither an administrator, nor a room-local agent manager, so **you cannot create new agents by yourself**."
+    "⚠️ You are neither a bot administrator, nor a room-local agent manager, so **you cannot create new agents by yourself**."
 }
 
 pub fn list_agents(command_prefix: &str) -> String {

src/strings/introduction.rs

@@ -23,20 +23,6 @@ fn purposes_intro() -> &'static str {
     "I can typically be used for the following purposes:"
 }
 
-fn no_text_generation_handler_agent() -> String {
-    format!(
-        "There is no configured handler agent which supports {} {}.",
-        AgentPurpose::TextGeneration.emoji(),
-        AgentPurpose::TextGeneration
-    )
-}
-
-fn introduction_outro(command_prefix: &str) -> String {
-    format!(
-        "You may also send a `{command_prefix} help` command message in this room for more information."
-    )
-}
-
 pub async fn create_on_join_introduction(
     name: &str,
     command_prefix: &str,
@@ -111,18 +97,17 @@ pub async fn create_on_join_introduction(
     message.push_str("\n\n");
 
     if got_text_generation_agent {
-        message.push_str(&simply_send_a_message(
+        message.push_str(&make_use_of_me_simply_send_a_message(
             command_prefix,
             room_config_context.text_generation_prefix_requirement_type(),
         ));
     } else {
-        message.push_str(&no_text_generation_handler_agent());
+        message.push_str(&make_use_of_me_agent_creation(
+            command_prefix,
+            room_config_context.text_generation_prefix_requirement_type(),
+        ));
     }
 
-    message.push_str("\n\n");
-
-    message.push_str(&introduction_outro(command_prefix));
-
     message
 }
 
@@ -130,17 +115,85 @@ pub fn create_short_introduction(name: &str) -> String {
     its_me(name)
 }
 
-fn simply_send_a_message(
+fn make_use_of_me_simply_send_a_message(
+    command_prefix: &str,
+    prefix_requirement_type: TextGenerationPrefixRequirementType,
+) -> String {
+    let message = r#"**To make use of me**:
+
+1. 👋 %send_a_message%
+2. 📖 %learn_more%
+"#;
+
+    message
+        .replace("%command_prefix%", command_prefix)
+        .replace(
+            "%send_a_message%",
+            &send_a_text_message(command_prefix, prefix_requirement_type),
+        )
+        .replace(
+            "%learn_more%",
+            &learn_more_from_usage_or_help(command_prefix),
+        )
+}
+
+fn make_use_of_me_agent_creation(
+    command_prefix: &str,
+    prefix_requirement_type: TextGenerationPrefixRequirementType,
+) -> String {
+    let message = r#"**To make use of me**:
+
+1. ☁️ **Choose an agent provider** (e.g. OpenAI, Mistral, etc). Send a `%command_prefix% provider` command to see the list.
+2. 🤖 %create_one_or_more_agents%
+3. 🤝 %set_new_agent_as_handler%
+4. 👋 %send_a_message%
+5. 📖 %learn_more%
+"#;
+
+    message
+        .replace("%command_prefix%", command_prefix)
+        .replace(
+            "%send_a_message%",
+            &send_a_text_message(command_prefix, prefix_requirement_type),
+        )
+        .replace(
+            "%learn_more%",
+            &learn_more_from_usage_or_help(command_prefix),
+        )
+        .replace(
+            "%create_one_or_more_agents%",
+            &create_one_or_more_agents(command_prefix),
+        )
+        .replace(
+            "%set_new_agent_as_handler%",
+            &set_new_agent_as_handler(command_prefix),
+        )
+}
+
+fn send_a_text_message(
     command_prefix: &str,
     prefix_requirement_type: TextGenerationPrefixRequirementType,
 ) -> String {
     match prefix_requirement_type {
         TextGenerationPrefixRequirementType::No => {
-            "**To make use of me, send a message** in this room (e.g. `Hello!`) and see me reply."
+            "**Send a text message** in this room (e.g. `Hello!`) and see me reply.".to_owned()
-                .to_owned()
         }
         TextGenerationPrefixRequirementType::CommandPrefix => {
-            format!("In this room, I'm configured to require the command prefix (`{command_prefix}`) for text messages.\n**To make use of me, send a prefixed text message** (e.g. `{command_prefix} Hello!`) and see me reply.")
+            format!("In this room, I'm configured to require the command prefix (`{command_prefix}`) for text messages. **Send a prefixed text message** (e.g. `{command_prefix} Hello!`) and see me reply.")
         }
     }
 }
+
+fn learn_more_from_usage_or_help(command_prefix: &str) -> String {
+    format!(
+        "**Learn more** by sending a `{command_prefix} usage` or `{command_prefix} help` command."
+    )
+}
+
+pub fn create_one_or_more_agents(command_prefix: &str) -> String {
+    format!("**Create one or more agents** in this room or globally. The provider help message will show you **🗲 Quick start** commands, but you may also send a `{command_prefix} agent` command to see the guide.")
+}
+
+pub fn set_new_agent_as_handler(command_prefix: &str) -> String {
+    format!("**Set the new agent as a handler** for a given use-purpose like text-generation, image-generation, etc. The agent-creation wizard will tell you how, but you may also send a `{command_prefix} config` command to see the guide (in the 🤖 *Handler Agents* section).")
+}

src/strings/provider.rs

@@ -25,10 +25,6 @@ pub fn invalid_configuration_for_provider(
     )
 }
 
-pub fn not_allowed() -> String {
-    "You are not allowed to see the providers list.".to_owned()
-}
-
 pub fn providers_list_intro() -> String {
     "The list of supported providers is below.".to_owned()
 }
@@ -39,7 +35,7 @@ pub fn help_how_to_choose_heading() -> String {
 
 pub fn help_how_to_choose_description(command_prefix: &str) -> String {
     let str = r#"
-If you're not sure which provider to start with, we **recommend OpenAI** as it's the most popular and has the **widest range of capabilities**.
+If you're not sure which provider to start with, **we recommend OpenAI** as it's the most popular and has the **widest range of capabilities**.
 
 You don't need to choose just one though. The bot supports **mixing & matching models** (by setting different handlers for different types of messages - see `%command_prefix% config`), so you can use multiple providers at the same time.
 "#;
@@ -55,13 +51,21 @@ pub fn help_how_to_use_heading() -> String {
 
 pub fn help_how_to_use_description(command_prefix: &str) -> String {
     let str = r#"
-- sign up for it
+1. 📝 **Sign up for it**
-- obtain an API key
+2. 🔑 **Obtain an API key**
-- create a new agent (see `%command_prefix% agent`)
+3. 🤖 %create_one_or_more_agents%
-- set the new agent as a handler for some types of messages (see `%command_prefix% config`)
+4. 🤝 %set_new_agent_as_handler%
 "#;
 
     str.replace("%command_prefix%", command_prefix)
+        .replace(
+            "%create_one_or_more_agents%",
+            &super::introduction::create_one_or_more_agents(command_prefix),
+        )
+        .replace(
+            "%set_new_agent_as_handler%",
+            &super::introduction::set_new_agent_as_handler(command_prefix),
+        )
         .trim()
         .to_owned()
 }

src/strings/usage.rs

@@ -12,7 +12,7 @@ If there's a text-generation handler agent configured (see `%command_prefix% con
 
 Whether the bot responds depends on the **💬 Text Generation / 🗟 Prefix Requirement** setting (see `%command_prefix% config status`).
 Sometimes, a prefix (e.g. `%command_prefix%`) is required in front of messages sent to the room for the bot to respond.
-For multi-user rooms, this setting defaults to "required"
+For multi-user rooms, this setting defaults to "required".
 
 Room messages start a threaded conversation where you can continue back-and-forth communication with the bot.
 

src/utils/text_to_speech.rs

@@ -5,12 +5,18 @@ use super::text::{block_quote, block_unquote};
 /// Creates a text message which is based on transcribed audio.
 /// This text message is prefixed with an emoji and blockquoted, to indicate that it is a transcription.
 /// To reverse the process, use `parse_transcribed_message_text()`.
+///
+/// It should be noted that in certain cases (Transcribe-only mode), transcriptions are posted as regular notice messages which do not include
+/// the `> 🦻` prefixing. That is, not every transcribed message will pass through here (intentionally).
 pub fn create_transcribed_message_text(text: &str) -> String {
     block_quote(&format!("{} {}", AgentPurpose::SpeechToText.emoji(), text))
 }
 
 /// Parses a transcribed message text, reversing the process done by `create_transcribed_message_text()`.
 /// If the provided text string does not match the expected format, None is returned.
+///
+/// It should be noted that in certain cases (Transcribe-only mode), transcriptions are posted as regular notice messages which do not include
+/// the `> 🦻` prefixing. This function will not handle these properly.
 pub fn parse_transcribed_message_text(text: &str) -> Option<String> {
     if !text.starts_with("> ") {
         return None;