Initial commit

docs/configuration/README.md

@@ -0,0 +1,48 @@
+## 🛠️ Configuration
+
+The bot's behavior is controlled by a combination of [static](#static-configuration) and [dynamic](#dynamic-configuration) configuration.
+
+
+### Static configuration
+
+The bot can be configured using a [YAML](https://en.wikipedia.org/wiki/YAML) configuration file as well as [environment variables](https://en.wikipedia.org/wiki/Environment_variable).
+
+When running the bot locally (during [🧑‍💻 development](../development.md)), the bot's configuration is read from the `var/app/config.yml` file.
+This file is created from the template found in [etc/app/config.yml.dist](../../etc/app/config.yml.dist).
+
+Certain keys can be left unset, in which case [📝 hardcoded defaults](../../src/entity/cfg/defaults.rs) would be used.
+
+Each configuration key found in the YAML configuration can be overridden by setting an environment variable (dots should be replaced with `_`). Example:
+
+- to override `command_prefix`, set an environment variable `BAIBOT_COMMAND_PREFIX`
+- to override `homeserver.server_name`, set an environment variable `BAIBOT_HOMESERVER_SERVER_NAME`
+
+The static configuration contains an `initial_global_config` key, which is used to populate the bot's global configuration (stored as [dynamic configuration](#dynamic-configuration)) the first time the bot starts. Modifying this subsequently will not have any effect. After initial global configuration creation, it's expected to be managed dynamically via chat commands.
+
+
+### Dynamic configuration
+
+Besides the bot's [static configuration](#static-configuration), **the bot can also be configured dynamically at runtime (via chat messages)**.
+
+This includes changes to [🔒 Access](../access.md), [🤖 Agents](../agents.md) and [🛠️ Room Settings](#room-settings).
+
+
+#### Room Settings
+
+Room Settings come from 3 different levels with priority in the following order (higher to lower):
+
+- 📍 per-room (`!bai config room ..` commands)
+- 🌐 globally (`!bai config global ..` commands)
+- 📝 as [hardcoded defaults](../../src/entity/cfg/defaults.rs)
+
+You can adjust the following settings per room and/or globally:
+
+- [💬 Text Generation](text-generation.md)
+- [🦻 Speech-to-Text](speech-to-text.md)
+- [🗣️ Text-to-Speech](text-to-speech.md)
+- [🖌️ Image Generation](image-generation.md)
+- [🤝 Handlers](handlers.md)
+
+Refer to the bot's help messages (as a response to a `!bai config` help command) for the most up-to-date information on what Room Settings can be configured.
+
+You can **get an overview of the configuration affecting the current room** (a mix of hardcoded defaults, agent defaults, global and room-level settings) by sending a `!bai config status` command to the room.

docs/configuration/handlers.md

@@ -0,0 +1,32 @@
+## 🤝 Handlers
+
+### Introduction
+
+You can use **different models in different rooms** (e.g. [OpenAI](../providers.md#openai) GPT-4o alongside [Llama](https://en.wikipedia.org/wiki/Llama_(language_model)) running on [Groq](../providers.md#groq), etc.)
+
+You can also use **different models within the same room** (e.g. [💬 text-generation](#-text-generation) handled by one [agent](./agents.md), [🦻 speech-to-text](#-speech-to-text) handled by another, [🗣️ text-to-speech](#️-text-to-speech) by a 3rd, etc.)
+
+The bot supports the following use-purposes:
+
+- [💬 text-generation](../features.md#-text-generation): communicating with you via text
+- [🦻 speech-to-text](../features.md#-speech-to-text): turning your voice messages into text
+- [🗣️ text-to-speech](../features.md#️-text-to-speech): turning bot or users text messages into voice messages
+- [🖌️ image-generation](../features.md#-image-generation): generating images based on instructions
+
+In a given room, each different purpose can be served by a different [provider](../providers.md) and model. This combination of provider and model configuration is called an [🤖 agent](../agents.md). Each purpose can be served by a different **handler** agent.
+
+See a [🖼️ Screenshot of an example room configuration](./screenshots/config-status-handlers.webp).
+
+
+### Configuring
+
+Handlers can be configured [dynamically](./README.md#dynamic-configuration):
+
+- either per-room (e.g. `!bai config room set-handler text-generation room-local/openai-gpt-4o`)
+- or globally (e.g. `!bai config global set-handler text-generation global/openai-gpt-4o`)
+
+The per-room configuration takes priority over the global configuration.
+
+There's also a `catch-all` purpose that can be used as a fallback handler for messages that don't match any other handler.
+
+💡 It's a good idea to globally-configure a powerful agent as a catch-all handler, so that the bot can always handle messages of any kind. You can then override individual handlers per room or globally.

docs/configuration/image-generation.md

@@ -0,0 +1,9 @@
+
+## 🖌️ Image Generation
+
+The Image Generation feature is not configurable at this moment.
+
+You may also wish to see:
+
+- [🌟 Features / 🖌️ Image Generation](../features.md#-image-generation) for a higher-level introduction to the Image Generation features
+- [📖 Usage / 🖌️ Image Generation](../usage.md#-image-generation) section for more details on how to use the bot for Image Generation in a room

docs/configuration/speech-to-text.md

@@ -0,0 +1,39 @@
+## 🦻 Speech-to-Text
+
+Below are some configuration settings related to Speech-to-Text.
+
+You may also wish to see:
+
+- [🌟 Features / 🦻 Speech-to-Text](../features.md#-speech-to-text) for a higher-level introduction to the Speech-to-Text features
+- [📖 Usage / 🦻 Speech-to-Text](../usage.md#-speech-to-text) section for more details on how to use the bot for Speech-to-Text in a room
+
+
+### 🪄 Flow Type
+
+Controls how voice messages sent by [👥 user](../access.md#-users) are handled.
+
+The following configuration values are recognized:
+
+- (default) `transcribe_and_generate_text`: the bot will turn [👥 user](../access.md#-users) voice messages into text and then generate text messages via [💬 Text Generation](../features.md#-text-generation). This is the default setting to allow for [Seamless voice interaction](../features.md#seamless-voice-interaction).
+
+- `ignore`: the bot will ignore all audio messages
+
+- `only_transcribe`: the bot will turn [👥 user](../access.md#-users) voice messages into text, but will **not** proceed with [💬 Text Generation](../features.md#-text-generation). Switching to this may be useful in some cases, as in [Transcribe-only mode](../features.md#transcribe-only-mode).
+
+Example: `!bai config room speech-to-text set-flow-type ignore` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+
+### 🔤 Language
+
+Lets you specify the language of the input voice messages, to avoid using auto-detection.
+Supplying the input language using a 2-letter code (e.g. `ja`) as per [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) may improve accuracy & latency.
+
+![Speech-to-Text Language setting usage example](../screenshots/speech-to-text-language.webp)
+
+In the above example screenshot, even without a language specified, the voice was understood correctly as [Bulgarian](https://en.wikipedia.org/wiki/Bulgarian_language), but was produced in latin, not [Cyrillic](https://en.wikipedia.org/wiki/Cyrillic_script), which is wrong.
+
+If different [👥 user](../access.md#-users) are using different languages, do not specify a language.
+
+💡 Certain models (like [OpenAI](../providers.md#openai)'s Whisper) may perform auto-translation if you specify a language, but you're speaking another one. You may abuse this side-effect for performing voice-to-text translation, but be aware that not all models behave this way.
+
+Example (setting it to Japanese): `!bai config room speech-to-text set-language ja` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))

docs/configuration/text-generation.md

@@ -0,0 +1,75 @@
+
+## 💬 Text Generation
+
+Below are some [🛠️ dynamic configuration settings](./README.md#dynamic-configuration) related to Text Generation.
+
+You may also wish to see:
+
+- [🌟 Features / 💬 Text Generation](../features.md#-text-generation) for a higher-level introduction to the Text Generation features
+- [📖 Usage / 💬 Text Generation](../usage.md#-text-generation) section for more details on how to use the bot for Text Generation in a room
+
+
+### 🗟 Prefix Requirement Type
+
+In Direct Message rooms with the bot (1:1 rooms), it most usually makes sense for the bot to respond to **all** of your messages, as shown on this [🖼️ screenshot](../screenshots/text-generation.webp).
+
+In group rooms (with multiple users), it may be more appropriate for the bot to only respond to messages that are **prefixed** with the command prefix (e.g. `!bai`), so that other chat exchange in the room will not trigger it. Such a setup is shown on this [🖼️ screenshot](../screenshots/text-generation-prefix-requirement.webp).
+
+There are exceptions to these rules, and you can configure the bot to respond only to prefixed messages in a 1:1 room, or to respond to all messages even in a multi-user group room.
+
+To support such use-cases, the bot has a `text-generation prefix-requirement-type` setting, which can be set to:
+
+- (default) `no`: indicates that the bot would not require a prefix and would respond to all messages
+
+- `command_prefix`: indicates that the bot would require that messages be prefixed with the command prefix (e.g. `!bai`) and would ignore all messages that are not prefixed
+
+By default, the bot is **auto-configured (upon joining a new room)** to use the `no` setting in rooms that only include 2 users (you and the bot), and `command_prefix` in rooms with more than 2 users. To prevent surprises, the bot will **not** adjust this setting subsequently. You can manually adjust it via `!bai config room set-prefix-requirement-type VALUE`.
+
+Example: `!bai config room text-generation set-prefix-requirement-type command_prefix` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+Regardless of this configuration, **the bot will also respond to messages which directly [mention](https://spec.matrix.org/latest/client-server-api/#user-and-room-mentions) the bot** (e.g. `@baibot`), even if they are not prefixed. An example of this can be seen on this [🖼️ screenshot](../screenshots/text-generation-prefix-requirement.webp).
+
+
+### 🪄 Auto Usage
+
+Text generation is enabled by default (the `text-generation auto-usage` setting being set to `always`), but can be set to:
+
+- (default) `always`: generate text for all messages (also see [🗟 Prefix Requirement Type](#-prefix-requirement-type))
+
+- `never`: never generate text for messages
+
+- `only_for_voice`: only generate text when the original user message was a voice message, later transcribed via [🦻 Speech-to-Text](../features.md#-speech-to-text)
+
+- `only_for_text`: only generate text when original user message was a text message
+
+Example: `!bai config room text-generation set-auto-usage only_for_voice` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+
+### ♻️ Context Management
+
+The bot also supports ♻️ **context management**, which automatically adjusts the message history length, etc.
+
+This feature relies on [tokenization](https://en.wikipedia.org/wiki/Large_language_model#Tokenization) performed by the [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) library which is [poorly well-maintained](https://github.com/zurawiki/tiktoken-rs/issues/50) and only works well for [OpenAI](../providers.md#openai) models.
+
+This setting is **disabled by default**, but can be enabled via `!bai config room context-management-enabled true` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings)).
+
+
+### ⌨️ Prompt Override
+
+You can override the [system prompt](https://huggingface.co/docs/transformers/en/tasks/prompting) configured at the [🤖 agent](../agents.md) level.
+
+Example (multi-line is supported):
+
+```
+!bai config room text-generation set-prompt-override You're a UI/UX expert. Everything you say needs to consider design and usability.
+
+Where appropriate, you'll mention best practices and common pitfalls.
+```
+
+A prompt override can also be set globally, see [🛠️ Room Settings](./README.md#room-settings).
+
+### 🌡️ Temperature Override
+
+You can override the [temperature](https://blogs.novita.ai/what-are-large-language-model-settings-temperature-top-p-and-max-tokens/#what-is-llm-temperature) (randomness / creativity) parameter configured at the [🤖 agent](../agents.md) level.
+
+Example: `!bai config room text-generation set-temperature-override 3.5` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))

docs/configuration/text-to-speech.md

@@ -0,0 +1,63 @@
+
+## 🗣️ Text-to-Speech
+
+Below are some configuration settings related to Text-to-Speech.
+
+You may also wish to see:
+
+- [🌟 Features / 🗣️ Text-to-Speech](../features.md#-text-generation) for a higher-level introduction to the Text-to-Speech features
+- [📖 Usage / 🗣️ Text-to-Speech](../usage.md#-text-generation) section for more details on how to use the bot for Text-to-Speech in a room
+
+
+### 🪄 Bot Messages Flow Type
+
+Controls how automatic text-to-speech functions for **messages sent by the bot**.
+
+The following configuration values are recognized:
+
+- (default) `on_demand_for_voice`: the bot will turn its own text messages into audio (voice) messages only after an allowed [👥 user](../access.md#-users) **reacts** to a bot's message with 🗣️. To make it easier for users to react without having to hunt for this emoji, the bot will automatically add a 🗣️ reaction to its own messages which are in response to a user audio (voice) message.
+
+- `on_demand_always`: the bot will turn its own text messages into audio (voice) messages only after an allowed [👥 user](../access.md#-users) **reacts** to a bot's message with 🗣️. To make it easier for users to react without having to hunt for this emoji, the bot will automatically add a 🗣️ reaction to **all of its own messages**.
+
+- `only_for_voice`: the bot will turn its own text messages into audio (voice) messages only if the original user message was a voice message. This is to allow for [Seamless voice interaction](../features.md#seamless-voice-interaction), where you can speak to the bot and then hear its responses
+
+- `never`: the bot will never turn its own text messags into audio (voice) messages
+
+- `always`: the bot will turn all its text messages into audio (voice) messages. This also allows for [Seamless voice interaction](../features.md#seamless-voice-interaction).
+
+Example: `!bai config room text-to-speech set-bot-msgs-flow-type never` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+
+### 🪄 User Messages Flow Type
+
+Controls how automatic text-to-speech functions for **messages sent by [👥 users](../access.md#-users)**.
+
+**Only works when automatic text-generation is disabled** (see [💬 Text Generation / 🪄 Auto Usage](./text-generation.md#-auto-usage)).
+
+The following configuration values are recognized:
+
+- (default) `never`: the bot will never turn [👥 user](../access.md#-users) text messages into audio (voice) messages
+
+- `on_demand`: the bot will turn [👥 user](../access.md#-users) text messages into audio (voice) messages if the text message receives a 🗣️ reaction
+
+- `always`: the bot will turn all [👥 user](../access.md#-users) text messages into audio (voice) messages. This is to allow for [Text-to-Speech-only mode](../features.md/#text-to-speech-only-mode).
+
+Example: `!bai config room text-to-speech set-user-msgs-flow-type always` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+
+### 🗲 Speed override
+
+The speed override setting lets you speed up/down speech relative to the default speed configured at the [🤖 agent](../agents.md) level (usually `1.0`).
+
+Values typically range from `0.25` to `4.0`, but may vary depending on the selected model.
+
+Example: `!bai config room text-to-speech set-speed-override 1.5` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))
+
+
+### 👫 Voice override
+
+The voice override setting lets you change the voice being used by the text-to-speech model configured at the [🤖 agent](../agents.md) level (usually `onyx` when using [OpenAI](../providers.md#openai)).
+
+Possible values (e.g. `onyx`) depend on the model you're using. For example, for [OpenAI](../providers.md#openai)'s Whisper model, [these voices](https://platform.openai.com/docs/guides/text-to-speech/voice-options) are available.
+
+Example: `!bai config room text-to-speech set-voice-override nova` (this can also be set globally, see [🛠️ Room Settings](./README.md#room-settings))