diff --git a/README.md b/README.md index 270fb3a..9ae618d 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ It's influenced by [chaz](https://github.com/arcuru/chaz), but does **not** use - Supports **different use purposes** (depending on the [☁️ provider](./docs/providers.md) & model): - - [πŸ’¬ text-generation](./docs/features.md#-text-generation): communicating with you via text (though certain models may "see" images as well) + - [πŸ’¬ text-generation](./docs/features.md#-text-generation): communicating with you via text (though certain models may "see" images as well). The [OpenAI provider](./docs/providers.md#openai) also supports [πŸ› οΈ built-in tools](./docs/features.md#️-built-in-tools-openai-only) (web search, code interpreter) - [🦻 speech-to-text](./docs/features.md#-speech-to-text): turning your voice messages into text - [πŸ—£οΈ text-to-speech](./docs/features.md#%EF%B8%8F-text-to-speech): turning bot or users text messages into voice messages - [πŸ–ŒοΈ image-generation](./docs/features.md#image-generation): creating and editing images based on instructions diff --git a/docs/features.md b/docs/features.md index 42414df..93df593 100644 --- a/docs/features.md +++ b/docs/features.md @@ -40,6 +40,23 @@ You may also wish to see: - [πŸ“– Usage / πŸ’¬ Text Generation](./usage.md#-text-generation) section for more details on how to use the bot for Text Generation in a room +#### πŸ› οΈ Built-in Tools (OpenAI only) + + + +The [OpenAI provider](./providers.md#openai) supports built-in tools that extend the model's capabilities: + +- [πŸ” Web Search](https://platform.openai.com/docs/guides/tools-web-search) (`web_search`): allows the model to search the web for up-to-date information. [πŸ–ΌοΈ Screenshot](./screenshots/text-generation-tools-web-search.webp) + +- [πŸ’» Code Interpreter](https://platform.openai.com/docs/guides/tools-code-interpreter) (`code_interpreter`): allows the model to write and execute Python code in a sandbox + +These tools are **disabled by default** and need to be explicitly enabled in the agent's `text_generation.tools` configuration. See the [OpenAI sample configuration](https://github.com/etkecc/baibot/blob/c70387b0c38d8d0f30bba2179a2a21a3710dbeaf/docs/sample-provider-configs/openai.yml#L12-L15) for reference. + +To enable tools on an existing dynamically-created agent, you need to [update the agent](./agents.md#updating-agents) to re-create it with the `text_generation.tools` section added and enable the tools you need + +πŸ’‘ **Note**: These tools run on OpenAI's infrastructure and may incur additional costs. Web search results include citations that are incorporated into the response. + + #### On-demand involvement In the following 2 cases, it's useful to involve the bot in conversations on-demand: diff --git a/docs/providers.md b/docs/providers.md index cd27132..7850d85 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -23,7 +23,7 @@ The list of supported providers is below. ### How to choose a provider -If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision), [πŸ–ŒοΈ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech). +If you're not sure which provider to start with, **we recommend [OpenAI](#openai)** as it's the most popular and has the **widest range of capabilities**: [πŸ’¬ text-generation](./features.md#-text-generation) (incl. vision, incl. [πŸ› οΈ tools](./features.md#️-built-in-tools-openai-only)), [πŸ–ŒοΈ image-generation](./features.md#️image-generation), [🦻 speech-to-text](./features.md#-speech-to-text), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech). You don't need to choose just one though. The bot supports [mixing & matching models](./features.md#-mixing--matching-models), so you can use multiple providers at the same time. @@ -47,7 +47,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo - πŸ†” Identifier: `anthropic` - πŸ”— Links: [🏠 Home page](https://www.anthropic.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/Anthropic), [πŸ‘€ Sign up](https://console.anthropic.com/), [πŸ“‹ Models list](https://docs.anthropic.com/en/docs/about-claude/models) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (incl. vision) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (incl. vision, no tools) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local anthropic my-anthropic-agent` - create a global agent: `!bai agent create-global anthropic my-anthropic-agent` @@ -61,7 +61,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo - πŸ†” Identifier: `groq` - πŸ”— Links: [🏠 Home page](https://groq.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/Groq), [πŸ‘€ Sign up](https://console.groq.com/login), [πŸ“‹ Models list](https://console.groq.com/docs/models) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision), [🦻 speech-to-text](./features.md#-speech-to-text) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools), [🦻 speech-to-text](./features.md#-speech-to-text) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local groq my-groq-agent` - create a global agent: `!bai agent create-global groq my-groq-agent` @@ -75,7 +75,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo - πŸ†” Identifier: `localai` - πŸ”— Links: [🏠 Home page](https://localai.io/), [πŸ“‹ Models list](https://localai.io/gallery.html) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local localai my-localai-agent` - create a global agent: `!bai agent create-global localai my-localai-agent` @@ -89,7 +89,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo - πŸ†” Identifier: `mistral` - πŸ”— Links: [🏠 Home page](https://mistral.ai/), [🌐 Wiki](https://en.wikipedia.org/wiki/Mistral_AI), [πŸ‘€ Sign up](https://auth.mistral.ai/ui/registration), [πŸ“‹ Models list](https://docs.mistral.ai/getting-started/models/) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local mistral my-mistral-agent` - create a global agent: `!bai agent create-global mistral my-mistral-agent` @@ -103,7 +103,7 @@ You don't need to choose just one though. The bot supports [mixing & matching mo - πŸ†” Identifier: `ollama` - πŸ”— Links: [🏠 Home page](https://ollama.com/), [πŸ“‹ Models list](https://ollama.com/library) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local ollama my-ollama-agent` - create a global agent: `!bai agent create-global ollama my-ollama-agent` @@ -120,7 +120,7 @@ For services which are not fully compatible with the OpenAI API, consider using - πŸ†” Identifier: `openai` - πŸ”— Links: [🏠 Home page](https://openai.com/), [🌐 Wiki](https://en.wikipedia.org/wiki/OpenAI), [πŸ‘€ Sign up](https://platform.openai.com/signup), [πŸ“‹ Models list](https://platform.openai.com/docs/models) -- 🌟 Capabilities: [πŸ–ŒοΈ image-generation](./features.md#️-image-creation), [πŸ’¬ text-generation](./features.md#-text-generation) (incl. vision), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) +- 🌟 Capabilities: [πŸ–ŒοΈ image-generation](./features.md#️-image-creation), [πŸ’¬ text-generation](./features.md#-text-generation) (incl. vision, incl. [πŸ› οΈ tools](./features.md#️-built-in-tools-openai-only)), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local openai my-openai-agent` - create a global agent: `!bai agent create-global openai my-openai-agent` @@ -137,7 +137,7 @@ Some of these popular services already have **shortcut** providers (leading to t This provider is just as featureful as the [OpenAI](#openai) provider, but is more compatible with services which do not fully adhere to the [OpenAI API spec](https://github.com/openai/openai-openapi/). - πŸ†” Identifier: `openai-compatible` -- 🌟 Capabilities: [πŸ–ŒοΈ image-generation](./features.md#️-image-creation), [πŸ’¬ text-generation](./features.md#-text-generation) (no vision), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) +- 🌟 Capabilities: [πŸ–ŒοΈ image-generation](./features.md#️-image-creation), [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools), [πŸ—£οΈ text-to-speech](./features.md#️-text-to-speech), [🦻 speech-to-text](./features.md#-speech-to-text) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local openai-compatible my-openai-compatible-agent` - create a global agent: `!bai agent create-global openai-compatible my-openai-compatible-agent` @@ -151,7 +151,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo - πŸ†” Identifier: `openrouter` - πŸ”— Links: [🏠 Home page](https://openrouter.ai/), [πŸ‘€ Sign up](https://openrouter.ai/), [πŸ“‹ Models list](https://openrouter.ai/models) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local openrouter my-openrouter-agent` - create a global agent: `!bai agent create-global openrouter my-openrouter-agent` @@ -165,7 +165,7 @@ This provider is just as featureful as the [OpenAI](#openai) provider, but is mo - πŸ†” Identifier: `together-ai` - πŸ”— Links: [🏠 Home page](https://www.together.ai/), [πŸ‘€ Sign up](https://api.together.ai/signup), [πŸ“‹ Models list](https://api.together.xyz/models) -- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision) +- 🌟 Capabilities: [πŸ’¬ text-generation](./features.md#-text-generation) (no vision, no tools) - πŸ—² Quick start: - create a room-local agent: `!bai agent create-room-local together-ai my-together-ai-agent` - create a global agent: `!bai agent create-global together-ai my-together-ai-agent` diff --git a/docs/screenshots/text-generation-tools-web-search.webp b/docs/screenshots/text-generation-tools-web-search.webp new file mode 100644 index 0000000..e2d9215 Binary files /dev/null and b/docs/screenshots/text-generation-tools-web-search.webp differ diff --git a/src/agent/provider/entity/agent_provider.rs b/src/agent/provider/entity/agent_provider.rs index 5d4f85b..5533531 100644 --- a/src/agent/provider/entity/agent_provider.rs +++ b/src/agent/provider/entity/agent_provider.rs @@ -69,6 +69,7 @@ impl AgentProvider { models_list_url: Some("https://docs.anthropic.com/en/docs/about-claude/models"), supported_purposes: vec![AgentPurpose::TextGeneration], text_generation_supports_vision: true, + text_generation_supports_tools: false, }, Self::Groq => AgentProviderInfo { id: Self::Groq.to_static_str(), @@ -80,11 +81,12 @@ impl AgentProvider { models_list_url: Some("https://console.groq.com/docs/models"), supported_purposes: vec![AgentPurpose::TextGeneration, AgentPurpose::SpeechToText], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::LocalAI => AgentProviderInfo { id: Self::LocalAI.to_static_str(), name: "LocalAI", - description: "LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.", + description: "LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.", homepage_url: Some("https://localai.io/"), wiki_url: None, sign_up_url: None, @@ -95,6 +97,7 @@ impl AgentProvider { AgentPurpose::SpeechToText, ], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::Mistral => AgentProviderInfo { id: Self::Mistral.to_static_str(), @@ -106,6 +109,7 @@ impl AgentProvider { models_list_url: Some("https://docs.mistral.ai/getting-started/models/"), supported_purposes: vec![AgentPurpose::TextGeneration], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::Ollama => AgentProviderInfo { id: Self::Ollama.to_static_str(), @@ -117,6 +121,7 @@ impl AgentProvider { models_list_url: Some("https://ollama.com/library"), supported_purposes: vec![AgentPurpose::TextGeneration], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::OpenAI => AgentProviderInfo { id: Self::OpenAI.to_static_str(), @@ -133,6 +138,7 @@ impl AgentProvider { AgentPurpose::SpeechToText, ], text_generation_supports_vision: true, + text_generation_supports_tools: true, }, Self::OpenAICompat => AgentProviderInfo { id: Self::OpenAICompat.to_static_str(), @@ -149,6 +155,7 @@ impl AgentProvider { AgentPurpose::SpeechToText, ], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::OpenRouter => AgentProviderInfo { id: Self::OpenRouter.to_static_str(), @@ -160,6 +167,7 @@ impl AgentProvider { models_list_url: Some("https://openrouter.ai/models"), supported_purposes: vec![AgentPurpose::TextGeneration], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, Self::TogetherAI => AgentProviderInfo { id: Self::TogetherAI.to_static_str(), @@ -171,6 +179,7 @@ impl AgentProvider { models_list_url: Some("https://api.together.xyz/models"), supported_purposes: vec![AgentPurpose::TextGeneration], text_generation_supports_vision: false, + text_generation_supports_tools: false, }, } } @@ -192,4 +201,5 @@ pub struct AgentProviderInfo { pub models_list_url: Option<&'static str>, pub supported_purposes: Vec, pub text_generation_supports_vision: bool, + pub text_generation_supports_tools: bool, } diff --git a/src/strings/provider.rs b/src/strings/provider.rs index 858d858..b2d0000 100644 --- a/src/strings/provider.rs +++ b/src/strings/provider.rs @@ -109,11 +109,21 @@ pub fn help_provider_details(id: &str, info: &AgentProviderInfo) -> String { let mut purpose_line = format!("{} {}", purpose.emoji(), purpose.as_str()); if let AgentPurpose::TextGeneration = purpose { + let mut extras = vec![]; + if info.text_generation_supports_vision { - purpose_line = format!("{} ({})", purpose_line, "incl. vision"); + extras.push("incl. vision"); } else { - purpose_line = format!("{} ({})", purpose_line, "no vision"); + extras.push("no vision"); } + + if info.text_generation_supports_tools { + extras.push("incl. tools"); + } else { + extras.push("no tools"); + } + + purpose_line = format!("{} ({})", purpose_line, extras.join(", ")); } capabilities.push(purpose_line);