Initial work on Vision support in text conversations and Image Editing
This is a huge patch which does some major refactoring like: - renaming "Image Generation" to "Image Creation" in most places, to better match its new command (`!bai image create`) - relocating image creation command (`!bai image` -> `!bai image create`), so it wouldn't conflict with the new image editing command (`!bai image edit`) - introducing a new image editing command (`!bai image edit`), which is meant to work only with the OpenAI provider, but doesn't fully work yet due to https://github.com/64bit/async-openai/issues/364, though a next patch will fix it - adding support for reading images off of Matrix conversations and forwarding them to text conversations. Works for OpenAI, but not for Anthropic yet (requires custom patches) and not for OpenAI-Compat (no support for images there) - relocating some utils around (base64, mime)
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
use crate::{agent::AgentPurpose, conversation::llm::Conversation};
|
||||
|
||||
use super::{
|
||||
ImageGenerationParams, SpeechToTextParams, SpeechToTextResult,
|
||||
ImageGenerationParams, ImageEditParams, SpeechToTextParams, SpeechToTextResult,
|
||||
entity::{
|
||||
ImageGenerationResult, PingResult, TextGenerationParams, TextGenerationResult,
|
||||
TextToSpeechParams, TextToSpeechResult,
|
||||
ImageGenerationResult, ImageEditResult, ImageSource, PingResult, TextGenerationParams,
|
||||
TextGenerationResult, TextToSpeechParams, TextToSpeechResult,
|
||||
},
|
||||
};
|
||||
|
||||
@@ -42,6 +42,13 @@ pub trait ControllerTrait {
|
||||
params: ImageGenerationParams,
|
||||
) -> impl std::future::Future<Output = anyhow::Result<ImageGenerationResult>> + Send;
|
||||
|
||||
fn create_image_edit(
|
||||
&self,
|
||||
prompt: &str,
|
||||
images: Vec<ImageSource>,
|
||||
params: ImageEditParams,
|
||||
) -> impl std::future::Future<Output = anyhow::Result<ImageEditResult>> + Send;
|
||||
|
||||
fn text_to_speech(
|
||||
&self,
|
||||
text: &str,
|
||||
@@ -166,6 +173,23 @@ impl ControllerTrait for ControllerType {
|
||||
}
|
||||
}
|
||||
|
||||
async fn create_image_edit(
|
||||
&self,
|
||||
prompt: &str,
|
||||
images: Vec<ImageSource>,
|
||||
params: ImageEditParams,
|
||||
) -> anyhow::Result<ImageEditResult> {
|
||||
match &self {
|
||||
ControllerType::OpenAI(controller) => controller.create_image_edit(prompt, images, params).await,
|
||||
ControllerType::OpenAICompat(controller) => {
|
||||
controller.create_image_edit(prompt, images, params).await
|
||||
}
|
||||
ControllerType::Anthropic(controller) => {
|
||||
controller.create_image_edit(prompt, images, params).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn text_to_speech(
|
||||
&self,
|
||||
text: &str,
|
||||
|
||||
Reference in New Issue
Block a user