Sticker generation was failing when using newer GPT image models
(gpt-image-1, gpt-image-1-mini, gpt-image-1.5). The issue occurred
because stickers requested 256x256 size, but these models only support
1024x1024, 1536x1024, 1024x1536, and auto.
To reproduce, send `!bai sticker Something` to an agent configured
with a GPT image model. The error was:
invalid_request_error: Invalid value: '256x256'. Supported values
are: '1024x1024', '1024x1536', '1536x1024', and 'auto'. (param: size)
(code: invalid_value)
The fix replaces the hardcoded 256x256 size override with a
`smallest_size_possible` flag, letting each provider determine the
appropriate sticker size based on the model being used.
The `openai_compat` provider still defaults to requesting 256x256 in all cases
(regardless of model name).
This is a huge patch which does some major refactoring like:
- renaming "Image Generation" to "Image Creation" in most places,
to better match its new command (`!bai image create`)
- relocating image creation command (`!bai image` -> `!bai image create`),
so it wouldn't conflict with the new image editing command (`!bai image edit`)
- introducing a new image editing command (`!bai image edit`), which
is meant to work only with the OpenAI provider, but doesn't fully work yet
due to https://github.com/64bit/async-openai/issues/364, though a next patch will fix it
- adding support for reading images off of Matrix conversations and forwarding them to
text conversations. Works for OpenAI, but not for Anthropic yet
(requires custom patches) and not for OpenAI-Compat (no support for
images there)
- relocating some utils around (base64, mime)
What previously was a generic error message like:
> ⚠️ Error: An error occurred while processing your message. Please try again.
.. now becomes a much more helpful error message like:
> ⚠️ Error: There was a problem performing image-generation via the room-local/my-openai-agent agent:
>
> invalid_request_error: Invalid value: 'standard'. Supported values are: 'low', 'medium', 'high', and 'auto'. (param: quality) (code: invalid_value)
Related to https://github.com/etkecc/baibot/issues/40
When speech-to-text/flow-type = `only_transcribe`, the bot will now send
text messages by default, not notices.
While notice messages may be less desirable with other bots in the room,
it's probably a better default for most people who enable "transcribe-only" mode.
This is an improvement related to https://github.com/etkecc/baibot/issues/14
Various clients (including newer versions of Element Web), do not like
it when the `body` field of the attachment is not a file name.
For images, a preview may not be shown and downloading the attachment
may suggest that the whole long text is used as a filename (which is odd).
There is value (improved accessibility, etc.)
in adding better descriptions (especially to generated images),
but given that it's currently problematic, I'm getting rid of it.
It's better and safer if we stick to using filenames.
The previous approach (implemented in dd1dd78312) was simple
(send typing notices for as long as the "controller" is running),
but this proved to be overly simplistic and unable to handle edge-cases:
- in multi-user rooms (or rooms with a prefix requirement), the bot
used to send a typing notice while "working", but its work consisted
of ignoring the message. So it then sent a "not typing" notice.
This is wasteful and otherwise problematic - certain clients (like nheko)
do not handle this "race" well.
- certain reactions (anything other than 🗣️ right now) are meant to be
ignored. There's no point in doing the same "typing / not typing"
dance
- there are other instances where the bot may do work, but doesn't (due
to configuration or lack of capabilities)
This new more fine-grained implementation of typing notices aims to:
- only send a typing notice if actual "slow work" will be done
- avoid stopping & restarting typing notices (wasteful) if a chain of work is to
be performed (processing voice messages and doing speech-to-text +
text-generation + ...). Rather, maintaining typing notice sending
throughout
Previously, the bot only had rudimentary typing notification support.
It used to send a single notification when starting a long task
and did not bother with notifications anymore.
By default matrix-rust-sdk gives these notifications a validity of 4
seconds, so it would expire shortly. If the bot takes longer to respond,
you'd see the typing notification expire and wonder if a response is
coming.
Another edge case is the bot sending an answer quicker and the typing
notice still being on. Some clients (like element-web) seem to hide the
typing notice when a new message comes, so they don't experience this as
problematic.
The reworked typing notification system should be robust:
- typing notices are sent continuously, until the bot finishes doing
work
- if the bot is performing multiple actions in a room (even for
different people), typing notices would continue to be sent until the
bot becomes idle
- as soon as the bot becomes idle, a "not typing anymore" notice is sent
to clear the state
