Allow for prompt caching to work by using baibot_conversation_start_time_utc instead of baibot_now_utc

This patch introduces a new `baibot_conversation_start_time_utc`
variable which indicates the time the conversation got started.

Using `baibot_now_utc` is still possible, but given that the current
time is a moving target, its use is in conflict with prompt caching.

Because the new `baibot_conversation_start_time_utc` prompt variable
is a more reasonable default, we're now using it in all sample configs.
This commit is contained in:
Slavi Pantaleev
2024-10-03 11:48:14 +03:00
parent db9422740c
commit 85e66406dc
25 changed files with 159 additions and 25 deletions

View File

@@ -77,11 +77,14 @@ Prompts may contain the following **placeholder variables** which will be replac
|---------------------------|-------------|---------|
| `{{ baibot_name }}` | Name of the bot as configured in the `user.name` field in the [Static configuration](./README.md#static-configuration) | `Baibot` |
| `{{ baibot_model_id }}` | Text-Generation model ID as configured in the [🤖 agent](../agents.md)'s configuration | `gpt-4o` |
| `{{ baibot_now_utc }}` | Current date and time in UTC | `2024-09-20 (Friday), 14:26:42 UTC` |
| `{{ baibot_now_utc }}` | Current date and time in UTC (⚠️ usage may break prompt caching - see below) | `2024-09-20 (Friday), 14:26:42 UTC` |
| `{{ baibot_conversation_start_time_utc }}` | The date and time in UTC that the conversation started | `2024-09-20 (Friday), 14:26:42 UTC` |
💡 `{{ baibot_now_utc }}` changes as time goes on, which prevents [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) from working. It's better to use `{{ baibot_conversation_start_time_utc }}` in prompts, as its value doesn't change yet still orients the bot to the current date/time.
Here's a prompt that combines some of the above variables:
> You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
> You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
### 🌡️ Temperature Override

View File

@@ -2,7 +2,7 @@ base_url: https://api.anthropic.com/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: claude-3-5-sonnet-20240620
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 8192
max_context_tokens: 204800

View File

@@ -2,7 +2,7 @@ base_url: https://api.groq.com/openai/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: llama3-70b-8192
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 4096
max_context_tokens: 131072

View File

@@ -2,7 +2,7 @@ base_url: http://my-localai-self-hosted-service:8080/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: gpt-4
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 4096
max_context_tokens: 128000

View File

@@ -2,7 +2,7 @@ base_url: https://api.mistral.ai/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: mistral-large-latest
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 4096
max_context_tokens: 128000

View File

@@ -2,7 +2,7 @@ base_url: http://my-ollama-self-hosted-service:11434/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: gemma2:2b
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 4096
max_context_tokens: 128000

View File

@@ -2,7 +2,7 @@ base_url: ''
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: some-model
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 4096
max_context_tokens: 128000

View File

@@ -2,7 +2,7 @@ base_url: https://api.openai.com/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: gpt-4o
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 16384
max_context_tokens: 128000

View File

@@ -2,7 +2,7 @@ base_url: https://openrouter.ai/api/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: mattshumer/reflection-70b:free
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 2048
max_context_tokens: 8192

View File

@@ -2,7 +2,7 @@ base_url: https://api.together.xyz/v1
api_key: YOUR_API_KEY_HERE
text_generation:
model_id: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
temperature: 1.0
max_response_tokens: 2048
max_context_tokens: 8192