diff --git a/docs/development.md b/docs/development.md index 6386954..8a808df 100644 --- a/docs/development.md +++ b/docs/development.md @@ -18,6 +18,27 @@ For local development, we run all dependency services in [🐋 Docker](https://w - (Optional) an API key for some Large Language Model [☁️ provider](./providers.md) (e.g. [OpenAI](./providers.md#openai)), though we recommend using [LocalAI](#localai) or [Ollama](#ollama) for local development +### Choosing a homeserver + +The development environment supports two homeserver implementations: + +- **[Continuwuity](https://continuwuity.org/)** (default) — lightweight, no external database required. Good for most development needs. +- **[Synapse](https://github.com/element-hq/synapse)** — the reference implementation, bundled with Postgres. Use this if you need Synapse-specific behavior. + +To choose a homeserver (optional — defaults to Continuwuity if skipped): + +```sh +just homeserver-init continuwuity # or: just homeserver-init synapse +``` + +The choice is stored in `var/homeserver` and affects all subsequent commands. + +> **Note:** If you switch homeservers after initial setup, you will need to: +> - Delete `var/app/local/` and/or `var/app/container/` (app config and data) +> - Delete `var/services/element-web/` (to regenerate its config) +> - Re-run the prepare and user registration steps + + ### Getting started guide Developing [locally](#running-locally) is possible, but requires a [Rust](https://www.rust-lang.org/) toolchain. @@ -28,11 +49,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency #### Running locally -1. Start the core dependency services (Postgres, Synapse, Element Web): `just services-start` -2. (Only the first time around) Prepare initial app configuration in `var/app/local/config.yml`: `just app-local-prepare` -3. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) -4. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` -5. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): +1. (Optional) Choose a homeserver: `just homeserver-init continuwuity` (or `synapse`). Default is `continuwuity`. +2. Start the homeserver and Element Web: `just services-start` +3. (Only the first time around) Prepare initial app configuration in `var/app/local/config.yml`: `just app-local-prepare` +4. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) +5. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` +6. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): - for [LocalAI](#localai): - Start services: `just localai-start` - Wait a while for LocalAI to start up. It has a lot of models to download. Monitor progress using `just localai-tail-logs` @@ -40,12 +62,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency - for [Ollama](#ollama): - Start services: `just ollama-start` - (Only the first time around) Pull the model configured in `agents.static_definitions` in the configuration file: `just ollama-pull-model gemma2:2b` -6. Start the bot: `just run-locally` -7. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` -8. Create a new room and invite `@baibot:synapse.127.0.0.1.nip.io` -9. When done, stop the bot (`Ctrl` + `C`) -10. Stop the core dependency services: `just services-stop` -11. (Optional) Stop additional services: +7. Start the bot: `just run-locally` +8. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` +9. Create a new room and invite `@baibot:continuwuity.127.0.0.1.nip.io` (or `@baibot:synapse.127.0.0.1.nip.io` if using Synapse) +10. When done, stop the bot (`Ctrl` + `C`) +11. Stop the services: `just services-stop` +12. (Optional) Stop additional services: - for [LocalAI](#localai): `just localai-stop` - for [Ollama](#ollama): `just ollama-stop` @@ -54,11 +76,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency You can avoid having a [Rust](https://www.rust-lang.org/) toolchain installed locally and build/run this in a container. -1. Start the core dependency services (Postgres, Synapse, Element Web): `just services-start` -2. (Only the first time around) Prepare initial app configuration in `var/app/container/config.yml`: `just app-container-prepare` -3. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) -4. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` -5. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): +1. (Optional) Choose a homeserver: `just homeserver-init continuwuity` (or `synapse`). Default is `continuwuity`. +2. Start the homeserver and Element Web: `just services-start` +3. (Only the first time around) Prepare initial app configuration in `var/app/container/config.yml`: `just app-container-prepare` +4. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) +5. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` +6. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): - for [LocalAI](#localai): - Start services: `just localai-start` - Wait a while for LocalAI to start up. It has a lot of models to download. Monitor progress using `just localai-tail-logs` @@ -66,12 +89,12 @@ You can avoid having a [Rust](https://www.rust-lang.org/) toolchain installed lo - for [Ollama](#ollama): - Start services: `just ollama-start` - (Only the first time around) Pull the model configured in `agents.static_definitions` in the configuration file: `just ollama-pull-model gemma2:2b` -6. Start the bot: `just run-in-container` -7. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` -8. Create a new room and invite `@baibot:synapse.127.0.0.1.nip.io` -9. When done, stop the bot (`Ctrl` + `C`) -10. Stop the dependency services: `just services-stop` -11. (Optional) Stop additional services: +7. Start the bot: `just run-in-container` +8. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` +9. Create a new room and invite `@baibot:continuwuity.127.0.0.1.nip.io` (or `@baibot:synapse.127.0.0.1.nip.io` if using Synapse) +10. When done, stop the bot (`Ctrl` + `C`) +11. Stop the services: `just services-stop` +12. (Optional) Stop additional services: - for [LocalAI](#localai): `just localai-stop` - for [Ollama](#ollama): `just ollama-stop` diff --git a/etc/app/config.yml.dist b/etc/app/config.yml.dist index 0122348..326dcb1 100644 --- a/etc/app/config.yml.dist +++ b/etc/app/config.yml.dist @@ -1,7 +1,7 @@ homeserver: # The canonical homeserver domain name - server_name: synapse.127.0.0.1.nip.io - url: http://synapse.127.0.0.1.nip.io:42020 + server_name: __HOMESERVER_SERVER_NAME__ + url: __HOMESERVER_URL__ user: mxid_localpart: baibot @@ -45,7 +45,7 @@ room: access: # Space-separated list of MXID patterns which specify who is an admin. admin_patterns: - - "@admin:synapse.127.0.0.1.nip.io" + - "@admin:__HOMESERVER_SERVER_NAME__" persistence: # This is unset here, because we expect the configuration to come from an environment variable (BAIBOT_PERSISTENCE_DATA_DIR_PATH). @@ -157,7 +157,7 @@ initial_global_config: # Space-separated list of MXID patterns which specify who can use the bot. # By default, we let anyone on the homeserver use the bot. user_patterns: - - "@*:synapse.127.0.0.1.nip.io" + - "@*:__HOMESERVER_SERVER_NAME__" # Controls logging. # diff --git a/etc/services/element-web/compose.yml b/etc/services/element-web/compose.yml new file mode 100644 index 0000000..d9c5263 --- /dev/null +++ b/etc/services/element-web/compose.yml @@ -0,0 +1,21 @@ +services: + element-web: + image: ghcr.io/element-hq/element-web:v1.12.9 + user: "${UID}:${GID}" + restart: unless-stopped + environment: + ELEMENT_WEB_PORT: 8080 + ports: + - "${SERVICE_ELEMENT_WEB_BIND_PORT_HTTP}:8080" + volumes: + - ./element-web/config.json:/app/config.json:ro + tmpfs: + - /var/cache/nginx:rw,mode=777 + - /var/run:rw,mode=777 + - /tmp/element-web-config:rw,mode=777 + - /etc/nginx/conf.d:rw,mode=777 + +networks: + default: + name: ${NETWORK_NAME} + external: true diff --git a/etc/services/core/element-web/config.json b/etc/services/element-web/config.json.dist similarity index 84% rename from etc/services/core/element-web/config.json rename to etc/services/element-web/config.json.dist index 71bae7b..6312bb7 100644 --- a/etc/services/core/element-web/config.json +++ b/etc/services/element-web/config.json.dist @@ -1,5 +1,5 @@ { - "default_hs_url": "http://synapse.127.0.0.1.nip.io:42020", + "default_hs_url": "__HOMESERVER_CLIENT_URL__", "default_is_url": "https://vector.im", "integrations_ui_url": "https://scalar.vector.im/", "integrations_rest_url": "https://scalar.vector.im/api", diff --git a/etc/services/core/compose.yml b/etc/services/synapse/compose.yml similarity index 61% rename from etc/services/core/compose.yml rename to etc/services/synapse/compose.yml index 8f62ad5..82fc025 100644 --- a/etc/services/core/compose.yml +++ b/etc/services/synapse/compose.yml @@ -23,25 +23,9 @@ services: - "${SERVICE_SYNAPSE_BIND_PORT_CLIENT_API}:8008" - "${SERVICE_SYNAPSE_BIND_PORT_FEDERATION_API}:8008" volumes: - - ../../etc/services/core/synapse/config:/config:ro + - ../../etc/services/synapse/config:/config:ro - ./synapse/media-store:/media-store - element-web: - image: ghcr.io/element-hq/element-web:v1.12.9 - user: "${UID}:${GID}" - restart: unless-stopped - environment: - ELEMENT_WEB_PORT: 8080 - ports: - - "${SERVICE_ELEMENT_WEB_BIND_PORT_HTTP}:8080" - volumes: - - ../../etc/services/core/element-web/config.json:/app/config.json:ro - tmpfs: - - /var/cache/nginx:rw,mode=777 - - /var/run:rw,mode=777 - - /tmp/element-web-config:rw,mode=777 - - /etc/nginx/conf.d:rw,mode=777 - networks: default: name: ${NETWORK_NAME} diff --git a/etc/services/core/synapse/config/homeserver.yaml b/etc/services/synapse/config/homeserver.yaml similarity index 100% rename from etc/services/core/synapse/config/homeserver.yaml rename to etc/services/synapse/config/homeserver.yaml diff --git a/etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.log.config b/etc/services/synapse/config/synapse.127.0.0.1.nip.io.log.config similarity index 100% rename from etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.log.config rename to etc/services/synapse/config/synapse.127.0.0.1.nip.io.log.config diff --git a/etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.signing.key b/etc/services/synapse/config/synapse.127.0.0.1.nip.io.signing.key similarity index 100% rename from etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.signing.key rename to etc/services/synapse/config/synapse.127.0.0.1.nip.io.signing.key diff --git a/justfile b/justfile index e5aceaa..e41f3dc 100644 --- a/justfile +++ b/justfile @@ -7,6 +7,8 @@ admin_password := "admin" bot_username := "baibot" bot_password := "baibot" +homeserver := `cat var/homeserver 2>/dev/null || echo continuwuity` + mise_data_dir := env("MISE_DATA_DIR", justfile_directory() / "var/mise") mise_trusted_config_paths := justfile_directory() / "mise.toml" @@ -14,6 +16,19 @@ mise_trusted_config_paths := justfile_directory() / "mise.toml" default: @just --list --justfile {{ justfile() }} +# Selects which homeserver implementation to use (continuwuity or synapse) +homeserver-init value: + #!/bin/sh + mkdir -p {{ justfile_directory() }}/var + echo {{ value }} > {{ justfile_directory() }}/var/homeserver + echo "" + echo "⚠️ If you had already prepared your app configuration (var/app/local/config.yml or var/app/container/config.yml)," + echo " you will need to update it manually or delete it and re-run the prepare step." + echo " You should also delete var/app/local/data and/or var/app/container/data," + echo " as old application state is not compatible across homeserver implementations." + echo "" + echo "⚠️ If Element Web was already prepared, delete var/services/element-web/ to regenerate its config." + # Builds and runs a development binary run-locally *extra_args: app-local-prepare RUST_BACKTRACE=1 \ @@ -73,9 +88,13 @@ docker-compose services_type *extra_args: -p {{ project_name }}-{{ services_type }} \ {{ extra_args }} -# Runs a docker-compose command against the core services -docker-compose-core *extra_args: - just docker-compose core {{ extra_args }} +# Runs a docker-compose command against the synapse services +docker-compose-synapse *extra_args: + just docker-compose synapse {{ extra_args }} + +# Runs a docker-compose command against the element-web services +docker-compose-element-web *extra_args: + just docker-compose element-web {{ extra_args }} # Runs a docker-compose command against the localai services docker-compose-localai *extra_args: @@ -89,17 +108,48 @@ docker-compose-ollama *extra_args: docker-compose-continuwuity *extra_args: just docker-compose continuwuity {{ extra_args }} -# Runs all core dependency components (in the background) -services-start: services-prepare (docker-compose-core "up" "-d") +# Runs the homeserver and Element Web (in the background) +services-start: services-prepare + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-start + just -f {{ justfile_directory() }}/justfile element-web-start -# Stops all core dependency components -services-stop: (docker-compose-core "down") +# Stops Element Web and the homeserver +services-stop: + just -f {{ justfile_directory() }}/justfile element-web-stop + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-stop -# Tails the logs for all running core services -services-tail-logs: (docker-compose-core "logs" "-f") +# Tails the logs for the homeserver and Element Web +services-tail-logs: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-tail-logs -# Prepares the core services for running -services-prepare: _prepare-var-services-env _prepare-var-services-postgres _prepare-var-services-synapse _prepare-container-network +# Prepares the homeserver and Element Web for running +services-prepare: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-prepare + just -f {{ justfile_directory() }}/justfile element-web-prepare + +# Runs Synapse (in the background) +synapse-start: synapse-prepare (docker-compose-synapse "up" "-d") + +# Stops Synapse +synapse-stop: (docker-compose-synapse "down") + +# Tails the logs for Synapse +synapse-tail-logs: (docker-compose-synapse "logs" "-f") + +# Prepares Synapse for running +synapse-prepare: _prepare-var-services-env _prepare-var-services-postgres _prepare-var-services-synapse _prepare-container-network + +# Runs Element Web (in the background) +element-web-start: element-web-prepare (docker-compose-element-web "up" "-d") + +# Stops Element Web +element-web-stop: (docker-compose-element-web "down") + +# Tails the logs for Element Web +element-web-tail-logs: (docker-compose-element-web "logs" "-f") + +# Prepares Element Web for running +element-web-prepare: _prepare-var-services-env _prepare-var-services-element-web _prepare-container-network # Runs LocalAI (in the background) localai-start: localai-prepare (docker-compose-localai "up" "-d") @@ -142,7 +192,7 @@ continuwuity-register-user username password: {{ justfile_directory() }}/etc/services/continuwuity/register-user.sh {{ justfile_directory() }}/var/services/env {{ username }} {{ password }} # Prepares the Continuwuity user accounts -continuwuity-users-prepare: +continuwuity-users-prepare: continuwuity-prepare just -f {{ justfile_directory() }}/justfile continuwuity-register-user "{{ admin_username }}" "{{ admin_password }}" just -f {{ justfile_directory() }}/justfile continuwuity-register-user "{{ bot_username }}" "{{ bot_password }}" @@ -159,16 +209,20 @@ app-local-prepare: _prepare-var-app-local-config_yml _prepare-var-app-local-data app-container-prepare: _prepare-var-app-container-config_yml _prepare-var-app-container-data # Prepares the user accounts -users-prepare: services-prepare +users-prepare: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-users-prepare + +# Prepares the Synapse user accounts +synapse-users-prepare: synapse-prepare just -f {{ justfile_directory() }}/justfile synapse-register-admin-user "{{ admin_username }}" "{{ admin_password }}" just -f {{ justfile_directory() }}/justfile synapse-register-regular-user "{{ bot_username }}" "{{ bot_password }}" # Starts a Postgres CLI (psql) -postgres-cli: services-prepare (docker-compose-core "exec" "postgres" "/bin/sh" "-c" "'PGUSER=synapse PGPASSWORD=synapse-password PGDATABASE=homeserver psql -h postgres'") +postgres-cli: synapse-prepare (docker-compose-synapse "exec" "postgres" "/bin/sh" "-c" "'PGUSER=synapse PGPASSWORD=synapse-password PGDATABASE=homeserver psql -h postgres'") -# Creates an administrator user -synapse-register-admin-user username password: services-prepare - just -f {{ justfile_directory() }}/justfile docker-compose-core \ +# Creates an administrator user on Synapse +synapse-register-admin-user username password: synapse-prepare + just -f {{ justfile_directory() }}/justfile docker-compose-synapse \ exec synapse \ register_new_matrix_user \ --admin \ @@ -177,9 +231,9 @@ synapse-register-admin-user username password: services-prepare -c /config/homeserver.yaml \ http://localhost:8008 -# Create a regular user -synapse-register-regular-user username password: services-prepare - just -f {{ justfile_directory() }}/justfile docker-compose-core \ +# Creates a regular user on Synapse +synapse-register-regular-user username password: synapse-prepare + just -f {{ justfile_directory() }}/justfile docker-compose-synapse \ exec synapse \ register_new_matrix_user \ --no-admin \ @@ -259,6 +313,22 @@ _prepare-var-services-synapse: mkdir -p var/services/synapse/media-store fi +_prepare-var-services-element-web: + #!/bin/sh + cd {{ justfile_directory() }}; + + if [ ! -f var/services/element-web/config.json ]; then + mkdir -p var/services/element-web + cp {{ justfile_directory() }}/etc/services/element-web/config.json.dist var/services/element-web/config.json + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's|__HOMESERVER_CLIENT_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/services/element-web/config.json + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's|__HOMESERVER_CLIENT_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/services/element-web/config.json + fi + fi + _prepare-var-services-ollama: #!/bin/sh cd {{ justfile_directory() }}; @@ -298,6 +368,15 @@ _prepare-var-app-local-config_yml: if [ ! -f var/app/local/config.yml ]; then mkdir -p var/app/local cp {{ justfile_directory() }}/etc/app/config.yml.dist var/app/local/config.yml + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/continuwuity.127.0.0.1.nip.io/g' var/app/local/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/app/local/config.yml + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/synapse.127.0.0.1.nip.io/g' var/app/local/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/app/local/config.yml + fi fi _prepare-var-app-local-data: @@ -315,7 +394,18 @@ _prepare-var-app-container-config_yml: if [ ! -f var/app/container/config.yml ]; then mkdir -p var/app/container cp {{ justfile_directory() }}/etc/app/config.yml.dist var/app/container/config.yml - sed --in-place 's/synapse.127.0.0.1.nip.io:42020/synapse:8008/g' var/app/container/config.yml + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/continuwuity.127.0.0.1.nip.io/g' var/app/container/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/app/container/config.yml + sed --in-place 's/continuwuity.127.0.0.1.nip.io:42030/continuwuity:6167/g' var/app/container/config.yml + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/synapse.127.0.0.1.nip.io/g' var/app/container/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/app/container/config.yml + sed --in-place 's/synapse.127.0.0.1.nip.io:42020/synapse:8008/g' var/app/container/config.yml + fi + sed --in-place 's/127.0.0.1:42026/ollama:11434/g' var/app/container/config.yml sed --in-place 's/127.0.0.1:42027/localai:8080/g' var/app/container/config.yml fi