From 1f2c65d2e633a8325f6840a374c391632170dd4a Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Wed, 18 Feb 2026 05:28:32 +0200 Subject: [PATCH] Refactor dev services to support homeserver choice (Continuwuity or Synapse) The dev environment previously hardcoded Synapse (bundled with Postgres and Element Web) in a monolithic etc/services/core/ directory. With Continuwuity now available as a lighter alternative (no external DB), this refactors the service layout so developers choose their homeserver once and everything derives from that choice. Continuwuity is the new default for its smaller footprint. Key changes: - Break etc/services/core/ into etc/services/synapse/ and etc/services/element-web/, each with their own compose.yml - Add `homeserver` variable in justfile (reads var/homeserver, defaults to continuwuity) - Add `homeserver-init` recipe to persist the choice - Use placeholders (__HOMESERVER_SERVER_NAME__, __HOMESERVER_URL__, __HOMESERVER_CLIENT_URL__) in config templates, resolved at prepare time based on the chosen homeserver - Make services-start/stop/prepare/tail-logs delegate to the chosen homeserver's recipes + element-web - Make users-prepare delegate to {homeserver}-users-prepare - Update docs/development.md for the new homeserver choice flow Co-Authored-By: Claude Opus 4.6 --- docs/development.md | 67 ++++++--- etc/app/config.yml.dist | 8 +- etc/services/element-web/compose.yml | 21 +++ .../config.json.dist} | 2 +- etc/services/{core => synapse}/compose.yml | 18 +-- .../{core => }/synapse/config/homeserver.yaml | 0 .../synapse.127.0.0.1.nip.io.log.config | 0 .../synapse.127.0.0.1.nip.io.signing.key | 0 justfile | 132 +++++++++++++++--- 9 files changed, 183 insertions(+), 65 deletions(-) create mode 100644 etc/services/element-web/compose.yml rename etc/services/{core/element-web/config.json => element-web/config.json.dist} (84%) rename etc/services/{core => synapse}/compose.yml (61%) rename etc/services/{core => }/synapse/config/homeserver.yaml (100%) rename etc/services/{core => }/synapse/config/synapse.127.0.0.1.nip.io.log.config (100%) rename etc/services/{core => }/synapse/config/synapse.127.0.0.1.nip.io.signing.key (100%) diff --git a/docs/development.md b/docs/development.md index 6386954..8a808df 100644 --- a/docs/development.md +++ b/docs/development.md @@ -18,6 +18,27 @@ For local development, we run all dependency services in [🐋 Docker](https://w - (Optional) an API key for some Large Language Model [☁️ provider](./providers.md) (e.g. [OpenAI](./providers.md#openai)), though we recommend using [LocalAI](#localai) or [Ollama](#ollama) for local development +### Choosing a homeserver + +The development environment supports two homeserver implementations: + +- **[Continuwuity](https://continuwuity.org/)** (default) — lightweight, no external database required. Good for most development needs. +- **[Synapse](https://github.com/element-hq/synapse)** — the reference implementation, bundled with Postgres. Use this if you need Synapse-specific behavior. + +To choose a homeserver (optional — defaults to Continuwuity if skipped): + +```sh +just homeserver-init continuwuity # or: just homeserver-init synapse +``` + +The choice is stored in `var/homeserver` and affects all subsequent commands. + +> **Note:** If you switch homeservers after initial setup, you will need to: +> - Delete `var/app/local/` and/or `var/app/container/` (app config and data) +> - Delete `var/services/element-web/` (to regenerate its config) +> - Re-run the prepare and user registration steps + + ### Getting started guide Developing [locally](#running-locally) is possible, but requires a [Rust](https://www.rust-lang.org/) toolchain. @@ -28,11 +49,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency #### Running locally -1. Start the core dependency services (Postgres, Synapse, Element Web): `just services-start` -2. (Only the first time around) Prepare initial app configuration in `var/app/local/config.yml`: `just app-local-prepare` -3. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) -4. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` -5. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): +1. (Optional) Choose a homeserver: `just homeserver-init continuwuity` (or `synapse`). Default is `continuwuity`. +2. Start the homeserver and Element Web: `just services-start` +3. (Only the first time around) Prepare initial app configuration in `var/app/local/config.yml`: `just app-local-prepare` +4. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) +5. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` +6. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): - for [LocalAI](#localai): - Start services: `just localai-start` - Wait a while for LocalAI to start up. It has a lot of models to download. Monitor progress using `just localai-tail-logs` @@ -40,12 +62,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency - for [Ollama](#ollama): - Start services: `just ollama-start` - (Only the first time around) Pull the model configured in `agents.static_definitions` in the configuration file: `just ollama-pull-model gemma2:2b` -6. Start the bot: `just run-locally` -7. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` -8. Create a new room and invite `@baibot:synapse.127.0.0.1.nip.io` -9. When done, stop the bot (`Ctrl` + `C`) -10. Stop the core dependency services: `just services-stop` -11. (Optional) Stop additional services: +7. Start the bot: `just run-locally` +8. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` +9. Create a new room and invite `@baibot:continuwuity.127.0.0.1.nip.io` (or `@baibot:synapse.127.0.0.1.nip.io` if using Synapse) +10. When done, stop the bot (`Ctrl` + `C`) +11. Stop the services: `just services-stop` +12. (Optional) Stop additional services: - for [LocalAI](#localai): `just localai-stop` - for [Ollama](#ollama): `just ollama-stop` @@ -54,11 +76,12 @@ In any case, you will need [🐋 Docker](https://www.docker.com/) as [dependency You can avoid having a [Rust](https://www.rust-lang.org/) toolchain installed locally and build/run this in a container. -1. Start the core dependency services (Postgres, Synapse, Element Web): `just services-start` -2. (Only the first time around) Prepare initial app configuration in `var/app/container/config.yml`: `just app-container-prepare` -3. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) -4. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` -5. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): +1. (Optional) Choose a homeserver: `just homeserver-init continuwuity` (or `synapse`). Default is `continuwuity`. +2. Start the homeserver and Element Web: `just services-start` +3. (Only the first time around) Prepare initial app configuration in `var/app/container/config.yml`: `just app-container-prepare` +4. (Only the first time around) [Prepare your configuration file](#prepare-your-configuration-file) +5. (Only the first time around) Prepare initial default Matrix user accounts (`admin` and `baibot`): `just users-prepare` +6. (Optional) Start additional services depending on which [agent provider you've chosen](#choosing-an-agent-provider): - for [LocalAI](#localai): - Start services: `just localai-start` - Wait a while for LocalAI to start up. It has a lot of models to download. Monitor progress using `just localai-tail-logs` @@ -66,12 +89,12 @@ You can avoid having a [Rust](https://www.rust-lang.org/) toolchain installed lo - for [Ollama](#ollama): - Start services: `just ollama-start` - (Only the first time around) Pull the model configured in `agents.static_definitions` in the configuration file: `just ollama-pull-model gemma2:2b` -6. Start the bot: `just run-in-container` -7. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` -8. Create a new room and invite `@baibot:synapse.127.0.0.1.nip.io` -9. When done, stop the bot (`Ctrl` + `C`) -10. Stop the dependency services: `just services-stop` -11. (Optional) Stop additional services: +7. Start the bot: `just run-in-container` +8. Go to http://element.127.0.0.1.nip.io:42025/ and login with `admin` / `admin` +9. Create a new room and invite `@baibot:continuwuity.127.0.0.1.nip.io` (or `@baibot:synapse.127.0.0.1.nip.io` if using Synapse) +10. When done, stop the bot (`Ctrl` + `C`) +11. Stop the services: `just services-stop` +12. (Optional) Stop additional services: - for [LocalAI](#localai): `just localai-stop` - for [Ollama](#ollama): `just ollama-stop` diff --git a/etc/app/config.yml.dist b/etc/app/config.yml.dist index 0122348..326dcb1 100644 --- a/etc/app/config.yml.dist +++ b/etc/app/config.yml.dist @@ -1,7 +1,7 @@ homeserver: # The canonical homeserver domain name - server_name: synapse.127.0.0.1.nip.io - url: http://synapse.127.0.0.1.nip.io:42020 + server_name: __HOMESERVER_SERVER_NAME__ + url: __HOMESERVER_URL__ user: mxid_localpart: baibot @@ -45,7 +45,7 @@ room: access: # Space-separated list of MXID patterns which specify who is an admin. admin_patterns: - - "@admin:synapse.127.0.0.1.nip.io" + - "@admin:__HOMESERVER_SERVER_NAME__" persistence: # This is unset here, because we expect the configuration to come from an environment variable (BAIBOT_PERSISTENCE_DATA_DIR_PATH). @@ -157,7 +157,7 @@ initial_global_config: # Space-separated list of MXID patterns which specify who can use the bot. # By default, we let anyone on the homeserver use the bot. user_patterns: - - "@*:synapse.127.0.0.1.nip.io" + - "@*:__HOMESERVER_SERVER_NAME__" # Controls logging. # diff --git a/etc/services/element-web/compose.yml b/etc/services/element-web/compose.yml new file mode 100644 index 0000000..d9c5263 --- /dev/null +++ b/etc/services/element-web/compose.yml @@ -0,0 +1,21 @@ +services: + element-web: + image: ghcr.io/element-hq/element-web:v1.12.9 + user: "${UID}:${GID}" + restart: unless-stopped + environment: + ELEMENT_WEB_PORT: 8080 + ports: + - "${SERVICE_ELEMENT_WEB_BIND_PORT_HTTP}:8080" + volumes: + - ./element-web/config.json:/app/config.json:ro + tmpfs: + - /var/cache/nginx:rw,mode=777 + - /var/run:rw,mode=777 + - /tmp/element-web-config:rw,mode=777 + - /etc/nginx/conf.d:rw,mode=777 + +networks: + default: + name: ${NETWORK_NAME} + external: true diff --git a/etc/services/core/element-web/config.json b/etc/services/element-web/config.json.dist similarity index 84% rename from etc/services/core/element-web/config.json rename to etc/services/element-web/config.json.dist index 71bae7b..6312bb7 100644 --- a/etc/services/core/element-web/config.json +++ b/etc/services/element-web/config.json.dist @@ -1,5 +1,5 @@ { - "default_hs_url": "http://synapse.127.0.0.1.nip.io:42020", + "default_hs_url": "__HOMESERVER_CLIENT_URL__", "default_is_url": "https://vector.im", "integrations_ui_url": "https://scalar.vector.im/", "integrations_rest_url": "https://scalar.vector.im/api", diff --git a/etc/services/core/compose.yml b/etc/services/synapse/compose.yml similarity index 61% rename from etc/services/core/compose.yml rename to etc/services/synapse/compose.yml index 8f62ad5..82fc025 100644 --- a/etc/services/core/compose.yml +++ b/etc/services/synapse/compose.yml @@ -23,25 +23,9 @@ services: - "${SERVICE_SYNAPSE_BIND_PORT_CLIENT_API}:8008" - "${SERVICE_SYNAPSE_BIND_PORT_FEDERATION_API}:8008" volumes: - - ../../etc/services/core/synapse/config:/config:ro + - ../../etc/services/synapse/config:/config:ro - ./synapse/media-store:/media-store - element-web: - image: ghcr.io/element-hq/element-web:v1.12.9 - user: "${UID}:${GID}" - restart: unless-stopped - environment: - ELEMENT_WEB_PORT: 8080 - ports: - - "${SERVICE_ELEMENT_WEB_BIND_PORT_HTTP}:8080" - volumes: - - ../../etc/services/core/element-web/config.json:/app/config.json:ro - tmpfs: - - /var/cache/nginx:rw,mode=777 - - /var/run:rw,mode=777 - - /tmp/element-web-config:rw,mode=777 - - /etc/nginx/conf.d:rw,mode=777 - networks: default: name: ${NETWORK_NAME} diff --git a/etc/services/core/synapse/config/homeserver.yaml b/etc/services/synapse/config/homeserver.yaml similarity index 100% rename from etc/services/core/synapse/config/homeserver.yaml rename to etc/services/synapse/config/homeserver.yaml diff --git a/etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.log.config b/etc/services/synapse/config/synapse.127.0.0.1.nip.io.log.config similarity index 100% rename from etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.log.config rename to etc/services/synapse/config/synapse.127.0.0.1.nip.io.log.config diff --git a/etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.signing.key b/etc/services/synapse/config/synapse.127.0.0.1.nip.io.signing.key similarity index 100% rename from etc/services/core/synapse/config/synapse.127.0.0.1.nip.io.signing.key rename to etc/services/synapse/config/synapse.127.0.0.1.nip.io.signing.key diff --git a/justfile b/justfile index e5aceaa..e41f3dc 100644 --- a/justfile +++ b/justfile @@ -7,6 +7,8 @@ admin_password := "admin" bot_username := "baibot" bot_password := "baibot" +homeserver := `cat var/homeserver 2>/dev/null || echo continuwuity` + mise_data_dir := env("MISE_DATA_DIR", justfile_directory() / "var/mise") mise_trusted_config_paths := justfile_directory() / "mise.toml" @@ -14,6 +16,19 @@ mise_trusted_config_paths := justfile_directory() / "mise.toml" default: @just --list --justfile {{ justfile() }} +# Selects which homeserver implementation to use (continuwuity or synapse) +homeserver-init value: + #!/bin/sh + mkdir -p {{ justfile_directory() }}/var + echo {{ value }} > {{ justfile_directory() }}/var/homeserver + echo "" + echo "⚠️ If you had already prepared your app configuration (var/app/local/config.yml or var/app/container/config.yml)," + echo " you will need to update it manually or delete it and re-run the prepare step." + echo " You should also delete var/app/local/data and/or var/app/container/data," + echo " as old application state is not compatible across homeserver implementations." + echo "" + echo "⚠️ If Element Web was already prepared, delete var/services/element-web/ to regenerate its config." + # Builds and runs a development binary run-locally *extra_args: app-local-prepare RUST_BACKTRACE=1 \ @@ -73,9 +88,13 @@ docker-compose services_type *extra_args: -p {{ project_name }}-{{ services_type }} \ {{ extra_args }} -# Runs a docker-compose command against the core services -docker-compose-core *extra_args: - just docker-compose core {{ extra_args }} +# Runs a docker-compose command against the synapse services +docker-compose-synapse *extra_args: + just docker-compose synapse {{ extra_args }} + +# Runs a docker-compose command against the element-web services +docker-compose-element-web *extra_args: + just docker-compose element-web {{ extra_args }} # Runs a docker-compose command against the localai services docker-compose-localai *extra_args: @@ -89,17 +108,48 @@ docker-compose-ollama *extra_args: docker-compose-continuwuity *extra_args: just docker-compose continuwuity {{ extra_args }} -# Runs all core dependency components (in the background) -services-start: services-prepare (docker-compose-core "up" "-d") +# Runs the homeserver and Element Web (in the background) +services-start: services-prepare + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-start + just -f {{ justfile_directory() }}/justfile element-web-start -# Stops all core dependency components -services-stop: (docker-compose-core "down") +# Stops Element Web and the homeserver +services-stop: + just -f {{ justfile_directory() }}/justfile element-web-stop + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-stop -# Tails the logs for all running core services -services-tail-logs: (docker-compose-core "logs" "-f") +# Tails the logs for the homeserver and Element Web +services-tail-logs: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-tail-logs -# Prepares the core services for running -services-prepare: _prepare-var-services-env _prepare-var-services-postgres _prepare-var-services-synapse _prepare-container-network +# Prepares the homeserver and Element Web for running +services-prepare: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-prepare + just -f {{ justfile_directory() }}/justfile element-web-prepare + +# Runs Synapse (in the background) +synapse-start: synapse-prepare (docker-compose-synapse "up" "-d") + +# Stops Synapse +synapse-stop: (docker-compose-synapse "down") + +# Tails the logs for Synapse +synapse-tail-logs: (docker-compose-synapse "logs" "-f") + +# Prepares Synapse for running +synapse-prepare: _prepare-var-services-env _prepare-var-services-postgres _prepare-var-services-synapse _prepare-container-network + +# Runs Element Web (in the background) +element-web-start: element-web-prepare (docker-compose-element-web "up" "-d") + +# Stops Element Web +element-web-stop: (docker-compose-element-web "down") + +# Tails the logs for Element Web +element-web-tail-logs: (docker-compose-element-web "logs" "-f") + +# Prepares Element Web for running +element-web-prepare: _prepare-var-services-env _prepare-var-services-element-web _prepare-container-network # Runs LocalAI (in the background) localai-start: localai-prepare (docker-compose-localai "up" "-d") @@ -142,7 +192,7 @@ continuwuity-register-user username password: {{ justfile_directory() }}/etc/services/continuwuity/register-user.sh {{ justfile_directory() }}/var/services/env {{ username }} {{ password }} # Prepares the Continuwuity user accounts -continuwuity-users-prepare: +continuwuity-users-prepare: continuwuity-prepare just -f {{ justfile_directory() }}/justfile continuwuity-register-user "{{ admin_username }}" "{{ admin_password }}" just -f {{ justfile_directory() }}/justfile continuwuity-register-user "{{ bot_username }}" "{{ bot_password }}" @@ -159,16 +209,20 @@ app-local-prepare: _prepare-var-app-local-config_yml _prepare-var-app-local-data app-container-prepare: _prepare-var-app-container-config_yml _prepare-var-app-container-data # Prepares the user accounts -users-prepare: services-prepare +users-prepare: + just -f {{ justfile_directory() }}/justfile {{ homeserver }}-users-prepare + +# Prepares the Synapse user accounts +synapse-users-prepare: synapse-prepare just -f {{ justfile_directory() }}/justfile synapse-register-admin-user "{{ admin_username }}" "{{ admin_password }}" just -f {{ justfile_directory() }}/justfile synapse-register-regular-user "{{ bot_username }}" "{{ bot_password }}" # Starts a Postgres CLI (psql) -postgres-cli: services-prepare (docker-compose-core "exec" "postgres" "/bin/sh" "-c" "'PGUSER=synapse PGPASSWORD=synapse-password PGDATABASE=homeserver psql -h postgres'") +postgres-cli: synapse-prepare (docker-compose-synapse "exec" "postgres" "/bin/sh" "-c" "'PGUSER=synapse PGPASSWORD=synapse-password PGDATABASE=homeserver psql -h postgres'") -# Creates an administrator user -synapse-register-admin-user username password: services-prepare - just -f {{ justfile_directory() }}/justfile docker-compose-core \ +# Creates an administrator user on Synapse +synapse-register-admin-user username password: synapse-prepare + just -f {{ justfile_directory() }}/justfile docker-compose-synapse \ exec synapse \ register_new_matrix_user \ --admin \ @@ -177,9 +231,9 @@ synapse-register-admin-user username password: services-prepare -c /config/homeserver.yaml \ http://localhost:8008 -# Create a regular user -synapse-register-regular-user username password: services-prepare - just -f {{ justfile_directory() }}/justfile docker-compose-core \ +# Creates a regular user on Synapse +synapse-register-regular-user username password: synapse-prepare + just -f {{ justfile_directory() }}/justfile docker-compose-synapse \ exec synapse \ register_new_matrix_user \ --no-admin \ @@ -259,6 +313,22 @@ _prepare-var-services-synapse: mkdir -p var/services/synapse/media-store fi +_prepare-var-services-element-web: + #!/bin/sh + cd {{ justfile_directory() }}; + + if [ ! -f var/services/element-web/config.json ]; then + mkdir -p var/services/element-web + cp {{ justfile_directory() }}/etc/services/element-web/config.json.dist var/services/element-web/config.json + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's|__HOMESERVER_CLIENT_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/services/element-web/config.json + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's|__HOMESERVER_CLIENT_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/services/element-web/config.json + fi + fi + _prepare-var-services-ollama: #!/bin/sh cd {{ justfile_directory() }}; @@ -298,6 +368,15 @@ _prepare-var-app-local-config_yml: if [ ! -f var/app/local/config.yml ]; then mkdir -p var/app/local cp {{ justfile_directory() }}/etc/app/config.yml.dist var/app/local/config.yml + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/continuwuity.127.0.0.1.nip.io/g' var/app/local/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/app/local/config.yml + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/synapse.127.0.0.1.nip.io/g' var/app/local/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/app/local/config.yml + fi fi _prepare-var-app-local-data: @@ -315,7 +394,18 @@ _prepare-var-app-container-config_yml: if [ ! -f var/app/container/config.yml ]; then mkdir -p var/app/container cp {{ justfile_directory() }}/etc/app/config.yml.dist var/app/container/config.yml - sed --in-place 's/synapse.127.0.0.1.nip.io:42020/synapse:8008/g' var/app/container/config.yml + + homeserver="{{ homeserver }}" + if [ "$homeserver" = "continuwuity" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/continuwuity.127.0.0.1.nip.io/g' var/app/container/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://continuwuity.127.0.0.1.nip.io:42030|g' var/app/container/config.yml + sed --in-place 's/continuwuity.127.0.0.1.nip.io:42030/continuwuity:6167/g' var/app/container/config.yml + elif [ "$homeserver" = "synapse" ]; then + sed --in-place 's/__HOMESERVER_SERVER_NAME__/synapse.127.0.0.1.nip.io/g' var/app/container/config.yml + sed --in-place 's|__HOMESERVER_URL__|http://synapse.127.0.0.1.nip.io:42020|g' var/app/container/config.yml + sed --in-place 's/synapse.127.0.0.1.nip.io:42020/synapse:8008/g' var/app/container/config.yml + fi + sed --in-place 's/127.0.0.1:42026/ollama:11434/g' var/app/container/config.yml sed --in-place 's/127.0.0.1:42027/localai:8080/g' var/app/container/config.yml fi