Production¶

The reference production setup is several identical servers sharing one Redis, deployed by GitHub Actions with rolling updates and automatic rollback. Requests reach the servers either through a load balancer of your own, or straight from Cloudflare (proxied DNS records, one per server), with a Caddy container on each server terminating TLS (see Cloudflare without a load balancer).

flowchart LR
    C[Clients] --> CF[Cloudflare / CDN]
    CF --> LB[nginx load balancer]
    LB --> A1[server 1<br/>telegram-me]
    LB --> A2[server 2<br/>telegram-me]
    LB --> A3[server 3<br/>telegram-me]
    A1 & A2 & A3 --> R[(Redis / Valkey)]
    A1 & A2 & A3 --> T[t.me]

Server preparation¶

Run deploy/bootstrap-server.sh once on each fresh Ubuntu server, as the user GitHub Actions logs in as:

ssh ubuntu@<server> 'bash -s' < deploy/bootstrap-server.sh

It is safe to run again. It:

• installs Docker Engine and the compose plugin from Docker's repository;
• adds the user to the docker group;
• adds 2 GiB of swap when there is none (SWAP_MB=0 skips it). This helps small machines such as 1 GB cloud instances;
• opens port 443 in the host firewall of Oracle Cloud images, which reject everything but SSH. The cloud security list or security group still decides who may connect.

Behind a load balancer of your own, allow it to reach port 8000 and nothing else from outside, for example with UFW:

sudo ufw default deny incoming
sudo ufw allow ssh
sudo ufw allow from 10.0.0.0/24 to any port 8000 proto tcp
sudo ufw enable

Nothing else needs to be installed: the workflow copies the compose file, the deploy script and the .env on every deployment.

How a deployment works¶

sequenceDiagram
    participant GH as GitHub Actions
    participant S1 as Server 1 (canary)
    participant SN as Servers 2…N
    GH->>GH: CI and Guards: lint, types, tests, contract, image, integration, deploy scripts
    GH->>GH: build & push linux/amd64 + linux/arm64
    GH->>S1: remote-deploy.sh <image:sha-…>
    S1->>S1: compose up --wait (health check), smoke test inside the container
    alt healthy and smoke test passed
        S1-->>GH: ok, remember image
    else otherwise
        S1->>S1: start previous image again
        S1-->>GH: failure → rollout stops, nothing else changed
    end
    GH->>GH: CANARY_SOAK seconds of real traffic on server 1
    GH->>S1: remote-deploy.sh --verify (same image, healthy, no restart, smoke test)
    loop each other server, one at a time
        GH->>SN: remote-deploy.sh <image:sha-…> (same checks)
    end
    GH->>GH: smoke test through the public URL
    Note over GH,SN: a failure after server 1 rolls back every server that took the release

• Pushing to main runs CI and Guards, builds the image (latest and sha-… tags) and deploys sha-… to every server listed in SERVERS_IPS, one after another (deploy/rollout.sh).
• deploy/remote-deploy.sh waits until the new container is healthy, then runs the smoke test (scripts/smoke_matrix.py, status codes and content of every endpoint, inside the container against live t.me, one retry). The health check alone only proves that the process answers /healthz; the smoke test proves that it serves pages through this server's configuration and Redis. When either fails, it restarts the previously deployed image (recorded in ~/telegram-me/.deployed-image) and fails.
• Server 1 is the canary: after it took the release it serves real traffic for CANARY_SOAK seconds (default 60) and is verified again: it must still run the image, be healthy, not have restarted or been killed for memory, and pass the smoke test. Only then are the other servers deployed.
• Last, the smoke test runs through the public URL (PUBLIC_URL, or https://PUBLIC_HOST), that is through Cloudflare and Caddy, with the CF_BYPASS_KEY header.
• When any of these checks fails after a server took the release, every server that took it is rolled back (remote-deploy.sh --rollback), so the fleet never stays split between two releases.
• The first 2.0 deployment replaces the 1.x container (named telegram-me, started with docker run). If 2.0 does not become healthy, the 1.x container is started again from its image and ~/env.app.
• Pushing a tag v2.0.0 additionally publishes 2.0.0, 2.0, 2 tags and a GitHub release with the notes from CHANGELOG.md.

Configuration¶

Secrets¶

Variables¶

Repository variables override the production .env written on every deployment:

Use the production environment in the repository settings to add required reviewers or restrict which branches may deploy.

Cloudflare without a load balancer¶

With proxied DNS records pointing at the servers themselves, Cloudflare connects to port 443 of each one. Set the PUBLIC_HOST variable, and every deployment then also runs the proxy service of compose.prod.yaml: Caddy (deploy/Caddyfile).

• Caddy terminates TLS and forwards to the API, which then listens on 127.0.0.1:8000 only.
• The client address comes from CF-Connecting-IP, trusted only from Cloudflare's ranges. Caddy passes it on in X-Forwarded-For, so rate limiting and logs see real clients.
• Certificate. Create an Origin CA certificate in the Cloudflare dashboard (SSL/TLS → Origin Server) and store it as the TLS_CERT and TLS_KEY secrets. Then use SSL mode Full (strict). Without them Caddy serves a self-signed certificate, which works with SSL mode Full only.
• Let only Cloudflare's ranges reach port 443 in the cloud firewall (security list or group), and nothing reach port 8000.

API clients are programs, not browsers. Cloudflare's bot protection (Bot Fight Mode, the "I'm Under Attack" level, Managed Challenge rules) answers them with a challenge page (403) instead of JSON, and that includes the deployment's public health check. Keep it off when programs are meant to use the API. When it stays on (a demo, say), add a WAF custom rule that skips it for requests whose header matches a long random key, for example any(http.request.headers["x-bypass-key"][*] eq "<key>"). Store the key as the CF_BYPASS_KEY secret for the deployment's public smoke test and health check, and pass it to the smoke test by hand with --header "x-bypass-key:$KEY".

Cloudflare picks one of the DNS records per connection. While one server restarts during a rolling deployment, some requests to it can fail for a few seconds.

Rolling back¶

Run the Deploy workflow manually (Actions → Deploy → Run workflow) with the tag to go back to, for example sha-0123456789ab (every build is kept in the registry). The same rolling procedure is used, with the same checks; uncheck smoke only while t.me itself is down, when no release can pass.

On a single server you can also roll back by hand, to the image deployed before the last one or to any other:

bash ~/telegram-me/remote-deploy.sh --rollback
cd ~/telegram-me
APP_IMAGE=docker.io/koval01/telegram-me:sha-0123456789ab \
  docker compose -f compose.prod.yaml up -d --wait

Standalone feed worker¶

To keep the feed refresher off the API processes, run the Deploy workflow with with_worker checked: every server then starts the worker service and the API runs with FEED_REFRESH_MODE=worker. Leader election in Redis guarantees a single active refresher across all servers either way.

Operations¶

cd ~/telegram-me
docker compose -f compose.prod.yaml ps
docker compose -f compose.prod.yaml logs -f --since 10m api
cat .deployed-image

Logs are JSON lines in production (LOG_FORMAT=json), rotated at 5 × 50 MB.