PR Preview Deployment — Comprehensive Guide

Preview environments are created on demand by commenting /deploy on a PR. Each environment is a fully isolated Kubernetes namespace on the office Mac Mini running all five services with its own MongoDB snapshot. Teardown is automatic on PR merge/close, manual with /destroy, and stale previews are auto-expired daily.

Architecture Overview
Network Path — GitHub → EC2 → Mac Mini
Components
PR Lifecycle
Branch Gates
Image Build Strategy
Environment Variables
MongoDB Seeding
URL Structure
Helm Chart & Per-Service Overrides
Mac Gateway (nginx TCP proxy)
Infra Auto-Sync to Mac
Webhook Receiver
GitHub Secrets Reference
Auto-Expiry of Stale Previews
Troubleshooting
Key Files Reference

Architecture Overview

┌─────────────────────────────────────────────────────┐
│  Any watched app repo (6 total)                     │
│  PR opened  → welcome comment                       │
│  /deploy    → build + deploy                        │
│  /destroy   → teardown                              │
│  PR closed  → auto teardown                         │
└──────────────────────┬──────────────────────────────┘
                       │ GitHub Webhook (HMAC-signed)
                       ▼
          hooks.pr.artistevolve.io
                       │ (K8s Ingress on Mac Mini)
                       ▼
     ┌─────────────────────────────────┐
     │  webhook-receiver               │
     │  Node.js / Express (K8s pod)   │
     │  namespace: shared              │
     │  ‣ Verify HMAC signature        │
     │  ‣ Check branch gates           │
     │  ‣ Post plan/welcome comment    │
     │  ‣ Dispatch deploy-pr.yml       │
     └─────────────┬───────────────────┘
                   │ workflow_dispatch → ae-infra main
                   ▼
     ┌─────────────────────────────────┐
     │  deploy-pr.yml (GitHub Actions) │
     │                                 │
     │  notify-pending                 │
     │    └─ commit status: pending    │
     │  resolve-refs                   │
     │    └─ pick branch per repo      │
     │  build-* (5 parallel jobs)      │
     │    └─ Docker build + GHCR push  │
     │  deploy                         │
     │    └─ bundle + SCP → EC2:2222   │
     │    └─ SSH exec deploy-pr.sh     │
     │  notify-result                  │
     │    └─ PR comment + commit status│
     └─────────────┬───────────────────┘
                   │ SSH port 2222
                   ▼
     ┌─────────────────────────────────┐
     │  EC2 jump host (13.204.230.168) │
     │  Port 2222 → Mac Mini           │
     │  (TCP relay via Tailscale)      │
     └─────────────┬───────────────────┘
                   │ Tailscale tunnel
                   ▼
     ┌─────────────────────────────────┐
     │  Mac Mini — OrbStack K8s        │
     │  deploy-pr.sh                   │
     │                                 │
     │  1. Download .env from S3       │
     │  2. kubectl create namespace    │
     │  3. Copy ghcr-pull-secret       │
     │  4. Create K8s secrets          │
     │  5. Sync Atlas → local MongoDB  │
     │  6. Clone source DBs → pr-N    │
     │  7. helm upgrade --install      │
     └─────────────┬───────────────────┘
                   │
                   ▼
     Kubernetes namespace: pr-<repo>-<N>
     ┌───────────────────────────────────────────────┐
     │  Ingress (nginx)                              │
     │  fe-from-<repo>-<N>.pr.artistevolve.io        │
     │  be-from-<repo>-<N>.pr.artistevolve.io        │
     │  train-from-<repo>-<N>.pr.artistevolve.io     │
     │  admin-fe-from-<repo>-<N>.pr.artistevolve.io  │
     │  admin-be-from-<repo>-<N>.pr.artistevolve.io  │
     │                                               │
     │  Pods                                         │
     │  ├─ artistevolve-fe   (Next.js static)        │
     │  ├─ genesis-web       (Node.js API)           │
     │  ├─ genesis-train     (Python FastAPI)        │
     │  ├─ fe-artistevolve-admin (Next.js static)    │
     │  └─ artist-admin      (NestJS API)            │
     │                                               │
     │  Shared (namespace: shared)                   │
     │  └─ MongoDB StatefulSet                       │
     │     databases: test-pr<N>  (web + admin API)  │
     └───────────────────────────────────────────────┘

Network Path

GitHub Actions cannot reach the Mac Mini directly — it is a private office machine. Traffic flows through two hops:

GitHub Actions runner
        │
        │  SSH -p 2222  (MAC_SSH_KEY)
        ▼
EC2 instance — 13.204.230.168
        │
        │  TCP relay / port-forward via Tailscale
        │  (configured on the EC2 jump host)
        ▼
Mac Mini — OrbStack Kubernetes
        │
        ├─ kubectl (OrbStack kubeconfig)
        ├─ helm (Homebrew)
        ├─ mongosh / mongodump / mongorestore (Homebrew)
        └─ aws CLI (for S3 .env re-download on Mac)

Why port 2222? The EC2 instance listens on its public port 2222 and tunnels the connection to the Mac Mini’s SSH port (22) through Tailscale. This keeps the Mac Mini off the public internet while letting GitHub Actions reach it with a single SSH hop using the MAC_SSH_KEY.

How the workflow connects:

yaml

# In deploy-pr.yml — deploy job
ssh-keyscan -p 2222 -H 13.204.230.168 >> ~/.ssh/known_hosts
scp  -P 2222 -i ~/.ssh/mac_key rajib@13.204.230.168:/tmp/infra-deploy.tar.gz
ssh  -p 2222 -i ~/.ssh/mac_key rajib@13.204.230.168 bash -l << 'ENDSSH'
  ...deploy-pr.sh...
ENDSSH

Components

Component	Repo / Path	Role
`webhook-receiver`	`infra/webhook-receiver/index.js`	Node.js Express app: verifies GitHub webhooks, posts PR comments, dispatches `deploy-pr.yml`
`k8s/shared/webhook-receiver.yaml`	`infra/k8s/shared/`	K8s Deployment + Service + Ingress for the receiver pod
`deploy-pr.yml`	`infra/.github/workflows/deploy-pr.yml`	Hub workflow: builds 5 images in parallel, SCPs bundle to Mac, runs `deploy-pr.sh`
`deploy-pr.sh`	`infra/scripts/deploy-pr.sh`	Shell script on Mac: namespace, secrets, MongoDB sync/clone, Helm
`helm/pr-env/`	`infra/helm/pr-env/`	Helm chart for all 5 services + per-host Ingress
`MongoDB StatefulSet`	`infra/k8s/shared/mongodb.yaml`	Single shared MongoDB (`mongo:7`) in `shared` namespace; 20 Gi PVC
`mac-gateway`	`infra/mac-gateway/`	`nginx:alpine` Docker Compose on Mac: TCP/TLS proxy for external MongoDB access (port 27017) + HTTP routing
`sync-mac-infra.yml`	`infra/.github/workflows/sync-mac-infra.yml`	Auto-deploys `helm/`, `scripts/`, `k8s/shared/`, `mac-gateway/` to Mac on every push to `main`
`oauth-proxy`	`infra/k8s/shared/`	Nginx SPA at `auth.pr.artistevolve.io` for OAuth redirect handling

PR Lifecycle

1. PR Opened

GitHub sends pull_request action: opened to hooks.pr.artistevolve.io/github.
webhook-receiver checks:
- Repo is in WATCHED_REPOS.
- pull_request.base.ref === PREVIEW_TARGET_BASE (develop).
- If PREVIEW_HEAD_BRANCH is non-empty, pull_request.head.ref must match it.
If checks pass → posts welcome comment with /deploy instructions and preview URLs.
reopened action is ignored — no duplicate welcome comment.
If genesis-aws-services → posts “Lambda-only, no K8s preview” message.

2. `/deploy` Comment

Comment must be exactly one line matching /deploy optionally followed by repo=branch pairs. Any extra lines, headings, or text causes the comment to be silently ignored.

/deploy
/deploy genesis-web=my-branch
/deploy fe-artistevolve-admin=preview-deployment genesis-train=fix-ai

Flow:

webhook-receiver receives issue_comment action: created.
Validates: single-line /deploy, PR is open, base=develop, and optional head-branch gate (only when PREVIEW_HEAD_BRANCH is set).
Adds 🚀 reaction to the comment.
Checks active deploys for the same PR (in-memory lock + GitHub Actions queued/in_progress check). If one exists, posts an “already running” comment.
Posts build plan table (service, branch, status: queued).
Dispatches deploy-pr.yml with inputs:
- action: deploy
- pr_number, head_sha, triggering_repo, github_repo
- partner_branches (e.g. fe-artistevolve-admin=preview-deployment)
- override_tags (bare words without =)

deploy-pr.yml job sequence:

notify-pending
   └─ Sets "PR Preview" commit status → pending on the app repo head SHA

resolve-refs
   └─ Triggering repo → head_sha
      Partner repos  → develop (or partner_branches override)

build-* (5 parallel jobs)
   ├─ build-artistevolve-fe
   ├─ build-fe-artistevolve-admin
   ├─ build-genesis-web
   ├─ build-artist-admin
   └─ build-genesis-train
   Each: checkout at resolved ref, pull .env.dev from S3, docker build + push :pr-N to GHCR

deploy
   ├─ Download backend .env files from S3 → envfiles/
   ├─ tar czf infra-deploy.tar.gz helm/ scripts/ envfiles/
   ├─ SCP tar to rajib@13.204.230.168:/tmp/infra-deploy.tar.gz (port 2222)
   └─ SSH: extract + run deploy-pr.sh (with env vars injected)

notify-result
   └─ Posts result table to PR (service, branch, build status, URL)
   └─ Sets commit status → success / failure

deploy-pr.sh steps on Mac:

bash

1. download_backend_env_from_s3   # re-downloads from S3 on Mac (fresh secrets)
2. kubectl create namespace pr-<repo>-<N>
3. Copy ghcr-pull-secret from shared → new namespace
4. Create K8s secrets: genesis-web-env, genesis-train-env, artist-admin-env
   (from .env files; strips comments/whitespace)
5. Sync Atlas → local MongoDB
   - db=fresh: re-download all 5 source DBs from Atlas (10–30 min)
   - default:  only download DBs missing locally (fast path, usually a no-op)
6. Clone source DBs into pr-N snapshots via MongoDB $out aggregation
   (runs BEFORE helm so app seeders don't block clone)
7. helm upgrade --install pr-<repo>-<N> helm/pr-env/ \
     --set prNumber=N --set repoName=<repo> \
     --set images.*.tag=pr-N --wait --timeout 10m

3. `/destroy` Comment

Comment must be exactly the single line /destroy (case-insensitive).

webhook-receiver validates: single-line /destroy, PR open, base=develop, and optional head-branch gate (only when PREVIEW_HEAD_BRANCH is set).
Adds 👀 reaction to the comment.
Posts teardown queued comment.
Dispatches deploy-pr.yml with action: destroy.

deploy-pr.sh destroy path:

bash

kubectl delete namespace pr-<repo>-<N> --ignore-not-found
# Drop only databases ending with -pr<N>:
mongosh → db.adminCommand('listDatabases') → drop all *.endsWith('-pr<N>')
# Source databases (test, genesis-rag, artist-admin, …) are NEVER dropped.

notify-result posts teardown result comment.

4. PR Closed / Merged

GitHub sends pull_request action: closed for both merge and abandon.

webhook-receiver checks base=develop and optional head-branch gate.
Dispatches deploy-pr.yml with action: destroy (same as manual /destroy).
notify-result posts teardown result on the (now closed) PR.

5. `/cancel` Comment

Exact single line /cancel. Adds 👎 reaction, dispatches deploy-pr.yml with action: cancel. The cancel action enters the same concurrency group as any in-progress deploy for that PR number, causing GitHub to cancel it. Posts a cancellation comment.

6. Push / Synchronize

Pushes to any branch and pull_request synchronize are ignored — no auto-rebuild on new commits. Comment /deploy again to refresh the preview after pushing.

7. Scheduled Expiry

expire-pr-previews.yml runs daily and removes stale pr-* namespaces older than 7 days (configurable on manual run). It uses the same destroy path (deploy-pr.sh ... destroy) so namespace and *-pr<N> DB snapshots are cleaned consistently.

Branch Gates

Configured as env vars on the webhook-receiver pod (k8s/shared/webhook-receiver.yaml):

Env var	Current value	Effect
`PREVIEW_TARGET_BASE`	`develop`	Only PRs targeting `develop` get preview actions
`PREVIEW_HEAD_BRANCH`	`""`	Empty means no source-branch restriction (set a branch name to enforce a head-branch gate)

These guard welcome comments and all preview commands. For /deploy and /destroy, branch-gate failures are replied to with a skip comment explaining the reason.

To enable a head-branch restriction (only allow previews from one source branch):

bash

# Edit k8s/shared/webhook-receiver.yaml: set PREVIEW_HEAD_BRANCH value to e.g. "preview-deployment"
# Commit + push to main → sync-mac-infra.yml applies it automatically

Image Build Strategy

All five images are built fresh on every /deploy using tag :pr-<N>. The triggering repo is checked out at its PR head SHA; all partner repos default to develop.

Frontend apps — static Next.js export

NEXT_PUBLIC_* variables are baked into the static HTML/JS at build time. The Dockerfile:

Declares ARG NEXT_PUBLIC_API_URL and ARG NEXT_PUBLIC_PR_ID.
Downloads dev .env.local from S3.
Strips NEXT_PUBLIC_API_URL and NEXT_PUBLIC_PR_ID from all env files (.env, .env.local, .env.development, .env.production) so the ARG values always win.
Sets ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL before npm run build.

App	Build args injected by `deploy-pr.yml`
`artistevolve-fe`	`NEXT_PUBLIC_PR_ID=<N>` `NEXT_PUBLIC_API_BASE_URL=https://be-from-<repo>-<N>.pr.artistevolve.io/api/v1` `NEXT_PUBLIC_ADMIN_API_URL=https://admin-be-from-<repo>-<N>.pr.artistevolve.io` `NEXT_PUBLIC_APP_URL=https://fe-from-<repo>-<N>.pr.artistevolve.io`
`fe-artistevolve-admin`	`NEXT_PUBLIC_PR_ID=<N>` `NEXT_PUBLIC_API_URL=https://admin-be-from-<repo>-<N>.pr.artistevolve.io/api`

Important: fe-artistevolve-admin’s NEXT_PUBLIC_API_URL must include /api because artist-admin uses NestJS setGlobalPrefix('api') — all routes are under /api/*.

Backend apps — no env in image

Backend images contain no secrets. At deploy time, deploy-pr.sh creates a K8s Secret per app from the S3 .env.dev file and Helm mounts it via envFrom: secretRef. Individual env entries in the Helm template override PR-specific values (DB connection strings, URLs, PR ID). K8s env always takes precedence over envFrom.

Source code per branch

Scenario	Branch used
Triggering repo	PR `head_sha`
Partner repos (no override)	`develop`
Partner repos (with `repo=branch`)	The specified branch
Partner repos (tag override)	The specified tag/sha

Environment Variables

S3 env file locations (bucket: `genesis-backend-resources`)

App	S3 path	Used as
`artistevolve-fe`	`env-variables/artistevolve-fe/.env.dev`	`.env.local` at Docker build time
`fe-artistevolve-admin`	`env-variables/fe-artistevolve-admin/.env.dev`	`.env.local` at Docker build time
`genesis-web`	`env-variables/.env.dev`	K8s Secret `genesis-web-env`
`genesis-train`	`env-variables/genesis-train/.env.dev`	K8s Secret `genesis-train-env`
`artist-admin`	`env-variables/artist-admin/.env.dev`	K8s Secret `artist-admin-env`

Per-service Helm overrides (set in `helm/pr-env/templates/`)

These env entries override whatever is in the K8s Secret:

genesis-web (genesis-web.yaml):

Variable	PR value
`PR_ID`	`<N>`
`NODE_ENV`	`dev`
`PORT`	`3000`
`MONGO_URL`	`mongodb://admin:<pw>@mongodb.shared.svc.cluster.local:27017/test-pr<N>?authSource=admin`
`FASTAPI_URL`	`http://genesis-train.pr-<repo>-<N>.svc.cluster.local/api/v1/agent`
`CLIENT_URL`	`https://fe-from-<repo>-<N>.pr.artistevolve.io`
`OAUTH_PROXY_URL`	`https://auth.pr.artistevolve.io`

genesis-train (genesis-train.yaml):

Variable	PR value
`PR_ID`	`<N>`
`NODE_ENV`	`development`
`PORT`	`8000`
`MONGODB_URI`	`mongodb://admin:<pw>@mongodb.shared.svc.cluster.local:27017/test-pr<N>?authSource=admin`

artist-admin (artist-admin.yaml):

Variable	PR value
`PR_ID`	`<N>`
`NODE_ENV`	`development`
`PORT`	`3000`
`MONGODB_URI`	`mongodb://admin:<pw>@mongodb.shared.svc.cluster.local:27017/test-pr<N>?authSource=admin`
`API_BASE_URL`	`https://admin-be-from-<repo>-<N>.pr.artistevolve.io/api`
`FRONTEND_URL`	`https://admin-fe-from-<repo>-<N>.pr.artistevolve.io`

Unified database: All three backend services point to the same test-pr<N> database (previously each had its own). This matches the dev Atlas setup where test is the single shared database.

MongoDB Seeding

Source databases

deploy-pr.sh manages exactly these five source databases:

bash

PR_SOURCE_DBS="artist-admin genesis-train genesis-web genesis-rag test"

These are synced from Atlas and cloned into *-pr<N> snapshots. They are never dropped — only the snapshot copies (*-pr<N>) are managed.

Atlas sync strategy

On every /deploy:

db=fresh flag set?
├─ YES → do_atlas_sync (all 5 source DBs)
│         mongodump --uri=<atlas-uri> --db=<each> --numParallelCollections=1
│         timeout: 60 min per database
│         mongorestore --uri=<local> --drop
│         WARNING: can take 10–30 min total
└─ NO  → sync_atlas_to_local (smart)
          for each DB in PR_SOURCE_DBS:
            if local objects == 0 → add to missing list
          if missing list empty → skip (fast path, ~0s)
          else → do_atlas_sync(missing_dbs only)

--numParallelCollections=1 is required — MongoDB Atlas cursors expire after ~10 minutes, and parallel dumps cause large collections to go idle and hit CursorNotFound errors.

The Atlas URI is read from envfiles/genesis-web.env (the MONGO_URL variable pointing to Atlas).

PR snapshot clone

Runs before helm upgrade — critical ordering. If Helm started first, genesis-web seeders would write into test-pr<N> and the clone would see non-zero documents and skip, leaving the PR with missing or wrong data.

The clone uses MongoDB’s $out aggregation (in-cluster copy, no network overhead):

javascript

// For each source DB (e.g. "test"):
const dest = src + '-pr' + prNumber;  // e.g. "test-pr51"
// Drop all non-system collections in dest first
// Copy each collection: sdb.getCollection(col).aggregate([{ $out: { db: dest, coll: col } }])

If a source DB is empty locally (e.g. Atlas sync was skipped), that DB is skipped for cloning.

Destroy cleanup

Only databases whose names end with -pr<N> are dropped:

javascript

// In mongosh on destroy:
if (name.endsWith('-pr' + pr)) db.getSiblingDB(name).dropDatabase();
else print('PR Mongo: preserving ' + name);

This ensures source databases (test, genesis-rag, etc.) and other PR snapshots are never touched.

URL Structure

Each PR environment has five dedicated subdomains, one per service. The subdomain encodes the triggering repo and PR number to avoid collisions:

https://fe-from-<repo>-<N>.pr.artistevolve.io/        → artistevolve-fe      (Next.js frontend)
https://be-from-<repo>-<N>.pr.artistevolve.io/        → genesis-web          (Node.js API)
https://train-from-<repo>-<N>.pr.artistevolve.io/     → genesis-train        (Python FastAPI)
https://admin-fe-from-<repo>-<N>.pr.artistevolve.io/  → fe-artistevolve-admin (Next.js admin UI)
https://admin-be-from-<repo>-<N>.pr.artistevolve.io/  → artist-admin         (NestJS admin API)

https://hooks.pr.artistevolve.io/                     → webhook-receiver     (GitHub webhooks)
https://auth.pr.artistevolve.io/                      → oauth-proxy          (OAuth redirects)

TLS is a wildcard cert *.pr.artistevolve.io — covers all PR subdomains automatically.

Each service gets its own Ingress rule (separate hosts, not paths). This is different from the diagram in some older docs that showed path-based routing — the actual Helm ingress template uses one rule per hostname.

Helm Chart & Per-Service Overrides

Chart location: infra/helm/pr-env/

Key values set by deploy-pr.sh:

bash

helm upgrade --install pr-<repo>-<N> helm/pr-env/ \
  --namespace pr-<repo>-<N> \
  --set prNumber=<N> \
  --set repoName=<repo> \
  --set mongoHost=mongodb.shared.svc.cluster.local \
  --set images.genesisWeb.tag=pr-<N> \
  --set images.artistevolveFe.tag=pr-<N> \
  --set images.genesisTrain.tag=pr-<N> \
  --set images.artistAdmin.tag=pr-<N> \
  --set images.feArtistevolveAdmin.tag=pr-<N> \
  -f /tmp/mongo-password.json \   # mongoPassword set via JSON values file (safe for special chars)
  --wait --timeout 10m

The mongoPassword is passed via a temp JSON values file rather than --set because --set breaks on special characters like @, #, ,.

Default resources per container (tuned for Mac Mini single-node):

yaml

requests: { memory: 128Mi, cpu: 50m }
limits:   { memory: 512Mi, cpu: 500m }

Mac Gateway

A Docker Compose nginx:alpine container (mac-local-router) runs on the Mac Mini as a local router/proxy.

File: infra/mac-gateway/docker-compose.yml

yaml

ports:
  - "8888:80"
  - "8443:443"
  - "27017:27017"    # MongoDB TCP/TLS proxy

MongoDB TLS proxy (mac-gateway/stream.d/mongo.conf):

nginx

stream {
  server {
    listen 27017 ssl;
    ssl_certificate  /etc/letsencrypt/live/pr.artistevolve.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/pr.artistevolve.io/privkey.pem;
    proxy_pass host.docker.internal:30017;  # → K8s NodePort for MongoDB
    proxy_timeout 300s;
  }
}

This allows external tools (e.g. MongoDB Compass on a dev laptop) to connect to pr.artistevolve.io:27017 over TLS and reach the K8s MongoDB. It also serves as the path for mongodump/mongorestore operations that run on the Mac itself (those connect via localhost:30017 directly to the NodePort).

Infra Auto-Sync to Mac

Workflow: infra/.github/workflows/sync-mac-infra.yml

Triggers on push to main when any of these paths change:

helm/**
scripts/**
k8s/shared/**
mac-gateway/**
.github/workflows/sync-mac-infra.yml
.github/workflows/deploy-pr.yml

What it does:

Downloads latest backend .env.dev files from S3.
Bundles helm/ scripts/ envfiles/ k8s/ mac-gateway/ into a tar.
SCPs to Mac via EC2 jump (port 2222) → extracts to ~/infra-runner/.
kubectl apply -f webhook-receiver.yaml + rollout status.
kubectl apply -f mongodb-external.yaml (NodePort for external MongoDB access).
Syncs mac-gateway/ configs + docker compose up -d + nginx -s reload.

This means: push to main → Mac is updated within ~2 minutes. No manual SSH needed for config changes.

Webhook Receiver

Source: infra/webhook-receiver/index.js
Image: ghcr.io/genesis-technologies-llc/pr-webhook-receiver:latest
Namespace: shared
Endpoint: hooks.pr.artistevolve.io/github (POST)

How it works

Receives application/json body (raw, before JSON parse) for HMAC verification.
Verifies x-hub-signature-256 using GITHUB_WEBHOOK_SECRET (timing-safe compare).
Checks WATCHED_REPOS — ignores any repo not in the list.
Dispatches based on event type:

GitHub event	Trigger	Action
`pull_request` `opened`	PR opened to `develop` (plus optional head gate if configured)	Post welcome comment
`pull_request` `closed`	PR merged or abandoned	Dispatch `destroy`
`pull_request` `reopened`	PR reopened	Ignored (no duplicate welcome)
`pull_request` `synchronize`	New commit pushed to PR	Ignored
`issue_comment` `created` `/deploy`	Single-line `/deploy` on open PR	Dispatch `deploy`
`issue_comment` `created` `/destroy`	Single-line `/destroy` on open PR	Dispatch `destroy`
`issue_comment` `created` `/cancel`	Single-line `/cancel`	Dispatch `cancel`
`push`	Any branch push	Ignored

Comment validation rules

/deploy: Must be the entire comment (single non-empty line). May be followed by repo=branch pairs on the same line. Multiple lines → silently ignored.
/destroy: Must be the entire comment, exactly /destroy, case-insensitive. No arguments. Any extra text/lines means the comment is ignored.
/cancel: Must be the entire comment, exactly /cancel, case-insensitive.

Concurrency

deploy-pr.yml uses concurrency: group: pr-preview-<N> cancel-in-progress: true. In addition, the receiver blocks duplicate /deploy comments when it detects an in-flight deploy (local lock + active run lookup) and asks users to /cancel first.

Updating the receiver

The receiver image is built and pushed by a separate workflow in infra that triggers on changes to webhook-receiver/. That workflow also refreshes ghcr-pull-secret in shared and runs kubectl rollout restart deployment/webhook-receiver.

GitHub Secrets Reference

All secrets live on the ae-infra repository.

Secret	Used in	Purpose
`MAC_SSH_KEY`	`deploy-pr.yml`, `sync-mac-infra.yml`, `build-webhook-receiver.yml`, `build-oauth-proxy.yml`, `expire-pr-previews.yml`	SSH private key used to reach the Mac via the EC2 jump host (`13.204.230.168:2222`)
`INFRA_PAT`	`deploy-pr.yml`, `webhook-receiver`, `expire-pr-previews.yml`	GitHub PAT for cross-repo checkout, workflow dispatch, PR comments, commit statuses, and PR-state checks
`MONGO_ROOT_PASSWORD`	`deploy-pr.sh`	MongoDB `admin` user password for the K8s StatefulSet
`AWS_ACCESS_KEY_ID`	`deploy-pr.yml`, `sync-mac-infra.yml`	AWS credentials to download `.env.dev` files from S3
`AWS_SECRET_ACCESS_KEY`	`deploy-pr.yml`, `sync-mac-infra.yml`	AWS credentials to download `.env.dev` files from S3

On each repo’s webhook config:

Field	Value
Payload URL	`https://hooks.pr.artistevolve.io/github`
Content type	`application/json`
Secret	Same value as `GITHUB_WEBHOOK_SECRET` K8s secret on Mac
Events	Pull requests + Issue comments + Pushes

Auto-Expiry of Stale Previews

Stale preview namespaces are auto-cleaned by infra/.github/workflows/expire-pr-previews.yml.

Schedule: daily at 02:00 UTC
Default expiry age: 168 hours (7 days)
Supports manual run inputs:
- max_age_hours to change cutoff
- dry_run=true to list expirations without deleting

Cleanup flow:

SSH to Mac (13.204.230.168:2222) and list pr-* namespaces with creation timestamps.
For each expired namespace, parse pr-<repo>-<N>.
Run deploy-pr.sh <N> destroy <repo> (deletes namespace + drops *-pr<N> DB snapshots).
If PR is still open, post an expiry comment telling developers to re-run /deploy.

This provides a safety net if a close/merge webhook is missed or destroy fails.

Troubleshooting

Symptom	Likely cause	Fix
`ImagePullBackOff`	`ghcr-pull-secret` not in namespace	Re-run deploy; or `kubectl get secret ghcr-pull-secret -n shared` to verify source exists
Pod `CrashLoopBackOff`	App misconfiguration or missing env	`kubectl logs -n pr-<repo>-<N> deploy/<app>`
Admin API returns 404	`NEXT_PUBLIC_API_URL` baked without `/api` — old build	Re-run `/deploy`; latest Dockerfile fix strips conflicting env vars
Login uses wrong users	Clone was skipped — PR DB had data from previous deploy	`kubectl exec` into mongo, drop `test-pr<N>`, then `/destroy` + `/deploy`
MongoDB collections empty	`sync_atlas_to_local` found local DBs but they’re empty in Atlas	Check Atlas cluster has data; use `/deploy db=fresh`
`CursorNotFound` during Atlas dump	Removed in recent fix (`--numParallelCollections=1`)	Update Mac’s `deploy-pr.sh` via push to `main` → `sync-mac-infra`
Backend using wrong MongoDB DB	PR_ID not injected or Helm override not applied	`kubectl exec -n pr-<repo>-<N> <pod> -- env \| grep MONGO`
Webhook 502 / not firing	nginx ingress timeout or pod not ready	Check `kubectl logs -n shared deploy/webhook-receiver`; ensure pod is Ready; check ingress controller logs
`/deploy` silently ignored	Comment has multiple lines or extra text	Comment body must be exactly one line starting with `/deploy`
`/deploy` says “already running”	A deploy is already queued/in-progress for that PR	Use `/cancel`, wait for cancellation comment, then comment `/deploy` again
Commit status stuck at pending	`notify-result` job failed	Check the `notify-result` job in the Actions run on ae-infra
`helm upgrade` timeout	App pod failing to start (bad image or crashloop)	Check pod logs in namespace; `--wait --timeout 10m` will fail the deploy job
Mac not getting latest scripts	`sync-mac-infra` not triggered or failed	Push to `main` on ae-infra, or trigger `sync-mac-infra` via workflow_dispatch
SSH connection timeout to Mac	EC2 → Tailscale tunnel down	Check Tailscale status on Mac; check EC2 port-forward; try `ssh -p 2222 rajib@13.204.230.168 echo ok`

Key Files Reference

File	Description
`infra/.github/workflows/deploy-pr.yml`	Hub workflow — 5 parallel builds + SSH deploy + commit status + PR comment
`infra/.github/workflows/sync-mac-infra.yml`	Auto-syncs infra config to Mac on push to `main`
`infra/.github/workflows/expire-pr-previews.yml`	Daily stale-preview cleanup (`pr-*` namespaces older than max age)
`infra/.github/workflows/build-webhook-receiver.yml`	Builds receiver image, refreshes `ghcr-pull-secret`, restarts webhook receiver
`infra/.github/workflows/build-oauth-proxy.yml`	Builds oauth-proxy image and restarts oauth-proxy deployment
`infra/scripts/deploy-pr.sh`	Runs on Mac — K8s namespace, secrets, MongoDB sync/clone, Helm
`infra/helm/pr-env/values.yaml`	Default Helm values; `deploy-pr.sh` overrides `prNumber`, `repoName`, image tags, `mongoPassword`
`infra/helm/pr-env/templates/ingress.yaml`	One Ingress rule per service hostname
`infra/helm/pr-env/templates/genesis-web.yaml`	Deployment + Service for genesis-web; sets `MONGO_URL`, `FASTAPI_URL`, `CLIENT_URL`, `OAUTH_PROXY_URL`
`infra/helm/pr-env/templates/artist-admin.yaml`	Deployment + Service for artist-admin; sets `MONGODB_URI`, `API_BASE_URL`, `FRONTEND_URL`
`infra/helm/pr-env/templates/genesis-train.yaml`	Deployment + Service for genesis-train; sets `MONGODB_URI`
`infra/k8s/shared/mongodb.yaml`	MongoDB StatefulSet (mongo:7) in `shared` namespace; 20 Gi PVC
`infra/k8s/shared/webhook-receiver.yaml`	Webhook receiver Deployment + ClusterIP Service + Ingress; sets `PREVIEW_TARGET_BASE` and `PREVIEW_HEAD_BRANCH`
`infra/k8s/shared/mongodb-external.yaml`	NodePort service exposing MongoDB on host port 30017
`infra/mac-gateway/docker-compose.yml`	nginx:alpine on Mac — HTTP on 8888, HTTPS on 8443, MongoDB TCP/TLS on 27017
`infra/mac-gateway/stream.d/mongo.conf`	MongoDB TLS stream proxy: `0.0.0.0:27017` → `host.docker.internal:30017`
`infra/webhook-receiver/index.js`	Node.js webhook handler — HMAC verify, branch gates, comment parsing, workflow dispatch
`fe-artistevolve-admin/Dockerfile`	Only on `preview-deployment` branch — not on `develop` (stage/prod use EC2 rsync)

Migration Guide — Moving from Mac Mini to EC2

This section covers everything needed to drop the Mac Mini and run the entire PR preview stack directly on an EC2 instance. No Tailscale, no jump-host indirection.

This section is a migration playbook (target-state guidance), not a description of the current production preview system.

When to do this

Mac Mini hardware failure or retirement
Team scales up and the single-node Mac becomes a bottleneck
You want the preview runner in the same AWS region as Atlas / stage / prod

What changes vs. the current setup

Concern	Mac Mini (current)	EC2 (target)
K8s runtime	OrbStack	k3s (recommended) or k0s
SSH path	EC2:2222 → Tailscale → Mac	Direct SSH to EC2:22 (or custom port)
Tailscale	Required	Not needed
Kubeconfig	OrbStack auto-configures	k3s writes `/etc/rancher/k3s/k3s.yaml`
MongoDB storage	Mac local SSD via OrbStack PVC	EBS gp3 volume mounted to EC2
TLS / wildcard cert	Managed on Mac via Let’s Encrypt + mac-gateway nginx	Managed on EC2 (same certbot or cert-manager in-cluster)
Public IP	EC2 static IP (13.204.230.168) — already used for jump	New/existing EC2 Elastic IP — DNS points here
`mac-gateway` Docker Compose	Needed for external Mongo access	Optional (keep if you want external MongoDB access; otherwise remove)

Step 1 — Provision EC2 instance

Recommended spec for a preview server running 5 services + MongoDB:

Type: t3.xlarge (4 vCPU, 16 GB RAM) minimum; t3a.2xlarge for comfortable headroom
OS: Ubuntu 24.04 LTS
Storage: Root 30 GB gp3 + separate data volume 50 GB gp3 (for MongoDB PVC)
Security group inbound: 22 (SSH), 80 (HTTP), 443 (HTTPS)
Elastic IP: Assign one — DNS wildcard *.pr.artistevolve.io must point here

Mount the data volume at /data for MongoDB PVC:

bash

sudo mkfs.ext4 /dev/xvdb
sudo mkdir -p /data
echo '/dev/xvdb /data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab
sudo mount -a

Step 2 — Install k3s

k3s is a single-binary Kubernetes distribution — no extra components needed.

bash

# On the EC2 instance
curl -sfL https://get.k3s.io | sh -s - \
  --disable=traefik \          # We use ingress-nginx instead
  --data-dir=/data/k3s \       # Store K8s data on the data volume
  --write-kubeconfig-mode=644

# Verify
sudo k3s kubectl get nodes

Install ingress-nginx (matches what OrbStack uses):

bash

sudo k3s kubectl apply -f \
  https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml

# Wait for controller to be ready
sudo k3s kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

Make kubectl and helm available without sudo:

bash

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER ~/.kube/config

# Install helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Step 3 — TLS wildcard certificate

Get a Let’s Encrypt wildcard cert for *.pr.artistevolve.io. Use DNS-01 challenge (required for wildcards):

bash

sudo apt install -y certbot python3-certbot-dns-route53   # if DNS is on Route 53

sudo certbot certonly \
  --dns-route53 \
  -d "*.pr.artistevolve.io" \
  -d "pr.artistevolve.io" \
  --email ops@artistevolve.io \
  --agree-tos \
  --non-interactive

If DNS is not on Route 53, use the corresponding certbot plugin (python3-certbot-dns-cloudflare, etc.) or do manual DNS challenge once and set up a cron for renewal.

Create a K8s TLS secret from the cert (needed by ingress-nginx):

bash

sudo k3s kubectl create namespace shared

sudo k3s kubectl create secret tls pr-tls \
  --namespace shared \
  --cert=/etc/letsencrypt/live/pr.artistevolve.io/fullchain.pem \
  --key=/etc/letsencrypt/live/pr.artistevolve.io/privkey.pem

# Renew hook — update the K8s secret after every renewal
cat > /etc/letsencrypt/renewal-hooks/deploy/update-k8s-secret.sh << 'EOF'
#!/bin/bash
kubectl create secret tls pr-tls \
  --namespace shared \
  --cert=/etc/letsencrypt/live/pr.artistevolve.io/fullchain.pem \
  --key=/etc/letsencrypt/live/pr.artistevolve.io/privkey.pem \
  --dry-run=client -o yaml | kubectl apply -f -
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/update-k8s-secret.sh

Alternatively, install cert-manager in-cluster and let it manage the wildcard cert automatically. That is more complex to set up but fully automated.

Update the Ingress templates in helm/pr-env/templates/ingress.yaml to use the TLS secret:

yaml

spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "{{ include \"pr-env.host.fe\" . }}"
        - "{{ include \"pr-env.host.be\" . }}"
        - "{{ include \"pr-env.host.train\" . }}"
        - "{{ include \"pr-env.host.adminFe\" . }}"
        - "{{ include \"pr-env.host.adminBe\" . }}"
      secretName: pr-tls
  rules:
    # ... existing rules unchanged ...

Do the same for k8s/shared/webhook-receiver.yaml ingress (add tls: block with secretName: pr-tls).

Step 4 — Apply shared K8s resources

bash

# GHCR pull secret (replace token with a PAT that has packages:read)
kubectl create secret docker-registry ghcr-pull-secret \
  --namespace shared \
  --docker-server=ghcr.io \
  --docker-username=<github-username> \
  --docker-password=<github-pat>

# MongoDB password secret
kubectl create secret generic mongodb-secret \
  --namespace shared \
  --from-literal=root-password=<your-password>

# Webhook receiver secrets
kubectl create secret generic webhook-receiver-secret \
  --namespace shared \
  --from-literal=webhook-secret=<same-webhook-secret-as-github-repos> \
  --from-literal=github-token=<infra-pat>

# Apply MongoDB StatefulSet and webhook receiver
kubectl apply -f k8s/shared/mongodb.yaml
kubectl apply -f k8s/shared/webhook-receiver.yaml
kubectl apply -f k8s/shared/mongodb-external.yaml   # optional — only if you want external Mongo access

Step 5 — Install `mongodump` / `mongorestore` / `mongosh`

bash

# MongoDB tools (needed by deploy-pr.sh for Atlas sync)
wget -qO - https://www.mongodb.org/static/pgp/server-7.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/7.0 multiverse" \
  | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt update
sudo apt install -y mongodb-mongosh mongodb-database-tools

Step 6 — Update `deploy-pr.yml` SSH target

Change every 13.204.230.168 reference and remove the -p 2222 (or keep a custom port — just make sure EC2 security group allows it).

In infra/.github/workflows/deploy-pr.yml, update the deploy job:

yaml

- name: Setup SSH keys
  run: |
    mkdir -p ~/.ssh
    echo "${{ secrets.EC2_SSH_KEY }}" > ~/.ssh/ec2_key   # renamed from MAC_SSH_KEY
    chmod 600 ~/.ssh/ec2_key
    ssh-keyscan -H ${{ secrets.EC2_HOST }} >> ~/.ssh/known_hosts 2>/dev/null || true

- name: Copy scripts + helm + env files to EC2
  run: |
    tar czf /tmp/infra-deploy.tar.gz helm/ scripts/ envfiles/
    scp -i ~/.ssh/ec2_key \
      -o StrictHostKeyChecking=accept-new \
      /tmp/infra-deploy.tar.gz \
      ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }}:/tmp/infra-deploy.tar.gz

- name: Run deploy script on EC2
  run: |
    ssh -i ~/.ssh/ec2_key \
      -o StrictHostKeyChecking=accept-new \
      -o BatchMode=yes \
      -o ServerAliveInterval=30 \
      -o ServerAliveCountMax=4 \
      ${{ secrets.EC2_USER }}@${{ secrets.EC2_HOST }} bash -l << 'ENDSSH'
        set -e
        mkdir -p ~/infra-runner
        tar xzf /tmp/infra-deploy.tar.gz -C ~/infra-runner
        export GITHUB_TOKEN='${{ secrets.INFRA_PAT }}'
        export GITHUB_REPO='${{ github.event.inputs.github_repo }}'
        export MONGO_ROOT_PASSWORD='${{ secrets.MONGO_ROOT_PASSWORD }}'
        export AWS_ACCESS_KEY_ID='${{ secrets.AWS_ACCESS_KEY_ID }}'
        export AWS_SECRET_ACCESS_KEY='${{ secrets.AWS_SECRET_ACCESS_KEY }}'
        export AWS_DEFAULT_REGION='ap-south-1'
        bash ~/infra-runner/scripts/deploy-pr.sh \
          '${{ github.event.inputs.pr_number }}' \
          '${{ github.event.inputs.action }}' \
          '${{ github.event.inputs.triggering_repo }}' \
          '${{ github.event.inputs.override_tags }}'
    ENDSSH

Do the same in sync-mac-infra.yml (rename to sync-ec2-infra.yml for clarity).

Step 7 — Update GitHub secrets

Remove the old Mac-specific secrets and add EC2 ones:

Old secret	New secret	Value
`MAC_SSH_KEY`	`EC2_SSH_KEY`	SSH private key for EC2 instance
`MAC_USERNAME`	`EC2_USER`	EC2 username (e.g. `ubuntu`)
`MAC_TAILSCALE_IP`	(remove)	No longer needed
—	`EC2_HOST`	EC2 public IP or hostname

All other secrets (INFRA_PAT, MONGO_ROOT_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) remain unchanged.

Step 8 — Point DNS to EC2

In your DNS provider, update the wildcard record:

*.pr.artistevolve.io  A  <EC2 Elastic IP>
pr.artistevolve.io    A  <EC2 Elastic IP>
hooks.pr.artistevolve.io  A  <EC2 Elastic IP>
auth.pr.artistevolve.io   A  <EC2 Elastic IP>

Wait for TTL to propagate, then verify:

bash

dig +short fe-from-artist-admin-1.pr.artistevolve.io
# Should return the EC2 Elastic IP

Step 9 — Test end-to-end

bash

# On EC2 — verify K8s is healthy
kubectl get nodes
kubectl get pods -n shared

# Check webhook receiver is reachable
curl https://hooks.pr.artistevolve.io/health
# Expected: OK

# Manual deploy test
bash ~/infra-runner/scripts/deploy-pr.sh 99 deploy artist-admin
kubectl get pods -n pr-artist-admin-99

Step 10 — Decommission Mac Mini

Once at least two PRs have deployed and torn down cleanly on EC2:

Delete Tailscale node from the Tailscale admin console.
Remove the EC2 port-2222 forwarding rule from the old EC2 (or terminate the jump host if it was dedicated).
Wipe the Mac Mini or repurpose it.
Archive / remove infra/mac-gateway/ if external MongoDB access is not needed on EC2.

PR Preview Deployment — Comprehensive Guide

Table of Contents

Architecture Overview

Network Path

Components

PR Lifecycle

1. PR Opened

2. /deploy Comment

3. /destroy Comment

4. PR Closed / Merged

5. /cancel Comment

6. Push / Synchronize

7. Scheduled Expiry

Branch Gates

Image Build Strategy

Frontend apps — static Next.js export

Backend apps — no env in image

Source code per branch

Environment Variables

S3 env file locations (bucket: genesis-backend-resources)

Per-service Helm overrides (set in helm/pr-env/templates/)

MongoDB Seeding

Source databases

Atlas sync strategy

PR snapshot clone

Destroy cleanup

URL Structure

Helm Chart & Per-Service Overrides

Mac Gateway

Infra Auto-Sync to Mac

Webhook Receiver

How it works

Comment validation rules

Concurrency

Updating the receiver

GitHub Secrets Reference

Auto-Expiry of Stale Previews

Troubleshooting

Key Files Reference

Migration Guide — Moving from Mac Mini to EC2

When to do this

What changes vs. the current setup

Step 1 — Provision EC2 instance

Step 2 — Install k3s

Step 3 — TLS wildcard certificate

Step 4 — Apply shared K8s resources

Step 5 — Install mongodump / mongorestore / mongosh

Step 6 — Update deploy-pr.yml SSH target

Step 7 — Update GitHub secrets

Step 8 — Point DNS to EC2

Step 9 — Test end-to-end

Step 10 — Decommission Mac Mini

2. `/deploy` Comment

3. `/destroy` Comment

5. `/cancel` Comment

S3 env file locations (bucket: `genesis-backend-resources`)

Per-service Helm overrides (set in `helm/pr-env/templates/`)

Step 5 — Install `mongodump` / `mongorestore` / `mongosh`

Step 6 — Update `deploy-pr.yml` SSH target