Skip to main content

Incident Recovery

This page is the operator playbook for the most common DaoFlow failures.

First Five Minutes​

Collect the state before you make changes:

cd /opt/daoflow
docker compose ps
docker compose logs --tail=200 daoflow
docker compose logs --tail=100 postgres redis temporal temporal-ui
curl http://127.0.0.1:3000/health
curl http://127.0.0.1:3000/ready

If the web UI is reachable, also capture:

  • the failing deployment or backup run ID
  • the server name involved
  • the last successful deployment or backup before the incident
  • the deployment state artifact export from the deployment details panel

Control Plane Will Not Start​

Check:

  1. .env contains non-empty BETTER_AUTH_SECRET, ENCRYPTION_KEY, and POSTGRES_PASSWORD; TEMPORAL_POSTGRES_PASSWORD is also required when DAOFLOW_WORKFLOW_PROFILE=temporal
  2. BETTER_AUTH_URL matches the public origin operators are actually using
  3. /var/run/docker.sock is mounted into the daoflow container
  4. postgres is healthy before daoflow starts
  5. the daoflow logs show whether startup stopped during database migration, owner bootstrap, or worker startup

Useful commands:

docker compose logs --tail=200 daoflow
docker compose logs --tail=200 postgres
docker compose run --rm -e DAOFLOW_RUN_MIGRATIONS_ONLY=true daoflow

If startup fails during owner bootstrap, correct the DAOFLOW_INITIAL_ADMIN_* values and restart daoflow.

If startup fails during migration, restore from the database backup taken before upgrade or fix the schema drift before starting the app. Do not set DAOFLOW_ALLOW_START_WITH_MIGRATION_FAILURE=true unless you intentionally want an emergency degraded process; it cannot bypass lineage mismatches or a recorded failed migration, and /ready remains unavailable when a bypass is allowed. After repairing a recorded failure, run migration-only mode once with DAOFLOW_RETRY_FAILED_MIGRATION=true, then remove that setting.

Destination-Key Rotation Or Migration Failure​

Destination-key rotation does not require changing the global ENCRYPTION_KEY. Keep that key unchanged for all other DaoFlow secrets. Set DAOFLOW_BACKUP_DESTINATION_ENCRYPTION_KEY to the new destination key and temporarily set DAOFLOW_PREVIOUS_BACKUP_DESTINATION_ENCRYPTION_KEY to the old destination key, then run:

docker compose run --rm -e DAOFLOW_RUN_MIGRATIONS_ONLY=true daoflow
docker compose up -d daoflow
curl http://127.0.0.1:3000/ready

Startup checks every backup-destination envelope, re-encrypts it transactionally with the current destination key, and migrates and clears legacy plaintext secrets. Mixed or undecryptable state fails closed and prevents production startup. If migration fails before commit, no partial rotation is persisted: restore the old destination key as DAOFLOW_BACKUP_DESTINATION_ENCRYPTION_KEY (or unset it if it previously inherited ENCRYPTION_KEY), remove the temporary previous-key variable, and restart. Do not change ENCRYPTION_KEY.

After /ready succeeds, test every destination before removing the previous key:

daoflow backup destination test --id <destination-id>

Remove DAOFLOW_PREVIOUS_BACKUP_DESTINATION_ENCRYPTION_KEY and restart only after all destination tests pass.

Recovery Bundles And Database Backups​

Recovery bundles and database backups contain encrypted database state or ciphertext only. They must never contain ENCRYPTION_KEY, DAOFLOW_BACKUP_DESTINATION_ENCRYPTION_KEY, or DAOFLOW_PREVIOUS_BACKUP_DESTINATION_ENCRYPTION_KEY. Keep these keys in an external secret store. Restoring a bundle requires the matching external keys: use the old destination key for state from before a successful rotation and the new destination key for state from after it.

Control-plane Recovery Procedure​

Before an incident, configure DAOFLOW_RECOVERY_ENCRYPTION_KEY in an external secret manager and create a verified bundle:

daoflow backup recovery plan --destination <destination-id> --json
daoflow backup recovery run --destination <destination-id> --dry-run --json
daoflow backup recovery run --destination <destination-id> --yes
daoflow backup recovery list --json

Use daoflow backup recovery inspect --bundle <bundle-id> --json to capture the application version, schema version, object paths, checksums, required secret names, and isolated verification evidence. A queued or running bundle is not a recovery point. Keep failed records and follow their reported next steps.

During disaster recovery, recover the external recovery key before attempting to read the sidecar manifest. The sidecar is deliberately deterministic so it can be found even when the original DaoFlow database catalog is lost. Live restore into a clean installation is a separate approval-gated procedure; do not write a bundle into a production database as an incident shortcut.

Offline clean-install restore​

The offline restore contract takes a local bundle, its signed manifest, and an external secrets file. Set the file mode before using it:

chmod 600 /secure/daoflow-recovery.env

The file needs BETTER_AUTH_SECRET, ENCRYPTION_KEY, DAOFLOW_RECOVERY_ENCRYPTION_KEY, any manifest-required optional key, DAOFLOW_RECOVERY_VERIFY_EMAIL, and DAOFLOW_RECOVERY_VERIFY_PASSWORD.

Plan against a new database and capture the returned plan hash:

daoflow backup recovery restore \
  --dir /srv/daoflow-recovery \
  --bundle ./bundle.dfr \
  --manifest ./latest.json \
  --external-secrets /secure/daoflow-recovery.env \
  --database-name daoflow_recovery_20260718 \
  --dry-run \
  --json

Run only that exact plan by passing its hash to --confirm and including --yes:

daoflow backup recovery restore \
  --dir /srv/daoflow-recovery \
  --bundle ./bundle.dfr \
  --manifest ./latest.json \
  --external-secrets /secure/daoflow-recovery.env \
  --database-name daoflow_recovery_20260718 \
  --confirm <exact-plan-hash-from-dry-run> \
  --yes \
  --json

This flow uses a new database only and retains the original database and configuration. A post-start verification failure rolls the configuration back automatically. If the target fails, record the failure and clean up only that failed target database before retrying; retain the original database, configuration, bundle, signed manifest, and external secrets file.

Deployments Fail Or Stall​

Use the deployment ID from the dashboard or CLI response:

daoflow logs --deployment <deployment-id> --json
daoflow status --json
daoflow doctor --json

Check:

  • target server SSH connectivity
  • Docker and Docker Compose availability on the managed host
  • whether the Compose deploy required a context upload and that the staging workspace is writable
  • whether the failure is in plan generation, artifact staging, Docker execution, or post-start health
  • whether the dashboard deployment details show a difference between declared config, frozen deployment input, and last observed live state

From the dashboard:

  1. open the failed service or deployment record
  2. expand the deployment details
  3. copy or download the deployment state artifact JSON
  4. compare the frozen deployment input with the live runtime section before changing anything

If Temporal mode is enabled, also inspect:

docker compose logs --tail=200 temporal temporal-ui daoflow

Emergency fallback:

  1. set DAOFLOW_ENABLE_TEMPORAL=false in .env
  2. docker compose up -d daoflow

That returns the system to the legacy in-process worker while you investigate Temporal separately.

Backups Fail​

daoflow backup list --json
daoflow backup destination test --id <destination-id>
daoflow backup run --policy <policy-id> --yes

Most backup failures reduce to one of:

  • destination credentials or bucket and path permissions
  • SSH or Docker access to the target host
  • not enough disk space in the staging or destination path

Failed backup runs are preserved as first-class records. Do not delete them until you have captured their error detail.

Restore Or Verification Fails​

daoflow backup restore --backup-run-id <run-id> --yes
daoflow backup verify --backup-run-id <run-id> --yes
daoflow backup download --backup-run-id <run-id> --json

Current product behavior is artifact-oriented:

  • restore requests resolve the backup run and download the artifact from the configured destination
  • success or failure is recorded in restore metadata, audit entries, and events
  • application-specific volume or database rehydration may still require manual operator steps

If you need manual recovery today:

  1. use daoflow backup download --backup-run-id <run-id> --json to discover the artifact path
  2. use your storage backend tooling or rclone to fetch the artifact
  3. restore the data with the application or database-specific procedure
  4. record the manual action in your incident notes

Compose State Recovery​

When a Compose-backed service looks wrong but the host is still reachable:

  1. open the service Compose tab to copy or download the DaoFlow-managed override layer
  2. open the latest deployment details and export the deployment state artifact JSON
  3. compare the declared config, frozen deployment input, and live runtime sections
  4. only then fall back to host-level docker compose ps, docker inspect, or manual file inspection

This keeps DaoFlow Compose-first while still giving operators a visible escape hatch into the exact state the control plane believes it manages.

Upgrade Regression​

If a newly pulled DaoFlow image regresses:

  1. pin DAOFLOW_VERSION back to the previous known-good tag
  2. docker compose pull && docker compose up -d
  3. if the database schema changed incompatibly, restore your pre-upgrade database backup before bringing the older image back online

See Upgrading for the normal upgrade path.

Escalation Checklist​

  • Captured docker compose ps
  • Captured logs from daoflow and the affected dependency
  • Recorded the failing deployment, backup, or restore ID
  • Confirmed whether Temporal mode was enabled
  • Verified whether the incident is control-plane-local or remote-target-specific