ChronoOpsDocumentation
Browse and search documentation

Backup recovery and upgrades

Distinguish database-only and full backups, protect encryption materials and establish a verified recovery path before upgrading.

For: Administrators responsible for maintenance and recoveryReviewed:
On this page

What to back up

ContentHandling
PostgreSQL business and audit dataConsistent database backup with restore verification
Job output, recordings and release artifactsBack up the relevant volumes or object storage
Active configuration, keys and licenseArchive separately with encryption, apart from database backups
Trusted Agent release catalogInclude it in recovery planning if signed upgrades are used

Choose the backup mode

A full backup uses maintenance mode. Enter a maintenance window and stop applications that write to the database and related volumes before running it, to avoid inconsistent database and file snapshots.

Database-only mode takes an online database snapshot but skips data volumes. It cannot by itself restore job output, recordings or release artifacts and is not a full backup.

Online database-only backup
CHRONOOPS_BACKUP_OUTPUT_DIR=/var/backups/chronoops \
CHRONOOPS_BACKUP_CONSISTENCY_MODE=database-only \
  bash scripts/delivery/backup/chronoops-backup.sh

Verify recoverability

  1. Check the manifest, checksums, database archive and applicable volumes. Keep a copy outside the source host’s failure domain.
  2. Confirm encrypted archives of keys, active configuration and the license are available to authorized maintainers only.
  3. Use the delivered restore tool’s verification mode and check migration versions, key tables and record counts in an isolated environment.
  4. After a recovery drill, test sign-in, assets, audit queries and a read-only job. Record actual timing instead of borrowing recovery targets from another environment.

Upgrade checklist

  1. Read the release notes and verify compatibility with the database, Agents and external components.
  2. Create and verify a fresh backup. Record current application versions, images and configuration, and assign a maintenance window and rollback owner.
  3. Verify the new package signature, review changes using the dry-run mode of tools/upgrade.sh, then follow its version-specific execution procedure.
  4. After upgrading, verify sign-in, assets, Agent status, read-only jobs and audit records before resuming schedules and external delivery.

When an upgrade fails

Preserve the failed step and current state before retrying. Do not repeatedly reinstall or reset keys. Application and database rollback are separate decisions; an older image does not automatically restore the old schema.

A production restore affects business data. Stop writers during maintenance, preserve the original database and volumes, validate restoration into new empty targets and then switch the service. Do not overwrite or delete the original state before verification.

Something differs from your environment?

Send your deployment version, page and a redacted description so we can investigate and update the guide.

weiwendi@aiops.red