Disaster recovery · Production operations
Recovering a production ClickHouse cluster from S3 backups
Restored a five-shard production cluster from a backup exceeding 7 TiB and later formalized full and incremental S3 backup routines.
- Organization
- Jet Games FZ-LLC
- Period
- 2024–2026
Context
A production incident required restoring a five-shard ClickHouse analytics cluster from a backup of more than 7 TiB.
Recovery
Performed the recovery hands-on with clickhouse-backup; backups were stored in Hetzner Object Storage (S3-compatible). Validated that the cluster was restored and back in operation. By my recollection the recovery took about 1–1.5 hours.
Follow-up
Established a recurring backup routine: weekly full and daily incremental backups to S3 with create_remote.
Outcome
Production operation restored; backup procedures made systematic.
Trade-offs and limitations
The root cause of the original incident is not recorded in the available notes and is not described here. The 1–1.5 hours is a recollection, not a measurement: where the clock started and whether it covers the full 7+ TiB restore is still to be confirmed. Scheduled test restores and a calculated recovery point are not claimed.
Technologies
- ClickHouse
- clickhouse-backup
- S3
- Hetzner Object Storage
- Linux