From legacy batch jobs to a production analytics platform

Context

The analytics function already existed — the work was to build its engineering foundation. The stack consisted primarily of older Airflow jobs pulling MySQL data into PostgreSQL in batches. Jobs were managed through FTP rather than a normal Git-based delivery workflow; critical data services lacked a dependable backup and operating model.

Constraints

The backend could not be relied on to emit application events on its own. Architecture changes had to be approved by the technical lead, and the new platform had to serve a working analytics team without making DevOps the single manual gatekeeper for every operation.

Decisions and implementation

Proposed CDC as a pragmatic answer to the available source systems: MySQL → Debezium → Kafka with Avro / Schema Registry → clustered ClickHouse, with a raw-data layer and downstream transformation and serving. Moved services to Git-managed CI/CD and reproducible deployment; modernized Airflow (Docker and Kubernetes deployments both existed; Kubernetes workers scaled with HPA); integrated Lakekeeper, OpenMetadata and Keycloak. Separated the analytics infrastructure into its own servers, networks and cloud resources; planned quarterly roadmaps and capacity and selected providers within budget; added Ansible with an inventory source of truth, backups, monitoring and runbooks.

Operating model

Worked with a technical lead and other DevOps teams. Established release coordination, on-call, postmortems and documented operational procedures — the goal was a platform that stays supportable when its principal engineer is unavailable.

Outcome

Analytics moved from scheduled batch dependency toward near-real-time ingestion, and gained a managed platform with clear ownership of production incidents, deployments and recovery.

Trade-offs and limitations

CDC was chosen because of the backend constraints, not because CDC is better in general. ReplacingMergeTree deduplicates eventually: depending on the query, a correct current state needs FINAL, argMax or another approach. The pipeline is not end-to-end exactly-once. Incident and recovery times were not measured before and after.

Technologies

• Kafka
• ClickHouse
• Airflow
• Kubernetes
• Debezium
• Linux
• GitLab CI
• Ansible
• Keycloak
• Backups