Flagship cases with full write-ups, and shorter engineering notes.
Platform architecture · Infrastructure ownership
Jet Games FZ-LLC · 2024–2026
Took ownership of the analytics department's infrastructure, replaced manually managed data jobs with an automated and observable platform, and established reliable day-to-day operations.
SRE · Reliability engineering
UNICORN LLC · Jet Games FZ-LLC · 2021–2026
Designed monitoring architectures and connected SLO-based alerting, incident response, on-call rotation and release policy to engineering decisions.
Developer platform · Internal tooling
Jet Games FZ-LLC · 2024–2026
Built a Go service with a Slack UI, PostgreSQL state and the Proxmox API so developers could provision and manage game development environments themselves.
Kubernetes · Delivery engineering
UNICORN LLC (ujin.tech) · 2021–2024
Built and maintained a Kubernetes cluster with Kubespray, Calico, ingress-nginx and Ceph, while operating CI/CD for backend and Unity workloads.
Disaster recovery · Production operations
Jet Games FZ-LLC · 2024–2026
Restored a five-shard production cluster from a backup exceeding 7 TiB and later formalized full and incremental S3 backup routines.
Incident response · Root cause analysis
Jet Games FZ-LLC · Sep 2026
Traced a Tableau Custom SQL failure to ClickHouse response-format precedence and shipped a targeted connector workaround while upstream fixed the Java driver.
Shorter notes without a page of their own.
Databases & data · 22 Sep 2026
Upgraded two shards × two replicas from 25.12.4.35 to 26.8.4.11 LTS one node at a time (order 4 → 2 → 3 → 1), watching replicas, the replication queue and ingestion after each node. The three Keeper nodes stayed unchanged; compatibility settings bridged the versions.
Limits: A risk-reduction procedure; downtime was not measured.
Databases & data
MySQL → Debezium → Kafka → ClickHouse with Avro and Schema Registry, a custom Java SMT, and a move from self-hosted Kafka to DigitalOcean Managed Kafka, with connector monitoring.
Limits: errors.tolerance=all keeps the connector running by skipping records it cannot process — it is not a reliability guarantee. ReplacingMergeTree deduplicates eventually; nothing here is exactly-once.
Incidents & recovery
A custom Debezium converter passed synthetic tests but failed in production: real MySQL binlog string columns arrived as byte[], not the assumed String. Lesson: tests have to cover the real integration path and wire representation, not only idealized fixtures.
Limits: A lesson, not a shipped process change.
Databases & data
NVMe and RAID replacement with recovery, S3 backups, statement diagnostics, and a cron job whose repeated GRANTs blocked queries — replaced with default privileges.
Limits: Downtime of the hardware work was not measured.
Databases & data · 6 Aug 2026
Moved Tableau Server 2024.2 on Debian to Ubuntu 24.04 with Tableau 2025.3.8 through a .tsbak backup; fixed permissions, the public host, ODBC and ingress; consolidated several sites.
Limits: Downtime was not recorded.
Platform & infrastructure
Airflow 2.10.4 and 3.x side by side, on Docker Compose and Kubernetes, with Celery, images built in GitLab, SSO, Slack alerts, StatsD → Prometheus metrics and dbt through Cosmos.
Limits: HPA scales the Kubernetes workers; not every Airflow 3.x workload runs in Kubernetes.
Platform & infrastructure
OIDC for Airflow, OpenMetadata, Lakekeeper and JupyterHub, separate service-to-service credentials, and access granted and revoked automatically.
Incidents & recovery · 13 Aug 2026
Investigation note · unresolved
Code 209 SOCKET_TIMEOUT in a distributed query; the load pointed to heavy remote system.tables reads and catalogs of PostgreSQL-engine databases.
Limits: No confirmed final resolution.
Incidents & recovery
Recovered a server's RAID10 and bootloader through Hetzner rescue mode; expanded a ClickHouse RAID10 array.
Platform & infrastructure
Ansible with DataGerry as the inventory source of truth, Proxmox with Ceph, self-hosted GitLab with Gitaly and LFS, and access tooling in Go.