Senior Data Engineer (Google Cloud Platform & Python)

Job Description:

Required Core Skills:

Python Engineering: 5+ years of hands-on experience building modular, reusable, production-grade Python applications, data pipelines, and processing frameworks.

Advanced BigQuery & SQL: Advanced proficiency in BigQuery SQL, including complex multi-table joins, analytical window functions, CTEs, incremental MERGE statements, and query execution plan tuning.

Airflow / Google Cloud Platform Cloud Composer: Deep expertise in developing production Airflow DAGs, runtime/dynamic configurations, task dependencies, failure handling, and production backfill management.

Google Cloud Platform Infrastructure: Strong operational experience across Google Cloud Platform core services: BigQuery, Cloud Storage (GCS), Pub/Sub messaging, and IAM role management.

Testing & Release Management: Proven experience with pytest, unit testing, mocking, Git version control, CI/CD pipeline integration, and automated deployment validation across multi-tier enterprise environments.

Preferred:

Lumi / ERIRP Framework: JSON-driven pipeline architecture, runtime configuration, reusable hooks, orchestration patterns, safety gates, validation controls, and operational notifications.

Data Reliability & Financial Processing: Idempotent reruns, late-arriving data handling, deduplication strategies, audit and reconciliation checks, data-quality controls, and financial precision/rounding.