Airflow Architect – Big Data Systems

Job Summary

Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.

Key Responsibilities

• Architect and implement large-scale data orchestration frameworks using Apache Airflow.

• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.

• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.

• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.

• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.

• Lead cloud-based data platform implementations on AWS

• Optimize pipeline performance, scalability, reliability, and cost efficiency.

• Establish CI/CD processes and DevOps practices for Airflow environments.

• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.

• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.

Required Skills

Airflow & Data Engineering

• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.

• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.

• Hands-on expertise in ETL/ELT development and data pipeline automation.

Big Data Technologies

• Apache Spark / PySpark

• Hadoop ecosystem

• Apache Kafka

• Data Warehousing concepts

• Data Lakes and Lakehouse architectures

• NoSQL databases

Programming

• Python (advanced)

• SQL (advanced)

• Shell scripting

• Core Java (preferred)

Cloud Platforms

• AWS (EMR, Glue, S3, Redshift)

Preferred Qualifications

• 10+ years of experience in data engineering and big data platforms.

• 3+ years of architecture experience designing enterprise data solutions.

• Experience with Kubernetes, Docker, and cloud-native deployments.

• Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.

• Experience leading architecture reviews and technology strategy discussions.

Success Metrics

• Pipeline reliability and SLA adherence

• Workflow performance optimization

• Reduced operational incidents

• Improved data quality and observability

• Scalable and reusable orchestration frameworks