Lead AWS Data Engineer

Title: Lead AWS Data Engineer

Experience: 15 Years

Job Type: Contract

Location: Remote

Key Responsibilities:

• Lead the design, development, and implementation of scalable, enterprise-grade data engineering solutions on AWS.
• Design and develop data lake architectures using Apache Iceberg, including table management, schema evolution, partition evolution, and ACID transactions.
• Develop and optimize large-scale data transformation and processing jobs using PySpark / Apache Spark.
• Build and maintain ETL/ELT pipelines using AWS Glue, including Glue Jobs, Data Catalog, and integration with Iceberg tables.
• Develop serverless data processing and event-driven solutions using AWS Lambda.
• Design and implement complex workflow orchestration using AWS Step Functions, including parallel execution, retries, error handling, and integration with Glue and Lambda.
• Integrate AWS services to build reliable, scalable, and fault-tolerant data pipelines.
• Optimize PySpark jobs, Glue workloads, and Iceberg table operations for performance and cost efficiency.
• Implement data quality checks, schema validation, monitoring, logging, and recovery mechanisms.
• Troubleshoot complex production issues across Iceberg, PySpark, Glue, Lambda, and Step Functions.
• Establish best practices for data architecture, code quality, security, governance, and deployment.
• Lead technical design reviews, code reviews, and architecture discussions.
• Mentor engineers, provide technical leadership, and collaborate with architects, DevOps teams, and business stakeholders.
• Drive root-cause analysis and resolution of critical data pipeline failures in production environments.

Qualifications:

• 15 years of overall IT experience, with extensive experience in Data Engineering.
• Strong hands-on experience designing and implementing data engineering solutions on AWS.
• Proven hands-on experience with Apache Iceberg, including Iceberg table architecture and data lakehouse concepts.
• Strong experience developing distributed data processing solutions using PySpark / Apache Spark.
• Extensive hands-on experience with AWS Glue for ETL development and data processing.
• Strong hands-on experience with AWS Lambda for serverless computing and event-driven integrations.
• Proven experience implementing workflow orchestration using AWS Step Functions.
• Experience integrating Glue, Lambda, Step Functions, and Iceberg into end-to-end data pipelines.
• Strong understanding of data lakes, lakehouse architecture, ETL/ELT, and distributed computing.
• Experience with data partitioning, schema evolution, incremental processing, and performance optimization.