Lead Data Engineer with Java, Python, Spark & AWS - Onsite - W2 only

**** Webcam interview; *** Long term project with the possible contract to hire ***Linkedin Must*** Onsite ****

Job Description

Seeking an experienced Lead Data Engineer with strong expertise in Java, Python, AWS, ETL/ELT pipelines, and Apache Spark.

Key Responsibilities

• Lead architecture and development of large-scale ETL/ELT data pipelines.
• Design data ingestion, transformation, validation, and reconciliation workflows.
• Develop distributed data processing solutions using Apache Spark.
• Design and implement data lake and lakehouse architectures.
• Optimize Spark workloads, including joins, caching, partitioning, and performance tuning.
• Develop Java-based services and REST APIs.
• Implement automated testing, CI/CD, monitoring, and production support.
• Drive engineering best practices and mentor development teams.
• Promote enterprise-approved AI-assisted engineering practices.

Required Skills

• 10+ years of software engineering experience.
• Strong hands-on Java development.
• Python proficiency.
• ETL/ELT pipeline development.
• Apache Spark and distributed processing.
• AWS cloud experience.
• Parquet, Avro, and data lake/lakehouse architecture.
• Snowflake or equivalent cloud data warehouse experience.
• AI-assisted development tools and responsible AI practices.

Preferred Skills

• Airflow, Dagster, or Control-M.
• Kafka and Spark Structured Streaming.
• Delta Lake, Apache Iceberg, or Hudi.
• Data governance and data quality.
• Financial services or regulated industry experience.
• Agentic AI and Generative AI exposure.