Senior Data Engineer

Looking for local candidates who are ready/willing to come down for a F2F interview.

Please refrain from applying to this job who are not local and not ready/willing to come down for a F2F interview

Job Title: Senior Data Engineer

Experience: 10 years

Job Summary

We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data platforms that support analytics, reporting, and machine learning initiatives. The ideal candidate should have strong experience with Python, Apache Spark, Scala, Linux, AWS cloud technologies, and the Apache big data ecosystem.

Key Responsibilities

Design, develop, and optimize scalable ETL/ELT data pipelines using Python, Scala, and Apache Spark.

Build and maintain batch and real-time data processing solutions.

Develop data ingestion frameworks for structured, semi-structured, and unstructured data from multiple sources.

Design and optimize data lakes and data warehouses on AWS.

Implement data quality checks, validation, monitoring, and error handling.

Work closely with Data Scientists, BI Developers, and Business Analysts to deliver high-quality data solutions.

Optimize Spark jobs for performance, memory utilization, and scalability.

Develop reusable data processing frameworks and automation scripts.

Monitor and troubleshoot production data pipelines.

Ensure data governance, security, and compliance with organizational standards.

Collaborate with DevOps teams to implement CI/CD pipelines for data engineering workflows.

Create technical documentation and maintain data dictionaries.