Senior Data Engineer

Job Summary

We are seeking a Senior Data Engineer with strong hands-on experience in Python and Apache Spark to design, develop, and optimize scalable data pipelines and distributed data processing solutions. The ideal candidate will have expertise in ETL/ELT development, data modeling, cloud technologies, and software engineering best practices. This is a hands-on role requiring the ability to build production-grade data solutions in an Agile environment.

Required Skills

• 10+ years of Data Engineering or Software Engineering experience.
• Strong hands-on Python experience (Required).
• Strong hands-on Apache Spark / PySpark experience (Required).
• Experience building scalable ETL/ELT pipelines.
• Strong knowledge of data modeling, SQL, distributed computing, and data warehousing.
• Experience with pipeline orchestration (Airflow, Databricks Workflows, or similar).
• Experience developing and consuming REST APIs.
• Strong understanding of software engineering principles, including object-oriented programming, design patterns, data structures, and algorithms.
• Experience with Git, CI/CD pipelines, automated testing, and Agile/Scrum.
• Excellent analytical, troubleshooting, and communication skills.

• AWS experience (S3, EC2, Glue, Lambda, EMR).
• Databricks experience and/or Apache Spark certification.
• Experience with Docker, Kubernetes, Delta Lake, Snowflake.
• Experience with R or SAS is a plus.

Key Responsibilities

• Design, develop, and optimize enterprise data pipelines using Python and Apache Spark.
• Build scalable ETL/ELT solutions for large datasets.
• Design and maintain data models supporting analytics and reporting.
• Optimize Spark jobs for performance, scalability, and reliability.
• Develop APIs and data services supporting enterprise applications.
• Implement automated testing, CI/CD, and data quality controls.
• Collaborate with architects, software engineers, data scientists, and business stakeholders to deliver high-quality data solutions.
• Participate in code reviews, technical design discussions, and production support.

Must-Have Skills (Knockout Requirements)

• Python
• Apache Spark / PySpark
• ETL/ELT Development
• SQL
• Data Modeling
• Distributed Computing
• CI/CD
• Agile Development