Lead Data Engineer

Position: Senior/Lead Data Engineer

Experience: 10+ Years
Employment Type: Contract W2 only

About the Role

We are looking for a highly experienced Data Engineer to design, build, and optimize scalable data platforms and pipelines. The ideal candidate has deep expertise across the modern data stack, strong architectural judgment, and the ability to lead data engineering initiatives end-to-end while mentoring junior engineers and collaborating closely with data science, analytics, and business teams.

Key Responsibilities

• Design, build, and maintain scalable, reliable ETL/ELT pipelines for structured and unstructured data
• Architect and optimize data warehouses, data lakes, and lakehouse solutions
• Lead the design of data models (dimensional, normalized, and denormalized) to support analytics and reporting
• Build and manage batch and real-time streaming data pipelines
• Ensure data quality, integrity, governance, and security across all pipelines and platforms
• Optimize pipeline performance, cost, and scalability across cloud environments
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Implement CI/CD practices for data pipelines and infrastructure-as-code
• Monitor, troubleshoot, and resolve production data pipeline issues (on-call/incident support as needed)
• Establish and enforce data engineering best practices, standards, and documentation
• Lead technical design reviews and mentor junior/mid-level data engineers
• Evaluate and integrate new tools/technologies to improve the data platform

Required Skills & Experience

• 10+ years of experience in data engineering, data warehousing, or related fields
• Strong programming skills in Python, Scala, or Java
• Expert-level SQL and experience with query optimization on large datasets
• Hands-on experience with big data technologies: Spark, Hadoop, Kafka, Hive
• Strong experience with cloud data platforms – AWS (Redshift, Glue, EMR, S3), Azure (Synapse, Data Factory, Databricks), or Google Cloud Platform (BigQuery, Dataflow, Dataproc)
• Experience with modern data warehouse/lakehouse platforms – Snowflake, Databricks, or similar
• Proficiency in orchestration tools – Airflow, Dagster, or similar
• Solid understanding of data modeling (Star/Snowflake schema, Data Vault) and dimensional design
• Experience with real-time/streaming architectures (Kafka, Kinesis, Flink, Spark Streaming)
• Strong knowledge of CI/CD, version control (Git), and infrastructure-as-code (Terraform/CloudFormation)
• Experience with containerization and orchestration (Docker, Kubernetes)
• Understanding of data governance, security, lineage, and compliance (GDPR/HIPAA as applicable)
• Proven experience architecting solutions from scratch and leading data engineering teams/projects
• Excellent problem-solving, debugging, and performance-tuning skills

Good to Have

• Experience with dbt (data build tool) for transformation workflows
• Exposure to MLOps and supporting ML pipelines/feature stores
• Experience with DataOps practices and data observability tools (Monte Carlo, Great Expectations)
• Knowledge of API development for data services
• Relevant cloud certifications (AWS Certified Data Analytics, Azure Data Engineer Associate, Google Cloud Platform Professional Data Engineer)
• Experience in a specific domain (Finance, Healthcare, Retail, etc. – customize as needed)

Educational Qualification

• Bachelor''s/Master''s degree in Computer Science, Data Engineering, Information Systems, or related field

Soft Skills

• Strong communication skills to work with cross-functional and non-technical stakeholders
• Leadership ability to mentor and guide junior engineers
• Ability to drive projects independently with minimal supervision
• Strong ownership mindset and attention to data quality/detail