Lead Data Engineer

Lead Data Engineer
Remote
Fulltime

Job Summary

We are seeking an experienced, technically strong Lead Data Engineer to design, develop, implement, and support enterprise-scale data engineering platforms and integration solutions.

The ideal candidate will bring deep hands-on experience in Python, PySpark, SQL, Apache Spark, Databricks, Snowflake, ETL/ELT, and modern Data Lakehouse architectures, along with the ability to guide engineering teams and drive technical delivery.

The role involves close collaboration with Data Architects, Data Engineering teams, AI/ML teams, business stakeholders, and technology leadership to deliver scalable, reliable, secure, and high-performance data solutions.

Key Responsibilities

1. Data Engineering & Pipeline Development

• Design, develop, and maintain scalable batch and real-time data pipelines using Python, PySpark, SQL, Spark, and ETL/ELT technologies.
• Build and optimize data ingestion, transformation, integration, and processing workflows across enterprise systems.
• Develop and maintain enterprise data warehouses, data lakes, databases, tables, and data processing systems.
• Write complex SQL queries and optimize database and pipeline performance.
• Troubleshoot production data issues and improve pipeline reliability, efficiency, and quality.

2. Data Architecture & Platform Engineering

• Design and implement data structures, schemas, data models, and integration frameworks.
• Work with modern cloud data platforms including Databricks, Snowflake, OCI, and similar technologies.
• Build and support Delta Lake, Lakehouse, and enterprise data warehouse solutions.
• Apply data engineering best practices to ensure scalability, performance, maintainability, and reliability.
• Evaluate technical designs, identify potential issues, and recommend improvements.

3. Technical Leadership & Delivery

• Provide technical guidance to data engineers and collaborate with Data Architects and engineering teams.
• Translate business requirements into technical specifications, implementation plans, and test plans.
• Coordinate technical activities across multiple teams and enterprise platforms.
• Support implementation, testing, deployment, and production stabilization.
• Manage competing technical priorities and ensure timely delivery of engineering solutions.

4. Data Governance, Quality & Security

• Collaborate with data governance, data quality, and information security teams.
• Implement data quality checks, validation processes, and monitoring frameworks.
• Support metadata management, data modeling, and data security requirements.
• Ensure pipelines meet applicable governance, security, and certification standards before production deployment.
• Ensure compliance with HIPAA regulations and appropriate handling of Protected Health Information (PHI).

5. Analytics, AI & Data Science Enablement

• Work closely with analytics, AI/ML, and data science teams to develop reliable datasets and data processing solutions.
• Prepare structured and unstructured data for predictive and prescriptive analytics.
• Support data transformation and integration requirements for machine learning and AI applications.
• Identify opportunities to improve automation, data processing, and analytical capabilities.

6. DevOps & DataOps

• Apply Agile, DevOps, and DataOps principles to data engineering development and delivery.
• Improve deployment processes, pipeline monitoring, validation, and production reliability.
• Support automation of engineering workflows and continuous improvement initiatives.

Required Qualifications

• 12+ years of relevant IT/data engineering experience, including at least 3 years in data modeling, schema design, and building and maintaining data processing systems.
• Strong hands-on programming experience with Python, PySpark, SQL, and Spark.
• Extensive experience designing, developing, and optimizing ETL/ELT pipelines.
• Experience with Databricks, Snowflake, Delta Lake, data warehouses, or comparable enterprise data platforms.
• Strong knowledge of data modeling, database development, data transformation, and large-scale data processing.
• Experience troubleshooting end-to-end data pipelines and resolving production data issues.
• Understanding of data governance, data quality, metadata, and security controls.
• Ability to collaborate with engineering, architecture, analytics, and business teams.
• Strong analytical, problem-solving, communication, and technical leadership skills.
• Bachelor's degree in Computer Science, Information Technology, or a related discipline preferred.