Quick overview
Job description
Data Engineer Lead - W2
As a Data Engineer Lead, you will play a pivotal role in designing, developing, and managing large-scale data pipelines and data lake patterns. You will lead a team of data engineers, driving the adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes. You will be responsible for building robust ETL/ELT pipelines, optimizing distributed processing workloads, and ensuring data quality, governance, and compliance.
Responsibilities:
- Execute creative software solutions, design, development, and technical troubleshooting with the ability to think beyond routine or conventional approaches to build solutions or breakdown technical problems.
- Drive team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes.
- Lead architecture and hands-on delivery of large-scale data pipelines (ETL/ELT) covering ingestion, transformation, validation, reconciliation, and publishing across curated layers.
- Design and operationalize data lake patterns, including partitioning strategies, data quality controls, lineage, governance, and reusable datasets/data products.
- Build and optimize distributed processing workloads using Apache Spark and modern storage/file formats (e.g., Parquet, Avro).
- Drive performance tuning across compute and storage (e.g., Spark tuning: shuffle, joins, caching, skew handling; and warehouse tuning where applicable).
- Implement engineering best practices: code quality, automated testing, CI/CD, observability (metrics/logs/traces), security-by-design, and operational readiness.
- Build and maintain Java-based services and components that support data workflows (e.g., ingestion services, orchestration helpers, APIs, data access layers).
- Develop REST APIs and integration components to enable downstream consumption and platform interoperability.
- Apply modern application engineering practices (clean architecture, SOLID principles, test automation, and secure coding practices).
- Apply knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
Requirements:
<>Required Skills:- Formal training or certification on software engineering concepts and 5+ years applied experience.
- Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.
- 10+ years of software engineering experience, with strong depth in data engineering/big data platforms.
- Strong hands-on development experience in Java plus proficiency in at least one data-focused language such as Python.
- Proven experience designing and building robust ETL/ELT pipelines and data integration frameworks.
- Strong experience with Apache Spark and distributed processing concepts (fault tolerance, partitioning, performance tuning).
- Strong understanding of data storage serialization and formats such as Parquet and Avro.
- Solid knowledge of data lake/lakehouse concepts and patterns (e.g., Medallion architecture), including data quality, metadata, and governance considerations.
- Experience working with cloud data warehouses such as Snowflake (loading/unloading patterns, performance basics) or equivalent.
- Ability to lead technical discussions, communicate clearly to varied stakeholders, and drive delivery in cross-functional environments.
- Experience with orchestration and workflow scheduling tools (e.g., Airflow, Dagster, Control-M, etc.).
- Exposure to streaming/event-driven processing (e.g., Kafka, Spark Structured Streaming) and incremental processing patterns (CDC, upserts).
- Experience with table formats / ACID layers (e.g., Delta Lake, Apache Iceberg, Hudi).
- Strong production operations mindset (on-call practices, SLAs/SLOs, observability, incident response).
- Experience in regulated environments (risk, audit, compliance, privacy, PII handling).
- Experience with cost optimization / FinOps practices for data platforms.
- Experience with working with Agentic AI / Gen AI solutions and capabilities will be an added advantage.
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- II

Senior Data Engineer / Software Engineer (SQL, Databricks, Java, Spark)Recently posted
Intuitive Insights LLC · Houston, TX · On-site · Posted 19 Nov 2026 🇺🇸
DockerOraclePL/SQL+18TechnologySalary not listed - GT
Senior Data Engineer with AIRecently posted
Goldenpick Technologies LLC · Dallas, TX · Hybrid · Posted 19 Nov 2026 🇺🇸
DockerNeo4jSQL+23TechnologySalary not listed - SS

Data Engineer with Autosys & StonebranchRecently posted
Softcom Systems · Weehawken Township, NJ · Hybrid · Posted 19 Nov 2026 🇺🇸
ShellETLNumPy+5TechnologySalary not listed - LO
Data Engineer – Cloud Security (AWS)Recently posted
Logicplanet, Inc. · Milpitas, CA · On-site · Posted 19 Nov 2026 🇺🇸
AWSEncryptionCDK+4TechnologySalary not listed - RD

Data Engineer (EDA)Recently posted
Randstad Digital · Irving, TX · Hybrid · Posted 19 Nov 2026 🇺🇸
SQLETLSnowflake+1Technology$69 - $70/hr - FL
Senior PostgreSQL DBARecently posted
Flexton Inc · Los Angeles, CA · Remote · Posted 19 Nov 2026 🇺🇸
MySQLOracleSQL+25TechnologySalary not listed - PS

Senior Snowflake Data Engineer (Informatica) Remote (Local)Recently posted
Paramount Software Solutions, Inc · Denver, CO · Remote · Posted 19 Nov 2026 🇺🇸
SQLGitETL+6TechnologySalary not listed - GA

Staff Data EngineerRecently posted Auto Apply
Gamechanger · GameChanger Remote - US · Remote · Posted 18 Nov 2026 🇺🇸
SQLAWSMLOps+8Technology$170k - $185k/yr - SP

Snowflake & Informatica Data EngineerRecently posted
Stellar Professionals LLC · United States · On-site · Posted 19 Nov 2026 🇺🇸
SQLGitETL+5TechnologySalary not listed - TH

Data EngineerRecently posted
The Hollister Group · Pittsburgh, PA · On-site · Posted 19 Nov 2026 🇺🇸
SQLAWSETL+7Technology$75k - $85k/yr - VN

Data EngineerRecently posted
Virtual Networx · Boston, MA · Hybrid · Posted 19 Nov 2026 🇺🇸
SQLETLDatabricks+4TechnologySalary not listed - EG

Data EngineerRecently posted
Eliassen Group · Pittsburgh, PA · On-site · Posted 19 Nov 2026 🇺🇸
JavaTechnologySalary not listed
