ETL Lead Tester - Databricks & PySpark

Position - ETL Lead Tester | Data Quality Lead | Databricks Testing Lead

Experience - 12+ Years

Location - Remote only on w2

Job Summary

We are seeking an experienced ETL Lead Tester with strong expertise in Databricks, PySpark, ETL/Data Warehouse Testing, and Cloud Data Platforms. The ideal candidate will lead end-to-end testing of enterprise data pipelines, validate large-scale data transformations, and ensure data quality across modern lakehouse architectures. Experience with Azure Databricks or AWS Databricks environments is highly preferred.

Key Responsibilities

• Lead ETL and Data Warehouse testing activities across multiple projects.
• Validate data ingestion, transformation, and loading processes developed using Databricks and PySpark.
• Design and execute end-to-end test strategies for large-scale data migration and modernization initiatives.
• Develop SQL-based validation scripts for source-to-target data verification.
• Validate batch and incremental data loads within cloud-based lakehouse architectures.
• Perform reconciliation of structured and semi-structured datasets.
• Validate Delta Lake/Iceberg tables, partitioning, and data quality rules.
• Execute regression, integration, functional, and system testing for enterprise data platforms.
• Collaborate with Data Engineers, Solution Architects, Business Analysts, and Product Owners.
• Participate in Agile ceremonies including Sprint Planning, Daily Standups, and Retrospectives.
• Mentor junior QA engineers and establish enterprise testing standards.
• Prepare test plans, test cases, traceability matrices, and test summary reports