Data Analyst Engineer

Key Responsibilities

Data Validation & Verification

·        Validate transformation logic against business rules and documented specifications

·        Perform source-to-target data reconciliation — verifying completeness, accuracy, and consistency

·        Identify data anomalies, silent failures, and drift in pipeline outputs

·        Build and maintain automated data validation suites that execute as part of pipeline runs

·        Conduct periodic data audits beyond automated checks

QA Process Definition & Governance

·        Define acceptance criteria for each ETL pipeline and transformation step — establishing what "correct" means in measurable, testable terms

·        Define Definition of Done (DoD) for all data deliverables — specifying when a pipeline output is considered production-ready

·        Create and maintain data quality test plans covering functional correctness, edge cases, regression, and performance

·        Design test cases for new transformations before they reach production (shift-left testing)

·        Establish data quality SLAs — freshness, completeness, and accuracy thresholds

·        Define entry and exit criteria for pipeline releases — what must pass before promoting changes

·        Maintain a defect taxonomy — categorizing data issues (schema drift, logic errors, source issues, timing issues) for root cause tracking and trend analysis

·        Define sign-off workflows — who approves what before data reaches the UI layer

Test Strategy & Frameworks

·        Design the overall data testing strategy:

o   Unit tests for individual transformation logic

o   Integration tests for end-to-end pipeline correctness

o   Regression tests for ongoing stability

o   Smoke tests for post-deployment verification

o   Negative tests for resilience (nulls, duplicates, out-of-range values, schema violations)

·        Build reusable, parameterized validation patterns applicable across multiple pipelines

·        Implement data contract validation — ensuring upstream sources meet agreed schemas and value constraints

·        Integrate quality gates into CI/CD pipelines for automated release blocking

Documentation & Traceability

·        Document expected behavior for each transformation (input, logic, expected output)

·        Maintain data quality runbooks — investigation and resolution procedures for common issues

·        Create traceability matrices linking business requirements to test cases to pipeline outputs

·        Document known data limitations, assumptions, and technical debt

·        Maintain a living catalog of data quality rules and their rationale

Monitoring, Observability & Reporting

·        Build data quality dashboards tracking pass/fail rates, trend analysis, and SLA compliance

·        Configure alerting for data quality threshold breaches

·        Track and report data quality metrics (DQ score, defect density, mean time to detect, mean time to resolve)

·        Conduct data quality retrospectives — identifying gaps and improving coverage

Collaboration

·        Participate in pipeline design reviews — flagging quality risks early in development

·        Work with product and business stakeholders to translate vague requirements into testable assertions

·        Collaborate with data engineers on pipeline improvements driven by quality findings

·        Support UI/frontend teams in verifying rendered data correctness

Required Qualifications

Technical Skills

Experience

·        6-10 years of combined experience in data engineering, data QA, or analytics engineering

·        Has owned data quality for at least one production system end-to-end (not just contributed)

·        Has defined acceptance criteria and quality gates that blocked defective releases

·        Has built automated validation suites that caught real production issues

·        Comfortable reading and reasoning about pipeline code (transformation logic, orchestration DAGs)

·        Experience working with curated/aggregated datasets that serve application UIs

·        Familiarity with AWS Glue, Redshift Spectrum, and AWS data pipeline services

Preferred Experience

·        Experience with BDD-style data testing (Given/When/Then for data transformations)

·        CI/CD integration for data quality — automated gates in deployment pipelines

·        Experience defining and tracking data SLAs/SLOs

·        Knowledge of regulatory or compliance data requirements

·        Performance testing for pipelines — verifying latency and throughput

·        Exposure to chaos engineering for data — intentionally injecting bad data to test resilience

·        Experience with pipeline orchestration tools (Glue Orchestrator, Step Functions, Airflow)

·        Experience with IAM permissions and Lake Formation access controls for data governance