ETL Lead - Software Engineering

Position:  Lead - Software Engineering

Location: Bellevue/ Frisco

Duration: 6 Months 

Work Type: Onsite

Job Description:

DATA PIPELINE DEVELOPMENT

·                  Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex).

·                  Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion).

·                  Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms.

·                  Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency.

·                  Provide technical leadership for multi-source, high-volume data integration pipelines.

PLATFORM & TOOLING

·                  Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics.

·                  Define best practices for:

·                  o Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency)

·                  o Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization)

·                  o dbt (modular design, testing frameworks, CI/CD integration, reusable components)

·                  Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring.

·                  Evaluate and drive tooling strategy and platform standardization across teams.

CLOUD INFRASTRUCTURE

·                  Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents.

·                  Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning.

·                  Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation).

·                  Ensure platforms are scalable, secure, and production-ready.

LANGUAGES & FRAMEWORKS

·                  Provide deep technical leadership in:

·                  o Advanced SQL (query tuning, execution optimization, complex transformations)

·                  o Python / PySpark for distributed data processing

·                  Guide teams on best practices, reusable frameworks, and performance optimization.

·                  Oversee development standards for Spark, Scala (where applicable), and automation scripting.

STREAMING & REAL-TIME

·                  Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming.

·                  Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance.

·                  Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting.

DATA QUALITY & TESTING

·                  Establish enterprise frameworks for data quality, validation, and observability.

·                  Define standards for:

·                  o Automated testing (unit, integration, regression)

·                  o Data validation (completeness, accuracy, consistency)

·                  o Data quality tools (dbt tests, Great Expectations, custom frameworks)

·                  Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines.

·                  Drive proactive data quality and governance practices across teams.

DATA MODELING SUPPORT

·                  Interpret and implement architect-defined enterprise data models (star, snowflake, data vault).

·                  Provide guidance on:

·                  o SCD (Type 1/2) strategies

·                  o Partitioning, clustering, and performance optimization

·                  Collaborate with architects to evolve scalable and reusable data models.

·                  Support semantic layer enablement for analytics and reporting.

DEVOPS & ENGINEERING PRACTICES

·                  Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps).

·                  Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines).

·                  Standardize environment promotion (dev → QA → prod) and release management.

·                  Drive adoption of engineering excellence practices including reusable frameworks and templates.

SECURITY & GOVERNANCE

·                  Lead implementation of enterprise-grade security and governance controls:

·                  o RBAC, row/column-level security

·                  o PII and CPNI compliance (TISS-310)

·                  Define standards for secrets management and secure pipeline design.

·                  Ensure data lineage, auditability, and compliance readiness across platforms.

FINANCE DOMAIN KNOWLEDGE

·                  Deep understanding of finance and revenue data domains, including:

·                  o Billing and revenue systems

·                  o GL structures and financial reporting

·                  o Revenue recognition and reconciliation

·                  o Period-end close cycles

·                  Guide engineering teams on accurate implementation of finance logic.

·                  Ensure high data integrity standards for regulated financial data.

SOFT SKILLS & COLLABORATION

·                  Act as a technical leader and escalation point across engineering teams.

·                  Partner with architects, product managers, analysts, and business stakeholders.

·                  Drive cross-team alignment and solution consistency.

·                  Communicate complex technical topics clearly to both technical and non-technical audiences.

·                  Lead incident reviews and ensure continuous improvement.

PRINCIPAL-LEVEL EXPECTATIONS

·                  Own and drive enterprise-level data engineering strategy and execution.

·                  Lead delivery of large, complex, multi-domain data platforms.

·                  Mentor senior engineers and define technical direction for the team.

·                  Drive tooling, architecture, and platform decisions across programs.

·                  Identify and lead technical debt reduction and modernization initiatives.

·                  Establish best practices, reusable components, and platform standards at scale.

·                  Influence cross-functional teams and leadership decisions on data platform strategy.