Senior Data Engineer

Job Title: Senior Data Engineer
Experience: 10+ Years
Employment Type: Contract W2 only

Job Summary

We are seeking a highly skilled Senior Data Engineer with 10+ years of experience in designing, developing, and maintaining scalable, high-performance data engineering solutions. The ideal candidate will have strong expertise in data architecture, ETL/ELT development, cloud data platforms, data modeling, and big data technologies. This role requires hands-on technical leadership, excellent problem-solving skills, and the ability to collaborate with architects, data scientists, business stakeholders, and cross-functional engineering teams to deliver reliable, secure, and cost-effective data solutions.

Key Responsibilities

• Design, develop, and maintain scalable, reliable, and high-performance data pipelines for batch and real-time data processing.

• Architect and implement end-to-end ETL/ELT workflows using modern data engineering tools and frameworks.

• Build and optimize data solutions on cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.

• Develop complex SQL queries, stored procedures, data transformations, and data models to support analytics and reporting requirements.

• Work with cloud data warehouses and lakehouse platforms such as Snowflake, Azure Synapse Analytics, Databricks, and Microsoft Fabric.

• Implement data ingestion, transformation, integration, and orchestration processes using tools such as Azure Data Factory, Apache Airflow, or equivalent technologies.

• Develop data processing and automation solutions using Python, PySpark, SQL, and other relevant programming languages.

• Implement incremental data loading, Change Data Capture (CDC), data replication, and real-time streaming solutions.

• Design and maintain data lakes, data warehouses, data marts, and enterprise data platforms.

• Establish data quality checks, validation frameworks, data lineage, metadata management, and monitoring mechanisms.

• Optimize data pipelines and queries for performance, scalability, reliability, and cost efficiency.

• Implement data security, access controls, encryption, governance, and compliance standards.

• Collaborate with data architects, business analysts, data scientists, BI developers, and application teams to translate business requirements into technical solutions.

• Implement CI/CD pipelines, version control, automated testing, and Infrastructure as Code practices for data engineering workflows.

• Troubleshoot complex production issues, identify root causes, and implement permanent solutions.

• Mentor junior and mid-level engineers, conduct code reviews, and establish engineering best practices.

• Lead technical discussions, contribute to architectural decisions, and drive continuous improvement across data engineering initiatives.

Required Technical Skills

1. Programming and Query Languages

• Advanced SQL, including complex joins, CTEs, window functions, query optimization, and stored procedures.

• Python for data processing, automation, API integration, and ETL/ELT development.

• PySpark and Apache Spark for large-scale distributed data processing.

2. Cloud Platforms

• Strong hands-on experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud Platform.

• Experience with cloud storage, data integration services, compute resources, identity management, and monitoring.

• Knowledge of cloud security, scalability, availability, and cost optimization.

3. Data Engineering and Integration

• Strong experience building enterprise-grade ETL/ELT pipelines.

• Expertise in batch processing, incremental loads, CDC, and streaming data integration.

• Experience with Azure Data Factory, Apache Airflow, AWS Glue, or similar orchestration tools.

• Familiarity with REST APIs, file-based ingestion, database integration, and event-driven architectures.

4. Data Warehousing and Lakehouse

• Experience with Snowflake, Azure Synapse Analytics, Databricks, Microsoft Fabric, Amazon Redshift, or equivalent platforms.

• Strong understanding of dimensional modeling, star and snowflake schemas, data marts, and data warehouse architecture.

• Experience with Delta Lake, data lakes, lakehouse architecture, partitioning, and schema evolution.

5. Big Data and Streaming Technologies

• Experience with Apache Spark and PySpark.

• Knowledge of Apache Kafka, Azure Event Hubs, AWS Kinesis, or similar streaming technologies.

• Understanding of distributed computing, parallel processing, and large-scale data transformation.

6. DevOps and DataOps

• Experience with Git, GitHub, GitLab, Azure DevOps, or equivalent version control and CI/CD tools.

• Knowledge of automated testing, deployment pipelines, workflow monitoring, and environment management.

• Familiarity with Docker, Kubernetes, Terraform, or other infrastructure automation tools is an advantage.

Required Experience and Qualifications

• 10+ years of overall experience in data engineering, data integration, data warehousing, or related technical roles.

• Strong hands-on experience designing and implementing complex, enterprise-scale data pipelines.

• Proven experience working with large datasets and optimizing data processing workloads.

• Experience delivering cloud-based data engineering solutions in production environments.

• Strong understanding of data architecture, data modeling, data governance, and data quality principles.

• Experience handling production support, performance tuning, and troubleshooting complex data issues.

• Demonstrated ability to lead technical initiatives and collaborate with cross-functional teams.

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

• Experience with Snowflake features such as Streams, Tasks, Dynamic Tables, and performance tuning.

• Experience with Microsoft Fabric, Azure Databricks, or Azure Synapse Analytics.

• Knowledge of metadata-driven pipelines, data lineage, and automated data quality frameworks.

• Experience implementing data mesh, domain-oriented data platforms, or modern lakehouse architectures.

• Familiarity with data catalog and governance tools such as Microsoft Purview or equivalent platforms.

• Experience with machine learning data pipelines and feature engineering workflows.

• Relevant cloud certifications in Azure, AWS, Google Cloud Platform, or Databricks are a plus.