W2-Fulltime -Azure Data Engineer (Azure Data Factory & Azure Data Lake)

Role: Azure Data Engineer (Azure Data Factory & Azure Data Lake)

Location: Remote

Duration: FTE

Must have:

• Hands-on experience with Azure Data Factory for designing, developing, and maintaining data pipelines.
• Expertise in Azure Data Lake Storage and enterprise-scale data lake management.
• Experience developing ETL/ELT workflows to ingest data from databases, APIs, flat files, and cloud applications.
• Knowledge of Mapping Data Flows, SQL, and Spark for data transformation.
• Experience with incremental data loads, Change Data Capture (CDC), and data validation.
• Ability to monitor, troubleshoot, and optimize data pipelines for performance and reliability.
• Experience with data modeling, data warehousing, and cloud data platform security and governance.
• Experience supporting migration of on-premises data solutions to Azure cloud environments.
• Niche Skill Combination: Azure Data Factory + Azure Data Lake Storage + ETL/ELT + Mapping Data Flows, SQL, and Spark. Candidates who combine these data engineering skills with CDC, pipeline optimization, and Azure cloud migration experience are particularly relevant.

Position Overview:

Build and maintain scalable, cloud-based data integration and storage solutions using Microsoft Azure services. This role focuses on developing reliable data pipelines, managing enterprise-scale data lakes, implementing ETL/ELT workflows, and supporting cloud-based analytics and data warehousing solutions.

The ideal candidate brings expertise in Azure Data Factory, Azure Data Lake Storage, data transformation technologies, and cloud data engineering practices.

Responsibilities:

·        Design, develop, and maintain data pipelines using Azure Data Factory.

·        Build and manage enterprise-scale data lakes using Azure Data Lake Storage.

·        Develop ETL/ELT workflows to ingest data from multiple sources, including databases, APIs, flat files, and cloud applications.

·        Implement data transformation processes using Mapping Data Flows, SQL, and Spark technologies.

·        Monitor, troubleshoot, and optimize data pipelines for performance and reliability.

·        Configure incremental data loads, Change Data Capture (CDC), and data validation processes.

·        Collaborate with data analysts, business stakeholders, and development teams to understand data requirements.

·        Design data models and support data warehousing solutions.

·        Implement security, governance, and access controls for cloud data platforms.

·        Create technical documentation and maintain data architecture standards.

·        Support the migration of on-premises data solutions to Azure cloud environments.

Requirements:

·        Expertise in Azure Data Factory for data pipeline development and maintenance.

·        Experience with Azure Data Lake Storage and enterprise-scale data lakes.

·        Experience developing ETL/ELT workflows for data ingestion from multiple sources.

·        Knowledge of Mapping Data Flows, SQL, and Spark technologies for data transformation.

·        Experience with incremental data loads, Change Data Capture (CDC), and data validation.

·        Ability to troubleshoot and optimize data pipelines for performance and reliability.

·        Knowledge of data modeling, data warehousing, and cloud data platform security and governance.

·        Experience supporting on-premises data migration to Azure cloud environments.