Data Engineer

We are looking for an experienced Data Engineer to support a data migration and integration programme within a relatively low maturity Azure environment.

The role will focus on designing and building robust data pipelines, migrating data from a wide range of legacy and enterprise source systems, and establishing scalable data flows into Azure Data Lake and Databricks. A key part of the role will also involve integrating data into a proprietary data transformation solution where applicable.

Key Skills & Experience

• Azure Data Engineering
• Azure Data Factory (ADF)
• Azure Data Lake
• Databricks
• Building ETL / ELT pipelines
• Large-scale data migration and integration
• Oracle and/or Oracle Fusion data sources
• SharePoint data extraction and integration
• Working with multiple and varied source systems
• Data transformation, mapping and validation
• Designing data pipelines in environments where architecture and processes are still developing

Key Responsibilities

• Design, build and maintain data migration and ingestion pipelines within Azure.
• Support the setup and development of the Azure Data Lake environment.
• Build integrations between source systems, Azure Data Lake and Databricks.
• Develop and manage pipelines using Azure Data Factory (ADF).
• Extract and migrate data from a variety of source systems, including:
  • Oracle
  • Oracle Fusion
  • Informatica
  • SharePoint, which will be an initial priority
  • Additional enterprise and legacy systems as the programme develops
• Help define scalable data ingestion, transformation and migration patterns for future source systems.
• Support data quality, validation and troubleshooting throughout the migration process.
• Work closely with technical and business stakeholders to understand source data, target structures and transformation requirements.
• Help improve the overall maturity of the Azure data environment, introducing appropriate engineering standards and best practices.