Senior Databricks Data Engineer and Data Modeller - 6 months contract - £650-750 per day (inside IR3

Senior Databricks Data Engineer and Data Modeller

The Senior Databricks Data Engineer and Data Modeler will join a team that will support the modernisation of investment management data capabilities currently delivered through our client's on-prem SQL Server and related Legacy components. The role will inventory the current environment, assess dependencies and data quality, redesign priority data structures, and migrate selected workloads into the approved Databricks architecture while supporting reporting, reconciliation and production needs.

Key responsibilities

• Inventory current state and related environments, including databases, schemas, tables, views, stored procedures, jobs, SSIS packages, feeds, reports, consumers, permissions and operational processes.
• Document owners, data lineage, business keys, relationships, calculations, transformations, refresh frequencies and direct or indirect downstream dependencies.
• Profile data for completeness, accuracy, duplication, referential integrity, timeliness, historical coverage, inconsistent definitions and redundant or obsolete logic.
• Work with business, accounting and technology SMEs to determine which structures, calculations and reports should be retained, redesigned, consolidated, expanded or retired.
• Convert, migrate or redesign T-SQL, stored procedures, functions, SSIS packages, jobs, ETL logic and reporting transformations.
• Design and implement Databricks data structures, Delta tables, Spark SQL transformations, views, pipelines, quality controls, reconciliation processes and monitoring.
• Create source-to-target mappings, phased migration plans, parallel-run validation, decommissioning criteria and technical support documentation.
• Support BAU priorities including SQL work, feed troubleshooting, reporting, testing, reconciliation and production issues.

Required experience

• Advanced SQL Server, T-SQL, Databricks, Spark SQL development experience.
• Strong relational, analytical and dimensional data-modelling skills and understanding of medallion architecture.
• Experience migrating SQL Server, SSIS, ETL, stored procedure or reporting workloads to Databricks
• Experience with profiling, metadata, lineage, dependency analysis, source-to-target mapping and data-quality controls.
• Ability to analyse Legacy logic and determine whether it should be migrated, refactored, redesigned, consolidated, replaced or retired.
• Strong technical documentation, communication and knowledge-transfer skills.

Preferred experience

• Commercial real estate, investment management and/or CML experience.
• Experience with Azure DevOps, Git-based source control and CI/CD.
• Python or PySpark development experience.
• Power BI, SQL Server Reporting Services, semantic models and paginated reports.
• Experience with platforms such as MRI, RealINSIGHT, Investran, Pereview, Aladdin, DealCloud, Argus Enterprise, Anaplan or eFront.