Quick overview
Job description
Build and operate the data pipelines that feed the Entity Hub. This role lands all six in-scope sources into Fabric, implements standardization and transformation logic, and maintains the data quality checks and monitoring that the entity resolution engine depends on. Reliable, observable ingestion is the foundation the entire programme rests on.
- Data Pipeline Development: Design, develop, and maintain scalable batch and streaming ETL pipelines using Databricks, PySpark, Python, and SQL.
- Lakehouse Implementation: Build and manage data solutions leveraging Delta Lake and medallion architecture principles to support trusted and reusable data products.
- Data Integration: Integrate data from enterprise applications, databases, APIs, files, and cloud platforms to enable unified analytics and reporting.
- Data Modeling & Transformation: Develop efficient data models and transformation frameworks that support business intelligence, advanced analytics, and AI/ML use cases.
- Platform Optimization: Monitor and optimize Databricks workloads, cluster configurations, and data processing performance to ensure scalability and cost efficiency.
- Data Quality & Governance: Implement data quality checks, monitoring, lineage, security controls, and governance practices in alignment with enterprise standards.
- DevOps & Automation: Contribute to CI/CD implementation, automated deployments, testing frameworks, and release management for data engineering solutions.
- Cross-functional Collaboration: Partner with business stakeholders, data architects, analytics teams, and AI engineers to deliver data products that support business outcomes.
- AI & Analytics Enablement: Enable downstream AI, machine learning, and Generative AI initiatives by building reliable, high-quality, and governed data foundations.
- Continuous Improvement & Innovation: Stay current with advancements in Databricks, cloud technologies, and modern data engineering practices to continuously enhance platform capabilities.
- Experience: 4 - 6 years of experience in data engineering, ETL development, and cloud-based data platforms.
- Technical Skills:
- Proficiency in Python, PySpark and SQL
- Strong expertise in Databricks Data Engineering capabilities, including: Databricks Lakehouse Architecture, ETL/ELT Design and Development, Delta Lake, Data Modeling, Batch and Streaming Data Pipelines, Spark Optimization and Performance Tuning
- Experience in designing and implementing: Medallion Architecture, Data Quality Frameworks, Data Governance Controls, Metadata and Lineage Solutions
- Experience with orchestration and workflow management tools such as Databricks Workflows, Azure Data Factory, or equivalent.
- Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation.
- Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architectures.
- Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or GenAI data preparation workflows (Preferred).
- Familiarity with data preparation for RAG, LLM, and AI/ML use cases (Preferred).
- Experience with infrastructure-as-code and containerization technologies (Preferred).
- Certifications: Relevant certifications in data engineering, such as Databricks Certified Data Engineer Associate/Professional, or similar. (Preferred not mandatory)
- Problem-Solving: Strong analytical and problem-solving skills, with a proven ability to design scalable data solutions and resolve complex data engineering challenges.
- Communication: Excellent communication skills with experience working directly with clients and business stakeholders to deliver data and analytics solutions.
- Stakeholder Management: Experience engaging with business stakeholders to understand data and analytics requirements and translate them into scalable data engineering solutions.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- GU

Survey Data Automation SpecialistRecently posted Auto Apply
Guidepoint · Gurugram · Hybrid · Posted 20 Nov 2026 🇮🇳
Microsoft ExcelMicrosoft PowerPointMarket Research+5Salary not listed - AG

Team Manager (Gurugram)Recently posted Auto Apply
Agoda · Gurugram · Hybrid · Posted 20 Nov 2026 🇮🇳
RecruitingMicrosoft ExcelMicrosoft PowerPoint+2Salary not listed - BP

PCS - Senior ManagerRecently posted Auto Apply
Bpmcpa · Bengaluru · Hybrid · Posted 20 Nov 2026 🇮🇳
RecruitingOnboardingCPA+1Salary not listed - RU
Director - Renewals and Quote Desk Strategy & OpsRecently posted Auto Apply
Rubrik · Bengaluru · Hybrid · Posted 20 Nov 2026 🇮🇳
RecruitingSQLMicrosoft Excel+11Salary not listed - LA
Operations CoordinatorRecently posted Auto Apply
Laundryheap · Bengaluru, Karnataka · On-site · Posted 19 Nov 2026 🇮🇳
Stakeholder ManagementMicrosoft ExcelGoogle WorkspaceOperations & Project ManagementSalary not listed - PA

Service Account Operation - Noida - Sales Operation Auto Apply
Paytmpayments · Noida · On-site 🇮🇳
Microsoft ExcelData AnalysisSalesSalary not listed - AI

Associate Servicing & Commissioning ManagerRecently posted Auto Apply
AirTrunk · Mumbai · Hybrid · Posted 19 Nov 2026 🇮🇳
ComplianceProcoreRoot Cause Analysis+1Salary not listed - WP

Head of Service Management and ImprovementRecently posted Auto Apply
WPP · India · Hybrid · Posted 19 Nov 2026 🇮🇳
Stakeholder ManagementCRMContinuous Improvement+7Salary not listed - PA

Cluster Manager - Manager - Varanasi Auto Apply
Paytmpayments · Varanasi · On-site 🇮🇳
ArticulateBusiness DevelopmentSalary not listed - PA

Cluster Manager - Manager/Senior Manager - QR - Mau Auto Apply
Paytmpayments · Mau · On-site 🇮🇳
ArticulateBusiness DevelopmentSalary not listed - CE

Operations ManagerRecently posted Auto Apply
Certifyos · Bengaluru · Remote · Posted 18 Nov 2026 🇮🇳
Microsoft ExcelGoogle WorkspaceProcess ImprovementOperations & Project ManagementSalary not listed - ON

Operations Manager - Customer ExperienceRecently posted Auto Apply
Onecom · Gurugram · Hybrid · Posted 18 Nov 2026 🇮🇳
RecruitingStakeholder ManagementOnboarding+5Operations & Project ManagementSalary not listed
