Senior Data Engineer / ML Platform Engineer (Azure Databricks & GenAI)

We are augmenting an internal team with contract contractors who have strong Azure Databricks and ML data platform skills. Our staff already owns platform standards, Unity Catalog

governance, and architecture; we need additional capacity to deliver lakehouse pipelines, catalogued engineering and enterprise datasets, and GenAI-ready data products (embeddings,

retrieval, batch enrichment) in parallel with other priorities.

Assignments may span multiple programs—not a single fixed product—so candidates should be comfortable ramping on new domains and codebases quickly.

● Design and build production data pipelines on Databricks (PySpark/SQL, Delta Lake, Unity Catalog, Workflows/Jobs) with idempotency, testing, monitoring, and clear lineage.

● Model bronze / silver / gold layers and data contracts (schemas, SLAs, quality checks) for analytics and downstream applications.

● Develop ML-oriented data products: feature stores or curated tables, embedding generation, chunking/indexing for RAG, batch classification/enrichment, and evaluation

of retrieval/data quality.

● Integrate with Azure services (ADLS Gen2, Key Vault, managed identities, private connectivity) following enterprise security and compliance patterns.

● Partner with application and ML teams to expose stable APIs, tables, and volumes that power agentic workflows (tool-backed assistants, multi-step orchestration) without

owning full UI or real-time serving unless scoped.

● Document data dictionaries, runbooks, and handoff materials; participate in code review and agile ceremonies with full-time staff.

● 5+ years in data engineering, including 2+ years production Databricks on Azure.

● Strong Python (PySpark, pandas); SQL; experience operationalizing jobs at scale.

● Hands-on Delta Lake and Unity Catalog (schemas, grants, environments, promotion patterns).

● Experience building pipelines that feed ML/GenAI use cases: embeddings, vector or hybrid search inputs, labeled datasets, batch inference outputs.

● Git, CI-friendly repo layout (notebooks + packages), unit/integration testing for data pipelines.

● Strong communication; effective in staff augmentation models (embedded with internal leads, shared standards).

● GenAI / agentic systems: RAG design, retrieval evaluation, metadata for grounding, guardrails-aware data prep.

● Multi-source integration: documents, PLM/BOM-style structured data, files in object storage, APIs, spreadsheets—normalized into a catalog.

● Exposure to containerized or microservice backends (for collaboration with app teams, not primary ownership).

● MLOps on Databricks (Model Serving, Feature Serving, experiment tracking).

● Stream or near-real-time ingestion (Structured Streaming, event-driven triggers).

● Infrastructure-as-code literacy (Terraform/Bicep)—often owned internally; ability to consume outputs is enough.

● Graph or highly relational domains (requirements traceability, product structure, hierarchies).

● Automotive or complex manufacturing data (helpful, not required).

● Platform expertise, Databricks/Azure standards, architecture review, and access to tooling and repos per assignment.

● Clear priorities from product/engineering owners; expectations to align with existing team patterns rather than invent a separate stack.