Quick overview
Job description
|
Role |
Lead Data Engineer/ Architect (Hands-On) |
|
Location |
Cary, NC (On-site / Hybrid) |
|
Experience |
12–18 Years – Visa Independent candidates – |
|
Employment |
Full-Time |
|
Customer |
ABOUT THE ENGAGEMENT
This is a senior hands-on leadership role. You will own the end-to-end technical design of the data and AI layers, build the reference implementations your engineers work from, and ship production-grade Python, Scala, and PySpark code every week. If you have not written or reviewed production code in the past year, this is not the right fit.
MUST-HAVE SKILLS & EXPERIENCE
Programming & Data Engineering
Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.
Strong SQL and data modelling — dimensional and normalised; schema design and data contract definition.
Databricks expertise — Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.
AI & Machine Learning
3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.
Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.
Architecture & Governance
12–18 years of total experience in data engineering, data platform delivery, or related disciplines.
Proven delivery of a medallion / lakehouse architecture at enterprise scale — not just familiarity with the concept.
Azure security and governance — Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
CI/CD and infrastructure as code — Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.
STRONGLY PREFERRED
Knowledge graphs and ontologies: RDF/SPARQL, property graphs (Neo4j), or graph modelling over a lakehouse.
Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale, including access control and ambiguity handling.
ML-based anomaly detection on time-series or transactional financial data.
Financial services or insurance domain knowledge: finance close, general ledger, subledger, reconciliation, or actuarial data.
LLMOps and MLOps: model versioning, prompt versioning, cost governance, and observability tooling.
Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.
dbt, Great Expectations, or similar data-quality and transformation tooling.
Workday, Prism, or Accounting Center exposure
Similar jobs
- CA

Data Engineer with Security Clearance
CACI · Arlington, VA · Hybrid · Posted 2 Nov 2026 🇺🇸
SQLGitApache Spark+9Technology$103.8k - $218.1k/yr - FO

Data Engineer IIRecently posted
FedEx Office · Plano, TX · Hybrid · Posted 18 Nov 2026 🇺🇸
MongoDBMySQLSQL+21Technology$9.0k/mo - SS
Data engineer with cybersecurityRecently posted
Sharpedge Solutions · Bellevue, WA · Hybrid · Posted 15 Nov 2026 🇺🇸
SnowflakeSplunkData Pipeline+7TechnologySalary not listed - IC

Sr Data EngineerRecently posted
Infinite Computer Solutions (ICS) · Atlanta, GA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLAWSETL+2TechnologySalary not listed - FS

Data EngineerRecently posted
FlairTech Solutions · United States · Remote · Posted 17 Nov 2026 🇺🇸
SQLAWSTableau+7TechnologySalary not listed - AT

Sr. Oracle DBARecently posted
Acadia Technologies, Inc. · Atlanta, GA · Hybrid · Posted 17 Nov 2026 🇺🇸
OraclePL/SQLSQL+2TechnologySalary not listed - ME

Customer Data Platform (CDP) ManagerRecently posted Auto Apply
Monster Energy · USA - Corona · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLLookerCRM+5$116.3k/yr - SA

Data Engineer with Security ClearanceRecently posted
Strategic Alliance Business Group LLC · Peterson SFB, CO · Hybrid · Posted 17 Nov 2026 🇺🇸
Microsoft ExcelMicrosoft OfficeMicrosoft PowerPoint+4Technology$95k - $135k/yr - SA

Senior Data Engineer with Security ClearanceRecently posted
Strategic Alliance Business Group LLC · Peterson SFB, CO · Hybrid · Posted 17 Nov 2026 🇺🇸
Microsoft ExcelMicrosoft OfficeMicrosoft PowerPoint+3Technology$104k - $147k/yr - AT
Database Engineer 2 with Security Clearance
Avid Technology Professionals · Linthicum Heights, MD · Hybrid · Posted 11 Nov 2026 🇺🇸
DjangoDockerFlask+9TechnologySalary not listed - BW

Data Engineer 3 - TS clearance required - Wash DC area with Security Clearance
Bow Wave LLC · Arlington, VA · Hybrid 🇺🇸
SQLGitAWS+14TechnologySalary not listed - MA

Expert Data Engineer with Security ClearanceRecently posted
Markon · Herndon, VA · Hybrid · Posted 17 Nov 2026 🇺🇸
ShellAWSSplunk+4Technology$187k - $213k/yr