Data Engineer (Databricks, Azure) - Hybrid Onsite with Security Clearance

Marathon TS is looking to hire an onsite Data Engineer (Databricks, Azure) to support a Federal Agency in Washington DC. We are looking for a Data Engineer to build and run the pipelines behind a generative AI program at a regulatory client. You will move very large document submissions into Databricks, deliver AI outputs to business applications, and bring reviewer feedback back to improve the workflows. What you'll do

• Build incremental, idempotent ingestion from Azure storage into Delta tables for very large document submissions, with metadata and text extraction at scale.
• Deliver AI outputs to the applications' operational data store (Azure SQL, PostgreSQL or Lakebase), and load reviewer edits back for prompt refinement.
• Load and reconcile historical data from multiple sources, including SQL databases and spreadsheets, into curated tables.
• Orchestrate jobs with Workflows or Lakeflow, with data quality checks, monitoring and alerting.
• Deploy Databricks assets through CI/CD (Asset Bundles) across development, test and production.
• Manage Unity Catalog permissions and keep compute costs down. What you bring
• 6+ years in data engineering, including 3+ years on Databricks: PySpark, Delta Lake, Workflows and Unity Catalog.
• Strong SQL and Python; CI/CD for data pipelines.
• Experience with large files and unstructured documents, not only tabular data.
• Azure storage and networking fundamentals.
• PostgreSQL or Databricks Lakebase, Azure SQL and Azure Data Factory. Nice to have
• Experience in regulated or confidential data environments. Working with us
• You will work inside a secure, private cloud environment with highly confidential data.
• You must be authorized to work in the United States and able to pass a background check.
• We are looking for people who can start within a few weeks.

#cjjobs Marathon TS is committed to the development of a creative, diverse and inclusive work environment. In order to provide equal employment and advancement opportunities to all individuals, employment decisions at Marathon TS will be based on merit, qualifications, and abilities. Marathon TS does not discriminate against any person because of race, color, creed, religion, sex, national origin, disability, age or any other characteristic protected by law (referred to as "protected status ").