Quick overview
Job description
Data Hub is a Databricks platform that brings transportation, safety, infrastructure and climate data from across CalSTA departments into one governed place. Analysts, dashboards, apps and business users rely on it. The first phase covers public state and federal data. The next phase adds internal, non-public data from the departments. We need a hands-on data architect and engineer to help build both phases.
What you will do
- Build and maintain automated pipelines that bring data from public portals, APIs and internal department systems into Delta tables.
- Assists the program manager in the platform's architecture, naming standards, and documentation.
- Work with the data hub governance group to make decisions on architecture, data sharing, data contracts, data ownership and data quality standards.
- Advise the data hub governance group on how sensitive data can be protected, including row-level and column-level access rules, and keep it separate from public data.
- Make the data reliable through monitoring, alerts and data quality checks.
- Set up datasets and documentation so data experts and business users can explore the data in plain language with AI tools.
- Write and maintain skill files (instruction files that guide AI coding assistants) so pipelines are built consistently.
- Help other CalSTA departments build their own pipelines using shared templates.
Required
- 7+ years in data engineering or a closely related role.
- Experience on large-scale, enterprise-wide data engineering or data architecture efforts.
- Strong Python and SQL.
- Experience with Apache Spark or a cloud data platform (Databricks, Snowflake, BigQuery or similar. Databricks is strongly preferred).
- Experience pulling data from REST APIs, databases or other source systems.
- Excellent communication skills, clear writing and good documentation habits.
Preferred
- Hands-on Databricks experience (Delta Lake, Unity Catalog, Jobs).
- Experience governing sensitive or confidential data, such as access controls, data masking and privacy requirements.
- Experience working within a data governance program, such as data contracts, data stewardship and cross-agency data sharing agreements.
- Experience writing skill files, prompts or instructions for AI coding assistants.
- Experience with GIS or location-based data.
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- LE

Data Engineer Intern with Security ClearanceRecently posted
Leidos · Fort Belvoir, VA · Hybrid · Posted 20 Nov 2026 🇺🇸
Prompt EngineeringTechnology$41.6k - $75.2k/yr - AG

Senior Data Engineer with AIRecently posted
AIT Global, Inc. · Dallas, TX · Hybrid · Posted 20 Nov 2026 🇺🇸
DockerNeo4jSQL+29TechnologySalary not listed - DD

Jr. Data EngineerRecently posted
Dexian DISYS · Durham, NC · Hybrid · Posted 20 Nov 2026 🇺🇸
DynamoDBOracleSQL+7TechnologySalary not listed - DO

INFORMATION TECHNOLOGY SPECIALIST (DATA MANAGEMENT) with Security ClearanceRecently posted
Department of Defense · Bremerton, WA · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLData AnalysisLinuxTechnologySalary not listed - MT

Data Engineer (ETL)Recently posted
MHK TECH INC · United States · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLScalaAWS+12TechnologySalary not listed - FA

Data EngineerRecently posted
Factspan Inc · Atlanta, GA · On-site · Posted 20 Nov 2026 🇺🇸
OracleSQLETL+6TechnologySalary not listed - NS
Sr Data EngineerRecently posted
Neural Strategic Solutions, Inc. · Wilmington, MA · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLT-SQLETL+3Technology$70/hr - TE
Data Engineer - Minneapolis, MN, Detroit, MI, Madison, WI, Columbus, OH, Chicago, IL.Recently posted
TechniPros, LLC · Minneapolis, MN · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLApache SparkAWS+8TechnologySalary not listed - DT
Data Engineer (W2 only)Recently posted
Digipulse Technologies, Inc · Durham, NC · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLETLSnowflake+3TechnologySalary not listed - DA

Data EngineerRecently posted
DatamanUSA, LLC · Denver, CO · Hybrid · Posted 20 Nov 2026 🇺🇸
GitETLSnowflake+4TechnologySalary not listed - GF
Senior Data Engineer - W2 (No C2C or 1099) - Onsite in Wilmington, MA (Posted SAM)Recently posted
Global Force USA · Wilmington, MA · Hybrid · Posted 20 Nov 2026 🇺🇸
SQLETLSQL ServerTechnologySalary not listed - PE

Sr. Data Engineer (OBI Advanced Analytic Method Augmentation) with Security ClearanceRecently posted
Prescient Edge · Washington, DC · Hybrid · Posted 18 Nov 2026 🇺🇸
DockerSQLETL+5Technology$160k - $180k/yr