Data Engineer Lead / Data Warehouse Architect

Data Engineer Lead / Data Warehouse Architect - State of Michigan DHHS - Hybrid Lansing, MI

\n\n

Long-term contract. Hybrid in Lansing, MI. Michigan residents or candidates willing to relocate. Apply through the Apply button.

\n\n

MI DHHS 168059 - Data Engineer Lead/Data Warehouse Architect (Level 3)

\n

Location: Lansing, MI (Hybrid). Michigan residents or willing to relocate to Michigan.

\n\n

Data Engineering Lead excels in data modeling, Lakehouse and Medallion Architecture concepts, and data/ML products delivery using an agile DevOps framework with a mastery of distributed data architectures essential for data, analytical and operational ML applications with demonstrated proficiency in frameworks including Spark, Kafka, Kubernetes, SageMaker, DBT, Jupyter Notebooks, Docker, SQL, and visualization tools such as Power BI/Tableau.

\n\n

Required Education: Bachelor's Degree

\n\n

Required Skills:

\n

1. Minimum 4 years of hands-on experience hydrating the enterprise Data Lake on a Cloud Service Provider (AWS, Azure, or Google) from relational, semi-structured and unstructured data sources by building Extract and Load Data Pipelines.

\n

2. Minimum 4 years hands-on experience transforming datasets in a Data Lake on a Cloud Service Provider to deliver Data Products with traceability of their attributes/features to the Critical Data Elements in the systems of record.

\n

3. Minimum 2 years experience enforcing Data Sharing Policies on Data Products so users see only the version of the Data Product they are entitled to at the time of subscription.

\n\n

Preferred Skills:

\n

- DevOps tools including GitHub/Jenkins/Terraform/YAML/DAGs/Docker/Databricks/SageMaker for building Data/ML pipelines to parquet destinations.

\n

- Promoting Data Engineering Standards and Best Practices among developers.

\n

- Implementing Data Governance and Data Security tools on a CSP to automate enforcement of Data Sharing Policies at the point of subscription.

\n\n

Roles and Responsibilities:

\n

Primary State of Michigan technology lead engaging with the UoM proof of concept team to support best practice adoption within the state hosting environment, and providing training to DAV data engineers.

\n

- Oversee the design, development, and maintenance of data pipelines for analytics and operational systems.

\n

- Architect and optimize data infrastructure, including storage, orchestration frameworks, and ETL/ELT processes.

\n

- Ensure data quality, reliability, and security through standards, monitoring, validation and governance.

\n

- Lead and mentor data engineering staff, review designs/code.

\n

- Collaborate with data scientists, analysts, product teams, and business stakeholders.

\n\n

Other Desired Skills:

\n

- BS in Computer Science / Data Science / Information Systems with 4+ years designing and implementing data, analytics, and AI solutions for large enterprises on AWS/Microsoft/Google.

\n

- 4+ years modeling relational data with Python/PySpark/SQL, tuning SQL, and loading multiple sources (relational DB, flat files, CSV, HTML, JSON) into Lakehouse Medallion architecture in parquet.

\n

- 4+ years in Python, R, Spark.

\n

- Designing and delivering Star-schema operational data stores for Power BI visualization (required).

\n\n

Key Languages: SQL, Python, R, Spark, Terraform, DAGs, YAML, Docker

\n

Other Technologies: Jupyter Notebooks, NoSQL DB, Graph DB, Parquet, Lakehouse/Medallion, Kafka, Kubernetes, Glue, Databricks, Power BI, Tableau, Redshift Spectrum, SageMaker, DBT, DataHub, Machine Learning, GitHub, DevOps