Systems Analyst III / Databricks Data Engineer

Systems Analyst III / Databricks Data Engineer

Client: Texas Health and Human Services Commission (HHSC)

Location: Austin, TX (Hybrid - 2 Days Onsite, 3 Days Remote)

Duration: 12+ Months (High possibility of extension)

Job Type: Contract (W2/1099/C2C)

Interview: Microsoft Teams / In-Person

Local Candidates Only

Job Summary

Texas Health and Human Services Commission (HHSC) is seeking an experienced Systems Analyst III / Databricks Data Engineer to design, develop, and optimize enterprise-scale data solutions using the Databricks platform. The ideal candidate will have extensive experience building scalable ETL/ELT pipelines, implementing modern Lakehouse architecture, and developing analytical dashboards that enable business decision-making.

This role requires deep expertise in Databricks, Apache Spark, Delta Lake, Lakeflow Declarative Pipelines (DLT), SQL, Python/Scala, data warehousing, and cloud-based data engineering practices. The selected consultant will collaborate with business stakeholders, architects, and technical teams to deliver secure, scalable, and high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable enterprise data solutions using the Databricks platform.
• Develop and optimize ETL/ELT pipelines using Apache Spark (PySpark or Scala).
• Build and maintain Lakehouse architectures utilizing Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Implement Lakeflow Declarative Pipelines (formerly Delta Live Tables) for production-grade data processing.
• Develop and schedule workflows using Lakeflow Jobs (formerly Databricks Workflows) or equivalent orchestration tools such as Apache Airflow.
• Design and maintain dimensional data models including Star and Snowflake schemas.
• Develop high-performance SQL queries for analytical workloads.
• Create interactive dashboards and applications using Databricks SQL Dashboards and Databricks Apps.
• Implement enterprise data governance, metadata management, data quality validation, and security best practices.
• Monitor pipeline performance and optimize workloads for scalability, reliability, and cost efficiency.
• Analyze business requirements and translate them into technical solutions.
• Perform system analysis, troubleshooting, and production support activities.
• Collaborate with business users, project managers, developers, and architects throughout the software development lifecycle.
• Document technical designs, workflows, data models, and operational procedures.
• Present technical findings and business insights to both technical and non-technical stakeholders.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
• Minimum 8 years of IT experience supporting the design, development, deployment, or delivery of technology solutions.
• Minimum 8 years of experience with Databricks.
• Strong expertise in Apache Spark using PySpark or Scala.
• Extensive experience building enterprise ETL/ELT pipelines.
• Strong knowledge of Delta Lake.
• Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables).
• Experience with Medallion Architecture (Bronze, Silver, Gold).
• Expertise in Lakehouse architecture.
• Strong experience with SQL.
• Strong programming skills in Python (preferred) or Scala.
• Experience designing dimensional data models (Star/Snowflake schema).
• Experience developing Databricks SQL Dashboards and Databricks Apps.
• Experience implementing data governance, data quality, security, and validation frameworks.
• Experience scheduling and orchestrating jobs using Lakeflow Jobs or Apache Airflow.
• Excellent verbal and written communication skills.
• Ability to work independently while collaborating across cross-functional teams.

Preferred Qualifications

• Databricks Certified Data Engineer (Associate or Professional).
• Experience working with Texas State agencies or other public sector organizations.
• Experience with Git-based version control.
• Experience implementing CI/CD pipelines for data engineering projects.
• Familiarity with DevOps best practices.
• Experience working in Agile/Scrum environments.

Technical Skills

Data Engineering

• Databricks
• Apache Spark
• PySpark
• Scala
• Delta Lake
• Delta Live Tables (DLT)
• Lakeflow Declarative Pipelines
• Lakeflow Jobs
• Databricks Workflows
• Data Lakehouse
• Medallion Architecture

Programming

• Python
• SQL
• Scala

Data Warehousing

• Dimensional Modeling
• Star Schema
• Snowflake Schema
• ETL
• ELT

Reporting & Analytics

• Databricks SQL
• Databricks Dashboards
• Databricks Apps

Data Management

• Data Governance
• Data Quality
• Data Validation
• Data Security
• Metadata Management

DevOps & Tools

• Git
• CI/CD
• Azure DevOps (preferred)
• Apache Airflow

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and presentation abilities
• Ability to interact with business and technical stakeholders
• Strong documentation skills
• Self-motivated and detail-oriented
• Ability to manage multiple priorities in a fast-paced environment