Sr Data Engineer

Job Title: Senior Data Engineer (Databricks)
Primary Location: Chicago, IL (Hybrid – 3 days onsite / 2 days remote)
Position Type: Full-Time, Direct Hire

OVERVIEW
TalentFish is casting a line for a Senior Data Engineer. This is a full-time, direct hire role in Chicago, IL, working a hybrid schedule with a minimum of three days onsite per week. This role exists as part of a multi-year initiative to transform our client's Data & AI function, building a modern data platform that enables speed-to-insight and makes the organization truly data-driven.

You'll be one of the core builders of a brand-new Azure Databricks platform, designing and delivering the pipelines, transformations, and models that everything else runs on. The environment is greenfield: Databricks is stood up, but no production pipelines exist yet, so you'll help set the standards from day one. This is a hands-on senior individual contributor role, not a people management, architecture, AI/LLM, or BI/reporting role. The architecture is defined; your job is to build it well and raise the technical bar of a growing team (expanding to 25–30) that is largely BI-focused today.

The work carries strong executive sponsorship and enterprise-wide visibility, with the chance to influence engineering practices across the organization. The interview process includes a live technical assessment covering Python, SQL, and practical data pipeline logic.

WHAT YOU BRING TO THE ROLE. (IDEAL EXPERIENCE)
Bachelor's degree in Computer Science, Engineering, Data Science, or a related field
6+ years of hands-on data engineering experience, with a track record of owning pipelines end to end: ingestion, validation, transformation, and governance
Real, hands-on Databricks experience (Azure preferred; AWS or Google Cloud Platform acceptable), including Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, and Workflows/Jobs
Strong coding skills in Python, SQL, and Apache Spark that go well beyond basic scripting
Proven experience designing and building scalable ETL/ELT pipelines, including reusable, metadata-driven pipelines
Experience with object storage, lakehouse engines, orchestration tools, and streaming or CDC pipelines
Solid data modeling, schema design, and performance tuning experience
Working knowledge of Git-based workflows, CI/CD, test automation, and monitoring and alerting
Strong troubleshooting skills and a solid grasp of engineering fundamentals: scalability, reliability, and maintainability
The ability to clearly explain what you built, why it mattered, and how it solved a business or platform problem
Preferred
Experience building or migrating a data platform from the ground up
Infrastructure as Code (Terraform, Bicep, ARM) and Governance as Code experience
Multi-cloud background with depth in Azure
Legacy data warehousing and/or Hadoop experience
Deep Databricks exposure beyond basic pipelines, such as enterprise-scale work and optimization
A track record of mentoring and upskilling peers
Agile delivery experience

WHAT YOU'LL DO. (SKILLS USED IN THIS POSITION)
Lead the design and build of scalable data pipelines and models in Azure Databricks on a brand-new platform with no production code yet
Own pipelines end to end, from ingesting source systems to transforming and modeling data to producing outputs for downstream analytics, BI, and AI/ML use cases
Refactor and migrate multiple legacy data warehouses into the centralized Databricks lakehouse
Work at the code level every day in Python, SQL, and Spark, building pipelines with Databricks Asset Bundles, Spark workflows, and API-driven jobs
Establish engineering standards for CI/CD (GitHub/Azure DevOps), automated data testing, monitoring, and alerting
Help shape and enforce standards for data modeling and governance
Troubleshoot and optimize performance across the platform
Mentor and coach junior engineers, conduct code reviews, and lead by example in design, coding, and testing practices
Partner with Data Engineering, BI, AI, Finance, and product owners, and help translate technical work into business impact