Quick Overview
Work Type
Hybrid
Level
Mid Senior
Job Description
Key Responsibilities
- Design the hybrid architecture connecting legacy job schedulers and modern cloud orchestration platforms, including dependency sequencing across both environments.
- Define and own the cross-platform SLA framework and the single escalation path for orchestration issues.
- Lead technical engagement with client engineering leadership, translating business requirements into an architecture roadmap.
- Direct the engineering team on bidirectional job triggering, failure propagation, and real-time dependency-status exchange between legacy and cloud workloads.
- Own delivery through dependency mapping, co-design, build and test, and stabilization phases.
Required Qualifications
- 10+ years in data engineering or platform architecture, including at least 3 years designing hybrid or multi-platform orchestration solutions.
- Hands-on experience with cloud data platform workflow engines (for example, Databricks Workflows, Google Cloud Composer, or Apache Airflow).
- Experience integrating legacy enterprise job schedulers (for example, IBM Tivoli Workload Scheduler, Control-M, or AutoSys) with modern cloud orchestration tools.
- Working knowledge of open table formats (Delta Lake, Iceberg), Databricks Lakebase, Azure Service Bus, REST and Pub/Sub-based integration patterns.
- Proven experience with event-driven architectures (pub/sub, streaming patterns) to enable high-throughput, loosely coupled message exchange.
- Experience leading API-led connectivity strategies and delivering RESTful and event-driven APIs (OpenAPI, webhooks) to expose reusable business capabilities and support real-time, asynchronous system integration.
- Experience defining and implementing Infrastructure-as-Code (IaC) and CI/CD strategies using Terraform and Git-based pipelines to standardize and automate deployments.
- Ability to define KPIs, metrics, and SLAs, and translate them into actionable dashboard views for technical and business stakeholders.
- Experience designing centralized, single-pane-of-glass dashboards that consolidate visibility across multiple, independently running systems.
- Demonstrated experience owning SLAs and escalation processes for production data pipelines.
- Strong client-facing communication skills; comfortable presenting architecture decisions to senior technical stakeholders.
- Experience leveraging agentic coding platforms (for example, GitHub Copilot or Claude Code) effectively.
Preferred Qualifications
- Prior experience on large-scale ETL modernization or cloud migration programs.
Skills
ETL
Airflow
Apache
Azure
Databricks
Git
Google Cloud
REST
Terraform
Similar jobs
Lead Databricks Solutions Architect with Security Clearance
ECS · Hampton, United States
3 hours ago$200k - $230k/yrSoftware Solutions Architect
Aroha Technologies · Lansing, United States
3 hours agoLead Databricks Solutions Architect
Electronic Consulting Services, Inc (ECS Federal) · Fairfax, United States
4 hours ago$200k - $230k/yrWorkday Solution Architect - Core HCM || REMOTE || W2 contract
CaritaTech LLC. · United States
4 hours agoSolutions Architect II, AWS National Security Solutions Architec with Security Clearance
Amazon · Denver, United States
4 hours ago$131.3k - $177.6k/yrAssurance Solutions Architect, AWS Security Assurance Services with Security Clearance
Amazon · Herndon, United States
4 hours ago$131.3k - $177.6k/yr