Databricks Tech Lead

Must Have Technical/Functional Skills

· 10+ years of experience in Data Engineering, Data Architecture, ETL Modernization, or Enterprise Data

Platform Migration.

· 6+ years of hands-on Databricks experience in large-scale enterprise environments with Data

Modernization or migration to Databricks expertise.

· Strong expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Databricks

Workflows/Jobs.

· Proven experience assessing and migrating legacy data platforms, ETL solutions, and data warehouses

to Databricks.

· Experience with Alteryx workflow assessment, migration planning, and modernization.

· Strong understanding of data modeling, data pipelines, source-to-target mappings, data quality, and

performance optimization.

· Experience designing scalable Lakehouse architectures, including Medallion Architecture and Delta

Lake.

· Knowledge of cloud data architecture, security, governance, monitoring, CI/CD, and DevOps practices.

· Hands-on experience with Azure Databricks, ADLS, Azure Data Factory, Purview, Key Vault, and Azure

DevOps.

· Ability to lead technical discovery workshops, define migration strategies, prepare architecture

documentation, and provide technical leadership throughout delivery.

Roles & Responsibilities

· Lead the assessment of current-state data platforms, including source systems, data warehouses, databases, ETL/ELT pipelines, reporting environments, and Alteryx workflows. · Define and drive the migration and modernization strategy for legacy data platforms to Databricks Lakehouse Architecture. · Analyze existing data assets, transformation logic, dependencies, source-to-target mappings, data quality requirements, and performance considerations. · Identify workloads suitable for lift-and-shift migration versus those requiring redesign, refactoring, consolidation, or retirement. · Design target-state data architecture leveraging Databricks, Delta Lake, Medallion Architecture, and cloud-native data services. · Lead the migration of legacy ETL, data integration, and analytics workloads to Databricks using PySpark, SQL, Databricks Workflows, and related technologies. · Assess and modernize Alteryx workflows and legacy data processing solutions into scalable Databricks-based implementations. · Define best practices for data ingestion, transformation, orchestration, security, governance, monitoring, metadata management, and lineage tracking. · Conduct technical discovery workshops and architecture reviews with business, data engineering, analytics, and platform stakeholders. · Create architecture documents, migration roadmaps, dependency assessments, implementation plans, effort estimates, and risk mitigation strategies. · Provide hands-on technical leadership during proof-of-concept, pilot, and large-scale migration initiatives. · Guide and mentor development teams on Databricks architecture, coding standards, performance tuning, and engineering best practices. · Collaborate with cross-functional teams to ensure migration and modernization initiatives align with business objectives and delivery priorities. · Establish CI/CD, DevOps, testing, and deployment standards for Databricks-based data solutions. · Ensure compliance with enterprise security, governance, data privacy, and operational standards throughout the migration lifecycle. · Act as the primary technical lead for architecture decisions, solution reviews, and stakeholder communications related to Databricks modernization programs