Sr Databricks Data Engineer :: W2 Position :: Remote

Databricks Data Engineer

Location: Remote
Candidate Preference: Please share only strong, highly qualified profiles.

Job Summary

We are seeking an experienced Databricks Data Engineer to own the end-to-end migration strategy, target architecture, and technical execution for moving legacy ETL workloads to the Databricks Lakehouse.

The ideal candidate will have strong hands-on expertise in Databricks, PySpark, Delta Lake, Unity Catalog, Photon, and DataStage, with proven experience modernizing legacy data pipelines, optimizing Spark workloads, and implementing scalable data engineering solutions.

Key Responsibilities

Architectural Strategy & Governance

• Design target Databricks Lakehouse architecture using Delta Lake, Photon, and Unity Catalog.
• Establish coding standards, refactoring guidelines, optimization benchmarks, and PySpark best practices.
• Analyze complex source-to-target dependencies and develop seamless dual-run/migration strategies.
• Lead the technical deployment and integration of specialized migration accelerators.
• Define scalable and maintainable architecture patterns for enterprise data migration.

Hands-on Engineering & Optimization

• Review automated output from migration tools and manually refactor complex legacy ETL logic into high-performance PySpark notebooks.
• Modernize legacy processing and eliminate inefficient patterns such as row-by-row processing and excessive lookups.
• Optimize PySpark workloads using advanced Spark techniques, including:
• Z-Ordering
• Partitioning
• Caching
• Spark optimization techniques
• Build robust Databricks Workflows and orchestrate complex DAGs based on source lineage and dependencies.
• Develop, test, debug, and deploy production-grade data pipelines.
• Troubleshoot performance, scalability, data quality, and migration issues.

Required Technical Skills

Core Platforms

• Databricks
• Delta Lake
• Unity Catalog
• Photon
• DataStage

Languages & Frameworks

• PySpark
• Python
• SQL
• Shell scripting

Cloud & DevOps

• AWS
• CI/CD pipelines
• Deployment automation and DevOps practices

Orchestration

• Apache Airflow
• Databricks Workflows