Data Engineer

Job Summary

We are looking for a Databricks Engineer with strong experience in building scalable data pipelines and migrating PySpark/Hive workloads from AWS EMR or legacy platforms to the Databricks Lakehouse. The ideal candidate should have expertise in Databricks, Delta Lake, Unity Catalog, and Spark technologies.

Key Responsibilities

• Design and develop scalable data pipelines using Databricks.
• Migrate PySpark, Hive, and legacy ETL workloads to Databricks Lakehouse.
• Build and optimize Spark SQL, PySpark, and Spark Declarative Pipelines (SDP).
• Implement Unity Catalog, data governance, and security best practices.
• Perform data validation, reconciliation, and quality assurance.
• Optimize performance, manage CI/CD pipelines, and support production deployments.
• Collaborate with architects, developers, and business teams.