Databricks Data Engineer

Job Summary:
Looking for an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines and data processing solutions using Databricks, PySpark, and SQL. The ideal candidate should have strong experience with data engineering, ETL/ELT, cloud data platforms, and modern data lake/lakehouse architecture.

Key Responsibilities:

• Design and develop scalable ETL/ELT data pipelines using Databricks.

• Build data processing solutions using PySpark and SQL.

• Develop and maintain Databricks notebooks, workflows, and jobs.

• Work with Delta Lake and implement Lakehouse architecture.

• Perform data ingestion, transformation, cleansing, and validation.

• Integrate data from various sources including databases, APIs, files, and cloud storage.

• Optimize Spark jobs and Databricks workloads for performance and cost.

• Implement data quality, governance, security, and monitoring practices.

• Collaborate with Data Architects, Developers, Analysts, and business teams.

• Troubleshoot production data pipeline issues and provide ongoing support.

• Participate in CI/CD and deployment processes for data engineering solutions.

Required Skills:

• 5+ years of experience in Data Engineering.

• Strong hands-on experience with Databricks.

• Strong PySpark and SQL skills.

• Experience developing ETL/ELT pipelines.

• Experience with Delta Lake / Databricks Lakehouse.

• Strong understanding of data warehousing and data modeling.

• Experience with cloud platforms such as Azure, AWS, or Google Cloud Platform.

• Experience with relational and/or NoSQL databases.

• Knowledge of Git and CI/CD.

• Strong troubleshooting and problem-solving skills.