Azure Databricks Engineer

We are seeking a highly skilled Senior Data Engineer with a strong background in Azure Databricks to join our dynamic team. The ideal candidate will have 5 7 years of experience in data engineering, with a proven track record of designing, building, and maintaining scalable data pipelines and data solutions. This role requires a deep understanding of data architecture and the ability to work collaboratively in a fast-paced environment.

Responsibilities:

• Design, develop, and maintain robust data pipelines using Azure Databricks.
• Collaborate with data scientists and analysts to understand data requirements and deliver high quality data solutions.
• Implement data integration processes and ETL workflows to ensure data accuracy and availability.
• Optimize data storage and retrieval processes for performance and scalability.
• Monitor and troubleshoot data pipeline performance and resolve issues as they arise.
• Ensure data governance and compliance with industry standards and best practices.
• Document data engineering processes and maintain clear communication with stakeholders.
• Stay updated with the latest trends and technologies in data engineering and Azure Databricks.

Mandatory Skills:

• Strong expertise in Azure Databricks, including experience with Spark and Delta Lake.
• Proficient in programming languages such as Python, Scala, or Java.
• Experience with data modeling, ETL processes, and data warehousing concepts.
• Solid understanding of SQL and experience with relational databases.
• Familiarity with cloud data services and architecture, particularly within the Azure ecosystem.
• Ability to work independently and manage multiple projects simultaneously.

Preferred Skills:

• Experience with data visualization tools such as Power BI or Tableau.
• Knowledge of machine learning concepts and frameworks.
• Familiarity with DevOps practices and CI/CD pipelines.
• Experience with data governance tools and practices.

Qualifications:

• Bachelor’s degree in computer science, Information Technology, or a related field.
• 5 7 years of experience in data engineering or a related role.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.