Senior Data Engineer, Databricks

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark


• Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment


• Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets


• Implement data quality checks, validation rules, and monitoring mechanisms


• Integrate structured, semi-structured, and unstructured data sources


• Ensure data security, governance, and compliance with banking regulations


• Optimize Spark jobs for performance and cost-efficiency in Azure


• Participate in code reviews, design discussions, and agile ceremonies


• Document data pipelines, workflows, and technical decisions

Requirements

• At least 2+ years of hands-on experience in Azure Databricks


• Strong proficiency in Apache Spark, especially PySpark


• Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake


• Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts


• Proficiency in Python and/or Scala for data processing


• Experience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferred


• Familiarity with SQL, NoSQL, and data lake architectures


• Knowledge of data governance, security, and compliance in financial services


• Strong analytical and problem-solving skills


• Excellent communication and stakeholder management abilities


• Ability to work collaboratively in cross-functional teams


• Agile mindset and experience in Scrum/Agile environments


• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field preferred


• Databricks Certified Data Engineer Associate/Professional certification is a plus


• Experience in banking or financial services is highly desirable


• Australian citizenship required


• Must be eligible to obtain security clearance

Core Competencies

Demonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services. Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.

Highest-signal resume keywords

• Azure Databricks


• Apache Spark


• Data Governance


• ETL/ELT Processes


• CI/CD Pipelines

ATS Optimization Keywords
Hard Skills

• Data Ingestion Pipelines


• PySpark


• Azure Data Factory


• SQL


• NoSQL


• Data Modeling


• Data Warehousing


• Python


• Scala


• Terraform

Soft Skills

• Analytical Skills


• Problem-Solving Skills


• Communication Skills


• Stakeholder Management


• Collaboration

Certifications & Qualifications

• Databricks Certified Data Engineer Associate


• Databricks Certified Data Engineer Professional

Industry Keywords

• Data Governance


• Data Security


• Compliance


• Banking Regulations


• Financial Services

Tools & Technologies

• Azure Synapse


• Azure Blob Storage


• Delta Lake


• Azure DevOps


• Git