Data Engineer

Senior Data Engineer (Databricks)

Experience Level: 7+ Years
Job Type: Full-time, Permanent
Location: Client Site (On-site - Brisbane)

Experience Level: 7+ Years
Job Type: Full-time, Permanent
Location: Client Site (On-site - Brisbane)

Role Overview

We are seeking an experienced Senior Data Engineer with strong hands on expertise in Databricks and AWS to design, develop, optimize, and support enterprise grade data engineering solutions. The ideal candidate will have extensive experience in building scalable data pipelines, implementing modern data transformation frameworks, and maintaining reliable cloud based data platforms.

The role combines platform engineering with production Data Business As Usual (BAU) support, requiring strong capabilities in monitoring data pipelines, resolving production incidents, maintaining data quality, and ensuring compliance with operational SLAs. A solid understanding of ITIL and IT Service Management (ITSM) practices is essential for maintaining platform stability and driving continuous improvement across production data environments.

Technical Competencies & Requirements 1. Data BAU, ITIL & ITSM (Critical)

• Strong hands on experience in Data BAU operations, supporting production data pipelines, monitoring data freshness, and resolving data quality alerts.

• Thorough understanding of ITIL frameworks and ITSM processes, including Incident, Problem, Change, and Release Management.

• Experience with ITSM tools such as ServiceNow or Jira Service Management for SLA tracking, incident management, escalation handling, and Root Cause Analysis (RCA) documentation.

2. AWS & Databricks Ecosystem

• Hands on experience with AWS cloud services, particularly Amazon S3.

• Advanced proficiency in Databricks development using PySpark, including Unity Catalog, Delta Live Tables (DLT)/Lakeflow Declarative Pipelines, Serverless Compute, SQL Warehouses, dashboards, and alerting.

• Strong understanding of performance optimization techniques, including Liquid Clustering, Z Ordering, and Spark query optimization.

• Practical experience managing deployments and workloads using Databricks Asset Bundles (DABs).

3. Data Ecosystem & Infrastructure

• Advanced experience with dbt (dbt Core or dbt Cloud) for developing and managing version controlled data transformations.

• Hands on experience building and supporting Kafka based streaming pipelines for real time and near real time data integration.

• Proven experience with Terraform for Infrastructure as Code (IaC), including provisioning and managing AWS and Databricks resources.

• Strong experience with GitLab for source control and implementing automated GitLab CI/CD pipelines.

• Good working knowledge of Unix/Linux environments, including scripting, troubleshooting, and operational support.

Preferred Qualifications

• Experience working in the Airline or Aviation domain will be highly regarded.

• Databricks certifications will be an added advantage.