Senior Data Engineer

• Position available in the following states: NSW, VIC
• Australian Citizenship Required

DFP Recruitment is seeking an a Senior Data Engineer who will design, build and maintain reliable data pipelines and data products that support data science, machine learning, artificial intelligence and advanced analytics. Working closely with data scientists, machine learning engineers, solution architects, platform teams and business stakeholders, you will translate complex data requirements into well-engineered datasets and services, taking solutions from exploration and prototyping through to secure, repeatable and supportable production delivery. You will apply strong software engineering, cloud and DevOps practices to ensure data quality, lineage, security, privacy and monitoring, supporting the responsible use of data.

Key duties and responsibilities

• Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases.
• Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources.
• Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines.
• Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria.
• Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting.
• Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency.
• Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions.
• Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions.
• Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements.

Technical skills

• AWS data engineering (S3, Glue, Lambda, Athena, EMR, Kinesis and Step Functions)
• Python and advanced SQL; ETL/ELT, data modelling, orchestration, quality testing, metadata and lineage; Git, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation; Docker, ECS or EKS; security, access controls, secrets management and monitoring.

Essential criteria

• Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services.
• Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data.
• Demonstrated application of modern software engineering and DevOps practices, including Git, code review, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation.
• Strong programming capability in Python, with experience developing maintainable, modular and testable production code. Experience with Java or Scala is also relevant.
• Advanced SQL skills, including query optimisation, performance tuning and working with large or complex datasets.
• Hands-on experience delivering cloud-based data solutions in containerised environments such as Docker, ECS or EKS, with a sound understanding of security, access controls, secrets management and operational monitoring.

Desirable criteria

• Experience supporting data science, machine learning, generative AI or document intelligence solutions, including data preparation, feature pipelines, vector or search-based retrieval, model outputs and evaluation datasets.
• Experience with one or more modern data platforms or processing technologies such as Snowflake, Databricks, Redshift, Spark or dbt.
• Knowledge of MLOps or DataOps practices, including experiment-to-production handover, reproducibility, model and data versioning, monitoring and automated deployment.
• SAS Base Programming

Applicants new to DFP may be asked to provide additional information including work rights status via a survey link - if requested, we ask that you provide this information in order to expedite your application.

DFP welcomes applications from Aboriginal and Torres Strait Islander people, people with diverse cultural and linguistic backgrounds and people with disability. In addition, DFP will provide reasonable adjustments for individuals with disability throughout the recruitment process. If you identify as a person with disability and require adjustments to the application, recruitment, selection and/or assessment process, please advise and indicate your preferred method of communication (email, phone, text) so we can keep in touch and meet your accessibility needs.