Senior Data Engineer (DataDivers)

DataDiversis hiring!

DataDivers, part of theRmkblefamily,is on the hunt for a dynamic and drivenSenior Data Engineerto join our growing team.

AboutDataDivers

DataDiversis one ofRmkble'sspecialist businesses, dedicated to all thingsdata andanalytics.Our expertise spans data strategy,data platforms, DataOps, advanced analytics, data science and AI, and building a data culture in our clients' organisations. We are exceptional problem solvers who turndata into real business insights.

As a results-driven organisation, we are committed to remarkable outcomes for our clients. We are well-known for always delivering, but not only do we deliver, we goabove and beyondto exceed expectations and build lasting relationships. Wedon'toperatein isolation to achieve this; we have strong partnerships with Microsoft, Snowflake, AWS, Databricks and Informatica, to name a few.

With a diverse client base, our people work across an exciting mix of organisations and with a variety of technology stacks. Our clients are both local and international, in the public and private sectors, and cover a vast cross-section of different domains, from health to government to mining to not-for-profits.

Our people are second to none in their field. They are true experts who are continuously learning and continuously giving back. Some deliver learning and mentoring programs; others are shaping future products, or tweaking frameworks and tools to advance new thinking. At the heart of what setsDataDiversapart is the support that our leaders and our people give to each other.

We believe that passion and curiosity lead to awesome possibilities for both you and for us. So come join the team and have a remarkable journey!

About the role

A SeniorData Engineerisaspecialistrole within theDataDiversbusiness.This rolepossessessignificant ICT technicalexpertiseinone or moreareas such asdata engineering,data analysis, datamanipulationand data visualisation.Data Engineersoftenleadteams in developing data solutions for client to meet their unique needs relating to understanding the data managed within their business.The role is primarily responsible for the design and implementation of data engineering solutions.

A Senior Data Engineeris responsible forunderstanding the client's data needs, assessing thequality of the client's data, using the data toprovideclient insights and presentviews on how that insight can bring benefit to the client'sorganisation.A Senior Data Engineer atDataDiversfacilitateslasting sustainabilityfor their clientsthrough thedelivery ofpragmatic solution outcomes.

Commonly, the Senior Data Engineeris required tolead a team of data engineers. In this role, theyare responsible forensuring the team members understand their role, manage the status of delivery, manage client relationships and communication, and provide support to more junior team members ifrequired.

Key Responsibilities:

Data Architecture Design:

• Design and implement data models and architectures that support business needs.

Evaluate and select appropriate datastorage solutions (e.g., databases, data lakes,lakehouses).

• Evaluate and selectappropriate datastorage solutions (e.g., databases, data lakes,lakehouses).

ETL Development:

• Build andoptimizeExtract, Transform, Load (ETLor ELT) processes for data integration from various sources.
• Ensure data quality and integrity throughout the ETLor ELTpipeline.

Data Pipeline Management:

• Create and manage data pipelines to ensuretimelyand efficient data flow.
• Monitor and troubleshoot pipeline performance and data anomalies.

Collaboration:

• Work closely with data scientists, analysts, and stakeholders to understand data requirements.
• Provide technical guidance and mentorship to junior data engineers.

Data Governance:

• Implement data governance best practices to ensure compliance and data security.
• Maintain documentation of data sources, pipelines, and architecture.

Performance Optimization:

• Analysesystem performance and make recommendations for improvements.
• Optimizedata storage and processing to reduce costs and improve efficiency.

Technology Evaluation:

• Stay current with emerging technologies and industry trends related to data engineering.
• Evaluate and recommend new tools and technologies to enhance data capabilities.

Skills

• Extensive experience in IoT architectures and ETL/ELT frameworks (Very High)
• Understanding of data modelling and database design principles (Very High)
• Ability to interpret technical requirements and translate into technical solutions and architectures (Very High)
• Extensive experience in deploying production ready data pipelines (Very High)
• Communication and collaboration (Very High)
• Skills Framework for the Information Age (SFIA) - (High)
• Business consulting experience (High)
• Conceptual thinking (High)
• Leadership and relationship management (High)
• Problem solving and trouble shooting (High)

Tools / Languages (Highly Proficient)

• Experience with Microsoft ADF, AWS Glue, AWS Lambda, Fivetran, DBT, Talend, Informatica, and other ETL/ELT tools
• Experience with one of AWS Redshift, Snowflake, Azure Synapse, Azure SQL, Microsoft SQL, Oracle, PostgreSQL, or MySQL
• Experience coding in SQL
• Proficiency in coding with Python, R, or similar
• Experience using cloud platforms including Azure, Snowflake, AWS, and GCP, as well as Kubernetes, Docker, and similar

Methodologies (Proficient)

• Extensive experience with Data Warehousing methodologies (e.g., Ralph Kimball, Data Vault)
• Broad understanding of CI/CD practices in Azure, AWS, or GCP
• Experience of data quality and data governance processes to ensure data accuracy and reliability

Experience:

• At least5years' experiencein a similar role.
• At least5years' experienceinclient deliverywithin a project/ consultancyenvironment.
• Considerableexperience inaleadershiprole(s)managingdesign and deliveryof data solutions.

Specific Qualifications:

The RmkbleData EngineerSenior)should have the following minimumqualifications:

• Undergraduate university degree inComputer Science /Statistics,Mathematics, Science, Engineering or Technology.
• Masters level degree preferred.