Data bricks Engineer

Job Title: Data bricks Engineer
Location: Coventry UK
Employment type: contract (Inside Ir35)

Detailed job Description:

Role:
Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
Experienced in working with AWS cloud services, supporting cloud-based infrastructure, deployment, configuration, monitoring, and application environments.
Develop the Pyspark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
Carry out data modelling in Databricks.
Expose the Delta tables to the visualization layer.
Ensure that the data pipelines are optimized.

Skillsets:

Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
Fluent in PySpark and Databricks SQL.
Proven experience building Databricks architecture patterns.
Familiarity with Unity Catalog for data governance and access control.