Start searching
Back to Jobs
Technology

Databricks Data Engineer

Recruit Professional ServicesBedminster, NJ🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationBedminster, NJ, United States
MongoDBOraclePL/SQLSQLETLAirflowAzureDatabricksGitPython

Job description

Databricks Data Engineer
Responsibilities:
  • Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc.
  • Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse.
  • Optimize data pipelines and data processing workflows for performance, scalability, and efficiency.
  • Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data.
  • Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams.
  • Build proof-of-concepts to determine viability of possible new processes and technologies.
  • Deploy and manage code in non-prod and prod environments.
  • Investigate and troubleshoot data related issues and fix or provide solutions to fix defects.
  • Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance.

Requirements:
  • Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field.
  • 4+ years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles.
  • Current/active Databricks Data Engineer/Analyst certification a BIG plus.
  • 3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices.
  • Solid continuous experience in Python.
  • 3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL.
  • 3+ years of solid experience writing SQL and PL/SQL code.
  • 2+ years of experience with Airflow.
  • 3+ years of experience working with relational databases (Oracle preferred).
  • 2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar.
  • 2+ years of cloud experience (Azure preferred).
  • Experience with CI/CD utilizing git/Azure DevOps.
  • Experience with storage formats including Parquet/Arrow/Avro.

#RecruitPS

Similar jobs

Apply