Back to Jobs
Engineering

Databricks/AWS Engineer

Newgen Technologies, Inc.United States🇺🇸United States

Why this role stands out

This hybrid role offers a fantastic opportunity to shape enterprise data solutions using cutting-edge Databricks and AWS technologies, perfect for a mid-senior architect who thrives on complex data modeling and clear communication. You'll contribute to a reputable company while developing your skills in a flexible environment.

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationUnited States
Unity

Job description

NewGen is seeking a Sr. Data Engineer with experience building enterprise data solutions in both Databricks (Unity Catalog, Hive Metastore) and AWS (AWS Glue, S3, RDS) and managing a data warehouse.

Required:

  • 8+ years of experience in designing, developing, deploying, and managing ETL processes and data pipelines utilizing comprehensive logging and robust error handling mechanisms to troubleshoot and resolve issues.
  • 3+ years of hands-on experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools using Databricks tools that include Delta Lake, Databricks Delta, and Unity Catalog.
  • 3+ years of hands-on experience writing Python and PySpark code for ETL scripts and custom transformations.
  • Experience in data modeling (e.g. understanding when to use dimensional vs relation data models) within a medallion architecture (Bronze, Silver, Gold).
  • Experience using various AWS services, including AWS Glue, Amazon S3, Amazon RDS.
  • Ability to design and implement data models, from initially defining schemas to optimizing data structures that support efficient queries and analytics.
  • Knowledge of performance optimization techniques for ETL jobs, such as partitioning, bucketing, caching, and troubleshooting to improve query performance.
  • Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements
  • Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Math, Physics, or Information Systems / Technology.

Similar jobs