Recently postedTechnology

Data Engineer (Databricks) -- Hybrid -- Only W2

Trebecon LLCBoston, MA🇺🇸United States

Quick overview

SeniorityMid Senior

Work modeOn Site

LocationBoston, MA, United States

SQLScalaApache SparkAWSETLApacheDatabricks

Job description

Role : Data Engineer (Databricks)
Location: Boston, MA – Hybrid (Local Candidates Preferred)
Experience: 10+ Years

Key Responsibilities

Design, develop, and maintain scalable data ingestion pipelines, ETL workflows, and data processing jobs.
Develop and optimize Apache Spark applications using Scala and PySpark.
Build data pipelines to ingest, transform, validate, and process structured and semi-structured data.
Write complex SQL queries for data transformations, validation, reconciliation, and performance optimization.
Configure and manage Databricks workspaces, clusters, jobs, and workflows.
Schedule, monitor, and troubleshoot Databricks jobs and data pipelines.
Optimize Spark applications for performance, scalability, reliability, and cost efficiency.
Integrate Databricks solutions with AWS services and cloud-based data platforms.
Implement data quality checks, error handling, logging, and monitoring.
Troubleshoot pipeline failures, data processing issues, and performance bottlenecks.
Collaborate with data architects, analysts, and engineering teams to translate business requirements into technical solutions.
Follow coding standards, data governance policies, and engineering best practices.
Support production deployments, maintenance, and continuous improvement.

Required Skills and Qualifications

10+ years of overall IT experience, with strong hands-on Scala/Spark data engineering experience.
Strong programming expertise in Scala and Apache Spark.
Advanced SQL skills, including complex queries, joins, aggregations, and query optimization.
Hands-on experience developing ETL pipelines, data ingestion workflows, and data processing jobs.
Experience with AWS services and cloud-based data engineering solutions.
Strong understanding of distributed computing, data transformations, and pipeline architecture.
Experience troubleshooting and optimizing Spark applications.
Strong analytical, problem-solving, and communication skills.
Ability to complete technical coding assessments and explain implementation decisions.

Preferred Skills

Hands-on experience with Databricks.
Databricks workspace and cluster configuration and management.
Databricks job scheduling and workflow management.
Experience building production-grade ETL and ingestion pipelines.
Familiarity with data quality validation, pipeline monitoring, and production support.