Why this role stands out
This hybrid role offers a fantastic opportunity to advance your career by designing and building cutting-edge data platforms using technologies like Spark, Kafka, and Databricks within a reputable consulting firm. You'll thrive here if you possess strong data engineering skills and a passion for solving complex data challenges, with the flexibility to work both remotely and in the office. Apply now to shape the future of data at West Coast Consulting LLC!
Quick overview
Job description
Job Description
Position: Senior Data Engineer / Data Platform Engineer
We are looking for a Senior Data Engineer with strong experience building and supporting large-scale data platforms, distributed data pipelines, and real-time data processing systems. The ideal candidate should be hands-on with Spark, Kafka, Databricks, Airflow, and cloud data technologies and comfortable working with high-volume enterprise data environments.
Key Responsibilities
Design, develop, and maintain scalable batch and real-time data pipelines. Build distributed data processing solutions using Apache Spark and Spark Structured Streaming. Develop streaming and event-driven data pipelines using Apache Kafka/Kafka Connect. Build and optimize ETL/ELT workflows using Python, Java, and SQL. Work with Databricks and modern lakehouse technologies such as Delta Lake or Apache Hudi. Develop and manage data workflows using Apache Airflow. Design data models, partitioning strategies, schema evolution, and data quality frameworks.
Monitor pipeline performance, data freshness, reliability, and operational health. Work with cloud platforms such as Google Cloud Platform or Azure and technologies including BigQuery, Cloud Storage, Dataflow, Kubernetes, or similar. Collaborate with analytics, ML, product, and infrastructure teams to build scalable data solutions. Participate in architecture discussions, code reviews, troubleshooting, and performance optimization. Implement CI/CD, automated testing, and engineering best practices for data platforms.
Required Skills
8+ years of experience in Data Engineering or related roles. Strong hands-on experience with Apache Spark / Spark Structured Streaming. Strong experience with Kafka and real-time/event-driven data pipelines. Strong experience with Databricks. Strong experience with Python and SQL; Java or Scala is highly preferred. Experience with Apache Airflow or similar workflow orchestration tools. Experience building large-scale ETL/ELT pipelines. Experience with cloud data platforms, preferably Google Cloud Platform or Azure. Experience with data lakes/lakehouse architectures.
Strong understanding of distributed systems, data modeling, performance tuning, and data quality. Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or similar. Preferred
Experience with Delta Lake or Apache Hudi. Experience with BigQuery, GCS, Dataflow, Cassandra, Presto/Trino, or similar technologies. Experience with Kubernetes and Docker. Experience with streaming data and CDC. Experience working with ML/analytics data platforms. Experience establishing data contracts, SLAs, monitoring, and observability. Enterprise-scale data platform experience with very large datasets.
Strong communication and stakeholder-management skills.
Similar jobs
- OR

Principal Core Infrastructure Engineer-Network CapacityRecently posted Auto Apply
Oracle · Seattle · Hybrid · Posted 16 Nov 2026 🇺🇸
OracleSQLAWS+8Technology$114.6k - $234.6k/yr - OR

Principal Core Infrastructure Engineer-Network CapacityRecently posted Auto Apply
Oracle · Nashville · Hybrid · Posted 16 Nov 2026 🇺🇸
OracleSQLAWS+8Technology$114.6k - $234.6k/yr - AF
Cloud DevOps Engineer with Security ClearanceRecently posted
Accenture Federal Services · Eglin AFB, FL · On-site · Posted 17 Nov 2026 🇺🇸
DockerGitAWS+20Technology$91.7k - $178.7k/yr - TE
Remote || US Technical SRE Architect - Multi Cloud Site ReliabilityRecently posted
Techridge, Inc. · United States · Remote · Posted 17 Nov 2026 🇺🇸
AWSSplunkDatadog+5TechnologySalary not listed - PT

AI LEAD PLATFORM ENGINEERRecently posted
PROCYON Technostructure · Chicago, IL · On-site · Posted 17 Nov 2026 🇺🇸
MLOpsGenerative AIAzure+2TechnologySalary not listed - DS
Junior Site Reliability Engineering (SRE)Recently posted
Decision Six Inc. · Alpharetta, GA · Hybrid · Posted 16 Nov 2026 🇺🇸
OracleShellAWS+8TechnologySalary not listed - RL

ForgeRock Platform EngineerRecently posted
Resource Logistics · Texas City, TX · On-site · Posted 16 Nov 2026 🇺🇸
SpringSpring BootOAuth+9TechnologySalary not listed - ES
Databricks Platform Engineer- Unity Catalog and DevOpsRecently posted
eSolutionsFirst, LLC · Philadelphia, PA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLGitApache Spark+12TechnologySalary not listed - TE

Google Cloud Platform DevOps & Data Engineer/Data ScientistRecently posted
Technogen, Inc. · Alpharetta, GA · On-site · Posted 17 Nov 2026 🇺🇸
API GatewayEncryptionBigQuery+11TechnologySalary not listed - EH

Senior Data Platform EngineerRecently posted Auto Apply
Ellipsis Health · San Francisco - Hybrid · Hybrid · Posted 17 Nov 2026 🇺🇸
GCPSQLGit+22Technology$150k - $170k/yr - AE
Network Automation Engineer with Security ClearanceRecently posted
Anonymous Employer · Chantilly, VA · Hybrid · Posted 17 Nov 2026 🇺🇸
AWSELKAnsible+12TechnologySalary not listed - PG

Network Automation Engineer with Security ClearanceRecently posted
PlanIT Group LLC · Chantilly, VA · Hybrid · Posted 17 Nov 2026 🇺🇸
Load BalancingAnsibleGo+6TechnologySalary not listed