Quick Overview
Job Description
Responsibilities
- Design and implementation of scalable big data pipelines for batch and real time
processing
- Build and operate streaming data platforms using Kafka
- Design and deploy cloud native data solutions across AWS
- Develop and manage containerized workloads using Kubernetes
- Enable data mesh architecture with domain oriented data products
- Design and implement data lake and lakehouse architectures
- Ensure data quality, reliability, and observability
- Implement governance capabilities including metadata and lineage
- Collaborate with analytics, AI, and business teams
- Optimize performance and cost efficiency
- Mentor engineering teams
Requirements
- 8+ years of experience in big data engineering
- Strong experience with Kafka
- Strong experience with Kubernetes
- Experience with AWS, Azure, and Google Cloud Platform
- Experience building data lakes or lakehouse platforms
- Experience with data mesh concepts
- Strong programming skills in Python, Scala, or Java
- Experience with Spark, Flink, or Beam
- Experience with Airflow or orchestration tools
- Understanding of data modeling and governance
- Experience with CICD and infrastructure automation
- Experience supporting AI and machine learning workloads including large language
models such as Claude or similar platforms
- Strong communication and leadership skills
Similar jobs
- SY
Senior Data Engineer with Credit Risk
Synechron
New York, NY🇺🇸$140k - $155k/yrHybrid6 weeks agoSQLSQL ServerETL+6Technology - RM
Data Engineer Software Developer - Rmantras with Security Clearance
NewRMantras
Scott AFB, IL🇺🇸On-siteYesterdaySAFeGCPSQL+29Technology - OP
Data Engineer (AWS, Spark)
NewOpenDataJobs
Washington, District of Columbia🇺🇸Hybrid10 hours agoDynamoDBSQLAWS+8Technology - RH
Data Engineer
NewRobert Half
Pittsburgh, PA🇺🇸HybridYesterdaySQLETLSnowflake+9Technology - AA
DATA ENGINEER
AaraTechnologies Inc
Virginia Beach, VA🇺🇸Hybrid7 weeks agoSQLAWSETL+1Technology - PA
Data Engineer (Snowflake)
NewProminence Advisors
United States🇺🇸Remote15 hours ago401kGCPOracle+22Technology