Quick overview
Job description
Key Responsibilities:
- Experience: 6 to 10 years Realtime experience on databricks is must.
- Collaborate as part of a development team to design and enhance large scale applications developed using Python, Spark & Pyspark .
- Realtime experience on databricks is must.
- Evaluates and plans software designs, test results and technical manuals using AWS.
- Confer with business units and development staff to understand both the business and technical requirements for producing technical solutions.
- Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
- Extend and enhance the business Data Lake.
- Create or implement solutions for metadata management.
- Solve for complex data integrations across multiple systems.
- Design and execute strategies for real-time data analysis and decisioning.
- Build robust data processing pipelines using AWS Services and integrate with multiple data sources.
- Translating client user requirements into data flows, data mapping, etc.
- Analyses and determines data integration needs and follows Agile practices.
Required Skills:
- At least 4+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python.
- At least 4 years of experience with Python, Spark & Pyspark.
- At least 3 years of experience working on AWS technologies.
- Experience of designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lamda, Kinesis using databases Aurora and Redshift.
- Experience with Spark programming (Pyspark or scala).
- Hands on experience with AWS components like (EMR, S3, Redshift, Lamdba, API Gateway, Kinesis ) in production environments.
- Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.
- Experience using ETL tools for data ingestion.
- Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle and DB.
- Ability to translate data needs into detailed functional and technical designs for development, testing and implementation.
Similar jobs
- ID
Sr AWS Data Engineer (with ETL, Python, AWS Glue, Lambda, Node.js & Microservices exp)Recently posted
Info Dinamica Inc · Malvern, PA · On-site · Posted 16 Nov 2026 🇺🇸
DynamoDBMicroservicesOracle+18TechnologySalary not listed - PG

Data Engineer – Databricks, PySpark, Cloud | Hybrid - 71701Recently posted
PRIMUS Global Services Inc. · Princeton, NJ · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+4TechnologySalary not listed - PH

Data EngineerRecently posted
Photon · Sunnyvale, CA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLAWSETL+6TechnologySalary not listed - GE

SQL Server DBARecently posted
Genesis10 · Charlotte, NC · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLShellMicrosoft Office+3Technology$53 - $61/hr - NT
Oracle DBA with Security ClearanceRecently posted
NorthHill Technology Resources · Fairfax, VA · Hybrid · Posted 17 Nov 2026 🇺🇸
MariaDBMongoDBOracle+6TechnologySalary not listed - IT

ETL LeadRecently posted
ITSYNTAX INC · Dallas, TX · Hybrid · Posted 16 Nov 2026 🇺🇸
OraclePL/SQLSQL+6Salary not listed - VI

Sr. Data EngineerRecently posted
Visa · Foster City, California · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+14Technology$131.9k - $204k/yr - VI

Staff Data EngineerRecently posted
Visa · Foster City, California · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLScalaApache Spark+13Technology$156.8k - $251.1k/yr - FA

Systems Engineer (Data Flow/NiFi) Level 2Recently posted Auto Apply
Fantom Corporation · Fort Meade · On-site · Posted 17 Nov 2026 🇺🇸
ApacheComplianceComputer Vision+5TechnologySalary not listed - LS

Senior Database EngineerRecently posted
London Stock Exchange Group · St Louis, MO · Hybrid · Posted 17 Nov 2026 🇺🇸
OracleShellAWS+5TechnologySalary not listed - KT

Senior Data EngineerRecently posted
Kforce Technology Staffing · Atlanta, GA · Hybrid · Posted 17 Nov 2026 🇺🇸
SQLETLAirflow+5TechnologySalary not listed - QT

Data Engineer LeadRecently posted
QUANTUM TECHNOLOGIES LLC · United States · Hybrid · Posted 17 Nov 2026 🇺🇸
DockerSQLGit+12Technology$83/hr