Data Engineer

Key Responsibilities

• Design and develop scalable batch and streaming data pipelines using Apache Spark and Scala.

• Build distributed data-processing solutions for high-volume structured and semi-structured datasets.

• Develop reusable data frameworks, transformation logic, and ingestion components.

• Optimize Spark applications for performance, scalability, partitioning, memory utilization, and execution efficiency.

• Build and maintain ETL/ELT pipelines across enterprise data platforms.

• Develop real-time or near-real-time data pipelines using technologies such as Kafka.

• Integrate data from APIs, databases, files, event streams, and cloud storage platforms.

• Partner with Data Architects, Data Scientists, Product teams, and Software Engineers to translate business requirements into technical data solutions.

• Implement data-quality, reconciliation, lineage, monitoring, and error-handling controls.

• Build cloud-native data solutions using AWS services.

• Support CI/CD, automated testing, code quality, infrastructure automation, and production deployments.

• Troubleshoot data-processing failures and performance bottlenecks across distributed systems.

• Participate in Agile delivery, code reviews, design reviews, and production support.

Required Qualifications

• 5+ years of experience in Data Engineering, Big Data Engineering, or Software Engineering.

• Strong hands-on development experience with Scala.

• Strong expertise with Apache Spark / Spark SQL.

• Experience building large-scale distributed data pipelines.

• Strong SQL skills.

• Experience with ETL/ELT and data transformation.

• Strong understanding of distributed-computing concepts.

• Experience with data modeling and large-scale data processing.

• Experience with Git, CI/CD, and automated deployment practices.

• Strong problem-solving and production troubleshooting capabilities.

Preferred Qualifications

• Experience with AWS services such as:

  • S3

  • EMR

  • Glue

  • Lambda

  • Redshift

  • Athena

  • Kinesis

  • IAM

  • CloudWatch

• Experience with Kafka or other event-streaming platforms.

• Experience with Databricks.

• Experience with Delta Lake, Iceberg, or similar modern data-lake technologies.

• Experience with Python / PySpark in addition to Scala.

• Experience with Hadoop ecosystem technologies such as Hive or HDFS.

• Experience with NoSQL technologies such as DynamoDB or Cassandra.

• Experience with Docker and Kubernetes.

• Experience with Terraform or other Infrastructure-as-Code technologies.

• Experience building secure data platforms in highly regulated environments.

• Banking or financial-services domain experience is preferred.

Key Technical Skills

Programming: Scala, SQL, Python
Big Data: Apache Spark, Spark SQL, PySpark
Streaming: Kafka, Kinesis
Cloud: AWS – S3, EMR, Glue, Lambda, Redshift, Athena
Data Platforms: Databricks, Delta Lake, Hadoop, Hive
Databases: PostgreSQL, Oracle, SQL Server, DynamoDB, Cassandra
DevOps: Git, Jenkins/GitHub Actions, CI/CD, Docker, Kubernetes, Terraform
Data Engineering: ETL/ELT, Data Modeling, Data Quality, Data Lineage, Performance Optimization
Methodology: Agile / Scrum

Note- All candidate who are legally authorized to work in US without sponsership can apply