Data Engineer

• Design and develop scalable batch and real-time data pipelines.

• Build data-processing solutions using Python, PySpark, Spark, and SQL.

• Develop and support streaming pipelines using Apache Kafka.

• Ingest, transform, validate, and publish large volumes of structured and semi-structured data.

• Develop reusable ETL/ELT frameworks and data integration components.

• Optimize Spark jobs, SQL queries, and data pipelines for performance and scalability.

• Integrate data from databases, APIs, files, messaging platforms, and cloud data services.

• Implement data-quality checks, reconciliation, exception handling, and monitoring.

• Support production data pipelines and perform root-cause analysis for failures and performance issues.

• Implement secure data-access patterns, encryption, masking, and role-based access controls.

• Participate in code reviews, testing, CI/CD deployments, and production releases.

• Collaborate with data architects, business analysts, and engineering teams to translate business requirements into technical solutions.

• Maintain technical documentation, data mappings, operational procedures, and support runbooks.

Required Skills

• 6+ years of Data Engineering experience.

• Strong hands-on experience with Python.

• Strong SQL and relational database experience.

• Experience with Apache Spark / PySpark.

• Experience with Apache Kafka and real-time/event-driven data processing.

• Strong understanding of ETL/ELT and enterprise data pipelines.

• Experience working with large-volume datasets and distributed processing.

• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.

• Experience with data warehouses/lakehouse platforms such as Snowflake, Databricks, Redshift, or similar.

• Strong understanding of data modeling and data warehousing concepts.

• Experience with Git and CI/CD processes.

• Strong troubleshooting and production-support experience.

Preferred Skills

• Banking / Financial Services experience.

• Experience with Databricks and Delta Lake.

• Kafka Streams / Confluent Kafka experience.

• Airflow or other workflow-orchestration tools.

• AWS services such as S3, Glue, EMR, Lambda, and Redshift.

• Data governance, lineage, security, and regulatory-data experience.

• Docker / Kubernetes exposure.

• Experience working in Agile/Scrum environments.