Big Data Engineer (Exp-7 Y) with PySpark exp-Full time-Hybrid-F2F time

• Exp  6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
• Unix SQL and Shell Scripting experience is a must have
• Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines
• Strong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts
• Expertise in SQL querying and complex joins
• Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
• Expertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
• Strong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption
• Expertise in Data quality checks and monitoring
•  Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
• Handson with Automation Framework Design for ETL and API
• SME in Data Analysis Database testing Messaging queues
• Experience with coding standards code reviews source management build processes CICD pipeline

Skills

Mandatory Skills : Apache Spark, Big Data Hadoop Ecosystem, Python, Python for DATA, SparkSQL