Big Data Lead with Databricks Experience (VISA INDEPENDENT CANDIDATES ONLY)

Responsibilities:

• Design & Implement Data ingestion and Data lakes-based solutions using Big Data Technologies.
• The Tech Lead should be highly proficient in the use of Big Data / Open-Source Technologies and standard techniques of Data Integration, Data Manipulation.
• Should be able to design and develop cost efficient and performant data pipelines in the cloud platform
• Create data environment to support our data analytics, reporting and data science teams
• Experience with integration of data from multiple data sources
• Knowledge of various Data Pipeline techniques and frameworks
• Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
• Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.

The Role offers:

• Great opportunities to learn various tools and technologies used in a sophisticated data architecture within the Business Intelligence and Analytics Data Services
• Gives an opportunity to showcase candidates strong analytical skills and problem-solving ability
• An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations
• Grow into a Technical architect role over a period

Essential Skills:

• 6+ Years hands on knowledge on SQL as well as SQL/NoSQL databases
• Proficient in programming languages such as Python, PySpark, Scala and Java
• Experience with Spark , Databricks
• Working knowledge of XML, ETL, API and Web Services
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
• Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or Google Cloud Platform)
• Experience in leading offshore teams