PySpark Consultant

Sharing a brief JD as below:

• 10+ years of relevant experience in Apps Development or systems analysis role with extensive Python and/or big data expertise
• Python/Pyspark Mastery: Expert-level capability in developing and maintaining complex data processing and analytical solutions using Python/Pyspark, including advanced scripting and framework development.
• Big Data Technologies & Platforms: Extensive experience (8+ years) architecting, designing, implementing, and managing solutions within distributed data processing platforms, specifically with the Hadoop ecosystem (preferably Cloudera distributions). Proficient in leveraging key big data components such as distributed file systems (e.g., HDFS), data warehousing solutions (e.g., Hive), data transformation frameworks (e.g., Pig), and data ingestion tools (e.g., Sqoop), alongside hands-on experience with NoSQL databases (preferably MongoDB).
• ETL Architecture & Development: Proven ability to architect, design, and implement highly scalable data pipelines. Extensive experience leveraging industry-standard ETL tools and frameworks for efficient data extraction, transformation, and loading into various relational databases and data warehouses, coupled with a strategic vision and planning experience for migrating to cloud-native, serverless ETL solutions.
• Data Architecture & Strategic Modeling: Understanding of advanced data modeling principles and practical experience in data warehouse design and development, ensuring data integrity, scalability, security, and optimal performance
• AI-Powered Development: Ability to leverage advanced AI tools, such as Devin, for efficient code refactoring, optimization, and identifying potential code improvements, thereby enhancing code quality and developer productivity.
• DevOps, Version Control & Containerization: CI/CD pipelines, Git, Docker and Kubernetes.
• Technical Leadership: Ability to act as a development lead, focusing on inculcating best practices, guiding junior developers, and taking the initiative to reengineer and re-architect generic frameworks to enhance efficiency and scalability.
• Extensive experience system analysis and in programming of software applications
• Experience in managing and implementing successful projects
• Subject Matter Expert (SME) in at least one area of Applications Development
• Ability to adjust priorities quickly as circumstances dictate
• Demonstrated leadership and project management skills
• Consistently demonstrates clear and concise written and verbal communication

Education:

• Bachelor’s degree/University degree or equivalent experience
• Master’s degree preferred