Lead Data Engineer

Role: Lead Data Engineer -- Location:(Charlotte)

We’re seeking an experienced pipeline-centric data engineer to put it to good use in building out ETL and Data Operations framework (Data Preparation / Normalization and Ontological processes).

Technical Skills:

•             Lead the design, development, and maintenance of scalable data pipelines and ETL processes, ensuring data integrity and accessibility for business intelligence and advanced analytics.

•             Architect and manage robust data platforms on the AWS ecosystem, leveraging services like Glue, PySpark, Apache Iceberg, IAM, S3, and Secrets Manager to build a secure and efficient data infrastructure.

•             Provide technical guidance and mentorship to a team of data engineers, fostering a culture of high performance and continuous learning.

•             Utilize deep expertise in various RDBMS (e.g., MySQL, Db2, PostgreSQL, Snowflake) and different data formats (e.g., JSON, Parquet) to drive strategic data initiatives.

•             Champion the adoption of modern data technologies such as Apache Iceberg with AWS Glue to optimize data lake performance and analytics capabilities.

•             Apply advanced proficiency in Generative AI to automate and streamline data analysis, development, and documentation processes.

•             Business Acumen and Stakeholder Communication

•             Translate complex business challenges into clear, data-driven solutions, effectively communicating technical concepts and project progress to both technical and non-technical stakeholders, including senior leadership.

•             Act as a key liaison between the data engineering team and business units, providing data-backed insights and recommendations that directly influence business strategy.

•             Manage concurrent projects in a dynamic, research-oriented environment, ensuring timely delivery and high-quality outcomes.

•             Experience in insurance domain preferrable

•             AWS Data Engineering certification good to have

Key Responsibilities

•             Collaborate with business analysts and stakeholders to translate business needs into comprehensive source-to-target (S2T) data mappings.

•             Lead the design and development of robust data pipelines, using your deep understanding of existing ETL frameworks to build scalable and efficient solutions.

•             Analyze and understand diverse source systems and data formats to create and validate synthetic datasets for testing and development.

•             Develop automated data validation tools using Python to ensure data integrity across various ETL layers.

•             Ensure Industry best practice is followed in all aspects of project.

•             Develop robust development testing approach to minimize the quality, integration and user testing issues.

•             Leverage generative AI to enhance data analysis, accelerate development, Unit Testing and streamline documentation.

•             Apply statistical analysis to large datasets, uncovering key patterns, identifying potential challenges, and generating actionable insights.

•             Partner with business leaders to understand their challenges and provide data-driven recommendations that improve processes and inform strategic decisions.

Qualification:

•             Somebody who has at least 12+ years of data engineering experience has played Lead Data Engineer role.

•             Bachelor’s degree (or equivalent) in computer science, information technology, engineering, or related discipline

•             Education qualification: Any degree from a reputed college