Senior Data Engineer

Description 

The Senior Data Engineer will play a critical implementation role on the Data Engineering and  Data Services team and be responsible for data pipeline solutions design and development, 

troubleshooting, and optimization tuning on the next generation data and analytics platform being developed with leading edge big data technologies in a highly secure cloud infrastructure. 

The Data Engineer will serve as a liaison to platform user groups ensuring successful  implementation of capabilities on the new platform. The Senior Data Engineer will also take a 

lead role on functional teams or projects. 

Senior Data Engineer Responsibilities: 

Deliver end-to-end data and analytics capabilities, including data ingest, data transformation, data science, and data visualization in collaboration with Data and Analytics stakeholder groups 

Design and deploy data pipelines to support analytics and client projects 

Develop scalable and fault-tolerant workflows 

Clearly document issues, solutions, findings and recommendations to be shared  internally & externally 

Demonstrate strong knowledge of data warehousing and data management concepts, including 3NF, star schema, Data Vault, Medallion Architecture, data governance, master data management, and reference data management. 

Learn and apply tools and technologies proficiently, including: 

Languages: SQL (standard and DB-specific), Python, Scala, Bash 

Framework: Apache Iceberg / Lakehouse, Spark, Kafka 

Data Platform: Snowflake or Databricks 

Tools/Products: dbt, Airflow, replication tools, semantic layer tools 

Gen AI: Agents, AI First Development 

Cloud Computing: AWS  

Performance optimization for queries and dashboards 

Develop and deliver clear, compelling briefings to internal and external stakeholders on findings, recommendations, and solutions 

Analyze client data & systems to determine whether requirements can be met 

Test and validate data pipelines, transformations, datasets, reports, and dashboards  built by team 

Develop and communicate solutions architectures and present solutions to both business and technical stakeholders 

Provide end user support to other data engineers and analysts 

Candidate Requirements 

* Expert experience in the following: 

o SQL, Python, PySpark. Other programming languages (R, Scala, SAS, Java, etc.) are a plus 

o Data and analytics technologies including SQL/NoSQL/Graph databases, ETL, and BI 

o Knowledge of CI/CD and related tools such as Gitlab, AWS CodeCommit, etc  

o AWS services including EMR, Glue, Athena, Batch, Lambda Cloudwatch, DynamoDB, EC2, Cloudformation, IAM and EDS 

* Solid scripting skills (e.g., bash/shell scripts, Python) 

* Proven work experience in the following: 

o Data streaming technologies 

o Data technologies including, Spark, Snowflake, dbt, etc. 

o Linux command-line operations 

o Networking knowledge (OSI network layers, TCP/IP, virtualization) 

* Candidate should be able to lead the team, communicate with business, gather and interpret business requirements 

* Experience with agile delivery methodologies using Jira or similar tools 

* Experience working with remote teams 

* AWS Solutions Architect / Developer / Data Analytics Specialty certifications, 

Professional certification is a plus 

* Bachelor Degree in Computer Science or relevant field, Masters Degree is a plus 

* 10-12 years of relevant experience or equivalent combination of experience and education

Optimize prompts, embeddings, context retrieval, and AI workflows for accuracy and performance.
Integrate enterprise systems including Microsoft Graph, Salesforce, ServiceNow, Jira, SharePoint, and other SaaS platforms.
Develop secure, scalable cloud-native applications on AWS, Azure, or Google Cloud Platform.
Containerize applications using Docker and deploy through Kubernetes and CI/CD pipelines.
Monitor AI application performance, latency, token usage, and model quality.
Follow AI governance, responsible AI, and security best practices.