Microsoft Fabric Data Engineer

Let’s create our future together at The AES Group!

About The AES Group:

The AES Group is a premier technology consulting company that has been bringing businesses and talent together for over 20 years to deliver the most innovative technology solutions that create the most positive impact on society. AES has helped over 40 business enterprises, including Fortune 500 companies, engage their customers, empower their employees, and transform their business operations with the power of cloud, data, AI, and other emerging technologies. 

Our client  is building a governed data and AI platform, integrating device, laboratory, partner, and document data into a unified foundation that supports regulatory reporting, advanced analytics, and AI-driven scientific insights

The Data Engineer is a hands-on builder responsible for developing data pipelines, API integrations, and AI infrastructure that bring structured and unstructured data into a governed Azure-based architecture. This delivery-focused role requires designing, coding, testing, and maintaining production-ready solutions across both AI document ingestion and structured ETL/ELT data engineering tracks.

Key Responsibilities

Data Engineering & Integration

• Design, build, and maintain data pipelines that integrate laboratory, quality, design, and partner data into Azure Fabric Lakehouse and PostgreSQL using a Bronze-Silver-Gold architecture.
• Develop ETL/ELT processes with data validation, harmonization, auditability, data quality controls, and regulatory-compliant lineage from source to governed datasets.
• Build AI-ready document ingestion pipelines and MCP connectors for systems such as SharePoint, Qdocs/Veeva, Jama, and TurboAC, supporting semantic search, retrieval, and enterprise LLM applications.
• Develop partner data ingestion workflows leveraging OCR, LLM-based extraction, confidence scoring, and exception handling for external CMO/CRO documentation.
• Implement monitoring, alerting, operational documentation, and data lineage across all pipelines and integrations.

API & Platform Engineering

• Develop and maintain RESTful API and instrument data connectors with secure authentication, error handling, schema management, and production-grade reliability.
• Integrate Azure OpenAI (or equivalent) services into retrieval-augmented generation (RAG) solutions using a vendor-agnostic architecture.
• Build and support cloud-native data solutions across Azure Fabric and AWS, including containerized services, CI/CD pipelines, infrastructure security, secrets management, and cross-cloud data movement.

Data Modeling & Governance

• Implement and maintain PostgreSQL and medallion-layer data models, including harmonization logic, referential integrity, lineage tracking, and governed data standards.
• Partner with Data Architects and WBWD stakeholders to evolve schemas, data structures, and integration patterns as business requirements mature

Basic Qualifications

• 5+ years of hands-on data engineering or backend engineering experience delivering production-grade data pipelines, integrations, and relational data models.
• Strong experience building API integrations and data connectors using REST APIs, OAuth, webhooks, and modern integration frameworks; experience with MCP or similar protocols preferred.
• Experience designing and implementing document ingestion and AI search solutions, including OCR, chunking, embeddings, vector databases, and retrieval-augmented generation (RAG) using Azure OpenAI, OpenAI, Claude, or similar technologies.
• Proven expertise delivering production workloads on Microsoft Azure, including Azure Fabric (Lakehouse, Data Factory, Pipelines, Delta Lake), Azure Functions, Storage, and AI Services.
• Hands-on AWS experience supporting data engineering solutions using services such as S3, Lambda, API Gateway, Glue, and RDS/Aurora.
• Strong proficiency in Python and/or PySpark for pipeline orchestration, data transformation, API development, testing, and CI/CD automation.
• Experience designing and supporting ETL/ELT pipelines with data validation, quality controls, monitoring, error handling, and operational reliability.
• Advanced knowledge of PostgreSQL, relational database design, schema modeling, query optimization, and medallion architecture (Bronze/Silver/Gold) data platforms.
• Experience implementing data lineage, auditability, and governance controls within regulated or compliance-sensitive environments, including familiarity with ALCOA+ principles

Preferred Qualifications

• Experience in pharmaceutical, biotechnology, or medical device development environments —working knowledge of regulated data requirements is a strong differentiator
• Familiarity with GxP data integrity requirements (21 CFR Part 11, ALCOA+) and their practical implications for data pipeline design, electronic records, and audit trail implementation
• Direct experience with any DDCS or WBWD systems: Oracle Agile PLM / Siemens Teamcenter, LabVantage LIMS, Darwin, Veeva Vault/QDocs, SmartLab/Biovia LES, NuGenesis NG9
• Microsoft Azure Data Engineer Associate or AWS Certified Data Engineer certification
• Experience with infrastructure-as-code (Bicep, Terraform) and GitOps deployment patterns for Azure-native data workloads
• Familiarity with LangChain, semantic chunking strategies, or embedding model selection for domain-specific scientific document retrieval
• Experience with Microsoft Fabric Unity Catalog, Delta Sharing, or governed data sharing patterns across workspaces
• Experience with Docker and Kubernetes-based orchestration for data pipeline services in production environments