Staff Machine Learning Engineer

ELLKAY started out providing connectivity solutions to laboratories and within a few years, grew to also provide data management solutions to ambulatory organizations. ELLKAY is now a trusted data management partner in five healthcare segments. ELLKAY’s solutions continue to serve laboratories and ambulatory practices and have expanded to empower hospitals and health systems, healthcare IT vendors, ambulatory practices, health plans, and other healthcare organizations with cutting-edge technologies and solutions that drive their growth and interoperability strategies.
Today, ELLKAY remains true to our core values, building strong partner relationships and offering unparalleled service and support while providing innovative, scalable solutions to the challenges our customers face in today’s data-rich world.
ELLKAY's experience, customer-focused approach, and reputation for innovation, speed, and accuracy differentiate ELLKAY as a premier partner for your interoperability needs and data management strategy.

Job Description:

ELLKAY is seeking a Staff Machine Learning Engineer, AI Systems Integration who can design, integrate, and operate production-grade, autonomous agentic systems across products, data pipelines, APIs, enterprise applications, and workflow engines.

This role sits at the intersection of machine learning engineering, backend systems, integration architecture, and cloud infrastructure. You will be responsible for building the core orchestration runtime, Model Context Protocol (MCP) gateways, enterprise context layers, and automated governance harnesses that allow autonomous and semi-autonomous AI agents to execute multi-step workflows safely and reliably across enterprise ecosystems.

Essential Duties & Responsibilities:

1. Enterprise Context Layer & Data Systems

• Design and implement unified Enterprise Context Layers that combine structured relational/NoSQL data and unstructured document stores into high-performance hybrid retrieval engines.
• Architect real-time streaming ingestion pipelines (e.g., Kafka, EventBridge) balancing recency (via fast-path memory/cache layers) with long-term semantic relevance (via asynchronous vector indexing).
• Enforce granular security policies by embedding Row-Level Security (RLS) and Role-Based Access Control (RBAC) directly into retrieval queries so agents only retrieve context authorized for the active user session.

2. Model Context Protocol (MCP) Gateway Architecture

• Design and maintain enterprise Model Context Protocol (MCP) gateways to seamlessly connect heterogeneous tools, REST/gRPC microservices, and enterprise databases to agent ecosystems.
• Implement dynamic tool discovery mechanisms, schema validation (using Pydantic v2), and authentication delegation (OAuth2 token exchange and dynamic scope validation).
• Build resilient tool execution handles featuring bounded retry loops, fallback routing, circuit breakers, and structured error reporting for agent self-correction during timeouts or malformed tool responses.

3. Agentic Execution Fabric & Orchestration

• Structure multi-agent orchestration runtimes supporting both hierarchical manager-worker state graphs and asynchronous, event-driven agent meshes.
• Architect state isolation models using thread-level persistence snapshots (Aurora PostgreSQL, DynamoDB) to maintain clear agent context boundaries.
• Implement execution circuit breakers (token/depth limits), human-in-the-loop (HITL) escalation boundaries for high-risk actions, and reflection loops for autonomous self-correction.
• Establish automated evaluation harnesses utilizing trajectory benchmarking, tool-selection precision evaluation, held-out test sets, and LLM-as-a-judge scoring to continuously govern system performance at scale.

4. Platform, Infrastructure & Developer Tools

• Build high-performance backend microservices using Python (FastAPI, Pydantic, AsyncIO) and write production-grade Infrastructure as Code using AWS CDK (TypeScript/Python).
• Deploy cloud-native agent runtime environments leveraging AWS Lambda, API Gateway, Step Functions, EventBridge, SQS, ECS Fargate, Aurora Serverless, and DynamoDB.
• Leverage modern AI development tools and agentic coding workflows (e.g., Claude Code, OpenAI Codex, GitHub Copilot) to maximize developer velocity, code quality, and implementation speed.
• Implement full observability across agent execution trajectories using structured logging, distributed tracing (OpenTelemetry, CloudWatch, X-Ray), and token/cost tracking.

Qualifications:

• 10+ years of progressive experience in software engineering, machine learning engineering, or distributed systems, with significant ownership of production platforms.
• Deep Agentic Systems Expertise: Hands-on experience building and deploying stateful agentic workflows, multi-agent frameworks (e.g., LangGraph, AutoGen, CrewAI, Bedrock Agents), dynamic tool/function calling, and agent memory systems.
• Context Layer & Security: Proven track record designing enterprise context engines, hybrid retrieval systems (dense/sparse vector search + re-ranking), and strictly enforcing user-level RLS/RBAC access control within vector/retrieval layers.
• MCP & Tool Integration: Experience architecting tool gateway solutions (such as MCP), contract/schema validation, authentication delegation, and robust error handling for external tool invocations.
• Orchestration & Resiliency: Deep knowledge of state isolation, multi-agent coordination patterns, loop circuit breakers, human-in-the-loop intervention protocols, and automated agent evaluation frameworks.
• Core Tech Stack: Advanced proficiency in Python (FastAPI, Pydantic, AsyncIO). Go/Rust a plus.
• Cloud & DevOps: Deep expertise with AWS services (Lambda, ECS Fargate, Step Functions, EventBridge, DynamoDB, Aurora), AWS CDK, Docker, and streaming/messaging systems (Kafka, EventBridge, SQS).
• AI-Assisted Development: Experience leveraging modern AI developer tools (Claude Code, Codex, Cursor, Copilot) to accelerate architecture design, prototyping, and production engineering.

Preferred Qualifications

• Degree in Computer Science, Machine Learning, Electrical Engineering, or a related quantitative field.
• Experience building or extending enterprise middleware, integration engines, or API gateways (e.g., Workato, MuleSoft, Apache Camel).
• Experience with automated trajectory evaluation tools (LangSmith, Arize, Phoenix, RAGAS) and LLM-as-a-judge methodologies.

Benefits:

ELLKAY offers a comprehensive and competitive benefit package that starts day one!

Including:

• Medical, Dental, and Vision benefits
• Employer-paid Life and LTD
• 401k w/ matching – once eligibility is met
• Work/life balance
• Paid Volunteer Program
• Flexible working hours
• Generous FTO
• Remote work options
• Employee Discounts
• Parental Leave
• Gym membership / Exercise class stipends

Our awesome culture includes:

• Working with talented, collaborative, and friendly people who love what they do
• Professional growth within
• Innovation environment
• On site in HQ Free daily lunches

Awards:

• 2025 Top Workplaces Employee Appreciation
• 2025 Top Workplaces Employee Well-Being
• 2025 Top Workplaces Professional Development
• 2025 Top Workplaces Leadership
• 2026 USA Today Top Workplaces
• 2026 NJ Top Workplaces

Additional information

At ELLKAY, we are committed to operating as a hybrid workplace, offering employees flexibility in how they structure their time between in-office and remote work. We recognize the significance of fostering connections, collaboration, and creativity within our office culture and its positive impact on our business. Our philosophy of operating as a hybrid workplace underscores our dedication to enabling employees to tailor work-life balance to their individual preferences. For those who do not live within 40 miles of one of our offices, we are open to considering remote work for candidates whose skills and experience strongly align with the role. While we prioritize a hybrid work environment for most roles, we understand the importance of flexibility and are open to remote work for specific positions and specialized skill sets.

For more information on our company, visit ;

ELLKAY LLC is a Smoke-Free Workplace.

ELLKAY, LLC provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

Compensation Range: $210,000-$230,000