Lead DevOps Engineer – AWS / Kubernetes / Helm

Role Description(W2 Hiring)

We are seeking a Lead DevOps CI/CD Engineer to manage platform deployment, infrastructure troubleshooting, enterprise integrations, security configurations, and environment-specific customizations for a Surveillance solution.

The ideal candidate will have strong hands-on experience with Kubernetes, Helm, Docker, cloud infrastructure, networking, storage, security, and production environment troubleshooting.

Key Responsibilities & Daily Tasks

1. Platform Deployment & Orchestration

• Deploy, upgrade, and maintain the Surveillance solution using provided Helm charts and Docker containers across UAT and Production environments.

• Execute deployments using Helm commands and manage custom values files for customer-specific environments.

• Manage high-availability scaling topologies and orchestrate multi-container pods, including Tomcat application dashboards paired with local OneTick network servers.

• Maintain backend architectures and ensure reliable platform deployment and operation.

2. Infrastructure Troubleshooting & Root-Cause Analysis

• Troubleshoot runtime failures within Kubernetes clusters, including volume mount failures, subPath initialization issues, and pod restarts.

• Analyze file system and shared storage permissions, ensuring container runtime users (runAsUser/securityContext) align with corporate NFS, AWS EFS, and SGW POSIX permissions.

• Troubleshoot storage access issues, including path traversal requirements and permissions such as 775/777.

• Investigate multi-container sidecar data synchronization issues, including rsync permission-denied errors (Code 13) within persistent memory spaces and shared directories.

• Identify and address security-related concerns onsite or proactively document product changes required for resolution by the OMD DevOps team.

3. Enterprise Integration & Security Configuration

• Configure and troubleshoot Federated Single Sign-On (SSO), integrating customer Identity Providers (IdP), including Azure AD, Okta, and Ping, with product brokerage using SAML 2.0 and OpenID Connect.

• Implement secure metadata exchanges, user attribute mapping, and group-based access control using enterprise Object IDs.

• Configure and troubleshoot secure connectivity for cloud-managed architectures, including CloudSQL Auth Proxies, database credential vaulting, and cloud storage integrations.

• Configure AWS Application Load Balancers (ALB), including ingress annotations, SSL/TLS termination, pass-through targets, and HTTP-to-HTTPS redirection.

4. Environment-Specific Customizations & Scaled Engineering

• Manage, isolate, and maintain tenant-specific deviations and infrastructure overrides within decoupled configuration directories.

• Maintain and calibrate customized environment layers for dedicated customer accounts.

• Design, deploy, and validate Load Balancer mappings, multi-region ingress architectures, and automated pod-scaling configurations.

5. Observability & Monitoring Maintenance

• Maintain localized log collection and log-shipping sidecars, including Grafana Promtail and Loki, to capture and store multi-container logs that do not natively pipe to standard output.

• Configure and maintain container-level resource requests, limits, and priority settings.

• Ensure platform dependencies are properly managed within infrastructure resource constraints.

6. Communication & Collaboration

• Independently troubleshoot infrastructure, networking, and environment-specific issues within customer tenancy.

• Identify and isolate application-related defects from infrastructure issues.

• Prepare detailed diagnostic reports for the remote engineering team, including:

  • kubectl describe pod and kubectl logs outputs.

  • Engine trace dumps and OS stack traces.

  • Product logs and supporting diagnostic information.

• Provide clear, actionable, and analytical bug reports to support remote engineering investigation and resolution.