Senior Unix/Linux System Admin

Senior Unix/Linux System Administrator

Contract right now is till the edge if Jan will most likely be extended

100% Remote

Not just any System Admin but specifically within Azure IOT edge

Must haves

• Azure IoT Edge
• Great Scripting
• Observability
• Virtual Machines
• LINUX-RHEL

Job Summary

We are looking for an experienced Unix/Linux System Administrator with strong systems automation experience to administer, operate, and automate our Linux-based server and edge computing environments, including Azure IoT Edge devices, at scale across a fleet of approximately 11,000 virtual machines. This role is responsible for day-to-day system administration, patching, automation, container operations, and production-grade observability (OpenObserve, PrometheGrafana) across on-premises and cloud-based disaster recovery systems. The ideal candidate is a hands-on operator who is comfortable scripting, troubleshooting, and maintaining CI/CD pipelines in a fast-paced, always-on environment.

Experience Required

10+ years of hands-on experience as a Unix/Linux System Administrator, with a strong track record of operating production Linux systems, containers, and CI/CD-driven automation workflows at scale.

Key Responsibilities

• Administer and support Azure IoT Edge devices and modules (edgeAgent/edgeHub) in production, including provisioning, patching, updates, and day-to-day troubleshooting.
• Plan and execute quarterly Azure IoT Edge upgrade and maintenance activities -- upgrading the IoT Edge runtime across the fleet within scheduled maintenance windows, with minimal disruption to store/edge operations.
• Perform L3-level Linux system administration: user/access management, kernel and OS-level troubleshooting, performance/capacity tuning, and root-cause analysis for recurring or high-severity incidents.
• Operate and maintain the observability stack (OpenObserve for logs, PrometheGrafana for metrics and dashboards) across edge, application, and cloud systems.
• Support disaster recovery execution between primary edge/on-prem infrastructure and cloud-based DR environments, including running failover/failback procedures and DR drills.
• Build, maintain, and troubleshoot CI/CD pipelines for application and infrastructure deployment.
• Write and maintain automation scripts/playbooks (Ansible/Puppet/Chef, shell, Python) to reduce manual operational work.
• Respond to alerts, triage incidents, and escalate per runbook/on-call procedures; keep runbooks and documentation current.
• Collaborate with cross-functional teams (security, networking, application) on day-to-day operational issues and change requests.

Mandatory Skills

• Strong hands-on, L3-level administration experience on RHE/Ubuntu: advanced troubleshooting and root-cause analysis across kernel, filesystem, storage, and networking issues, plus performance/capacity tuning, security hardening, and user/permission management in production.
• Proven ability to independently diagnose and resolve complex, escalated (L3) production issues -- performing deep root-cause analysis rather than surface-level fixes, and driving permanent remediation for recurring incidents.
• Hands-on experience operating and supporting Azure IoT Edge runtime (edgeAgent/edgeHub), edge modules, and device provisioning/management in production.
• Strong knowledge and hands-on experience running and troubleshooting Docker containers in production, including on resource-constrained edge hardware.
• Working experience with CI/CD pipelines, building and troubleshooting automated build/deploy workflows.
• Mandatory hands-on experience building and maintaining log aggregation, search, and dashboards using OpenObserve for production infrastructure telemetry.
• Mandatory hands-on experience instrumenting metrics pipelines with Prometheus and building actionable dashboards and alerts in Grafana across on-premises and cloud environments.
• Working knowledge of configuration management tools (Ansible/Puppet/Chef) for automating repeatable system administration tasks.
• Proficiency in shell scripting and at least one general-purpose language (Python/Java/C#/Go) for automation and tooling.
• Working knowledge of the full IoT lifecycle: device provisioning, commissioning, remote device management, module/firmware updates, and decommissioning.

Preferred Skills

• Experience migrating from legacy monitoring tools (e.g. Nagios) to modern observability stacks (OpenObserve, PrometheGrafana).
• Exposure to MQTT broker administration in high-throughput IoT environments.
• Experience administering NGINX (reverse proxy/ingress) configuration and tuning.
• Familiarity with certificate lifecycle management (issuance, rotation, expiry monitoring/alerting).
• Experience with CI/CD orchestration platforms.
• Knowledge of edge-computing constraints in distributed/remote environments (limited compute, intermittent connectivity).
• Experience following runbooks and participating in L1/L2/L3 incident escalation models.
• Experience working with cross-functional teams within and outside the organization.