Senior Splunk & Observability Engineer

Job Title: Senior Splunk & Observability Engineer
Location: Lafayette, LA
Type: Permanent Full Time
Work Model: Hybrid – onsite and remote

Responsibilities
• Support enterprise scale observability platforms within a complex, AWS-based production environment.
• Manage and support Splunk Enterprise running on AWS, covering the full data lifecycle including source onboarding, ingestion, indexing, search, dashboards, alerts, retention, and recovery.
• Develop and optimize SPL queries to troubleshoot ingestion and performance issues, enhance platform stability, support upgrades, vulnerability remediation, and data recovery after service interruptions.
• Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
• Integrate with AWS, Amazon CloudWatch, and Open Telemetry; develop Python-based automation for observability and platform operations.
• Support platform upgrades, patching, configuration changes, and vulnerability mitigation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tools for infrastructure and automation deployment.
• Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
• Participate in an on-call rotation for after-hours support related to production incidents, upgrades, and planned changes.

Requirements
• 5+ years of hands-on Splunk Enterprise administration and engineering experience in large production environments.
• Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, dashboards, alerts, and recovery.
• Advanced SPL query development and performance optimization skills.
• Hands-on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
• Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.
• Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases.
• Strong understanding of Open Telemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration.
• Experience building or maintaining Open Telemetry agent and collector packages.
• Working knowledge of Python for automation and operational tooling.
• Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.
• Strong troubleshooting skills including ingestion failures, data volume issues, platform outages, and data recovery.
• Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes.
• Ability to lead technical troubleshooting calls, communicate clearly, and coordinate across cloud, application, security, database, and support teams.
• Educational Requirements: Bachelor's degree in computer science, Information Systems, or a related field.

System One, and its subsidiaries including Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.

#LI-
Ref: #404-IT Pittsburgh