Sr Splunk Engineer with Security Clearance

Position: Senior Splunk Engineer
Location: Alexandria, VA (Hybrid)
Clearance: Active TS/SCI Required Position Overview
We are seeking a Senior Splunk Engineer (5+ Years Experience) to support the hands-on execution, configuration, and validation of an enterprise log ingestion pipeline for TRMC. Working alongside the Lead SME Engineer, you will directly configure remote Cribl edge collectors, central Cribl Stream worker clusters, Splunk indexer peers, and Splunk Enterprise Security (ES). You will be key in onboarding priority data sources, establishing control plane management, and executing simulated failover/replay testing.

Key Responsibilities
Cribl Edge & Cluster Build: Deploy Cribl Stream worker nodes across 3 remote sites (2 workers per site) and central clusters (1 Leader + 2–3 Workers). Configure persistent queues (~125 GB/site), in-line edge reduction, and store-and-forward transport.

Splunk Indexer & Search Tier Configuration: Deploy 4-node Splunk indexer clusters with 1 Cluster Manager (RF3/SF2), set up index-by-retention policies, configure license metering, and deploy ad-hoc/ES search heads.

Data Source Onboarding: Execute the five-gate onboarding process for 10 priority data sources—verifying sourcetypes, timestamp accuracy, field extractions, and mapping data to Splunk Common Information Model (CIM) standards for ES visibility.

Control Plane Management: Assist in configuring Cribl Leader fleet configs, Splunk Deployment Server app/TA distribution, License Manager alerting, and Monitoring Console health dashboards.

Testing & Replay Validation: Validate end-to-end telemetry flows, link-loss failovers, persistent queue backfills, and sample replay from S3/Azure Blob archive storage.

Documentation Support: Support the creation of as-built component inventories, configuration baselines, and operational runbooks.

Required Qualifications
Clearance: Active Top Secret / SCI Experience: 5+ years of experience in cyber data engineering, SIEM administration, or Splunk infrastructure operations.

Splunk Suite: Strong hands-on proficiency with Splunk Enterprise, Indexer Clustering, Enterprise Security (ES), ITSI, and SOAR.

Cribl Stream: Practical hands-on experience configuring Cribl Stream routes, pipelines, worker groups, and destinations.

Linux/Infrastructure: Solid Linux systems administration skills across VMware and Azure environments.

Preferred Qualifications
Splunk Enterprise Certified Admin or Power User.
Certified Cribl Stream User/Admin.
Experience working in DoD CAC-enabled environments and DoD logging standards.