Recently postedTechnology

Lead Systems Engineer

Hadiamondstar Software Solutions LLCWashington, DC🇺🇸United States

Quick overview

SeniorityMid Senior

Work modeHybrid

LocationWashington, DC, United States

DockerDynamoDBMicroservicesAWSAPI GatewaySplunkBashCloudFormationDatadogGrafanaPagerDutyPrometheusPythonTerraformRESTKubernetesCI/CDIAM

Job description

Summary

We seek a hands-on technical leader to architect, implement, optimize, and govern enterprise observability across AWS infrastructure, applications, microservices, containers, and hybrid environments. The role owns Datadog and AWS-native monitoring strategy, improves reliability and incident response, automates telemetry deployments, and partners with SRE, DevOps, security, and application teams.

Key Responsibilities

Observability Architecture & Engineering

Design enterprise monitoring for metrics, logs, traces, events, and service health.
Standardize dashboards, alerts, service-level indicators (SLIs), service-level objectives (SLOs), and instrumentation across AWS accounts and regions.
Provide end-to-end visibility for distributed systems, microservices, serverless, and containers.

Datadog Platform Ownership

Lead Datadog Infrastructure Monitoring, APM, Log Management, RUM, Synthetic Monitoring, service maps, and distributed tracing.
Build AWS/Kubernetes/database integrations, dashboards, anomaly detection, alert routing, and incident integrations.
Govern tagging, RBAC, data ingestion, cardinality, retention, and platform cost; automate with Datadog APIs and Terraform.

AWS Cloud Monitoring

Implement observability for EC2, EBS, Auto Scaling, ECS, EKS, Lambda, RDS/Aurora, DynamoDB, S3, CloudFront, API Gateway, ELB, VPC, Route 53, SQS, SNS, and EventBridge.
Use CloudWatch metrics/logs/alarms, Application Signals, CloudTrail, AWS Config, and X-Ray where appropriate.
Architect centralized, multi-account and cross-region AWS monitoring.

SRE, Reliability & Incident Management

Partner with SRE/DevOps teams to define SLIs, SLOs, error budgets, and actionable alerts.
Improve MTTD and MTTR through proactive detection, root cause analysis, post-incident reviews, and noise reduction.
Integrate observability with ServiceNow, PagerDuty, and collaboration platforms.

Automation, Performance & Leadership

Build monitoring-as-code using Terraform/CloudFormation, Python, Bash, AWS CLI, REST APIs, and CI/CD.
Optimize telemetry pipelines, sampling, log retention, infrastructure utilization, and observability spend.
Lead architecture reviews, engineering standards, mentorship, platform roadmaps, and cross-team delivery.

Required Qualifications

Bachelor’s degree in a technical field or equivalent practical experience.
10+ years in systems, infrastructure, or cloud engineering; 5+ years hands-on AWS; 3+ years hands-on Datadog or equivalent observability platform.
Advanced Datadog skills: infrastructure monitoring, APM, log management, dashboards, alerting, integrations, and tracing.
Strong AWS knowledge, including CloudWatch, CloudTrail, X-Ray, networking, compute, databases, containers, and serverless.
Experience with OpenTelemetry, Docker/Kubernetes, ECS/EKS, Terraform or CloudFormation, Python/Bash, APIs, and CI/CD.
Demonstrated production troubleshooting, incident management, SRE practices, and technical leadership.

Preferred Qualifications

AWS Solutions Architect – Professional or DevOps Engineer – Professional; relevant Datadog certification.
Prometheus, Grafana, OpenSearch, Splunk, OpenTelemetry Collector, or Fluent Bit experience.
Multi-account enterprise AWS architecture; IAM, security, compliance, and data governance.
AIOps, event correlation, automated remediation, or regulated-industry experience.

Success Measures

Broader observability coverage and adoption of standardized dashboards and alerts.
Reduced MTTD, MTTR, false positives, and redundant notifications.
Higher SLO compliance and automated monitoring deployments.
Improved Datadog and CloudWatch cost efficiency.
Assess AWS/Datadog monitoring gaps and establish an enterprise roadmap.
Standardize tagging, instrumentation, dashboards, alerting, and monitoring-as-code.
Expand critical workload coverage and deliver measurable reliability and cost improvements.