Principal Engineer - DevOps

Roles & Responsibilities 

• Work with the engineering teams to design, build, monitor and maintain the system for high availability  and scalability. 
• Define SLIs and SLOs with the engineering teams, track them, and use error budgets to guide reliability  work and release decisions. 
• Take part in design reviews and make architecture decisions that improve reliability, scalability, security  and operability before systems reach production. 
• Build security into the infrastructure and delivery pipelines, including access control, secrets  management, network policies and image and dependency scanning. 
• Work with the security team to track vulnerabilities and audit findings, and close them within agreed  SLAs. 
• Write production-grade software, deployment frameworks, tracers, monitors and auto remediation tools,  and automate the processes. 
• Follow an AI-first approach to operations. Use AI tools and agents for incident triage, log and alert  analysis, runbook execution, change review and documentation, and build our own where off-the-shelf  tools fall short. 
• Manage cloud infrastructure as code (Terraform / Pulumi) and keep environments consistent, reviewable  and reproducible. 
• Lead and participate in performance, load and stress tests; identify bottlenecks, opportunities for  optimization and capacity planning. 
• Participate in the on-call rotation and lead incident response as needed. Write postmortems and follow  through on the fixes. 
• Identify repetitive operational work and engineer it away. 
• Review existing processes and recommend changes or institute new processes as necessary, including the  areas of monitoring, alerting, upgrades, and tuning, etc. 
• Create quality project documentation of architecture designs, implementation plans, design documents,  runbooks, test plans, etc. 

Mandatory Skills 

• Excellent programming skills in at least one language (Go, Python, Java, Node or similar), with production  code you can walk us through. Experience with Go is a plus. 
• Experience applying SRE practices: defining SLIs and SLOs, error budgets, and using them to drive design  and prioritization decisions.
• Experience designing and building reliable, scalable systems on Google Cloud Platform (GKE, Compute  Engine, Cloud Storage, Pub/Sub, VPC, IAM). Working knowledge of AWS or other clouds. 
• Experience with infrastructure as code using Terraform or Pulumi. 
• Experience with Container and Cloud technologies like Docker/Kubernetes. 
• Experience with CI/CD using Argo CD, Jenkins, helm charts etc. 
• Experience in performance diagnostics, capacity planning, performance architecture design, performance  tuning, performance monitoring. 
• Experience with the use, maintenance and configuration of monitoring, metrics and logging infrastructure (Datadog, Grafana, Prometheus, New Relic, etc.) 
• Experience leading incident response and writing blameless postmortems, with a focus on removing the  cause rather than handling the symptom. 
• Comfortable working with modern nosql databases and big data platforms (MongoDB, Redis, etc.) 
• Total experience of 10+ years, with a software development background and 3+ years working with GCP,  Kubernetes and large-scale distributed systems.