Software Engineer - CLI Tools Building

Role : Software Engineer - CLI Tools & Data Pipelines

Location: USA Remote (EST or PST)

In this role, you will first build automated tooling and a regression test harness to continuously evaluate the quality of our PII Rewrite Service. Alongside this, you will design a fault-resilient command-line tool to automate the creation of Owned Test Accounts (OTAs) and orchestrate the upload of massive amounts of sanitized data from our internal storage (CNS) using AI.

KEY RESPONSIBILITIES:

1. Golden Dataset & Auto-Evaluation Pipeline

• Dataset Conversion Tooling: Build the conversion tools required to transform human-rater annotated evaluation results into a structured, machine-interpretable "Golden Dataset" using Protocol Buffers.
• Regression Test Harness: Build an Auto-Eval Harness that treats this dataset as a regression test suite to automatically evaluate the Rewrite Service and calculate key metrics like PII Recall, PII Precision, and Rewrite Consistency.
• Multi-Format & Multi-Language Scaling: Ensure the extraction pipeline and evaluation harness support core Workspace assets (Docs, Sheets, Slides, Gmail, PDFs, MS Office, Calendar, Chats) across multiple languages (English, Spanish, Portuguese, Japanese, Korean).

2. OTA Provisioning & Data Ingestion Automation:

• CLI Tool Development: Develop the smb ota_provisioner, a robust command-line tool (in C++, Python, or Go) to orchestrate OTA check, creation, and data upload workflows.
• Automated Provisioning: Interface with backend systems via Stubby (e.g., TestaccountService) to dynamically create and assign Prod Gaia OTAs and Workspace (Dasher) accounts.
• High-Volume API Ingestion: Ingest sanitized SMB data from Colossus (CNS) and upload it to the OTAs in parallel by leveraging and extending the internal ota_lifecycle tool.
• Rate Limiting & Resilience: Implement robust error handling and retry policies with exponential backoff to respect strict write limits on downstream APIs (Gmail, Drive).

Minimum Qualifications:

• Strong programming experience in C++, Python, or Go.
• Experience building robust, fault-tolerant Command Line Interface (CLI) tools and data processing pipelines.
• Familiarity with defining and parsing structured data schemas (e.g., Protocol Buffers, JSON, TextFormat).
• Experience working with RPC frameworks (e.g., Stubby/gRPC) and REST APIs.
• Solid understanding of system design principles, particularly around handling API rate limits and implementing exponential backoff mechanisms.