Research Crawling Engineer

About the Role

Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.

What You'll Do

• Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.

• Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.

• Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.

• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

• Build datasets for research and model training in collaboration with research teams.

• Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.

• Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

• Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.

• Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.

• Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.

• Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites.

• Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable.

• Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus.

• Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial.

Compensation & Benefits

Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote.