Software Engineer, Web Crawling
exa
Exa is a startup focused on building a search engine from scratch to support AI applications, featuring a massive-scale infrastructure for web crawling and advanced embedding models. The Web Crawler Engineer will be responsible for developing a Google-scale web crawler, optimizing systems, and ensuring high-quality knowledge retrieval.
Qualification
- Extensive experience in building and scaling web crawlers
- Proficiency in high-performance programming languages such as C++ or Rust
- Strong optimization skills for system performance
- Passion for improving knowledge retrieval systems
- Ability to work in a fast-paced startup environment
Responsibility
- Build and scale web crawlers to handle extensive web data
- Optimize system performance for high efficiency
- Design systems to manage dynamic content and anti-bot measures
- Implement intelligent crawl scheduling and prioritization algorithms
- Collaborate with teams to enhance crawling strategies and infrastructure
