Connect Web Crawler to ZenSearch
Crawl and index any website or web application. Turn public and internal web content into a searchable knowledge base.
What ZenSearch indexes from Web Crawler
Any Website
Crawl documentation sites, marketing pages, help centers, internal wikis, or any website accessible over HTTP. Static and JavaScript-rendered pages are supported.
Depth Control
Set the maximum crawl depth to control how many links deep the crawler follows. Index just the top-level pages or crawl the entire site hierarchy.
robots.txt Compliance
The crawler respects robots.txt rules and meta robots directives. Excluded pages are never fetched or indexed.
Intelligent Rate Limiting
Configurable request rate and concurrency limits prevent the crawler from overloading target servers. Respects Crawl-delay and Retry-After headers.
Change Detection
Content hashing detects pages that have changed since the last crawl. Unchanged pages are skipped, making re-crawls fast and efficient.
URL Pattern Filtering
Include or exclude pages based on URL patterns. Focus the crawl on specific sections of a site while ignoring irrelevant areas.
Up and running in three steps
Enter the website URL and configure crawl depth, rate limits, and URL filters. ZenSearch begins crawling immediately.
Pages are downloaded, cleaned, parsed into structured units, and vectorized. The crawler detects navigation, footers, and boilerplate to extract only meaningful content.
Your team searches across all crawled pages with AI-powered semantic search. Re-crawls run on a schedule to keep content fresh.
Questions your team can finally answer
Once Web Crawler is connected, your team can ask natural-language questions and get cited answers instantly.
“How do I configure SAML SSO in the vendor's admin panel?”
Searches the vendor's crawled documentation site and returns the specific configuration steps for SAML SSO setup.
“What are the API rate limits for the payments provider?”
Finds the rate limit section in the payments API documentation and returns the per-endpoint limits and retry guidance.
“What accessibility standards does our public site meet?”
Searches crawled pages from your company website for accessibility statements, WCAG references, and compliance certifications.
Explore related integrations
Confluence
Notion
S3Ready to search your Web Crawler data?
Connect Web Crawler in minutes. No credit card required.
Start Free