Cloudflare Launches Single API Call Website Crawler in Open Beta, Returning Content in Multiple Formats

Cloudflare launches a powerful new /crawl endpoint in open beta, enabling developers to crawl entire websites with a single API call and retrieve content in HTML, Markdown, or structured JSON formats, with advanced controls like depth limits, URL filtering, and incremental crawling available on both free and paid plans.

• Cloudflare launches a new /crawl endpoint for Browser Rendering, now in open beta, allowing entire websites to be crawled with a single API call that returns content in HTML, Markdown, and structured JSON formats.
• The endpoint runs crawl jobs asynchronously, automatically discovers pages via sitemaps and links, and supports advanced controls like crawl depth limits, URL filtering, incremental crawling, and a static mode for faster fetching of non-dynamic sites.
• Operating as a well-behaved bot, the crawler respects robots.txt directives and AI Crawl Control by default, cannot bypass Cloudflare bot detection or captchas, and is available on both the Workers Free and Paid plans.