URL Crawling

What is the URL Crawling tool?

A tool that fetches the body content of a specific web page as text.

When you enter a URL, it retrieves the body content of that web page as plain text.

It goes beyond simple text extraction — it can also visually analyze what the page actually looks like with its vision analysis feature, something most crawling tools can't do.

Key Features

• Quick Analysis: A standard text crawling feature that quickly extracts the page's body text.
• Vision Analysis: Analyzes the page by looking at the actual screen rather than just the text, extracting key information accurately without missing anything. Just add "vision analysis" to your request to use it.

When can you use it?

The URL Crawling tool is especially useful in these situations.

• When you want to receive the entire content of a web page as text
• When you want to analyze multiple web pages together
• When you want to use web content as data
• When you want to understand the content of a shared link

For example, you can naturally ask "https://www.example.com Scrape the entire content of this web page", "https://example.com/article Get the content of this page", or "https://en.wikipedia.org/wiki/AI Extract the body of this page". The AI will then use this tool to generate the result.

How to use it

Usage is simple. Just enter your request naturally in the input box like this.

URL + [Get content · extract · crawl expression]

Here are some examples:

• Entire body: "https://www.example.com Scrape the entire content of this web page"
• Article extraction: "https://example.com/article Get the content of this page"
• Wiki content: "https://en.wikipedia.org/wiki/AI Extract the body of this page"
• Content collection: "https://example.com Crawl the content of this site"
• Vision Analysis: "https://www.example.com Tell me the headlines on this page with vision analysis"

Practical request checklist

For a stronger URL Crawling request, state the goal, input, scope, and output format together. Adding one more layer of constraints beyond the basic examples reduces follow-up questions and makes the result easier to use.

• Give the URL or site, search terms, and the final outcome you want.
• Set the page scope, country, language, sort order, and stopping condition.
• State permitted actions and require a stop before login, purchase, posting, or form submission.

Review, save, and reuse the result

Treat the URL Crawling result as a draft or a snapshot taken at query time. Check the following before carrying it into the next task.

• Record the source URLs, collection time, and any dynamic content that could not be captured.
• Review any action that changes external state—login, purchase, posting, or submission—before it runs.
• Follow site terms, robots policies, privacy requirements, and your access permissions.