SSRF and Chatbot Web Scrapers: How Fetching URLs Safely Protects Your Servers

Many AI assistants can read a web page: you paste a link and the platform fetches it, extracts the text and adds it to the knowledge base. It is a convenient feature and a classic security hazard. When a server fetches whatever address a user supplies, an attacker can supply an address that points not to a public website but to something inside the platform's own network: an internal admin service, a database console, a cloud provider's metadata endpoint that hands out credentials. The server dutifully fetches it and returns the contents. This attack is called server-side request forgery, or SSRF, and it has been behind some of the most damaging cloud breaches. This guide explains SSRF in plain terms, why web-fetching features are exposed, how attackers try to bypass simple defences, and the layered protections a careful platform uses, including the ones Crescelo applies. It also gives you questions to ask any vendor whose product fetches URLs on your behalf.

SSRF in one paragraph

Normally, a browser asks a website for a page. In an SSRF attack, the attacker gets a server to make the request on their behalf, to a place the attacker cannot reach directly but the server can. Servers often sit inside private networks where internal services trust each other and are not exposed to the internet. A URL fetcher that follows any address becomes a bridge into that network.

Why chatbot platforms are exposed

• They deliberately fetch user-supplied URLs (knowledge sources, web search, link previews, webhooks).
• They run in cloud environments with metadata services and private networks.
• They may follow redirects, so a harmless public address can bounce to an internal one.
• They may resolve hostnames, so a public-looking name can point to a private address.
• Customers can supply custom AI endpoints, adding another fetch path.

Every feature that turns a user-provided address into a server-side request is a potential SSRF path.

How simple defences fail

Robust defences work on resolved addresses, at connection time, for every hop.

The layered defence

1. Restrict schemes. Allow only http and https.
2. Resolve and check the address. Look up the hostname and reject any that resolves to private, loopback, link-local, reserved or metadata ranges, in IPv4 and IPv6, including IPv4-mapped IPv6.
3. Check every redirect. Repeat the checks for each hop, and limit the number of redirects.
4. Connect to the checked address. Use the resolved IP for the connection so DNS cannot change between check and use.
5. Limit size and time. Cap response size and set short timeouts, so a hostile server cannot exhaust resources.
6. Restrict ports and content types where possible.
7. Isolate the fetcher. Run it with minimal network privileges, ideally in a separate environment with no route to internal services.
8. Do not return raw responses. Extract text only, and never echo internal error details to the user.
9. Log and alert on blocked attempts.

What Crescelo does

• Validates external URLs before fetching them for knowledge sources, rejecting addresses that point to internal, loopback or cloud-metadata ranges.
• Resolves hostnames and checks the resulting addresses, including IPv6 and IPv4-mapped forms.
• Applies the same protection to custom AI provider addresses, so a workspace cannot point its assistant at an internal service.
• Requires secure (HTTPS) addresses for outbound integrations such as webhooks.
• Rate limits the endpoints that fetch, and limits body sizes.

No single measure is perfect, which is why they are layered. We also keep security tests for the private-address checks. If you find a bypass, please tell us through a support ticket.

Questions to ask any vendor

• How do you prevent your URL fetcher from reaching internal or metadata addresses?
• Do you check resolved IPs, including IPv6, and every redirect?
• How do you handle DNS rebinding?
• Do custom endpoints and webhooks get the same protection?
• How is the fetcher isolated from your internal network?
• Do you log and alert on blocked requests?

Answers such as 'we block localhost' are a warning sign. Look for descriptions of layered defences.

If you are building your own

1. Treat every user-supplied URL as hostile.
2. Use a well-tested library or proxy for outbound requests.
3. Resolve, validate and pin the IP, and revalidate on redirects.
4. Deny by default: allow-list where you can.
5. Remove cloud metadata access for the fetching service, or require tokens (for example, use the stricter metadata service versions cloud providers offer).
6. Add automated tests with known bypass strings.
7. Review any new feature that fetches a URL.

What this means for your knowledge base

Adding a web page as a knowledge source should be safe: the platform fetches a public page and reads its text. You can help by adding only public pages you control or trust, avoiding addresses that require login and being suspicious of unusual addresses in shared instructions. If a URL ever returns content that looks like an internal system, tell the vendor.

Common mistakes

• Blocking a few hostnames and calling it done.
• Checking the URL once and ignoring redirects.
• Trusting DNS at check time only.
• Forgetting that custom endpoints and webhooks also fetch URLs.
• Running the fetcher with access to internal services it does not need.
• Returning raw fetched content or error text to the user.

Each of these has appeared in real incidents. Layered checks, applied to every hop, close most of the gaps, and regular testing with known bypass strings keeps them closed as the code changes.

Frequently asked questions

See how your data is protected, the knowledge base feature, how a bring-your-own-key vault works, securing your chatbot account and bring your own key explained, and try adding a page in the playground.

Further reading

• OWASP Top 10 for Large Language Model Applications: The main security risks of AI assistants, including prompt injection.
• Google Search Central: creating helpful, people-first content: What Google looks for in useful pages.