Home / Knowledge hub / Crawling
Knowledge hub
Crawling
Following links and forms automatically to map out what an app exposes.
What it means
Crawling, also called spidering, is a program following the links and forms of a site to build a map of its pages and inputs. It starts at one address, reads each response, finds more addresses to visit, and repeats until nothing new turns up or a limit is reached.
Search engines do this to index the web. Security tools do it to learn what there is to test: which pages exist, which parameters they take, which forms post where, and which cookies are set. Everything tested later depends on how complete that map turns out to be.
Why it matters
A page that was never found is a page that was never tested. If the crawler stops at the login screen, or cannot read a button that only works through JavaScript, whole sections of the app go unchecked and the report looks cleaner than the truth. That is a direct cause of false negatives.
It also shows you what an outsider sees. Crawling a public site can reveal old pages, test paths and files nobody meant to publish. Some are linked from places the team forgot about, such as an old sitemap, a footer link or a script that still references a retired page.
How it shows up
Crawlers struggle with the same things. Pages behind a sign-in need valid sessions, which means the tool must log in and stay logged in. Single-page apps build their links in the browser, so a plain HTTP crawler sees an empty shell. APIs often have no links at all and need a schema such as an OpenAPI file.
Responsible crawlers respect scope and rate. A crawl can click 'delete' or 'send' on a live system if nobody stops it, so exclude those actions. A site's robots.txt (RFC 9309) asks well-behaved crawlers to stay away from some paths, but it is a request and not access control. Pair crawling with forced browsing to find pages no link points to.
Questions people ask
Crawling, answered
Is crawling the same as scanning?
No. Crawling finds what exists. Scanning sends test input to find weaknesses in what was found. Most tools do the first to feed the second.
If the crawl is poor, the scan has little to work with.
Does robots.txt stop a crawler?
No. RFC 9309 describes it as a way for site owners to ask crawlers to avoid paths, and it relies on the crawler choosing to comply.
Never use it to hide sensitive pages. It can point attackers straight to them.
Can crawlers handle JavaScript apps?
Only if they drive a real browser engine. Basic crawlers read the first HTML and miss everything the script adds later.
Check how a tool handles single-page apps before trusting its map.
Can crawling damage a live site?
It can, if it follows links that trigger actions or sends requests too fast. Set a rate limit and exclude logout, delete and checkout paths.
Use test accounts when crawling while signed in.
How do we find APIs a crawler misses?
Use the API's own description, such as an OpenAPI file, plus traffic from your app and a review of the code's routes.
Source access is the surest way to list every endpoint that exists.
Sources
Where this comes from
Related
Terms that sit next to this one
Attack surface
Everything about your application an outsider can touch.
Forced browsing
Typing the address of a page you were never linked to and finding it opens.
Security test coverage
How much of your application a security test actually reached.
DAST
Tools that attack a running application from the outside.
Shadow API
An API running in production that nobody has documented, tracked or secured.
What did the crawl miss?
In white-box mode we read your code while testing the running app, so routes with no visible link are still found and tested. Findings point to the file and line. Book a pilot on one app.
Or start with a $199 pilot on one application: thirty days, success criteria agreed before day one, credited against the annual if you convert.