Home / Knowledge hub / Scraping
Knowledge hub
Scraping
Automatically copying large amounts of data from your site or API.
What it means
Scraping is the automated collection of content from a website or API. A script requests page after page, or record after record, and stores what it finds. Search engines scrape the web as a matter of course, so the technique is not wrong in itself.
It becomes a problem when the volume, the target or the purpose goes beyond what you intended: copying your whole catalogue, harvesting customer or seller details, or pulling prices every minute to undercut you.
Why it matters
Scraped data can be resold, used to clone your service, fed to competitors or turned into phishing lists. If the data includes personal details, you may have a data breach to explain even though no system was broken into.
There is a quieter cost: load. Heavy scraping slows your site for real customers and raises hosting bills. Teams often notice it only when a page becomes slow, and by then the data is long gone.
How it shows up
It shows up on listing pages, search, profiles, pricing and any API with sequential identifiers or large page sizes. The usual weaknesses are no limits on request volume, APIs that return more fields than the page shows, and enumeration through numbered records.
Reduce the value of what can be taken: return only needed fields, require sign-in for sensitive data, use unguessable identifiers and apply rate limits by account as well as address. Test by scripting a pull of your own data from outside and counting what comes back before anything stops it.
Questions people ask
Scraping, answered
Is scraping illegal?
It depends on what is taken, how, and where you are. Public data, terms of service, personal data rules and access barriers all matter. Take legal advice for a specific case. Technically, you can still reduce what is exposed.
How is scraping different from a data breach?
A breach usually involves getting past a control. Scraping can collect data that your own pages and API hand out freely. The harm can be similar, which is why the exposure itself needs controls.
Can I stop scraping completely?
No. If a person can see it, a script can usually fetch it. The aim is to raise the cost, limit volume, and keep sensitive data behind sign-in and authorisation.
Does robots.txt stop scrapers?
No. It is a request that polite crawlers follow. It does not enforce anything, and hostile scrapers ignore it.
How do we test for scraping risk?
Write a script that walks your listings and API, then see how many records it gets, how fast, and what fields come back. Then try the API with a different account to see whether it returns more than it should.
Sources
Where this comes from
Related
Terms that sit next to this one
Bot traffic
Automated visitors that scrape, stuff credentials or buy stock before real customers.
Enumeration
Working out what exists by asking repeatedly and watching the answers.
Unguessable ids
Using long random ids instead of counters, which hides records but does not protect them.
Excessive data exposure
An API sends whole records and trusts the page to show only the safe fields.
Pagination abuse
Asking for huge pages or walking every page to pull data out in bulk.
What can a script pull in a minute?
We check what an API hands over, how much of it and to whom, including records belonging to other accounts. A finding shows the requests and the data returned. Book a pilot on one app.
Or start with a $199 pilot on one application: thirty days, success criteria agreed before day one, credited against the annual if you convert.