Some websites try to block automated visits. A proxy routes your scraper's requests through a different network path, which often gets past that blocking so the scraper can read the page.
When you need a proxy
Consider a proxy when:
- A scraper that works on other sites returns nothing on this one.
- A job fails in a way that suggests the site refused the request.
- The site loads fine for you in a normal browser but not for the scraper.
Testing whether the proxy is the issue
When a run fails, you can retry it with or without a proxy. That is the fastest way to find out whether the proxy is helping or whether the problem lies elsewhere. If a run only succeeds with a proxy enabled, leave it on for that scraper.
Do not enable a proxy by default. Proxied requests can be slower, so use one only for sites that actually need it.