53
If the SiteAnalyzer scanning outcome is limited to one page of the site with the 403 Forbidden response code, try changing the User-Agent in the program settings. This often solves the problem and allows SiteAnalyzer to fully scan the site.
There are often preventive measures for certain search robots upon huge web sites. Therefore, some sites block little-known search robots like the one of SiteAnalyzer, while others block user agents which allegedly fake the Yandexbot and Googlebot robot. As a result, when you scan a site using SiteAnalyzer, the site may prohibit you from scanning.
The first sign that you have been blocked is that all pages are displayed with the 403 response code (HTTP 403 Forbidden). Sometimes it takes a few minutes for the 403 pages to start working, sometimes longer.
Therefore, if you encounter such problem, try to change the User-Agent:
1. In SiteAnalyzer, go to Settings > User-Agent.
2. In the drop-down list, change the current user agent («SiteAnalyzerbot» is installed by default) to Yandexbot, Googlebot Mobile or Chrome.
3. Restart the scan.
4. Repeat this process until the site starts scanning correctly.
Thus, alternately changing the current user agent, you can find the same User Agent that is not blocked by the site. This allows you to get the necessary scan data without contacting the help desk of IT specialists.
Tip:
If you are constantly blocked and you have access to the site development team, find out if you can use a unique user agent that they can use to identify you and disable the locks.
If this is a site that you regularly work on, it will probably be easier to whitelist your IP address. However, changing the user agent is an excellent short-term solution for situations where you cannot scan correctly.
Other articles:
