Facebook Twitter Telegram Linkedin Product Hunt

Most complete comparison of 21 Desktop Web Crawlers

Comparison of functionality and speed of SEO audit software in 2020

Comments
 113
2019-08-20 | Time to read: 19 minutes
Facebook
Author: Simagin Andrey

Updated at: 09.11.2020

In this review, we would like to conduct a detailed test of SiteAnalyzer, comparing its speed and functionality with the main competitors.

Comparison of modern web crawlers

Key features of SiteAnalyzer:

  • high speed of scanning sites (over 1,000 pages per minute)
  • scanning sites of almost any size (from 1 million pages or more)
  • analysis of more than 50 main SEO parameters
  • 60+ preset filters to find internal optimization errors
  • flexible filtering of the analyzed data by any parameters
  • calculation of the internal page weight of the site to optimize linking (PageRank)
  • the ability to graphically display reference links (visualization of data on a graph)
  • the ability to conduct full-fledged audits and upload them to Excel, PDF and HTML format

We will not dwell on the description of all the innovations and capabilities, since they are all described in detail in the documentation for working with the program. However, I would like to dwell on the first two (the speed of work and the volume of scanning), since in fact these are the main parameters by which the quality of the work of such programs and online services is assessed.

For comparison, we decided to test 21 of the most popular desktop SEO crawlers.

Screaming Frog SEO Spider icon
Netpeak Spider icon
Forecheck icon
Sitebulb icon
WebSite Auditor icon
Comparser icon
Visual SEO Studio icon
Xenu's Link Sleuth icon
SiteAnalyzer icon
Beam Us Up

Darcy SEO Checker
LinkChecker
PageWeight Desktop
Webbee
WildShark SEO spider
Site Visualizer
RiveSolutions SEO Spider
IIS SEO Toolkit
Website Link Analyzer
A1 Website Analyzer

seoBOXX WebsiteAnalyser

List of web crawlers participating in the review:

  1. Screaming Frog SEO Spider 12.6
  2. Netpeak Spider 3.9
  3. Forecheck 0.9.1
  4. Sitebulb 2.5.5
  5. WebSite Auditor 4.37.8
  6. Comparser 1.0.133
  7. Visual SEO Studio 1.9.5.10
  8. Xenu's Link Sleuth 1.3.9
  9. Beam Us Up 1.1.0
  10. Darcy SEO Checker 1.0.1
  11. LinkChecker 9.3
  12. PageWeight Desktop 3.0.5
  13. Webbee 3.0
  14. WildShark SEO spider 1.6.3
  15. Site Visualizer 3.0
  16. RiveSolutions SEO Spider 1.0.9
  17. IIS SEO Toolkit 1.0
  18. Website Link Analyzer 2.3
  19. A1 Website Analyzer 10.1
  20. seoBOXX WebsiteAnalyser 1.3.1
  21. SiteAnalyzer 2.3

Brief description of crawlers and their capabilities

Screaming Frog SEO Spider (United Kingdom)

Perhaps the most popular site analysis software among SEO specialists. Differs in high speed of work and simple interface. Pros: it is possible to visualize link links on the graph. Cons: no functionality for calculating and displaying internal PageRank for pages, no flexible filtering of data.

Screaming Frog SEO Spider

Screaming Frog SEO Spider

Price: £149 per licence, per year
OS: Windows / macOS / Linux
Demo: demo version parse first 500 pages of the site
Website: screamingfrog.co.uk

Netpeak Spider (Ukraine)

One of the top harvesters with many parameters for checking technical characteristics and errors on the site. Differs in high speed of work. The functionality is significantly richer than that of the Screaming Frog, flexible filter settings. Pros: there is functionality for calculating and displaying the internal PageRank for each page of the site. Cons: there is no possibility to visualize reference links on the graph.

Netpeak Spider

Price: $14 per licence, per month ($117 per year)
OS: Windows
Demo: yes, but without the ability to export reports
Website: netpeaksoftware.com

Forecheck (Germany)

Another "combine" with beautiful reports and many settings. The speed of work is not the highest. It works unstable: it hung tightly twice on ~4000 pages and once crashed with an error. Pros: many scanner settings, the ability to check an arbitrary list of URLs, a beautiful HTML report. Cons: somewhat confusing interface and "crashes" on medium-sized sites (unstable).

Forecheck

Price: $49 per licence, per month ($490 per year)
OS: Windows
Demo: demo version parse first 10,000 pages or up to 10 MB of data, some of the information is hidden
Website: forecheck.com

Sitebulb (United Kingdom)

Software has a beautiful design and decent speed of work, and also allows you to watch the dynamics of changes from audit to audit (stores the history of scans in the database). The program analyzes many parameters during the audit, but in our opinion, this number is less than that of more advanced competitors. It is possible to visualize link links on the graph, while there is no functionality for calculating and displaying the internal PageRank for pages. Pros: the interface is replete with many beautiful graphs. Cons: relatively low speed of work, as well as a beautiful, but at the same time not very user-friendly interface, which in design corresponds closer to a website than to desktop software, hence the problems with the usability.

Sitebulb

Sitebulb

Price: £25 per licence, per month (£300).
OS: Windows / macOS
Demo: free trial version for 14 days
Website: sitebulb.com

WebSite Auditor (Belarus)

Program has many tools for detailed site analysis. It differs in a good speed of work, however, on a large number of site pages (~30,000), the program starts to slow down terribly: all 3 times it hangs strongly on ~40,000 pages, after which a message about insufficient memory and a request to restart the program appeared. In our opinion, WebSite Auditor has a not quite familiar interface (perhaps the least convenient of similar "combine programs"), which is rather a big drawback. Also, the disadvantages include the inconvenience of working with data (they open in small windows that are not convenient to work with). Pros: the ability to create customized reports, visualization of link links on the graph (while there is no functionality for calculating and displaying the internal PageRank for pages, which is very strange). In general, we can conclude that working with large sites in WebSite Auditor can cause certain problems...

WebSite Auditor

WebSite Auditor

Price: $124 per program
OS: Windows / macOS / Linux
Demo: demo version parse first 500 pages of the site, there is no possibility of saving projects
Website: seopowersuite.ru

Comparser (Russia)

Russian product from renowned SEO specialist Alexander Alaev. The program is not very fast, but it cannot be called slow. The functionality is also incomplete compared to competitors. However, the program has its own tricks in the form of parsing of Yandex and Google search engines, as well as the ability to check the indexing of site pages in search engines based on parsed pages. Of the minuses: recently, the author refused to support the "old" versions of Windows: Windows XP / Vista.

Comparser

Price: $26 per license
OS: Windows
Demo: demo version parse first 150 pages of the site
Website: parser.alaev.info

Visual SEO Studio (Italy)

Program has a considerable set of features, however, the interface, in our opinion, is not entirely convenient and has an old-fashioned design from the 2000s. And if you look closely, then the functionality does not shine with a rich set of functions. One of the advantages is the ability to scan multiple sites at the same time, but we would not recommend using this, since everything is very slow. On small sites the speed of the program is average, on sites of large sizes the scanner works very slowly.

Visual SEO Studio

Price: €149.00
OS: Windows / macOS
Demo: trial free version for 15 days
Website: visual-seo.com

Xenu's Link Sleuth (Germany)

A kind of progenitor and, probably, the very first audit program. The functionality is very scarce and is limited only to checking broken links on the site and nothing more. The last update of the program was in 2010. It works, not fast, but it copes with its function of finding broken links. Since 2010, no new versions have been released. Bottom line: the old man should have a well-deserved rest for a long time, but some still use it. Strange, but habits are habits)

Xenu's Link Sleuth

Price: freeware
OS: Windows
Website: home.snafu.de/tilman/xenulink.html

Darcy SEO Checker (USA)

Program has a very meager set of checked SEO parameters (title, description, h1-h2, HTTP page headers). The only plus is the display of site load speed scores by Google Speed Insights. At the moment, the program is not being updated or supported, the program's website is not working. The latest version dates back to 2015.

Darcy SEO Checker

Price: $99 per license
OS: Windows
Demo: demo version parse first 25 pages of the site, and there is also no way to export the results
Website: not available

LinkChecker (Germany)

A program for checking broken links in HTML code and CSS files. Scans the site and displays only the pages of the site with errors. The list of crawled URLs is not displayed. It works very slowly. From the pros: it shows in which line of code the error is located. Otherwise, the program is probably useless for the needs of SEO. Judging by the history of changes on the github, the program was last updated in 2014.

LinkChecker

Price: freeware
OS: Windows / Linux
Website: wummel.github.io/linkchecker/

PageWeight Desktop (Russia)

Previously popular program for calculating the "weight" of pages and setting up interlinking (I myself once used it). In fact, apart from calculating the "weight" of the pages of the site, the program is nothing more remarkable. In appearance, the crawl speed is high, but it is high due to the fact that only link links and page titles are saved in the database. The program does not store any more data and does not use it in its work. So in fact, besides calculating PageRank, it has nothing more to boast about. On the other hand, this is its main purpose. And nothing more. Of the minuses: judging by the half-abandoned forum with unanswered questions from users and rare version updates (the latest version dates back to 2016), the program is in stagnation and whether it has ceased to be interesting for the authors, if they just do not have the resources and new thoughts on its further development.

PageWeight Desktop

Price: $26 per license
OS: Windows
Demo: free Lite version, which is very slow and shows incorrect results of calculating page weight
Website: page-weight.ru

Beam Us Up (Portugal)

It is the result of co-creation by a group of 4 people living in England, Portugal and Serbia. Pros: it works pretty fast, is free, supports Windows, Linux and MacOS platforms, the report contains information about duplicate pages, 302 redirects, pages with short or missing meta tags. Disadvantages: apart from the site pages itself, it no longer parses anything (hence the high speed), some sites on HTTPS refused to parse. However, for a quick quick analysis it will do. Another big drawback: the project is no longer supported or developed.

Beam Us Up

Price: freeware
OS: Windows
Website: beamusup.com

Webbee (Pakistan)

Not entirely successful attempt to create a copy of Screaming Frog. Pros: good speed of work, there are all the basic parameters for an SEO audit, the ability to check an arbitrary list of URLs. Cons: visualization of data on charts looks creepy, there could be more analyzed parameters, there is no way to filter data by arbitrary fields. The last release was released in 2015.

Webbee

Price: $15 per licence, per month ($129 per year)
OS: Windows / macOS
Demo: demo version scans up to 500 website pages
Website: eebew.com

WildShark SEO spider (United Kingdom)

An unpretentious program with a terribly simple interface, made apparently "on the column". There are practically no settings. Cons: the interface, even without parsing images and analyzing external links, scans very slowly on 16 threads (practically the worst), there is no way to save projects. Pros: it displays Rich Snippets, it is possible to generate reports in Excel, as well as the simplest data filtering (I still could not fully understand whether it works or not). In conclusion, we will say that, in our opinion, there are much better quality representatives from free software than this craft.

WildShark SEO spider

Price: freeware
OS: Windows
Website: wildshark.co.uk

Site Visualizer (Russia)

A small program for site auditing. Although it does not shine with speed, it has interesting advantages: visualization of the sitemap (it shows on the map where which link leads, but the implementation is not so hot), as well as the ability to create interesting reports (with the ability to edit them by changing the SQL queries to the database)... Cons: few parameters to check, low speed. In general, the software can be called useful due to the ability to create custom reports.

Site Visualizer

Price: $69 per licence
OS: Windows
Demo: 30-day fully functional trial version
Website: site-visualizer.com

RiveSolutions SEO Spider (Russia)

A fairly young software that, on the one hand, has a partial set of classic SEO tools for this kind of programs, on the other hand, it has a number of dubious, in our opinion, functions, such as, for example, "Yandex Metric Finder" (search for metric counters on the pages site). Judging by experience, this is a completely unnecessary thing, although a lot of space is allocated on the site for the description of this function. On the other hand, if someone needs it, then it is necessary. Disadvantages: huge size of the distribution kit (85 MB in the archive and 250 on the disk), not quite well thought-out interface + there is no possibility of custom filtering by arbitrary fields. From the pros: as I wrote above – a good set of classic SEO tools for this kind of programs and the ability to generate reports. In our opinion, while the program looks unfinished and "damp". At the moment, the program is abandoned by the developers, its development is not planned.

RiveSolutions SEO Spider

Price: $78 per licence, per year
OS: Windows
Demo: demo version parse first 500 pages of the site, there is no option to save the project to disk
Website: rivesolutions.com

IIS SEO Toolkit (USA)

A little-known free SEO analytic tool for technical website audit from Microsoft itself! The interface of the IIS SEO Toolkit is simple, but generally pleasant, optimization errors are grouped and presented in the form of intuitive reports, each error is accompanied by a small description. Pros: sufficiently high speed of work, search for duplicates, analysis of the correctness of the layout and lengths of meta tags, displaying errors in HTML markup, unclosed tags, frequency tables of words and bigram tables, displays pages with the largest number of errors, incoming and outgoing links, has an intuitive clear interface. Cons: outdated interface, stable operation only under Windows Vista / Server 2008/7 (XP and Windows 8-10 are not officially supported). During the audit of 100k pages, an error occurred related to the problem of database indexing. There were no errors for 20k. Those, when analyzing large projects, the program may experience problems.

IIS Search Engine Optimization (SEO) Toolkit

Price: freeware
OS: Windows
Website: iis.net

Website Link Analyzer (Germany)

This program has a not quite standard interface, more similar to a browser one, while it contains a classic set of functions for programs of this kind: search for broken links, displaying 301 redirects, exporting data to Excel and CSV, generating Sitemap. The program has been developing since 2015, but judging by the version history, it is not very active, so it is premature to talk about any development potential to a truly professional SEO tool.

Arclab Website Link Analyzer

Price: $49 per licence, per year
OS: Windows
Demo: free trial version for 7 days
Website: arclab.com

A1 Website Analyzer (Denmark)

The interface is similar to Visual SEO Studio – it looks a bit overloaded, so it will be difficult to figure it out right away. At the same time, in addition to collecting basic information about the site, the program has the ability to validate HTML and CSS code through the W3C service, search in text and HTML code for all scanned pages, the ability to analyze the density of keywords. Cons: low scanning speed and ill-conceived interface. Pros: the presence of the Russian language in the interface (although, and that is localized crookedly).

A1 Website Analyzer

Price: $69 per licence, per year
OS: Windows
Demo: free trial version for 30 days
Website: microsystools.com

seoBOXX WebsiteAnalyser (Germany)

I tried to run the program on 3 different computers, but I still couldn't test the demo version – at startup the program crashed with some kind of "UI Theme" error.

WebsiteAnalyser, error

At the same time, judging by the screenshots of the interface on the program website, it has at least a standard set of functions for this kind of programs (checking for the presence of meta tags, correctness of h1 headers, statistics of internal and external links, content analyzer, generating Sitemap.xml, creating reports)...

WebsiteAnalyser

Price: $39,99 per licence, per year
OS: Windows
Demo: free trial version for 30 days
Website: seoboxx.com

SiteAnalyzer (Russia)

In terms of the set of functions, SiteAnalyzer is practically not inferior to competitors, and also has a high speed of crawling sites, comparable to the speed of paid counterparts. It is possible to visualize link links on the graph, as well as functionality for calculating and displaying the internal PageRank for each page of the site. In addition to the preset filters, you can manually filter data in any format. From the pros: the ability to store all projects in one list, as well as automatic recording of all site data in the database, with the ability to automatically display them after restarting the program without unnecessary manipulations. I also note the ability to export reports to Excel, the small size of the distribution kit (9 MB in the archive), portability and free distribution, which is an important factor.

SiteAnalyzer

SiteAnalyzer

Price: freeware
OS: Windows
Website: site-analyzer.pro

Testing Web Crawlers

After a short acquaintance with the functionality and prices, let's proceed directly to testing the programs.

When testing crawlers, the following parameters were used:

  • tested site: open.spotify.com (huge number of HTML pages and practically no images, loads quickly, optimized for heavy loads)
  • number of scanning threads: 10 (or the value available in the program, but not more than 10)
  • computer configuration: Windows 10 x64, 8 GB RAM, CPU Intel Core i5 3.33 GHz, HDD

To test the number of scanned data, we chose the URI parameter, which contains not only text / html pages, but also images, scripts, styles and other data, since theoretically there may not be so many pages on the site, but the content on these pages is in the form There can be many times more images, styles and other links, so we will rely on URI counting.

Web crawler test results

After numerous tests, we received the following results:

Software Name
Time to scan 100 pages
Time to scan 1,000 pages
Time to scan 10,000 pages
Time to scan 100,000 pages
Wide range of audited parameters
Flexible data filtering
Crawling arbitrary URLs
Page Rank calculation
Data visualization on a Graph
Freeware
Screaming Frog SEO Spider
0:00:08
0:00:45
0:05:35
1:03:30
+
-
+
-
+
-
Netpeak Spider
0:00:04
0:00:30
0:04:53
0:55:11
+
+
+
+
+
-
Forecheck
0:00:15
0:01:12
0:08:02
1:36:14
+
-
+
-
-
-
Sitebulb
0:00:08
0:01:26
0:16:32
2:47:54
+
-
+
-
+
-
WebSite Auditor
0:00:07
0:00:40
0:05:56
2:36:26
+
-
-
-
+
-
Comparser
0:00:12
-
-
-
-
-
+
-
-
-
Visual SEO Studio
0:00:15
0:02:24
0:24:14
4:08:47
-
-
-
-
-
-
Xenu
0:00:12
0:01:22
0:14:41
2:23:32
-
-
-
-
-
+
Darcy SEO Checker
0:00:04
0:00:31
0:05:40
0:58:45
-
-
-
-
-
-
LinkChecker
0:00:29
0:00:52
0:03:22
0:52:04
-
-
-
-
-
+
PageWeight Desktop
0:00:06
0:00:56
0:17:40
4:23:15
-
-
-
+
-
-
Beam Us Up
0:00:08
0:01:03
0:10:18
1:43:03
-
-
-
-
-
+
Webbee
0:00:10
0:01:58
-
-
-
-
-
-
-
-
WildShark SEO spider
0:00:28
0:07:20
-
-
-
-
-
-
-
+
Site Visualizer
0:00:11
0:01:58
0:38:15
-
-
-
-
-
-
-
RiveSolutions SEO Spider
0:00:06
0:00:49
0:08:14
1:55:19
-
-
-
-
-
-
IIS SEO Toolkit
0:00:03
0:00:46
0:07:08
1:02:26
-
-
-
-
-
+
Website Link Analyzer
0:00:09
0:02:38
0:24:56
4:33:41
-
-
-
-
-
+
A1 Website Analyzer
0:00:24
0:05:32
0:53:15
8:42:11
-
-
-
-
-
+
seoBOXX WebsiteAnalyser
0:00:12
0:01:15
0:17:31
3:51:08
-
-
-
-
-
-
SiteAnalyzer
0:00:06
0:00:22
0:06:47
2:04:36
+
-
+
+
+
+

Note: when scanning 100 and 1,000 pages, there is no point in focusing much on the difference in algorithms for crawling pages by crawlers in different programs. But the scanning speed of 10,000 and 100,000 pages is already indicative, as it reflects the more or less stable speed of crawlers at long distance.

Summary table with final test results:

Software Name
Final Score
Free Distribution
Download size (Mb)
Netpeak Spider
92
-
11
Screaming Frog SEO Spider
89
-
228
WebSite Auditor
75
-
203
SiteAnalyzer
72
+
9
Sitebulb
70
-
162
Forecheck
68
-
12
RiveSolutions SEO Spider
62
+
85
Visual SEO Studio
52
-
10
Comparser
51
-
7
Site Visualizer
47
-
6
A1 Website Analyzer
45
-
8
seoBOXX WebsiteAnalyser
43
-
15
IIS SEO Toolkit
42
+
1
Website Link Analyzer
40
-
3
PageWeight Desktop
38
-
5
Beam Us Up
30
+
22
Webbee
28
-
48
Darcy SEO Checker
25
-
27
WildShark SEO spider
25
+
4
Xenu
18
+
1
LinkChecker
9
+
11

Final conclusions

To unambiguously identify the winner in terms of crawling speed was a very difficult task due to the fact that at different points in time of crawling one site by one program, the crawl speed could vary from 40 to 60 seconds per 1000 pages, respectively, for 100,000 URIs the final result could differ for tens of minutes, so for the final calculation, we took the averaged data for several scans (for some programs, the number of tests reached about a dozen and could take up to several days).

This is a word about the quality of testing. Logically, for the most popular scanners, we allocated more time for tests, and the most "simple" scanners only needed one run.

Summing up the results of testing 21 SEO crawlers, we can conclude that the final TOP-5 for the best of the best in 2020 turned out to be like this:

Thank you for your attention! We are waiting for your comments and feedback on the work of these programs!

Rate this article
4.9/5
4

Other articles:



<< Back

Our Clients