Web scraping: how to parse and extract data from a website for free?

It’s not a rare case when a webmaster, marketing expert, or SEO specialist needs to extract data from site pages and display it in a comfortable form for further processing. This could be parsing prices in an online store, getting the number of likes, or extracting reviews from resources you’re interested in.

Most technical site audit software collect only the H1 and H2 header content by default, however, if, for instance, you want to collect the H5 headers, you will have to extract them separately. They usually reach out to web scrapers to avoid the routine manual work of parsing and extracting data from the HTML pages.

Web scraping is a process of automated data extraction from the site pages following certain rules.

Web scraping application fields:

• Tracking the prices of goods in online stores.
• Extracting descriptions of goods and services, getting the number of goods and pictures in a data sheet.
• Extracting contact information (email addresses, phone numbers, etc.).
• Collecting data for marketing research (likes, shares, ratings).
• Extracting specific data from the HTML-page code (searching for analytics systems, checking if there is a micro-markup).
• Monitoring ads.

The basic web scraping methods are parsing methods using XPath, CSS selectors, XQuery, Regex, and HTML templates.

• XPath is a query language for XML/XHTML document elements. To access elements, XPath uses DOM navigation by describing the path to the desired element on the page. Using it, you can get the value of an element by its ordinal number in the document, extract its text content or internal code, check for specific elements on the page. XPath description >>
• CSS selectors are used to find an element of its part (attribute). CSS is syntactically alike to XPath, but in some cases, CSS locators show faster performance and are more descriptive and concise. The only drawback of CSS is that it only works in one direction, i.e. deeper into the document. XPath works both ways (for example, you can search for a parent element by a child). CSS and XPath Comparison table >>
• XQuery is based on XPath. XQuery imitates XML, which allows you to create nested expressions in a way that is not possible in XSLT. XQuery description >>
• Regex is a search language for extracting values from a set of text strings that match the required conditions (regular expression). Regex description >>
• HTML templates is a language for extracting data from HTML documents, which is a combination of HTML markup aimed to describe the search template for the fragment, as well as functions and operations for extracting and transforming data. HTML templates description >>

As a rule, parsing is used to cope with tasks that are difficult to handle manually. This can be web scraping of product descriptions in a new online store, scraping in marketing research to monitor prices or ads (for instance, for apartment sales). For SEO optimization tasks, highly specialized tools are usually used with built-in parsers and all the necessary settings for extracting the main SEO parameters.

BatchURLScraper

There are many tools used for scraping (extracting data from websites), but most of them are paid and cumbersome, which limits their availability for mass use.

So, we decided to develop a simple to use and free tool called BatchURLScraper, designed specially to collect data from a URL list and adapted to export the results to Excel.

The software interface is quite simple and has only 3 tabs:

• The "URL List" tab is designed to add parsing pages and display the results of data extraction with their subsequent export.
• In the "Rules" tab, you can configure scoping rules using XPath, CSS locators, XQuery, Regex, or HTML templates.
• The "Settings" tab contains general program settings (number of threads, User-Agent, etc.).

We also added a module for debugging rules.

Using the built-in rule debugger, you can quickly and easily get the HTML content of any site page and test the operation of queries, and then use the debugged rules for parsing data in BatchURLScraper.

Let’s consider the examples of parsing configuration for different variants of data extraction.

Examples of extracting data from site pages

Since BatchURLScraper allows you to extract data from an arbitrary list of pages, which may contain URLs from different domains and, accordingly, different types of site, we will use all five scraping variants as examples to test data extraction: XPath, CSS, Regex, XQuery, and HTML templates. The list of test URLs and rule settings are located in the software package, so you can test them all this yourself using presets (preset parsing settings).

Data extraction mechanics

1. An example of scraping using XPath.

For example, in an online mobile phone store, we need to extract prices from the product card pages, as well as an indication of the item availability in stock (available or not).

To extract prices, we need to:

• Go to the product card.
• Highlight the price.
• Right-click on it and click "Inspect".
• In the window that opens, find the element responsible for the price (it will be highlighted).
• Right-click on it and choose Copy > Copy XPath.

In the same way you can extract an indication of the product availability on the site.

Since ordinary pages usually have the same template, it only takes to perform the operation of obtaining XPath for one such typical product page to parse the prices of the entire store.

Next, we add rules one by one in the list of program rules and paste the previously copied XPath element codes from the browser into them.

2. Then we need to determine the availability of a Google Analytics counter using Regex or XPath.

• XPath:
  • Open the source code of any page using Ctrl-U, then search for the text "gtm.start" in it, look for the UA -... identifier in the code, and then by using the display of the element code we copy its XPath and paste it into a new rule in BatchURLScraper.
• Regex:
  • Searching for a counter using regular expressions is even easier: you just need to insert the data extraction rule code ['](UA -.*?)['].

3. Extract contact email address using CSS.

It’s quite simple to do it. If there are hyperlinks like "mailto:" on the site pages, you can extract all email addresses from them.

To do this, we add a new rule, select the CSSPath in it, and insert the rule a[href^="mailto:"] into the data extraction rule code.

4. Extract values into a list or in a table using XQuery.

Unlike other selectors, XQuery allows you to use loops and other programming language features.

For example, using the FOR statement, you can get the values of all LI lists. Example:

Or you can figure out if there is email on the site pages:

• if (count(//a[starts-with(@href, 'mailto:')])) then "Email is available" else "No email available"

5. Using HTML templates.

In this language, you can use XPath/XQuery, CSSpath, JSONiq, and regular expressions to extract data as functions.

Test table:

For example, this template searches for a table with the id="t2" attribute and extracts the text from the second column of the table:

• <table id="t2"><template:loop><tr><td></td><td>{text()}</td></tr></template:loop></table>

Extracting data from second line:

• <table id="t2"><tr></tr><tr><template:loop><td>{text()}</td></template:loop></tr></table>

While this template calculates the sum of the numbers in the table column:

• <table id="t2">{_tmp := 0}<template:loop><tr><td>{_tmp := $_tmp + .}</td></tr></template:loop>{result := $_tmp}</table>

Therefore, we were able to extract almost any data from the site pages using an arbitrary list of URLs, including pages on different domains.

The following is a table with the most common rules for extracting data.

Examples of code for extracting data

In the table below, we have compiled a list of the most common data extracting options that can be retrieved using various types of extractors.

Other articles:

• Comparison of 20 desktop Web Crawlers
• How to Find Duplicate Content on Your Site
• Bulk Page Speed Test with Google API