Twitter Facebook Linkedin Product Hunt
SiteAnalyzer SEO Tools

Check the Meta Robots Noindex / Nofollow

Also check the X-Robots-Tag HTTP-headers

URLs List (MAX 50) ↓

Captcha: refresh

 
(code shown in the picture)


SiteAnalyzer SEO Tools browser Extension, for Chrome and Firefox

What Does the Meta Robots Tool Do?

With the help of Meta Robots Tool you can find pages forbidden to indexing. Check if your webpage is using the robots meta tag or the X-Robots-Tag HTTP header to instruct search engines not to show your website in search results pages.

Meta Robots

Meta Robots is a meta tag that allows you to set up instructions for indexing a site. Its advantages are reliability and ease of installation. The tag is written between <head> and </head> in the page code. It can not only prohibit indexing, but also prohibit search robots from clicking on internal and external links, save the document in the cache, save some excerpts from the document and put them in a snippet, and much more.

Example:

  • <meta name="robots" content="noindex, nofollow"/>

X-Robots-Tag

X-Robots-Tag is an HTTP header that provides guidance on indexing a site to search engines.

Using X-Robots-Tag is considered productive and useful for optimizing the site. This element gives search engines instructions that are understandable without loading the main body of the document. In this regard, his instructions for search engines are more authoritative, because systems spend less resources for analyzing documents, which means they save the crawling budget allocated for the site.

This element is set at the level of the server headers, so the instructions in it have priority rules.

Example (PHP programming language):

  • header("X-Robots-Tag: noindex, nofollow");


0 comments

You must be logged to leave a comment.



Popular SEO Tools / All Tools


Text Relevance

Text Semantic

Redirect Checker

CMS Detector

Title & H1 Checker

Find Subdomains

Bulk Site Indexing
             

URL HTTP Status Code

Sitemap Generator

Robots.txt Checker

Domain Age Checker

Text Unique Checker

UULE Generator

Link Extractor
             
     

Our Soft
     

<< Back

Our Clients