Facebook Twitter Telegram Linkedin

Check the Meta Robots Noindex / Nofollow

Also check the X-Robots-Tag HTTP-headers

URLs List (MAX 50) ↓

Captcha: refresh

 
(code shown in the picture)


What Does the Meta Robots Tool Do?

With the help of Meta Robots Tool you can find pages forbidden to indexing. Check if your webpage is using the robots meta tag or the X-Robots-Tag HTTP header to instruct search engines not to show your website in search results pages.

Meta Robots

Meta Robots is a meta tag that allows you to set up instructions for indexing a site. Its advantages are reliability and ease of installation. The tag is written between <head> and </head> in the page code. It can not only prohibit indexing, but also prohibit search robots from clicking on internal and external links, save the document in the cache, save some excerpts from the document and put them in a snippet, and much more.

Example:

  • <meta name="robots" content="noindex, nofollow"/>

X-Robots-Tag

X-Robots-Tag is an HTTP header that provides guidance on indexing a site to search engines.

Using X-Robots-Tag is considered productive and useful for optimizing the site. This element gives search engines instructions that are understandable without loading the main body of the document. In this regard, his instructions for search engines are more authoritative, because systems spend less resources for analyzing documents, which means they save the crawling budget allocated for the site.

This element is set at the level of the server headers, so the instructions in it have priority rules.

Example (PHP programming language):

  • header("X-Robots-Tag: noindex, nofollow");


Other languages: Беларускі 简体中文 Čeština Français Deutsch Italiano 日本語 Қазақша 한국어 Polski Português (BR) Srpski Español Türkçe Українська


Popular SEO Tools / All Tools


Text Relevance

Text Semantic

CMS Detector

Sitemap Generator

Find Subdomains

Reverse IP Lookup

Bulk Site Indexing

<< Back

Our Clients