|
URLs List (MAX 50) ↓ Captcha: refresh
|
What Does the Meta Robots Tool Do?
With the help of Meta Robots Tool you can find pages forbidden to indexing. Check if your webpage is using the robots meta tag or the X-Robots-Tag HTTP header to instruct search engines not to show your website in search results pages.
Meta Robots
Meta Robots is a meta tag that allows you to set up instructions for indexing a site. Its advantages are reliability and ease of installation. The tag is written between <head> and </head> in the page code. It can not only prohibit indexing, but also prohibit search robots from clicking on internal and external links, save the document in the cache, save some excerpts from the document and put them in a snippet, and much more.
Example:
- <meta name="robots" content="noindex, nofollow"/>
X-Robots-Tag
X-Robots-Tag is an HTTP header that provides guidance on indexing a site to search engines.
Using X-Robots-Tag is considered productive and useful for optimizing the site. This element gives search engines instructions that are understandable without loading the main body of the document. In this regard, his instructions for search engines are more authoritative, because systems spend less resources for analyzing documents, which means they save the crawling budget allocated for the site.
This element is set at the level of the server headers, so the instructions in it have priority rules.
Example (PHP programming language):
- header("X-Robots-Tag: noindex, nofollow");
Popular SEO Tools / All Tools
 Text Relevance |
 Text Semantic |
 Redirect Checker |
 CMS Detector |
 Title & H1 Checker |
 Find Subdomains |
 Bulk Site Indexing |
 URL HTTP Status Code |
 Sitemap Generator |
 Robots.txt Checker |
 Domain Age Checker |
 Text Unique Checker |
 UULE Generator |
 Link Extractor |
 SiteAnalyzer |
