Errors in SEO: «Blocked by Robots.txt»

Errors List

1. 3xx Redirected Pages
2. 4xx Error Pages: Client Error
3. 5xx Error Pages: Server Error
4. Bad AMP HTML Format
5. Bad Base Tag Format
6. Blocked by Meta Robots
7. Blocked by Robots.txt
8. Blocked by X-Robots-Tag
9. Broken Images
10. Broken Pages
11. Broken Redirect
12. Canonical Chain
13. Canonical URL Contains Another Host
14. Canonicalized Pages
15. Compliant AMP Pages
16. Compliant URLs with Query
17. Contains Flash
18. Contains Lorem Ipsum
19. Duplicate Descriptions
20. Duplicate H1
21. Duplicate Pages
22. Duplicate Text
23. Duplicate Titles
24. Emojis or Special Characters in Description
25. Emojis or Special Characters in Title
26. Endless Redirect
27. External Dofollow Links
28. External Nofollow Links
29. External Redirect
30. GA: Compliant Pages w/o Traffic
31. GA: Max Bounce Rate
32. GA: Non-Compliant Pages with Traffic
33. GSC: Compliant Pages w/o Clicks
34. GSC: Compliant Pages w/o Impressions
35. GSC: Non-Compliant Pages with Impressions
36. Hreflang: Duplicate Alternate URLs
37. Hreflang: Duplicate Language Codes
38. Hreflang: Inconsistent Language Code in Confirmation Links
39. Hreflang: Incorrect Language Codes
40. Hreflang: Links to Non-Compliant URLs
41. Hreflang: Missing Alternate URLs
42. Hreflang: Missing Confirmation Links
43. Hreflang: Missing Self-Reference
44. Hreflang: Relative Links
45. HTTPS → HTTP Hyperlinks
46. HTTPS → HTTP Redirects
47. Hyperlinks with Empty Anchor Text
48. Identical Canonical URLs
49. Images Without Alt Attributes
50. Images Without Title Attributes
51. Internal Nofollow Links
52. Links to Localhost
53. Links with Bad URL Format
54. Long Server Response Time
55. Max Content Size
56. Max Description Length
57. Max External Links
58. Max H1 Length
59. Max HTML Size
60. Max Image Size
61. Max Internal Links
62. Max Redirections
63. Max Title Length
64. Max URL Length
65. Min Content Size
66. Min Text/HTML Ratio
67. Missing Internal Links
68. Missing or Empty Description
69. Missing or Empty H1
70. Missing or Empty Robots.txt File
71. Missing or Empty Title
72. Missing Structured Data
73. Mixed Content
74. Multiple Descriptions
75. Multiple H1
76. Multiple Titles
77. Nofollowed by Meta Robots
78. Nofollowed by X-Robots-Tag
79. Non-Compliant Canonical URL
80. Non-HTTPS Protocol
81. PageRank: Dead End
82. PageRank: Missing Outgoing Links
83. PageRank: Orphan
84. PageRank: Redirect
85. Percent-Encoded URLs
86. Query Contains Question Mark
87. Redirect Blocked by Robots.txt
88. Redirect Chain
89. Redirects with Bad URL Format
90. Refresh Redirected
91. Same Title and Description
92. Same Title and H1
93. Self-Referencing Hyperlinks
94. Short Description
95. Short H1
96. Short Title
97. Spelling Mistakes
98. URLs with Unwanted Special Characters
99. URLs with Uppercase Characters
100. URLs with UTM Parameters
101. URLs with Wrong Hyphenation

Importance: Medium

Error «Blocked by Robots.txt» Description

Indicates URLs disallowed in the robots.txt file.

The Importance of the Problem

A robots.txt file can disallow search robots from crawling specific website pages. Thus, they will not take part in search.

Robots.txt is mostly used to block pages unimportant for search (e.g. internal search, shopping cart, sign up pages, etc.) to save robot's crawling resources.

If a website contains a lot of links pointing to URLs that are disallowed from crawling, useful pages may get less link weight, lower rankings, thereby less traffic.

How to Fix Error

Important pages have to be allowed for crawling by search robots.

It's recommended to reduce the number of links pointing to pages disallowed from crawling.

However, if links are useful for user navigation through the website (e.g. links pointing to shopping cart or sign up pages), do not delete them.