Errors in SEO: «Missing or Empty Robots.txt File»

Errors List

1. 3xx Redirected Pages
2. 4xx Error Pages: Client Error
3. 5xx Error Pages: Server Error
4. Bad AMP HTML Format
5. Bad Base Tag Format
6. Blocked by Meta Robots
7. Blocked by Robots.txt
8. Blocked by X-Robots-Tag
9. Broken Images
10. Broken Pages
11. Broken Redirect
12. Canonical Chain
13. Canonical URL Contains Another Host
14. Canonicalized Pages
15. Compliant AMP Pages
16. Compliant URLs with Query
17. Contains Flash
18. Contains Lorem Ipsum
19. Duplicate Descriptions
20. Duplicate H1
21. Duplicate Pages
22. Duplicate Text
23. Duplicate Titles
24. Emojis or Special Characters in Description
25. Emojis or Special Characters in Title
26. Endless Redirect
27. External Dofollow Links
28. External Nofollow Links
29. External Redirect
30. GA: Compliant Pages w/o Traffic
31. GA: Max Bounce Rate
32. GA: Non-Compliant Pages with Traffic
33. GSC: Compliant Pages w/o Clicks
34. GSC: Compliant Pages w/o Impressions
35. GSC: Non-Compliant Pages with Impressions
36. Hreflang: Duplicate Alternate URLs
37. Hreflang: Duplicate Language Codes
38. Hreflang: Inconsistent Language Code in Confirmation Links
39. Hreflang: Incorrect Language Codes
40. Hreflang: Links to Non-Compliant URLs
41. Hreflang: Missing Alternate URLs
42. Hreflang: Missing Confirmation Links
43. Hreflang: Missing Self-Reference
44. Hreflang: Relative Links
45. HTTPS → HTTP Hyperlinks
46. HTTPS → HTTP Redirects
47. Hyperlinks with Empty Anchor Text
48. Identical Canonical URLs
49. Images Without Alt Attributes
50. Images Without Title Attributes
51. Internal Nofollow Links
52. Links to Localhost
53. Links with Bad URL Format
54. Long Server Response Time
55. Max Content Size
56. Max Description Length
57. Max External Links
58. Max H1 Length
59. Max HTML Size
60. Max Image Size
61. Max Internal Links
62. Max Redirections
63. Max Title Length
64. Max URL Length
65. Min Content Size
66. Min Text/HTML Ratio
67. Missing Internal Links
68. Missing or Empty Description
69. Missing or Empty H1
70. Missing or Empty Robots.txt File
71. Missing or Empty Title
72. Missing Structured Data
73. Mixed Content
74. Multiple Descriptions
75. Multiple H1
76. Multiple Titles
77. Nofollowed by Meta Robots
78. Nofollowed by X-Robots-Tag
79. Non-Compliant Canonical URL
80. Non-HTTPS Protocol
81. PageRank: Dead End
82. PageRank: Missing Outgoing Links
83. PageRank: Orphan
84. PageRank: Redirect
85. Percent-Encoded URLs
86. Query Contains Question Mark
87. Redirect Blocked by Robots.txt
88. Redirect Chain
89. Redirects with Bad URL Format
90. Refresh Redirected
91. Same Title and Description
92. Same Title and H1
93. Self-Referencing Hyperlinks
94. Short Description
95. Short H1
96. Short Title
97. Spelling Mistakes
98. URLs with Unwanted Special Characters
99. URLs with Uppercase Characters
100. URLs with UTM Parameters
101. URLs with Wrong Hyphenation

Importance: Low

Error «Missing or Empty Robots.txt File» Description

Indicates compliant URLs that belong to a host with an empty or missing robots.txt file. Note that different hosts (subdomains or http/https protocols) may contain different robots.txt files.

The Importance of the Problem

A robots.txt file informs search robots which pages and files on the site can and cannot be crawled. Search engines consider it important for robots.txt file to be present on a site. If it's missing, website crawling may be inefficient.

Note that robots.txt works for the host where it's located. That's why the file located at http://site.com/robots.txt does not affect crawling of https://site.com.

How to Fix Error

Add the robots.txt file with crawling recommendations to the root of all hosts on your site.