What it is
robots.txt is a plain text file that lives at the very root of a website (at an address like yourdomain.com/robots.txt) and gives instructions to search engine crawlers — the automated programs that visit websites to read and index their pages. It typically says things like "you're welcome to crawl all of this site" or lists specific sections that shouldn't be crawled, such as an internal admin area or a search-results page that doesn't need to show up in Google.
It's a set of polite requests rather than a security barrier — well-behaved crawlers (like Google's and Bing's) respect it, but it doesn't actually block anyone from accessing a page directly if they already have the address.
Why it matters
Without a robots.txt file, most crawlers will simply assume they're free to crawl everything on the site, which is often fine for a small site with nothing to exclude. But its absence also means you have no say at all in the matter — if there ever is a section you don't want crawled (an internal tool, a staging area accidentally left public, duplicate content generated by site search), there's no file in place to say so.
Search engines also check for this file as one of the first things they do when discovering a new site, and it's commonly used to point crawlers toward your sitemap, making your other pages easier to find and index faster.
How to fix it
- Create a plain text file named exactly
robots.txt(all lowercase). - At minimum, include
User-agent: *followed byAllow: /to explicitly allow all crawlers to access your whole site. - If there are specific sections you don't want indexed, add a line like
Disallow: /admin/for each one. - Add a line pointing to your sitemap, e.g.
Sitemap: https://yourdomain.com/sitemap.xml, so crawlers can find your other pages faster. - Upload the file to your site's root folder (not inside any subfolder) so it's reachable at
yourdomain.com/robots.txt— most website builders have a dedicated settings field for this instead of requiring a file upload. - Visit that address in a browser afterward to confirm the file loads and displays your rules as plain text.