What is a robots.txt and why it matters

A robots.txt file is something you create to tell search engines, typically Google, which parts of your website they should crawl or avoid crawling.

In simple terms, the robots.txt file tells search engines whether a page should be indexed (“index”) or not indexed (“no-index”).

It can also indicate whether they should follow or not follow links on certain pages.

How you can use robots.txt on your website

Imagine you have some hidden pages on your site that are only accessible if someone has a direct link. These are pages you probably don’t want showing up in Google search results. In that case, you would use the “no-index” directive in your robots.txt file to keep them out of the search index.

This would typically be added to the page’s HTML using a meta robots tag.

Det er måske en smule langhåret, og medmindre man er programmør, vil man også sidde og rette i denne fil på en lidt nemmere facon.

Of course, this can get a bit technical. Unless you’re a developer, you’ll probably want a simpler way to manage these settings.

Two tools that can help you

One option is to use a plugin like Yoast SEO for WordPress. This way, you don’t need to write any code—you just tick a few boxes to tell search engines how to handle your pages.

Another helpful tool is Google’s own robots.txt tester. It’s a bit harder to find in the new version of Search Console, and that might not be a coincidence. I’ll explain why below.

Is robots.txt still relevant?

As Google becomes increasingly sophisticated, the robots.txt file isn’t as crucial as it used to be. Google is now quite good at figuring out which parts of a site to crawl and which ones to ignore. Also, directives like “no-index” are now being phased out in how Google interprets content.

You can find more about Google’s current view on robots.txt in their own documentation.