when i open my robots.txt file it says: User-agent: * Disallow: / . what does this mean? does it block all robots? if so, my site might not be showing up on google!
what's the correct way to write robots.txt? which files should i 'disallow' and which ones should i leave open?
if i write robots.txt incorrectly will it show an error in Search Console, or will it just silently hide the site?
robots.txt explanation: User-agent: * = all robots, Disallow: / = disallow '/' (entire site). so it blocks ALL robots — meaning my site is hidden from Google, Bing, everyone. DELETE this file!
correct robots.txt: leave it blank (don't write anything) or write this: User-agent: * Disallow: /admin/ Disallow: /private/ Allow: / Sitemap: https://example.com/sitemap.xml
this means: block admin and private folders, allow everything else, tell google about the sitemap.
Search Console notification: if robots.txt completely disallows the site, Search Console will show 'Excluded by robots.txt' in coverage.
Correction: I misremembered the figure, it was a bit lower.
robots.txt examples: 1) disallow wordpress install dir: Disallow: /wp-admin/ Disallow: /wp-includes/. 2) private files: Disallow: /private/. 3) test page: Disallow: /test.html. 4) block all bots: Disallow: / (if you want, allow Googlebot specifically: Allow: / for Googlebot).
last year our developer messed up and disallowed the entire site in robots.txt. tbh we got zero google traffic for 1 month the site was basically dead. we removed the disallow from robots.txt and 2 weeks later google started crawling again traffic came back.
lesson: chcek your robots.txt most of the time disallowing the whole site is an accidental error.
the robots.txt file needs to be in the site root (example.com/robots.txt). if the file is missing bots won't have issues, but if it exists and is wrong, that's the problem. remove that 'Disallow: /' you wrote immediately it's hiding the whole site.
robots.txt: if you write Disallow: /, you're basically telling google to 'go away'. google: ok bye. 3 months out of the index, zero traffic, regret. write it correctly or leave it alone.
robots.txt status codes: 200 = site ok. 404 = no robots.txt (no problem, default allow). 403 = access denied. User-agents: * = all bots, Googlebot = Google robots, Bingbot = Bing.
I agree and Id like to emphasize that. Mistakes made on the robots txt error side are usually reversible but expensive.
If you have questions write them; I'll answer as best I can.
I felt relieved reading this answer, so it's not just me. Just because everyone does it doesn't mean it's right.
Hope this helps.
Doki · SEO consulting · 2026
i have no experience with robots txt error, so I'm asking. tbh hasty decisions become decisions you have to fix six months later.
hope this helps.
Doki · Corporate website · 2023
I'm a small business, let me explain from my side. If rankings stay the same but clicks drop the issue isn't your page, but the search result appearance.
Correct me if I'm wrong.
i have no experinece with robots txt error, so Im asking. like when making a decision, first look at what data you have on hand.
Doki · Log management setup · 2025
Yes, that's exactly how it is with robots txt error. Processes without records never improve, because you don't know what to fix.
Good luck with that.
The discussion got scattered, let me summarize. Don't hesitate to ask; those who don't ask always pay more.
Proven by experience.
I've been dealing with this for a long time. When making decisions, write down the worst-case scenario too, not just the best.
If you post the result here, it will help others too.