One line of code, and a whole pub estate goes invisible to AI
A single robots.txt setting is quietly blocking every photo and menu on a shared pub website platform from Google Images and every AI crawler. How to check your own site in two minutes.
We found this by accident, checking why a pub’s interior photos never showed up anywhere a search engine or an AI system could see them. The photos existed, properly taken, properly hosted. They were just invisible.
The cause was one line in a technical file called robots.txt, sitting on the shared platform behind a whole group of pub websites. That one line told every crawler, Google’s included, and every AI system that reads the web to answer a question, to ignore the entire folder where photos and menu documents were stored. Not one pub’s mistake. Every pub on that platform, blocked the same way, at the same time, by the same setting.
Nobody did this on purpose
This is the boring, unglamorous kind of fault that does the most damage: a default nobody remembers agreeing to, set once, years ago, by whoever built the platform, never revisited since. It doesn’t announce itself. Nothing breaks that anyone would notice by looking at the site. It just quietly removes you from every place an AI system looks for evidence.
The fix, once found, is small: one exception added to the file, for the folder that should be public. At the time of writing it has not been made. The photos don’t need retaking. The menus don’t need rewriting. They just need to stop being invisible.
Why this matters more with AI in the loop
A human customer who lands on your page can still see the photo even if a crawler can’t. An AI system answering a question on a customer’s behalf never lands on the page at all. It only knows what its crawler was allowed to read. A blocked folder isn’t a cosmetic problem for that customer. It’s the whole answer, missing.
What leaders should do next
- Open your own site’s
robots.txtin a browser today. It’s a plain text file at yoursite.com/robots.txt. If a folder holding photos or documents shows as blocked, ask why. - Test one photo and one menu document from a fresh, logged-out browser, the way a crawler actually sees your site, not the way you see it while signed in.
- If you’re on a shared platform across several sites, ask whether the fix needs applying once at platform level or once per site. One fix at the top is worth far more than a hundred edits underneath it.
- Add “can a machine actually read this” to your website checklist. It’s a different question from “does it look right,” and it’s the one that decides whether AI answers ever mention you.