I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links.
Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects.
I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the quest for frontier models.
I've also seen Google indexing pages with random values in the path that don't get linked to statically (server asks for the URL then redirects to it immediately). I'm pretty sure they index straight out of the Chrome address bar.
Yep. I remember a similar story as GP described from a friend back in 2008. The site he was working on that wasn't linked to yet was suddenly indexed after he checked out what it looked like in the fancy new "Chrome" browser that Google had just released, causing some moderate panic on his end.
Well, the angle is kind of important here. The company gets their name in the news, they have a reasonable explanation why they were scraping around, and we end up with a story about innovative tech company whiz-kids who made a funny discovery, while it was the webdevs on the other side that goofed up.
Imagine a private individual just scraped the website (or simply clicked 'view source') for no reason in particular and then told people about it... They'd be labeled an uber-haxxor, face a civil lawsuit asking for ridiculous damages while being threatened with a prison sentence over CFAA violations. Hell, that might even drive some people to suicide.
Yeah that happens all the time. Anyone/thing with popular public releases has fans/journeys scraping the website looking for unreleased material or scoops.
In the early days one of the high profile soaps in the UK published their "catch up" summaries for the week ahead which you could get just by editing the date in the URL. But back then not so many people were looking, so they were doing it for months...
Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects.
I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the quest for frontier models.
[1] https://developers.cloudflare.com/cache/advanced-configurati...
This is on the devs and feels like a very basic leak which could have exploited in the non LLM world as well.
Imagine a private individual just scraped the website (or simply clicked 'view source') for no reason in particular and then told people about it... They'd be labeled an uber-haxxor, face a civil lawsuit asking for ridiculous damages while being threatened with a prison sentence over CFAA violations. Hell, that might even drive some people to suicide.
In the early days one of the high profile soaps in the UK published their "catch up" summaries for the week ahead which you could get just by editing the date in the URL. But back then not so many people were looking, so they were doing it for months...
https://news.ycombinator.com/item?id=48902814
See also
Zhihu (Chinese Reddit): https://www.zhihu.com/question/2060133066643879544/answer/20...
Reddit: https://www.reddit.com/r/math/comments/1urv4id/comment/oxak6...
Interestingly, if true, it will also be the first time an MIT PhD graduate has won the Fields Medal.