> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Anubis's default 1-week token lifetime may not be enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
In theory, proof of work that is used to mine a cryptocurrency could be a solution.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill.
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting.
On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
I have a medium-sized Discord server of web sysadmin people (mostly wiki operators) that came together after I wrote a similar blog post [1] about how the LLM scrapers/resproxies are making it suck to run wikis. Not sure if there's other private communities out there, but feel free to email me if you want to join
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage.
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
That's missing the title and URL, as I suspected it would, though the submitter UID is available.
To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!
In other words, they don't care at all. For them, residential bandwidth is completely free.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
>sell access to the user's internal residential network
That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):
if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) {
LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId);
throw new IllegalArgumentException("Hacking? The tunnel host resolved ip is internal");
}
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, and 127.0.0.0 do not work. It will not connect to people's private file servers on their network.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
[1]: https://news.ycombinator.com/item?id=30175269
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)
and most of the time, its on marketing product pages like in framework main site, which can be cached.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
I believe that's the HN application itself, not a WAF in front of it.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.
... What?!
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Millions per hour is tens per second though; perhaps the fix is performance improvements
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Who’s doing it, are they even using the data?
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW.
Securing signup forms is another issue. Maybe related. But not what I was referring to.
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
Solved by making money off it.
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
Project is here on Github: https://github.com/UninvitedActivity/UninvitedActivity
[1] - https://weirdgloop.org/blog/clankers
Is that a side effect of whatever you are doing?
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
[1] - https://news.ycombinator.com/item?id=48416693
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
https://github.com/HackerNews/API
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
... it's starting to make sense, but ...
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
<https://news.ycombinator.com/front?day=2008-11-05>
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
"A Year on Hacker News" <https://news.ycombinator.com/item?id=31295219>
"Top Hacker News commenters of 2021" <https://news.ycombinator.com/item?id=29778994>
"What Happened This Year on Hacker News (2021)" <https://news.ycombinator.com/item?id=29769470>
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
OK, looking at a recent dead atomic128 comment:
So userID is visible.And from a current dead submission in the New queue:
That's missing the title and URL, as I suspected it would, though the submitter UID is available.To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
[2]: https://docs.mossdef.org/adblock-fast/
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
The article was called "The one we're commenting on"
Does that sound like your typical self-hosted blog?
unethical yes but really raises the question as to what we see is real or not
No they really don't, dishonest founders do that.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
??? shift is an extra key to press
most shibboleths are subtle like that
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
brdtnet.com
luminatinet.com
bright-sdk.com
luminati.io
As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year
- i know some people who use "smart lightbulbs" and other nonsense
- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
In other words, they don't care at all. For them, residential bandwidth is completely free.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
We agree that it would be great if it was even more widely used.
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Edit: If you don't want to be banned, you're welcome to email [email protected] and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.
Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.
<https://news.ycombinator.com/item?id=47116093> (LLM but not PF).
<https://news.ycombinator.com/item?id=47095664> (a16h)
<https://news.ycombinator.com/item?id=46695693> (vuln exploits) 2026-1-20
<https://news.ycombinator.com/item?id=46280602> (???, but not PF)
<https://news.ycombinator.com/item?id=46195234> (Monero / Dark Web) 2025-12-8
<https://news.ycombinator.com/item?id=45894305> and <https://news.ycombinator.com/item?id=45826273> (Tor hidden service) Nov 2025
That's from the past 20 comments.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
HN's prime directive is "anything that gratifies one's intellectual curiosity": <https://news.ycombinator.com/newsguidelines.html> and many, many, many dang comments.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Occasional alts are explicitly permitted, though not to engage in abuse (e.g., mutual admiration societies, sock-puppetry). See: <https://news.ycombinator.com/item?id=9963551> <https://news.ycombinator.com/item?id=9823379> (both against sock puppetry) and <https://news.ycombinator.com/item?id=9122086> and <https://news.ycombinator.com/item?id=7504621> (on where throwaways are/aren't permitted).
Where HN does favour persistent accounts the stated claim is to foster community, rather than for nefarious tracking purposes: <https://news.ycombinator.com/item?id=18082346> and <https://news.ycombinator.com/newsguidelines.html>. From that last:
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
I wasn't aware of this project. Thanks for the heads up.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
Really makes you think, what we're feeding them...
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.
LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.
>will now check for NetNut-infected apps
Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, and 127.0.0.0 do not work. It will not connect to people's private file servers on their network.