> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
[0] https://git.zx2c4.com/cgit/about/
Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python
https://github.com/mthcht/Masquerade-Spoofer
Great for AI scrapers, bad for hosters and everyone else.