Background: I’m using Ungoogled Chromium, which I do not keep up to date because non-Debian packages are a big hassle to maintain. I expect this risk to be mine alone.
Since websites are secretive¹ about their reasons for blocking, it has been quite painful (as a Tor user) that many services are not blocking Tor, but in fact blocking old versions of Chrome. WTF? Why does a website block old Chrome versions? My gut thought was the website wants to nanny users, to try to force users to secure themselves (“for the user’s own protection”), because surely a user with old chrome does not realistically threaten the server’s security.
The answer: it turns out that AI bots are strangely using old Chrome versions for a user-agent header. So webmasters are reacting by blocking old versions of Chrome. What I don’t understand is why devs of AI bots would try to masquerade as old Chrome versions.
And to a lesser extent I wonder why webmasters are reckless enough to take this collective-punishment-is-ok stance. It would be like a store owner banning customers wearing blue hats from their shop because some shoplifters were caught wearing blue hats.
¹ Exceptionally, codeberg.org actually respects our dignity enough to preduce an error message that says they are blocking old chrome. But they are the exception.
Yeah, it’s completely normal.
The AI bots often use curl, which is commonly blocked by UA filtering, so they started sidestepping that by setting a fake chrome UA (usually the latest chrome version as of model training cutoff).
You just set a fake UA of your own to fix it.
Unless I am using frogfind.com, which discriminates against recent UAs because it is meant to serve those with old gear.
The UA setting has become relevant to enough instances of enshitification that perhaps we really need a DB that tracks which UA gets the best treatment per website. I would like a browser that chooses the UA string that will minmize enshification. I suspect the answer may be a string that impersonates Google’s crawler would be ideal, particularly with paywalls.
Unless you wanna make life hard on yourself for no good reason, use your real UA where possible and set the closest-to-accurate UA as a site-specific override where needed.
Fake googlebots are extremely easy to spot and will get treated as malicious. Cloudflare will flag you if you’re spoofing browser identity in any obvious way.
Unless you wanna make life hard on yourself for no good reason, use your real UA where possible
I’m not sure you understand the transparency problem. Codeberg is the /only/ service that I have ever encountered that honestly, transparently states that they blocked me on the basis of old chrome. All other blocks on the order of 10s or 100s per day give no reason. As a Tor user, I assume Tor is the problem. But now there are increasingly non-Tor criteria emerging. I cannot investigate every blockade. Hence why establishing a UA string db would be useful, which could then be incorporated into browsers.
Cloudflare will flag you if you’re spoofing browser identity in any obvious way.
I boycott Cloudflare. CF already blocks most of my access attempts and I am grateful for it. What disturbs me is when I discover that I am on a CF website that whitelisted Tor because such cases could lead to me accidentally patronising a Cloudflare supporter. I know there are plugins that would help me avoid those instances but if CF has another trigger to block me from their side, then it is mutually appreciated.
If your point is transparency, and the given justification is to block scrapers, then it makes sense they wouldn’t be transparent about it. You don’t want to give your adversary instructions about how to bypass your security.
That could be a plausible reason for opacity if it were true, but I don’t think it’s true because blocking Tor (for example) is trivially discovered and the non-transparency only frustrates legit users. The opacity of an anti-tor policy has no effect on adversaries. Yet a vast majority of tor-hostile sites are sloppy about informing legit users.
Infosec that relies on secrecy is already out on a limb in despiration. And in this situation, the collateral damage to friendlies based on arbitrary factors creates a heightened need to inform the legit users. An AI scraper bot is not going to be coded to know where to look for such announcements because that’s actually more effort for the attacker than manual trial and error. Note as well we are talking about relatively unsophisticated adversaries considering they were too lazy to code the user-agent header to dynamically blend better in the first place. (They will fix that in the future).
Nonetheless, your point cannot be disregarded because those who take the crude approach of blocking on the basis of UA string are likely not thinking all this through. Perhaps some have consciously made the decision to not inform the legit users, but considering the widespread pattern of not informing users, it’s sloppy disregard for the most part.
What that database would tell you is that you should use the latest closest-to-accurate UA for your browser and OS.
But you seem to want to soapbox and relish in your self-inflicted struggles here, so I’ll leave you to play with your tin foil in peace.
What that database would tell you is that you should use the latest closest-to-accurate UA for your browser and OS.
If you actually mean the tool that uses the database, sure, but whatever the “closest” is requires a db as input to derive what that would be.
But you seem to want to soapbox and relish in your self-inflicted struggles here, so I’ll leave you to play with your tin foil in peace.
Although consumers should often be blamed for enshitification, this is not one of those cases. If an anti-enshitification tool would state: “this service judges you by your UA”, then I would protest accordingly so as to not be part of the problem of rewarding enshitification.
What I’ve noticed is that, when I can’t log in to a site because I’m using a “wrong” browser, if I just keep trying, it’ll eventually let me in.
You know what you’re doing. Your less technical friend or relative may for some reason have broken auto updates on chrome.
Probably not what you’re asking but sometimes companies are required to block insecure browser versions due to governmental certification requirements, and also because they could be held responsible if they allowed users to access their services with known insecure browsers and something then happened.
What we used to do at my job was to display a big red warning banner to users with outdated or unsupported browsers, without strictly preventing them from using the site. This resulted in some users (who apparently couldn’t read) submitting support tickets when the site didn’t work with their three year old browser, then getting angry when we wouldn’t “fix” the bug. More importantly tho, this led to a failure in an external audit. I won’t go into the details, but basically the only sensible way to meet that specific requirement was to block all outdated browsers (by user-agent matching)
Very interesting, thank you for sharing
What I don’t understand is why devs of AI bots would try to masquerade as old Chrome versions.
It’s because the bot maintainers don’t update it much. Once it works for their purposes, they just don’t touch it any more until it breaks. Chrome on the other hand gets a lot of updates and automatically increases some version number every time.
