Parts will be cached but it’s per instance. For a 100 user instance, what are the chances that a recent cache of All page 74 is available? Bots are crawling every page, and most pages are content from last month, last year, or 3 years ago.
I would expect Lemmy instances to have some limiters set for guest connections, especially for ones coming from datacenters. Like, it should be easy and senseful to only link and cache up to All page 10, or Hot page 5, and refuse guest loads of anything beyond that.
I dont believe it is that easy to tell if an IP is coming from a data center
The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.
I mean, adding the IP filter means you are getting a number of bots filtered out by a very simple decision making (originating IP address/segment) even before you have to apply any further processing, so it’s still an advantage even if that filter by itself only removes a non-majority portion of bot traffic. It means you are left with more / better processing capacity to filter the rest.
I guess the problem is that you’ve gotta be super careful. If you just block all data centres, you won’t be able to federate with most of the fediverse. It’s got to only be applied to the front end website, not to the API and especially not to the federation endpoints.
Tools like Anubis do the filtering pretty well anyway. Some 75% of traffic we get to the website is blocked without even getting a challenge, because of their list of IPs, user agents, etc.
Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.
As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.
Parts will be cached but it’s per instance. For a 100 user instance, what are the chances that a recent cache of All page 74 is available? Bots are crawling every page, and most pages are content from last month, last year, or 3 years ago.
I would expect Lemmy instances to have some limiters set for guest connections, especially for ones coming from datacenters. Like, it should be easy and senseful to only link and cache up to All page 10, or Hot page 5, and refuse guest loads of anything beyond that.
deleted by creator
The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.
I mean, adding the IP filter means you are getting a number of bots filtered out by a very simple decision making (originating IP address/segment) even before you have to apply any further processing, so it’s still an advantage even if that filter by itself only removes a non-majority portion of bot traffic. It means you are left with more / better processing capacity to filter the rest.
I guess the problem is that you’ve gotta be super careful. If you just block all data centres, you won’t be able to federate with most of the fediverse. It’s got to only be applied to the front end website, not to the API and especially not to the federation endpoints.
Tools like Anubis do the filtering pretty well anyway. Some 75% of traffic we get to the website is blocked without even getting a challenge, because of their list of IPs, user agents, etc.
Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.
As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.