Notice that flashed on wiki access today - too fast to read - was it important?
Arch Devs, On access of https://wiki.archlinux.org today, a dialog flashed to the screen immediately before the wiki main page was rendered, but the dialog flashed too quickly to read. Was it important or require any action? May have just been a wiki update note, dark mode looks a bit different? -- David C. Rankin, J.D.,P.E.
Hello. I'm not a developer and new to the mailing list. That is Anubis, a open source solution to stop AI crawlers. It's been deployed by a lot of projects as of recently. Thank you for your message. Best of wishes.
On access of https://wiki.archlinux.org today, a dialog flashed (…)
Hello, Many sites are currently falling victim to crawlers from LLM companies. The bots mercilessly hammer the servers, causing enormous load. This leads to service disruption to normal users. Since many of them own hundreds of IP addresses, it’s not possible to throttle them based on requests alone. Multiple targets, Arch Wiki included, decided to deploy a client-side proof-of-work mechanism: Anubis from Techaro.⁽¹⁾⁽²⁾ This runs a small program, that does “expensive” operation in the browser. It’s nothing for normal users, even if we count all users together. But it’s a notable cost to companies doing billions requests. Cheers ____ ⁽¹⁾ https://techaro.lol/ ⁽²⁾ Screenshot of what you didn’t se: https://0x0.st/8VRo.png
Hi,
Anubis from Techaro. This runs a small program, that does “expensive” operation in the browser. It’s nothing for normal users
Visiting the Arch Linux wiki, my browser achieves 2.8 kH/s at difficulty 4, and the test finally completes after 40 seconds. This matches other Anubis-wrapped sites. If it's meant to adjust according to the speed of the user's PC, then it doesn't do too well. -- Cheers, Ralph.
On Sat, 2025-04-26 at 10:04 +0100, Ralph Corderoy wrote:
the test finally completes after 40 seconds
I tried two time. The first time it was like a flash, but I was able to notice the word "success". The second time it was slow, still less than a second, but the word "success" stood still for a fraction of a second, it wasn't just a fleeting flash. In any case, it is not a problem for me, as I have JavaScript activated anyway.
Hi Ralf,
the test finally completes after 40 seconds ... In any case, it is not a problem for me, as I have JavaScript activated anyway.
I'm not sure why you're mentioning JavaScript; I too have it enabled. The 40 seconds is not due to a lack of JavaScript, but the assumption of a faster PC. All four cores here go full pelt, the balance being 80% userspace, ~15% kernel, and ~5% idle. Fan noise increases. -- Cheers, Ralph.
On 26/04/2025 12:09, Ralph Corderoy wrote:
Hi Ralf,
the test finally completes after 40 seconds ... In any case, it is not a problem for me, as I have JavaScript activated anyway.
I'm not sure why you're mentioning JavaScript; I too have it enabled. The 40 seconds is not due to a lack of JavaScript, but the assumption of a faster PC. All four cores here go full pelt, the balance being 80% userspace, ~15% kernel, and ~5% idle. Fan noise increases.
Providing system details then quite useful like CPU model/memory. Also note that once the challenge succeeded you won't get it for another week unless you remove cookies.
On Sat, 2025-04-26 at 11:09 +0100, Ralph Corderoy wrote:
I'm not sure why you're mentioning JavaScript; I too have it enabled. The 40 seconds is not due to a lack of JavaScript, but the assumption of a faster PC. All four cores here go full pelt, the balance being 80% userspace, ~15% kernel, and ~5% idle. Fan noise increases.
Hi Ralph, some users disable JavaScript. The posted screenshot say, that without JavaScript, the check won't work at all, see https://0x0.st/8VRo.png . FWIW I'm using a model 6.191.5 "13th Gen Intel(R) Core(TM) i3-13100" 4-core CPU, 32 GB RAM, slow copper cable DSL. $ pacman -Q linux; cat /proc/cmdline; pacman -Q waterfox-bin linux 6.14.3.arch1-1 BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/disk/by-label/m1.archlinux ro ipv6.disable=1 kvm.enable_virt_at_load=0 zswap.enabled=0 transparent_hugepage=never waterfox-bin 6.5.6-3 I allow cookies, but delete them again. Regards, Ralf
On Sat, 26 Apr 2025 at 12:07, Ralf Mardorf <ralf-mardorf@riseup.net> wrote:
On Sat, 2025-04-26 at 11:09 +0100, Ralph Corderoy wrote:
I'm not sure why you're mentioning JavaScript; I too have it enabled. The 40 seconds is not due to a lack of JavaScript, but the assumption of a faster PC. All four cores here go full pelt, the balance being 80% userspace, ~15% kernel, and ~5% idle. Fan noise increases.
Hi Ralph,
some users disable JavaScript. The posted screenshot say, that without JavaScript, the check won't work at all, see https://0x0.st/8VRo.png .
does this then break it for people with screen readers, eg? or other adaptive technology? implications with ADA and similar?
On 4/26/25 14:01, Andy Pieters wrote:
does this then break it for people with screen readers, eg? or other adaptive technology?
implications with ADA and similar?
Shouldn't be that different from the checks by reCAPTCHA or Cloudflare. There is landing page (which will add some delay), but user will end up on the actual page
Visiting the Arch Linux wiki, my browser achieves 2.8 kH/s at difficulty 4, and the test finally completes after 40 seconds. This matches other Anubis-wrapped sites.
Some anti-malware features may break Anubis or reduce performance. This is because it’s indistinguishable from mining malware. In the details it states: “Please note that Anubis requires the use of modern JavaScript features that plugins like JShelter will disable. Please disable JShelter or other such plugins for this domain.” Normally I find such demands offensive and a security anti-pattern. But here we have transparency from all involved, all reacting to a known and widely recognized emergency situation, developed and deployed with the intention of saving us from more abusive solutions, and with a clear declaration of trying to address this issue. If it’s not an add-on, I have no idea. Even on my E5300 the task finishes within seconds and the worst I see is 6 kH/s.
If it's meant to adjust according to the speed of the user's PC, then it doesn't do too well.
That would defeat the purpose. The adversary could just feign being a slow machine. This is basically a denial-of-service counter-attack, tuned against targets which rely on being able to process a lot of information cheaply and finish each request within a timeout. Anubis hits both conditions with one stone.
On 4/26/25 1:59 AM, mpan wrote:
On access of https://wiki.archlinux.org today, a dialog flashed (…)
Hello,
Many sites are currently falling victim to crawlers from LLM companies. The bots mercilessly hammer the servers, causing enormous load. This leads to service disruption to normal users. Since many of them own hundreds of IP addresses, it’s not possible to throttle them based on requests alone.
Multiple targets, Arch Wiki included, decided to deploy a client-side proof-of-work mechanism: Anubis from Techaro.⁽¹⁾⁽²⁾ This runs a small program, that does “expensive” operation in the browser. It’s nothing for normal users, even if we count all users together. But it’s a notable cost to companies doing billions requests.
Cheers ____ ⁽¹⁾ https://techaro.lol/ ⁽²⁾ Screenshot of what you didn’t se: https://0x0.st/8VRo.png
Yes, The world has been living through the record breaking 1.33 Million host botnet attack during the past month, and especially the last week as a repurposed brute-force campaign (my fail2ban had over 330 IPs in the dovecot jail - normal is about 15, with 95% of the new compromised IPs coming from LATNIC). This appears to be a continuation of the March record breaking botnet DDOS campaign "Eleven11bot". Things are likely only to get worse with the prevalence of AI aiding the bad guys. I recently saw the current internet described as a "foaming-boiling septic tank" which seems to be an apt description. -- David C. Rankin, J.D.,P.E.
The world has been living through the record breaking 1.33 Million host botnet attack during the past month, and especially the last week as a repurposed brute-force campaign (my fail2ban had over 330 IPs in the dovecot jail - normal is about 15, with 95% of the new compromised IPs coming from LATNIC).
This appears to be a continuation of the March record breaking botnet DDOS campaign "Eleven11bot". David’s message appeared as a reply to mine and appears to suggest we’re in agreement. But I don’t know, what he’s on about.
Anubis doesn’t protect against DDoS attacks. Neither it is a response to a botnet, “AI aiding the bad guys,” or any other attack. It deals with crawlers. Call them negligence, ignorance, or indifference. But certainly not purposeful disruption.
participants (8)
-
Andy Pieters
-
David C Rankin
-
Jelle van der Waa
-
Matej Dujava
-
mpan
-
Qtra Gabriex
-
Ralf Mardorf
-
Ralph Corderoy