[Lurker-users] [andy@strugglers.net: Bug#574568: lurker: "zap" link to delete email from archive is javascript GET, causes bad interactions with crawlers]

[Jonas M. <jo...@fr...>]

hey,

got following bugreport recently against lurker in the debian bts.

greetings,
 jonas

----- Forwarded message from Andy Smith <an...@st...> -----

Date: Fri, 19 Mar 2010 03:16:06 +0000
From: Andy Smith <an...@st...>
To: Debian Bug Tracking System <su...@bu...>
Subject: Bug#574568: lurker: "zap" link to delete email from archive is javascript GET, causes bad interactions with crawlers
Reply-To: Andy Smith <an...@st...>, 57...@bu...

Package: lurker
Version: 2.1-13
Severity: minor

By default, lurker includes a javascript image link on each email that
looks something like this:

javascript:trash('http://lists.example.com/lurker/zap/20100318.113155.0e0de092.en.html');

Because this link just appears to a non-human client like any other page
link, crawlers such as googlebot will attempt to follow it.

At the very least this causes log spam of "Password:" prompt and then
failed password notification, and of course the wasted bandwidth of
having the bots follow all these links, which appear on every page of
the archive.

It may be possible to keep bots out with a suitable robots.txt, e.g.:

User-agent: *
Disallow: 
Crawl-delay: 5
Disallow: /lurker/zap/

However, this seems to have limited effect even against googlebot.

In general using simple GET URLs for things which have an action (i.e.,
deleting an email from the archive) is bad form. This should really be
done as a proper form via POST, then bots would ignore it.

It might also be nice to document a simple way to remove the link
entirely. This looks promising:

http://www.terpstra.ca/lurker/message/20060423.233328.bd5efdb8.en.html

Cheers,
Andy

-- System Information:
Debian Release: 5.0.4
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-xen-686 (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages lurker depends on:
ii  adduser                3.110             add and remove users and groups
ii  apache2                2.2.9-10+lenny6   Apache HTTP Server metapackage
ii  apache2-mpm-prefork [h 2.2.9-10+lenny6   Apache HTTP Server - traditional n
ii  debconf [debconf-2.0]  1.5.24            Debian configuration management sy
ii  libc6                  2.7-18lenny2      GNU C Library: Shared libraries
ii  libgcc1                1:4.3.2-1.1       GCC support library
ii  libmimelib1c2a         4:3.5.9-5         KDE mime library
ii  libstdc++6             4.3.2-1.1         The GNU Standard C++ Library v3
ii  passwd                 1:4.1.1-6+lenny1  change and administer password and
ii  ucf                    3.0016            Update Configuration File: preserv
ii  xsltproc               1.1.24-2          XSLT command line processor
ii  zlib1g                 1:1.2.3.3.dfsg-12 compression library - runtime

lurker recommends no packages.

Versions of packages lurker suggests:
ii  gnupg                     1.4.9-3+lenny1 GNU privacy guard - a free PGP rep
ii  mailman                   1:2.1.11-11    Powerful, web-based mailing list m

-- debconf information excluded

-- 
http://bitfolk.com/ -- No-nonsense VPS hosting

"It is I, Simon Quinlank.  The chief conductor on the bus that is called
 hobby." -- Simon Quinlank



----- End forwarded message -----