Please take not that it will evaluate EVERY SINGLE REQUEST through a Deny policy in Apache’s mod_access – it does not halt your performance much if you’re just running a site with few requests. But I only suggest this solution as a temporary one until you have a better integration. I am currently working on a Django middleware that will only check the client’s IP address in case it is a POST request directed at a certain URL.

Since I’ve been told that Apache doesn’t use /etc/hosts.deny, you should probably use this method instead, unless you have a webserver that actually uses hosts.deny. Another method would be to add the IP addresses to iptables, if you’re running that — this is probably more efficient than using Apache configuration policies.

This method creates a file that you can include in httpd.conf (to cover ALL your virtual hosts). The overall goal is to avoid .htaccess files since they are evaluated at runtime. Thus, you won’t have to configure each site, and you save a but of CPU time.

1. Copy the script below to a file on your server, eg. /usr/sbin/stopforumspam.py. You might want to read it quickly as it will pretty much explain itself…
2. Make the file executable, eg. chmod +x /usr/sbin/stopforumspam.py
3. Add it to your crontab for automatic execution each night, eg. crontab -e and then insert the line 0 0 * * * /usr/sbin/stopforumspam.py && /etc/init.d/apache2 reload
4. By default, we will ban an entire class C subnet if more than 5 IP addresses are in this space. You can configure this behavior by giving stopforumspam.py a single argument, ie. /usr/sbin/stopforumspam.py 10 would mean that at least 10 IPs have to be within the class C subnet to qualify it for a ban.
5. By default stopforumspam.py creates /etc/apache2/stopforumspam.conf – you can change this by editing the script.
6. Add Include /etc/apache2/stopforumspam.conf to /etc/apache2/httpd.conf (Debian/Ubuntu).
7. Caution! Make sure that you DO NOT change the Order Deny,Allow option in VirtualHost directives or .htaccess files as this unblock the deny policies.

#!/usr/bin/python
 
import re
import urllib
import zipfile
import sys
 
if len(sys.argv) > 1:
    SUBNET_THRESHOLD = int(sys.argv[1])
else:
    SUBNET_THRESHOLD = 5
 
DOWNLOAD_ZIP = "http://www.stopforumspam.com/downloads/listed_ip_7.zip"
ZIP_FILENAME = "listed_ip_7.txt"
 
HTTPD_CONFIG_INCLUDE = "/etc/apache2/stopforumspam.conf"
 
# For security purposes we test that each line is actually an IP address
IP_MATCH = re.compile(r"^(\d+)\.(\d+)\.(\d+)\.(\d+)$")
 
filename, headers = urllib.urlretrieve(DOWNLOAD_ZIP)
 
z = zipfile.ZipFile(filename)
ips = z.read(ZIP_FILENAME)
 
ips = ips.split("\n")
 
# Remove non-ip members
ips = filter(lambda ip: IP_MATCH.match(ip), ips)
 
def get_ip_segments(ip):
    segments_match = IP_MATCH.search(ip)
    return [int(segments_match.group(i)) for i in range(1,5)]
 
def convert_ip_to_number(ip):
    numeric_value = 0
    ip_segs = get_ip_segments(ip)
    for i in range(4):
      numeric_value += ip_segs[i]*(255**(4-i))
    return numeric_value
 
# Sort everything first
ips.sort(key=convert_ip_to_number)
 
subnets = {}
 
for ip in ips:
    ip_segs = get_ip_segments(ip)
    key = (ip_segs[0], ip_segs[1], ip_segs[2])
    if not key in subnets.keys():
        subnets[key] = [ip]
    else:
        subnets[key].append(ip)
 
final_list = []
 
for subnet, subnet_ips in subnets.items():
    if len(subnet_ips) > SUBNET_THRESHOLD:
        # Ban the whole subnet
        final_list.append(".".join(map(str, subnet)) + ".0/24")
    else:
        final_list = final_list + subnet_ips
 
print ""
print "Lengh of original list: %d" % len(ips)
print "Lengh of final list: %d" % len(final_list)
 
apache_conf_file = file(HTTPD_CONFIG_INCLUDE, "w")
apache_conf_file.write("\n")
apache_conf_file.write("    Order Deny,Allow\n")
 
for entry in final_list:
    apache_conf_file.write("    Deny from %s\n" % entry)
 
apache_conf_file.write("\n")
 
apache_conf_file.close()

Use the following Python script to maintain a file in the hosts.deny syntax so that your Debian/Ubuntu box (or other Linux server) is kept undisturbed by forum spammers — COMPLETELY undisturbed, as the spammer is disallowed all access to the system.

1. Copy the script to a file on your server, eg. /usr/sbin/stopforumspam.py
2. Make the file executable, eg. chmod +x /usr/sbin/stopforumspam.py
3. Add it to your crontab for automatic execution each night, eg. crontab -e and then insert the line 0 0 * * * /usr/sbin/stopforumspam.py
4. Add the path to the file containing the ip list to /etc/hosts.deny by inserting this line: ALL: /etc/hosts.deny.stopforumspam

The script itself is here, modify it as you need.

#!/usr/bin/python
 
import re
import urllib
import zipfile
 
DOWNLOAD_ZIP = "http://www.stopforumspam.com/downloads/listed_ip_7.zip"
ZIP_FILENAME = "listed_ip_7.txt"
 
HOSTS_DENY = "hosts.deny.stopforumspam"
 
# For security purposes we test that each line is actually an IP address
IP_MATCH = re.compile(r"\d+\.\d+\.\d+\.\d+")
 
filename, headers = urllib.urlretrieve(DOWNLOAD_ZIP)
 
z = zipfile.ZipFile(filename)
ips = z.read(ZIP_FILENAME)
 
deny_file = file(HOSTS_DENY, "w")
 
for ip in ips.split("\n"):
    if IP_MATCH.match(ip):
        deny_file.write("%s\n" % ip)
 
deny_file.close()

Warning: g_object_ref: assertion `G_IS_OBJECT (object)' failed
  gtk.main()
Warning: instance of invalid non-instantiatable type `AtkSelection'
  gtk.main()
Warning: g_signal_emit_valist: assertion `G_TYPE_CHECK_INSTANCE (instance)' failed
  gtk.main()
(...)
Warning: instance of invalid non-instantiatable type `'
  gtk.main()

In most versions of GTK window decorators, this caused a crash – in a few lucky cases, it didn’t! My (wrong) assumption was that calling connect_signals was only of interest, if I actually had signal handlers to connect. This is also the case when looking at the docs. For instance:

The function connect_signals and variants thereof can MUST be used to connect handlers to the named signals in the description.

(…)

For each of handlers that cannot be found, a RuntimeWarning is issued.

However, when the warning was issued, GTK would crash anyways. This behavior was most like caused by dereferencing and deleting GTK widgets, which caused GTK to look for an on_delete_event handler as some sort of default behavior. Since I have a program that adds and removes GTK widgets during runtime, my program needed to handle these signals, even if the event handler would simply be empty. To make a long story short, you should ALWAYS make sure to call connect_signals no matter what, as it will connect core event handlers that you may not be aware of. This is how it should look:

class YourObject():
    def __init__(self):
        glade = add_from_file("x.glade")
        win = glade.get_object("window")
        win.show_all()
        # ALWAYS DO THIS:
        glade.connect_signals(self)
 
    def on_delete_event(self, *args):
        pass

And even if the above may not fail, here is something that will potentially fail if you don’t connect the signals. Please beware, that using the same on_delete_event handler for two different objects is a bad strategy.

Always take care that widgets are removed and deleted, and that delete events may be invoked at very separate places in your application. You should also be aware that the python garbage collector could be the one initiating these events, causing those strange c library reference errors.

I’ve often been frustrated that using settings.DATE_FORMAT does not give a localized date. Granted that the name of a month may be localized, but the format string does not change. So let’s start out by modifying settings.py. We wrap our default date format in a ugettext so the makemessages command will detect it, and we need to make it a dummy function, because the i18n library cannot be imported in settings.py due to circularity (it depends on settings.py).

ugettext = lambda s: s
DATE_FORMAT = ugettext('N j, Y')

Run compilemessages and type in your localized date formats. Now we need a template filter that uses a localized format for calling the Django date format function. This is really simple:

from django.template.defaultfilters import stringfilter
from django.utils import dateformat
from django.utils.translation import ugettext
from django.conf import settings
 
@register.filter()
def localdate(value):
    """Format date with localized date format"""
    format = ugettext(settings.DATE_FORMAT)
    return dateformat.format(value, format)

And done. Using the filter is straight forward:


Date: {{ my_date|localdate }}


Google Code project page

Demo website

Hierarchy and relations
First of all, as in the Trac wiki system, I chose to create a system for hierarchy, meaning that it’s possible to create an article and then create sub-articles. The hierarchy does not support multiple inheritance, because it needs to be basis for the permission system. That’s where the relation system comes in place: All articles can contain symmetrical relations to any other articles in the hierarchy.

Parsing
Python and Django supports Markdown pretty much out of the box, so it’s an obvious choice to use this for parsing. The HTML features of normal Markdown have been removed, so all HTML is escaped in django-simple-wiki. And parsing is static, so every time a revision is created, the contents are passed and stored. This means that the contents of the article itself are not supposed to be dynamic. On the other hand, it is desirable to avoid parsing contents for every page hit. The parsing area of the application is only a few lines of code, and can be expanded if further parsing needs to be done, or someone wants to replace Markdown completely. For instance, if no parsing is done and HTML escaping is disabled, the wiki becomes a very simple CMS.

Curious issues
There are a few out standing problems:

• Permission system is related to User entries in the Django auth system. But maybe this is too much of an annoyance, if the project already has groups setup in the existing auth system. On the other hand, other users would be bothered to setup both wiki groups and user groups, if the permission system was linked to user groups. And directly linking articles to user groups would require wiki-related groups to be created directly in the auth system.
• Since relations are symmetrical, what should happen if one article is locked, but a user modifies it’s relations by deleting them from related articles?
• Title editing: The title can only be created once, since it is coupled to the ‘slug’ of the article. A user can deliberately create a completely different title, which is fine, but should subsequent editing be allowed, which would add complexity to the revision system?
• Article deletion: When an article is deleted from the backend it shouldn’t worry anyone. But if the feature is added to the frontend, we would want to handle maliciousness etc. But should we really store all these files and revisions? Should we alert admins, so they can do the final cleanup?

I changed the display from a normal fixed table with 3 tracks to a gtk.Layout with a gtk.VBox containing n tracks. In a Layout widget it’s possible to freely place and move child widgets, so by detecting motion events on the edges of the Layout you can emulate scrolling by moving a child widget accordingly with Layout.move(widget, x, y). Unfortunately I get a rather nasty blinking effect when scrolling too fast, and I don’t have an explanation for this, so I’d be glad to hear from anyone who can help.

def track_layout_scroll(self, widget, event):
    time_step = 10 #msecs
    ycoord = event.y
    accel_factor = 10 #how many pixels to scroll at the edge
    edge_distance = 100.0 #pixels
    layout_size = widget.get_size()
    top_dist = edge_distance - ycoord
    bot_dist = edge_distance - layout_size[1] + ycoord
 
    if top_dist > 0:
        accel = -1 - (top_dist / edge_distance) * accel_factor
    elif bot_dist > 0:
        accel =  1 + (bot_dist / edge_distance) * accel_factor
    else:
        accel = 0.0
 
    if self.scroll_event_id:
        gobject.source_remove(self.scroll_event_id)
 
    if not accel == 0.0:
        self.scroll_event_id = gobject.timeout_add(time_step, self.do_scrolling, accel, widget)
 
def do_scrolling(self, accel, layout_widget):
    step = int(1*accel)
    if step == 0:
        return
    vbox_size = self.vbox.size_request()
    layout_size = layout_widget.get_size()
    scroll_height = vbox_size[1]-layout_size[1]
    if self.scroll_y + step < 0:
        self.scroll_y = 0
    elif self.scroll_y + step > scroll_height:
        self.scroll_y = scroll_height
    else:
        self.scroll_y += step
 
    self.track_layout.move(self.vbox, 0, -self.scroll_y)
    return self.scroll_y > 0 and self.scroll_y < scroll_height

The code is related to my Rhythmbox plugin, but I’m sure you get the idea. Also please note, that track_layout_scroll has to receive notify_motion_event from the Layout widget, and that you have to set a size for the Layout widget with set_size().

Update: By playing around with time_step (lowering it to be more exact) and slowing down the acceleration, I managed to almost make the white flashes disappear.