Testing and tracking down Warnings in Django

Warnings are often suppressed through many layers of control mechanisms in Python and Django. However, you should really be aware of these and clean them up once in a while!

In other cases, the Warnings reflect an inconsistent state, such as if you are mixing naive and timezone-aware variables. It can be a real pain to track down RuntimeWarning because it doesn’t leave a nice stack trace. The scenario could be that you have assigned a naive DateTime to a field, and the db layer is complaining at runtime.

However, you can run python in a mode where Warning-types raise exceptions with full stack traces. Use the following syntax:

python -W error:"":RuntimeWarning:django.db.models.fields:0 manage.py runserver
  • ‘error’ means that python should raise an exception – this is what we want!
  • ‘RuntimeWarning’ means to trigger when such is raised – you can also simply put ‘Warning’ to trigger for all Warning types.
  • ‘django.db.models.fields’ means activate for this module (you need the full path). Notice that simply putting ‘django’ does *not* activate for all of django, you need to put the specific module or simply ‘”"‘ to enable for all of Python (which might render some interesting Warning goodies that you have never seen!).
  • ’0′ means any line (you probably don’t need)

You can read more in Python Documentation chapter: Warning Control.

Posted in django, Python | Leave a comment

Django: Log and block brute force attempts

Here is a very simple mechanism for wrapping a decorator around your views to protect them against brute force attempts. For instance, if you have a secret file download available only with the right secret (/view/id/secret-hash/), you expose your view to simple brute force attempts.

Simply put, this decorator will log a 404 response object or Http404 exception, count pr. IP and return status=400 and send you an email whenever a new block is put into place.

Model

class IllegalLookup(LogModifications):
    """Log and block illegal lookups"""
    created = models.DateTimeField(
        verbose_name=_(u'created'),
        auto_now_add = True,
    )
    modified = models.DateTimeField(
        verbose_name=_(u'created'),
        auto_now = True,
    )
    ip_address = models.CharField(
        max_length=16,
        null=True,
        blank=True,
        verbose_name=_(u'IP address'),
    )
    path = models.CharField(
        max_length=255,
        null=True,
        blank=True,
        verbose_name=_(u'path'),
        help_text=_(u'First attempted path is always logged'),
    )
    count = models.PositiveIntegerField(
        default=1,
    )
 
    @classmethod
    def log_lookup(cls, ip_address, path):
        try:
            now = timezone.now()
            expired = now - timedelta(minutes=settings.BLOCK_EXPIRY)
            lookup = cls.objects.get(ip_address=ip_address,
                modified__gte=expired)
            lookup.count += 1
            lookup.save()
        except cls.DoesNotExist:
            # Delete old entries first
            cls.objects.filter(ip_address=ip_address).delete()
            lookup = cls.objects.create(ip_address=ip_address,
                path=path)
            lookup.save()
 
    @classmethod
    def is_blocked(cls, ip_address):
        try:
            now = timezone.now()
            expired = now - timedelta(minutes=settings.BLOCK_EXPIRY)
            lookup = cls.objects.get(ip_address=ip_address,
                modified__gte=expired)
            if lookup.count == settings.BLOCK_ATTEMPTS:
                mail_admins("IP blocked", "{0} is now blocked, IllegalLookup id: {1}".format(ip_address, lookup.id))
            if lookup.count > settings.BLOCK_ATTEMPTS:
                return True
        except cls.DoesNotExist:
            pass
        return False

Decorator

def log_and_block(func):
 
    def _log_and_block(request, *args, **kwargs):
        remote_ip = request.META.get('REMOTE_ADDR', None)
        if IllegalLookup.is_blocked(remote_ip):
            return HttpResponse('%s is blocked' % remote_ip,
                status=400)
 
        is_404 = False
        is_exception = False
        try:
            return_object = func(request, *args, **kwargs)
            if return_object.status_code == 404:
                is_404 = True
        except Http404:
            is_404 = True
            is_exception = True
 
        if is_404:
            if remote_ip:
                IllegalLookup.log_lookup(remote_ip,
                    request.META.get('PATH_INFO', ''))
 
            if is_exception:
                raise
 
        return return_object
 
    return _log_and_block

Usage

Now, simply wrap the decorator around your view:

@log_and_block
def my_view(request, id, secret_hash):
    object = get_object_or_404(models.MyModel, id=id, secret_hash=secret_hash)
Posted in django | Leave a comment

Share an internet connection: A nice little script for quickly getting the task done.

This script uses iptable forwarding and dnsmasq to share an internet connection with full relay of remote DNS servers and a local DHCP server. Before trying the script, here is the over all steps:

  • You connect to the internet in your normal fashion. For instance with a 3G dongle and your network manager applet.
  • Make sure that nothing is running on port 53, run netstat -tlnp to debug
  • READ THE SCRIPT before starting and know which interfaces you are using. EXTERNAL is the one connected to the internet and INTERNAL is the one you are sharing the connection via.
  • Do not let network-manager mange the INTERNAL interface!
  • You might have to adjust the script, especially the iwconfig part, as different interfaces may have different ways of configuring the WEP key
#!/bin/bash
 
if [ ! `whoami` = "root" ]
then
  echo "Only root"
  exit
fi
 
DHCP='yes'
EXTERNAL=ppp0
INTERNAL=wlan0
INTERNAL_HOST_IP='192.168.10.1'
SHARE_TO_WLAN='yes'
WEPKEY=abe1234567
SSID='mynetwork'
DNSMASQ_CONFIG='dhcp-range=192.168.10.20,192.168.10.255,12h'
DNSMASQ_CONFIG_FILE='/etc/dnsmasq.d/shareconnection'
 
if [[ $DHCP -eq 'yes' ]]
then
# DHCP SERVER (enable below lines)
	apt-get install dnsmasq
	echo $DNSMASQ_CONFIG > $DNSMASQ_CONFIG_FILE
        echo "listen-address=192.168.10.1" >> $DNSMASQ_CONFIG_FILE
        echo interface=$INTERNAL >> $DNSMASQ_CONFIG_FILE
	/etc/init.d/dnsmasq stop
fi
 
echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o $EXTERNAL -j MASQUERADE
/sbin/iptables -A FORWARD -i $EXTERNAL -o $INTERNAL -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i $INTERNAL -o $EXTERNAL -j ACCEPT
 
ifconfig $INTERNAL down
ifconfig $INTERNAL $INTERNAL_HOST_IP
iwconfig wlan0 mode ad-hoc
iwconfig wlan0 essid $SSID
iwconfig wlan0 key $WEPKEY
ifconfig $INTERNAL up
 
if [[ $DHCP -eq 'yes' ]]
then
	/etc/init.d/dnsmasq start
fi
Posted in Computers, Ubuntu, Web | Leave a comment

To Save Everything, Click Here: The Folly of Technological Solutionism

There is a new book coming up with a very remarkable abstract, to be released on March 5, 2013:

In the very near future, “smart” technologies and “big data” will allow us to make large-scale and sophisticated interventions in politics, culture, and everyday life. Technology will allow us to solve problems in highly original ways and create new incentives to get more people to do the right thing. But how will such “solutionism” affect our society, once deeply political, moral, and irresolvable dilemmas are recast as uncontroversial and easily manageable matters of technological efficiency? What if some such problems are simply vices in disguise? What if some friction in communication is productive and some hypocrisy in politics necessary? The temptation of the digital age is to fix everything—from crime to corruption to pollution to obesity—by digitally quantifying, tracking, or gamifying behavior. But when we change the motivations for our moral, ethical, and civic behavior we may also change the very nature of that behavior. Technology, Evgeny Morozov proposes, can be a force for improvement—but only if we keep solutionism in check and learn to appreciate the imperfections of liberal democracy. Some of those imperfections are not accidental but by design.

Arguing that we badly need a new, post-Internet way to debate the moral consequences of digital technologies, To Save Everything, Click Here warns against a world of seamless efficiency, where everyone is forced to wear Silicon Valley’s digital straitjacket.

New Republic has a wonderful chapter from it, Why Social Movements Should Ignore Social Media.

Posted in Debate, Web | 8 Comments

First impressions of GTK3 migration in Python

Yesterday, I succeeded in moving a GTK2 project* with a custom Cairo widget to GTK3. It’s about 750 lines of Python and took ~5 hours to migrate, mainly because my method was about googling and solving all Exceptions one by one. That’s not very advisable.

You can read a more thorough introduction to GTK3 and Python here -> python-gtk-3-tutorial.rtfd.org

The first thing, you need to know, is that the Python GTK3 bindings are

Step 1:

GTK3/Glib uses different libraries from GTK2, and those libraries conflict with the old ones. So the first part of a migration is to replace ALL occurrences of those imports in your whole project.

Examples of old stuff:

import gtk
import gobject
import gtk.gdk
import cairo
import pangocairo
 
gtk.gdk.threads_init()
 
# The below was used on systems with both GTK-1 and GTK-2 and is obsolete.
# pygtk uses static gobject inheritance different from GTK3.
# Short version: In GTK3, don't do this!
import pygtk
pygtk.require("2.0")

The example above is now:

from gi.repository import Gtk
from gi.repository import GObject
from gi.repository Gdk
from gi.repository import cairo
from gi.repository import PangoCairo
 
Gdk.threads_init()

Step 2:

There are a bunch of changes in the structure of the library. Most constants like gtk.CONSTANT_NAME have changed to something like Gtk.SubLibrary.NAME. For instance, gtk.gdk.LEAVE_NOTIFY_MASK is now Gdk.EventMask.LEAVE_NOTIFY_MASK; gtk.STATE_NORMAL is Gtk.StateFlags.NORMAL; and gtk.EXPAND is Gtk.AttachOptions.EXPAND.

So basically, have a look at everything that’s capilized. The following script can help you:

Step 3:

But last thing is that the library has a bunch of changes in the API. Classes, methods, properties, arguments and return values have all changed. So expect the unexpected!

In this regard, you might also find that your project is too complicated for 100% manual conversion. In that case, have a look at:

Examples and tutorials

The following examples are available as of now:

* The project is a fullscreen plugin for Rhythmbox.

Posted in Python, Uncategorized | Tagged , , , | Leave a comment