Notes on Django caching

I’ve spent the last few weeks playing around with django-page-cms. It’s going well so far. But I am having an issue with some customisations I made and the caching that is used. So here my notes so far on the subject. These are really for my own reference, but I thought others might find them helpful. The Django Project documentation describes the caching in detail as well.

Activating caching

You can turn on caching by using CACHE_BACKEND in the settings file.

Types of caching

There are five types of caching that come with Django:

1. Memcached

This uses the Memchached caching framework and is entirely memory based. Memcached can be downloaded from here: http://danga.com/memcached/. Memchached needs to be installed separately.

2. Database

This uses your database for caching. You need to create a table in your database using this command:

python manage.py createcachetable [cache_table_name]

and then use this setting:

CACHE_BACKEND = 'db://my_cache_table'

3. Filesystem chaching

This uses the file system for caching. You can set it up using the following setting:

CACHE_BACKEND = 'file:///var/tmp/django_cache'

Note: you need the three forward slashes after file and your webserver needs read/write access to the directory listed.

4. Local-memory caching

This provides the speed of memory caching if you don’t have the capability for running Memcached. It is multi-process and thread safe. However since each process therefore has it’s own chache, it is not bery memory efficient and best used in development environement only. Set it up like this:

CACHE_BACKEND = 'locmem:///'

5. Dummy caching

Used for development purposes to simulate caching, but without actually using a cache. Set it up like this:

CACHE_BACKEND = 'dummy:///'

You can also create your own custom chache mechanism if you wish.

CACHE_BACKEND arguments

There are three arguments:

1. timeout: how long to chache the info for
2. max_entries: maximum number of items to cache
3. cull_percentage: what percentage to cull when max_entries is reached. eg: 3 means 1/3

Choosing what to cache

Caching a whole site

This can be done with middleware. See the docs for more info.

The per-view cach

This lets you cache the output of specific views. This is easy to setup:

@cache_page(60 * 15)
from django.views.decorators.cache import cache_page

def slashdot_this(request):

Template fragment caching

This lets you cache blocks within a template. You can do this using the following code:

{% load cache %}
{% cache 500 sidebar request.user.username %}
    .. sidebar for logged in user ..
{% endcache %}

{% load %} is the Django mechanism for loading custom tag libraries/
{% cache 500 sidebar request.user.username%} .. {% endcache %} defines the block to be cached. 500 is the length of the cache in seconds and any following items make up the key for the cache.

API caching

This lets you cache items programmatically. The core functions are:

cache.set('my_key', 'hello, world!', 30)
cache.get('my_key')

where my_key is the key for the item in the cache, ‘hello world!’ is the content being cached and 30 is the number of seconds to cache it for.

Further information on caching

You can also vary your cache depending on the request headers or use the cache_control decorator to control how private data is handled, whether to check for new versions and how many times a page can be chached.

Creating a view for the menu

I now have a very basic menu in my dummy application. Now I want to display it to the end user, which is what the 3rd part of the Django tutorial covers.  The tutorial starts off with desiging the URLs that the end user will see. Well for the end user what I am creating is essentially a website. So my URLs will be something like:

http://www.example.com/node1/node1.1

etc. Except that node1 and node1.1 will be meaningful titles such as “About us” and “Staff”.  These could be structured to contain the following sort of URL:

http://www.example.com/about-us/staff

The system should then bring back the menu expanded to the correct sub menu and with the correct menu item selected.  Thus my URLs are quite straightforward in one sense and I will only need one regular expression pattern for now (essentially matching everything after the domain name). But I also need to figure out how to then split the path into it’s component sub-sections. Ideally I would like to end up with a urlpattern that results in a call which looks something like this:

view_menu(request=<HttpRequest object>, subsection1='about-us', subsection2='staff', subsection3=...)

And that there is no limit to the depth of the tree structure. The easier option is to just do this:

view_menu(request=<HttpRequest object>, path='about-us/staff/')

and then work out the subsection details in the view_menu function. I also think that if I store the path with the node in the database then this would be much more efficient (although much more complex to keep up to date).  So going with the latter idea, my url pattern will be (essentially matching anything for now!):

(r'^(?P.*)/$', 'contentedweb.cwcms.views.menu')

Now I need to code up the corresponding view for this URL pattern. But in order to keep the code in my public facing views clean and nimble (as these are the ones that are going to be called most often) I need to make a change to the node model to include the path to the node:

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)
    path = models.CharField(max_length=1000)

    def __unicode__(self):
        return self.title

So I have added the path field to the model. For now I will make do with having to populate this field manually, but further down the line this field will be populated and maintained by the model not by the user creating the menu.

Next I need to bring back the relevant part of the menu from the database and display it in HTML indicating which menu item is selected. So I created a basic menu in the admin site and came up with this view (well essentially it’s copied from the tutorial…):

from django.shortcuts import render_to_response
from contentedweb.cwcms.models import Menu, Node

def menu(request, path):
    main_menu = Menu.objects.get(id=1)
    main_menu_items = Node.objects.filter(menu=1, parent=None)

    return render_to_response('cwcms/menu.html', {'main_menu': main_menu,'main_menu_items': main_menu_items,'path': path})

and this template:

<html>
<body>
{% if main_menu %}
    <h1>{{ main_menu.title }}</h1>
    <ul>
    {% for node in main_menu_items %}
        <li{% ifequal node.path path %} class="selected"{% endifequal %}><a href="/{{ node.path }}">{{ node.title }}</a></li>
    {% endfor %}
    </ul>
{% else %}
    <p>Menu could not be found</p>
{% endif %}

</body>
</html>

Creating this, admittedly very basic, template was straight forward. There were only two problems I encountered. Firstly, I had tried to use the filter method to return the main menu info:

main_menu = Menu.objects.get(id=1)

But then the template didn’t work because main_menu was not actually a “Menu” object, but a list of “Menu” objects containing one element in the list. Secondly I was trying to use something like this in the templating system:

{% if node.path == path %}

Which threw an error because the templating system uses slightly different control flow statements. The correct form for the above code is:

{% ifequals node.path path %}

I am not sure why the templating system should use a different set of statements in this respect, but anyway, the documentation quickly came to the rescue.

Now I need to cope for the case that the path does not exist and raise a 404 error. So I modified the view to include a try-except construct:

    try:
        main_menu = Menu.objects.get(id=1)
        main_menu_items = Node.objects.filter(menu=1, parent=None)
        node = get_object_or_404(Node, path=url_path)
    except Node.DoesNotExist:
        raise Http404

Which basically checks that the path being looked for actually exists in the database. Adding in the 404 template is also straight forward. All you need to do is create a 404.html template in the root of the templates folder and Django will handle the rest.

Well I now have a very basic menu system in place. Obviously there is lots to do but right now I am concentrating on working myself through the tutorials. Once I have covered all 4 I will start expanding on my little app.

Modifying the default Django admin site

Those of you following my posts, anybody?, know that I am experimenting with creating my own little CMS using Django. I am starting by following the Django tutorials. I got through the first 1 and 3/4 tutorials before needing to do some research into how to get my recursive relationship key working in the admin site. It turns out I was a little naive regarding how the model system works in Django. Anyway, I finally got this sorted out and now I am back to continuing with the last part of the second tutorial.

The initial aim of my app is to be able to create multiple menu structures. So far I have two models, a menu model:

class Menu(models.Model):
    title = models.CharField(max_length=100)
    description = models.CharField(max_length=1000)
    in_use = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

    def is_in_use(self):
        return self.in_use == True

and a node model (which might turn into a “page” model in the future):

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

This has given me the basic admin site that you get from using django.contrib.admin. Customising the admin site is also straightforward. By adding a few lines to the Menu class in my admin.py file I added filtering, searching and the ability to generate multiple nodes whilst adding a menu.

For filtering:

list_filter = ['in_use']

For searching:

search_fields = ['title']

And for multiple nodes I created this class:

class NodeInline(admin.TabularInline):
    model = Node
    extra = 5

And then appended this line to my MenuAmdin class:

inlines = [NodeInline]

Finally I also modified the data brought back in my list of menus:

list_display = ('title', 'in_use')

I also copied the base_site.html file into my own templates directory and modified it to display my own title in the admin system. I will be playing around with the ability to customise the backend more in the future as this is one the things I really like about Django. It’s so easy to modify the backend.

So far I have written maybe 50 lines of “code” (well really it’s more like templating instructions) and have a fully functioning admin site. This is great, but also a little worrying as I am sure I will hit a major road block in the future with my limited Python knowledge. Still, so far so good.

Recursive relationships in Django

In my previous post I stated how I had created a recursive relationship in one mof my models. The code looks something like this:


parent = models.ForeignKey('self', blank=True, null=True)

This works great, but the problem I now faced was how to enter the root element. Even though I had sent blank and null to True I was not able to enter my root node using the default admin interface. So this is the first issue I need to solve outside of the tutorial documentation.

Ok, after some searching around it is clear that I need to update the database schema to reflect the model change. Obviously this won’t happen automatically. So I thought I could use the syncdb statement that was used to create the schema in the first place:


python manage.py syncdb

But, as it clearly says in the documentation, syncdb only creates new tables. It does not modify existing tables. The solution seems to be to run the following the statement:

python manage.py sql APP_TITLE


To see what Table schema your model is expecting and to then make the change manually in your database. Not ideal, but manageable. They are working on ways to evolve the schema via Django, but that will be for a future version I guess.

This is the first disapointing thing about Django I’ve encountered so far. The idea seems to be that you get your schema right the first time, but this seems be unrealistic to me. Especially in an agile development environment where you are likely to make small changes at a time.

My first Django app

Today I followed the first two pages of the Django tutorial with one small difference. Instead of recreating the polling app I decided to start right away on my Contented Web app (CW).

Creating the Django project and my app was straight forward – although I must admit I don’t understand the technical nuances of everything that I am following. I think partially this is due to the Unix environment I am using.

The models I created are essentialy enough for creating a hierarchical menu structure. This seemed straightforward and I quickly got the hang of how to define and alter my models, including a recursive relationship.

What blew me away though was the admin site that was created and how easily I could modify the look and feel of the forms through the use of python tuples and lists. Pure class. As far as I can tell the whole admin site can be remodelled without affecting the functionality. This is great. One of the annoying things about most CMS tools is that whilst they enable you to create great looking websites, the admin interface always seems to be forgotten. And there seems to be little thought given to making it customisable.

So after about 3 hourse of playing around with Django and following the first two tutorials I now have a fully functioning admin interface for my models, including user management and access rights.