Notes on Django caching

I’ve spent the last few weeks playing around with django-page-cms. It’s going well so far. But I am having an issue with some customisations I made and the caching that is used. So here my notes so far on the subject. These are really for my own reference, but I thought others might find them helpful. The Django Project documentation describes the caching in detail as well.

Activating caching

You can turn on caching by using CACHE_BACKEND in the settings file.

Types of caching

There are five types of caching that come with Django:

1. Memcached

This uses the Memchached caching framework and is entirely memory based. Memcached can be downloaded from here: http://danga.com/memcached/. Memchached needs to be installed separately.

2. Database

This uses your database for caching. You need to create a table in your database using this command:

python manage.py createcachetable [cache_table_name]

and then use this setting:

CACHE_BACKEND = 'db://my_cache_table'

3. Filesystem chaching

This uses the file system for caching. You can set it up using the following setting:

CACHE_BACKEND = 'file:///var/tmp/django_cache'

Note: you need the three forward slashes after file and your webserver needs read/write access to the directory listed.

4. Local-memory caching

This provides the speed of memory caching if you don’t have the capability for running Memcached. It is multi-process and thread safe. However since each process therefore has it’s own chache, it is not bery memory efficient and best used in development environement only. Set it up like this:

CACHE_BACKEND = 'locmem:///'

5. Dummy caching

Used for development purposes to simulate caching, but without actually using a cache. Set it up like this:

CACHE_BACKEND = 'dummy:///'

You can also create your own custom chache mechanism if you wish.

CACHE_BACKEND arguments

There are three arguments:

1. timeout: how long to chache the info for
2. max_entries: maximum number of items to cache
3. cull_percentage: what percentage to cull when max_entries is reached. eg: 3 means 1/3

Choosing what to cache

Caching a whole site

This can be done with middleware. See the docs for more info.

The per-view cach

This lets you cache the output of specific views. This is easy to setup:

@cache_page(60 * 15)
from django.views.decorators.cache import cache_page

def slashdot_this(request):

Template fragment caching

This lets you cache blocks within a template. You can do this using the following code:

{% load cache %}
{% cache 500 sidebar request.user.username %}
    .. sidebar for logged in user ..
{% endcache %}

{% load %} is the Django mechanism for loading custom tag libraries/
{% cache 500 sidebar request.user.username%} .. {% endcache %} defines the block to be cached. 500 is the length of the cache in seconds and any following items make up the key for the cache.

API caching

This lets you cache items programmatically. The core functions are:

cache.set('my_key', 'hello, world!', 30)
cache.get('my_key')

where my_key is the key for the item in the cache, ‘hello world!’ is the content being cached and 30 is the number of seconds to cache it for.

Further information on caching

You can also vary your cache depending on the request headers or use the cache_control decorator to control how private data is handled, whether to check for new versions and how many times a page can be chached.

There’s a Django app for everything!

I haven’t written anything in quite a while as I’ve been too busy hacking out annoying ASP.Net code in my working life.   That doesn’t mean I haven’t been reading up on Django, especially looking into what apps have already been written for Django.  And I am quite amazed. I am sure I can “cobble” together a professional, industrial strength web based CMS through the use various Django apps without really having to learn how to program python, beyond understanding the syntax that is.

My ultimate aim is to create an app that helps people write clear, meaningful, accessible content that is semantically marked up. However I have decided to first just get a basic CMS up and running with which I can manage a couple of my own personal sites.  I am hoping this will give me a deeper understanding of Django. The target audience for my CMS are people wanting to create small websites with maybe upto two dozen pages on them. What I want is the following:

• Basic admin features. Three levels:
  1. Admin: complete control over website
  2. Editor: can add/remove pages and modify content
  3. Author: can only write content
• Standard site administration enbaling the creation and editing of pages
• Complete version history
• Some form of tagging. Not sure exactly how this will be used, but I would like to be able to tag content
• Some kind of multilingual functionality. Not sure yet how this should look, but it is anyway low priority.
• Some way of managing more than one site from the same install

Well, there seems to be a Django app for each of these:

• For the admin features: django.contrib.auth should do the trick.
• For basic page features there were a bunch of options, but this looked the most suitable for my needs:django-page-cms
• For versioning: django-reversion
• For tagging: django-tagging
• For multilingual stuff django-page-cms seems to do the trick. But there is also django-multilingual which looks interesting
• For multiple sites, there is the “sites” framework that ships with Django: django.contrib.sites.models

This post is my “back of the envelope” sketch. There’s lots of interesting tools and apps listed here. Now I need to figure out how to combine them into something meaningful.

Doing my own thing in Django

I have so far followed the first three parts of the Django tutorial. These have been very helpful in getting me started with Django. The fourth part of the tutorial covers form processing and generic views. Having read through the tutorial it is clear to me that neither of these are of immediate interest to me. So for now I am going to go ahead and start doing my own thing.

For now I want to concentrate on making it as easy as possible for a user to create and modify a menu structure (or a taxonomy structure). My one chief gripe with many of the CMSs I’ve experimented with is how difficult it is to create and maintain large menu systems. I don’t claim to have the answers, but I want to experiment a little with this. Rather than having to add in each menu item, one by one I would like to offer users the ability to import them from essentially any kind of format, eg: XML, CSV, YAML or a simple indented text file. For example a menu could be created in an HTML text area using the following structure:

Home
About Us
  Contact Us
  Staff
Products
  Widget 1
    Overview
    Details
  Widget 2
Services
  Service 1
  Service 2

I am hoping that processing something like this is straight forward in Python as the language itself requires correct indentation. You could further enhance this with a +/- qualifier to indicate whether the node is visible or not (a common menu structure requirement) and also some for of indicating what the URL path of the node should be (if it is different to the node title), eg:

Home
About Us
  -Contact Us
  +Staff
Products
  Widget 1 | widget-one
    Overview
    Details
  Widget 2 | widget-two
Services
  Service 1
  Service 2

In the above menu the “Contact Us” node would not be displayed in a menu whereas the “Staff” node and all the other nodes would be. I have added the plus to “Staff” for illustration purposes, but the + can be implied. Also the URL of the node “Widget 1″ would be “http://www.example.org/products/widget-one/”, whereas the one of “Staff” might default to “http://www.example.org/about-us/staff/”.

In order to get going with this, I need to add a couple of fields to the node model:

    position = models.CharField(max_length=1000)
    parents = models.CharaField(max_length=1000)

Both these fields are comma delimited strings. The “position” field stores the overall position of the node in the menu. So for the “Staff” node above, the position field might be “01,01″. By padding to the left with 0s it means I can order the nodes by the position field and get the entire menu back in one query with the correct hierarchy intact. The number of digits has to be consistent for the ordering to work, so the above example allows for up to 100 child nodes for each node. Whilst this is a restriction, if you need more than 100 child nodes in your menu, you should probably consider redesigning the menu structure!

The other field stores all the parent ids of the node. So the parents field of the “Details” page of “Widget 1″ might be something like this: “5,6″ (assuming the nodes were added in the order listed and “Products” therefore has the id 5). You might also want to include the id of the “Details” page itself. It doesn’t matter as long it is consistent.

I have used both these fields in previous work and find them a very efficient way of maintaining info about a hierarchy. They are however quite complex to maintain as you have to update them throughout the menu whenever you move an item. Still, in most scenarios menus are not changed much once a site goes live so it is more important that viewing the menu hierarchy is quick rather than maintaining it.

Using these two fields, there is no real need for the “parent” and “sorting” field. However I like to keep them in as they make certain queries more straightforward. They are also useful as a backup in case something goes wrong with the “parents” and “position” data.

My node model now looks something like this:

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)
    path = models.CharField(max_length=1000)
    position = models.CharField(max_length=1000)
    parents = models.CharField(max_length=1000)

    def __unicode__(self):
        return self.title

The next entry will deal with how to maintain these two extra fields without the end user having to know about them.

If any readers have other ideas on how to maintain a hierarchy I would be very interested in hearing them. I know the above solution is very common but I also believe there are probably more efficient methods of storing hierarchical information in a database.

Creating a view for the menu

I now have a very basic menu in my dummy application. Now I want to display it to the end user, which is what the 3rd part of the Django tutorial covers.  The tutorial starts off with desiging the URLs that the end user will see. Well for the end user what I am creating is essentially a website. So my URLs will be something like:

http://www.example.com/node1/node1.1

etc. Except that node1 and node1.1 will be meaningful titles such as “About us” and “Staff”.  These could be structured to contain the following sort of URL:

http://www.example.com/about-us/staff

The system should then bring back the menu expanded to the correct sub menu and with the correct menu item selected.  Thus my URLs are quite straightforward in one sense and I will only need one regular expression pattern for now (essentially matching everything after the domain name). But I also need to figure out how to then split the path into it’s component sub-sections. Ideally I would like to end up with a urlpattern that results in a call which looks something like this:

view_menu(request=<HttpRequest object>, subsection1='about-us', subsection2='staff', subsection3=...)

And that there is no limit to the depth of the tree structure. The easier option is to just do this:

view_menu(request=<HttpRequest object>, path='about-us/staff/')

and then work out the subsection details in the view_menu function. I also think that if I store the path with the node in the database then this would be much more efficient (although much more complex to keep up to date).  So going with the latter idea, my url pattern will be (essentially matching anything for now!):

(r'^(?P.*)/$', 'contentedweb.cwcms.views.menu')

Now I need to code up the corresponding view for this URL pattern. But in order to keep the code in my public facing views clean and nimble (as these are the ones that are going to be called most often) I need to make a change to the node model to include the path to the node:

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)
    path = models.CharField(max_length=1000)

    def __unicode__(self):
        return self.title

So I have added the path field to the model. For now I will make do with having to populate this field manually, but further down the line this field will be populated and maintained by the model not by the user creating the menu.

Next I need to bring back the relevant part of the menu from the database and display it in HTML indicating which menu item is selected. So I created a basic menu in the admin site and came up with this view (well essentially it’s copied from the tutorial…):

from django.shortcuts import render_to_response
from contentedweb.cwcms.models import Menu, Node

def menu(request, path):
    main_menu = Menu.objects.get(id=1)
    main_menu_items = Node.objects.filter(menu=1, parent=None)

    return render_to_response('cwcms/menu.html', {'main_menu': main_menu,'main_menu_items': main_menu_items,'path': path})

and this template:

<html>
<body>
{% if main_menu %}
    <h1>{{ main_menu.title }}</h1>
    <ul>
    {% for node in main_menu_items %}
        <li{% ifequal node.path path %} class="selected"{% endifequal %}><a href="/{{ node.path }}">{{ node.title }}</a></li>
    {% endfor %}
    </ul>
{% else %}
    <p>Menu could not be found</p>
{% endif %}

</body>
</html>

Creating this, admittedly very basic, template was straight forward. There were only two problems I encountered. Firstly, I had tried to use the filter method to return the main menu info:

main_menu = Menu.objects.get(id=1)

But then the template didn’t work because main_menu was not actually a “Menu” object, but a list of “Menu” objects containing one element in the list. Secondly I was trying to use something like this in the templating system:

{% if node.path == path %}

Which threw an error because the templating system uses slightly different control flow statements. The correct form for the above code is:

{% ifequals node.path path %}

I am not sure why the templating system should use a different set of statements in this respect, but anyway, the documentation quickly came to the rescue.

Now I need to cope for the case that the path does not exist and raise a 404 error. So I modified the view to include a try-except construct:

    try:
        main_menu = Menu.objects.get(id=1)
        main_menu_items = Node.objects.filter(menu=1, parent=None)
        node = get_object_or_404(Node, path=url_path)
    except Node.DoesNotExist:
        raise Http404

Which basically checks that the path being looked for actually exists in the database. Adding in the 404 template is also straight forward. All you need to do is create a 404.html template in the root of the templates folder and Django will handle the rest.

Well I now have a very basic menu system in place. Obviously there is lots to do but right now I am concentrating on working myself through the tutorials. Once I have covered all 4 I will start expanding on my little app.

Modifying the default Django admin site

Those of you following my posts, anybody?, know that I am experimenting with creating my own little CMS using Django. I am starting by following the Django tutorials. I got through the first 1 and 3/4 tutorials before needing to do some research into how to get my recursive relationship key working in the admin site. It turns out I was a little naive regarding how the model system works in Django. Anyway, I finally got this sorted out and now I am back to continuing with the last part of the second tutorial.

The initial aim of my app is to be able to create multiple menu structures. So far I have two models, a menu model:

class Menu(models.Model):
    title = models.CharField(max_length=100)
    description = models.CharField(max_length=1000)
    in_use = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

    def is_in_use(self):
        return self.in_use == True

and a node model (which might turn into a “page” model in the future):

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

This has given me the basic admin site that you get from using django.contrib.admin. Customising the admin site is also straightforward. By adding a few lines to the Menu class in my admin.py file I added filtering, searching and the ability to generate multiple nodes whilst adding a menu.

For filtering:

list_filter = ['in_use']

For searching:

search_fields = ['title']

And for multiple nodes I created this class:

class NodeInline(admin.TabularInline):
    model = Node
    extra = 5

And then appended this line to my MenuAmdin class:

inlines = [NodeInline]

Finally I also modified the data brought back in my list of menus:

list_display = ('title', 'in_use')

I also copied the base_site.html file into my own templates directory and modified it to display my own title in the admin system. I will be playing around with the ability to customise the backend more in the future as this is one the things I really like about Django. It’s so easy to modify the backend.

So far I have written maybe 50 lines of “code” (well really it’s more like templating instructions) and have a fully functioning admin site. This is great, but also a little worrying as I am sure I will hit a major road block in the future with my limited Python knowledge. Still, so far so good.

Making database schema changes for Django

As I found out in my previous post it is not possible to modify your database schema via Django. This means once you have created the schema for your models, the only thing you can do is add more models using:

python manage.py syncdb

If you make a change to a model you are on your own.

When I defined my first model for a node (as in a node in a hierarchy) the model was something like this:

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self')
    is_visible = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

The problem with this is that this model does not allow for a root node, since the parent column is a foreign key on itself, but the root node must by definition not have a parent (that is parent = NULL in the database). So I changed the model to this:

class Node(models.Model):
    title = models.CharField(max_length=100)
    sort_order = models.IntegerField(default=1)
    menu = models.ForeignKey('Menu')
    parent = models.ForeignKey('self', blank=True, null=True)
    is_visible = models.BooleanField(default=True)

    def __unicode__(self):
        return self.title

This allows the parent column to be null and is fine for the model, but now I need to make the change to the schema manually in sqlite.

So first I run this script to see what table schema the model is expecting:

python manage.py sql cwcms

Which brings back this:

CREATE TABLE "cwcms_node" (
    "id" integer NOT NULL PRIMARY KEY,
    "title" varchar(100) NOT NULL,
    "sort_order" integer NOT NULL,
    "menu_id" integer NOT NULL REFERENCES "cwcms_menu" ("id"),
    "parent_id" integer NULL,
    "is_visible" bool NOT NULL
)
;

I then need to log in to sqlite and make the change to the schema to reflect this. This is easily done with the manage.py script:

python manage.py dbshell

Next, because sqlite only supports basic ALTER TABLE syntax (ie you can’t change column definitions) you essentially need to create your new table schema using a different name. Then copy the contents from the old table into the new one, delete the old table and then rename the new table to the name of the old table (ie the one that Django is expecting). The series of statements would look something like this:

CREATE TABLE "cwcms_node_new" (
    "id" integer NOT NULL PRIMARY KEY,
    "title" varchar(100) NOT NULL,
    "sort_order" integer NOT NULL,
    "menu_id" integer NOT NULL REFERENCES "cwcms_menu" ("id"),
    "parent_id" integer NULL,
    "is_visible" bool NOT NULL
);

INSERT INTO cwcms_node_new SELECT * FROM cwcms_node;

DROP TABLE cwcms_node;

ALTER TABLE cwcms_node_new RENAME TO cwcms_node;

Another way of doing this if you’re still early on in development is to simply get the commands needed to clear the database using manage.py:

python manage.py sqlclear cwcms

And run them on the your sqlite database and use syncdb to recreate your whole database. In either case it is probably worth creating scripts to populate your database with some dummy default data for testing purposes.

And now, after around 6 hours work from the initial install of Django I have a fully functioning admin interface that allows me to create any number of menus with any kind of hierarchy.

Recursive relationships in Django

In my previous post I stated how I had created a recursive relationship in one mof my models. The code looks something like this:


parent = models.ForeignKey('self', blank=True, null=True)

This works great, but the problem I now faced was how to enter the root element. Even though I had sent blank and null to True I was not able to enter my root node using the default admin interface. So this is the first issue I need to solve outside of the tutorial documentation.

Ok, after some searching around it is clear that I need to update the database schema to reflect the model change. Obviously this won’t happen automatically. So I thought I could use the syncdb statement that was used to create the schema in the first place:


python manage.py syncdb

But, as it clearly says in the documentation, syncdb only creates new tables. It does not modify existing tables. The solution seems to be to run the following the statement:

python manage.py sql APP_TITLE


To see what Table schema your model is expecting and to then make the change manually in your database. Not ideal, but manageable. They are working on ways to evolve the schema via Django, but that will be for a future version I guess.

This is the first disapointing thing about Django I’ve encountered so far. The idea seems to be that you get your schema right the first time, but this seems be unrealistic to me. Especially in an agile development environment where you are likely to make small changes at a time.

My first Django app

Today I followed the first two pages of the Django tutorial with one small difference. Instead of recreating the polling app I decided to start right away on my Contented Web app (CW).

Creating the Django project and my app was straight forward – although I must admit I don’t understand the technical nuances of everything that I am following. I think partially this is due to the Unix environment I am using.

The models I created are essentialy enough for creating a hierarchical menu structure. This seemed straightforward and I quickly got the hang of how to define and alter my models, including a recursive relationship.

What blew me away though was the admin site that was created and how easily I could modify the look and feel of the forms through the use of python tuples and lists. Pure class. As far as I can tell the whole admin site can be remodelled without affecting the functionality. This is great. One of the annoying things about most CMS tools is that whilst they enable you to create great looking websites, the admin interface always seems to be forgotten. And there seems to be little thought given to making it customisable.

So after about 3 hourse of playing around with Django and following the first two tutorials I now have a fully functioning admin interface for my models, including user management and access rights.

Getting Django

So, my first post is about downloading and installing Django. Makes sense.

My development environment, by the way, is Ubuntu running in VMWare player on Windows Vista. I decided to go with Unix for my Django work as it seems that that is what most of the documentation is targeted at. I had tried Virtual PC 2007, but be warned, Ubuntu on Virtual PC 2007 does not work out of the box.

I followed the steps on the “How to install Django” page. Ubuntu already shipped with the latest version of Python so there was no need to install that.  I did not install Django on Apache as this a development environment and I will simply be using the inbuilt python webserver and I chose to go for SQLite as the database which also comes as part of Django – mainly so I could get going asap. Getting this far took me about an hour. Most of which was time spent getting to grips with Ubuntu.