User-Testing the Django Tutorial

Building on PyCon

I attended Karen Rustad’s inspiring talk at PyCon 2012 about improving the Django tutorial. In her talk Karen did an excellent job of identifying the strengths and weaknesses of the existing tutorial. But she also went beyond just a simple critique, and laid out a pretty clear plan for overhauling the tutorial. Her suggestions point at an approach to building tutorials that could be helpful in making better tutorials for all projects, and with less effort as well. One of her suggestions during the talk was to conduct user testing on people as they go through the tutorial. Last summer I took copious notes as I tried to build my first django project while consulting the tutorial. Looking through these notes on the plane trip home, I realized I had basically conducted user-testing on myself as I went through the tutorial for the first time.

I have pretty strong motivations for helping to clean up the tutorial. I have built one mostly-functional django project which currently lives on an internal server at the high school where I teach. I would like to develop this project further, host it publicly, and refine my skills to the point where I can whip out a prototype django project in just a few hours. I need to make some documentation for myself as I do this work, so why not clean up the tutorial in the process? In my work as a teacher I also pay attention to how young people, especially high school students, learn to use projects like django. Most high school students don’t have access to teachers who can guide them through learning something like django. Finding a way to build better tutorials for django and other projects could have a huge impact on the recruitment and retention of good developers. If we can improve the independent learning process about technical topics, we should see a crop of better-trained CS students in just a few years.

Profile of me as a tutorial user
One of the critical steps Karen mentioned was to define the intended audience for a tutorial. Am I totally new to web development? Or am I an experienced developer who has used other languages and frameworks, looking to quickly get up to speed in django? I am right on the middle of this spectrum. My father was a software developer in the 1970s and 80s, so I have been around programming all my life. I studied physics in undergrad, where I got a smattering of Fortran, Pascal, and C. My biggest project was writing a 3d plotting program in C for fun in the early 90s. In 2002 I built a personal photo website using just html and javascript. In 2005-6 I tried to build a web app using php/ mysql/ javascript, but didn’t know how to market my product effectively. It was a good failure in that I learned basic java and mysql through the project. In 2011, I needed to build a database app for my school, and after a bit of research I settled on learning django for the project. That is, roughly, how I found myself sitting in front of the django tutorial for the first time last summer.

Unwittingly user-testing myself
I ran through the tutorial, building out parts of the poll app on my own system. But I’ve done enough tutorials that I pretty quickly jumped into building a prototype of my own project while following along with the tutorial steps. I took detailed notes as I tried to build my prototype, including questions I had, outside resources I consulted, and decisions I made for my own project. Reading back through these notes now with an eye towards improving the tutorial, I find myself identifying the same strengths and weaknesses in the tutorial that Karen talked about. First I’ll share the raw text of my notes, then some specific ideas about how to improve the tutorial.

Raw text of user notes
I have included my notes here verbatim, rather than cleaning anything up before posting. If you take the time to read through any of this you will undoubtedly identify some misunderstandings and bad ideas, and that is the point of sharing this. I have a pretty good sense of my own ignorance, and I know which of my own questions need to be answered before putting a project into production. The project I was working on dealt with keeping track of competencies, which are a form of learning standards. In the text, “CDB” refers to “competency database system”.


We can clearly see a number of gaps in the tutorial, and how I tried to fill in those gaps for myself.

  • I used an outside resource that I’m sure many of us are familiar with, the Salty Crane post about setting up virtualenv. I had pretty significant questions about how to properly set up my development environment. For example: “Is the virtual environment worthwhile right now? Or jump into django on my system as-is? Use virtual environment until proves too complicated.”
  • I had many questions about how to configure my installation of mysql to work with django.
  • I thought a lot about how to deploy on apache, when I should have been focusing on just using the development server at first. I knew my project would only be meaningful if I could move it onto some kind of production server, so I was trying to figure out if I would be able to accomplish this after investing the time to build a django project. A link with a clean description of deployment options would have been very helpful at this point, freeing me to focus on learning with the development server.
  • There are more notes than just these. I split my notes over several files, so this is just the first batch of notes. The rest are more focused on implementation details, and captured less of the thinking than this set did.

What’s next?
There are some pretty straightforward ways to improve the tutorial, following the suggestions Karen laid out. If we do this well, the django tutorial could serve as an exemplar for other projects. As we heard in a number of talks, building a tool to manage documentation is more interesting than writing the documentation itself. So I think I might prototype a tutorial-building tool that incorporates much of the following. Here is what I took as Karen’s “blueprint” for improving the tutorial:

  • tutorial sketch

    Rough sketch of an approach to building tutorials. On the tutorial landing page, you identify what kind of user you are - noob, mid level, or experienced. Everyone sees the same code, but the level of detail in the explanations you see corresponds to the kind of user you have identified yourself as.

    Identify the audience for the tutorial. Each reader who arrives at the tutorial should get to select what kind of tutorial user they are, and see explanations that match their level of experience and prior knowledge. Everyone sees the same code, but the explanations that accompany each code sample match the level you have selected. So if you’re a noob, you see paragraphs of explanatory text that cover django specifically, along with some explanations of general web development issues. If you’re an experienced developer you see mostly code, with just enough explanation to understand how django is addressing issues you should already understand.

  • Identify environment setup steps that need to be taken. If we all use virtualenv, and consider that best practice, then we need to make it part of the tutorial.
  • Provide links to outside resources where appropriate. If the Salty Crane post is really the best resource for setting up virtualenv, then let’s link to it. We can put a disclaimer in front of the link if necessary, but that’s better than leaving noobs to try and google information they are not even sure they should be looking for.
  • Prominently post where to ask for help. In the noob version, we might even link to a guide for how to set up an IRC client, and how to use a mailing list. It is probably safe to assume that many noobs have not used anything other than forums for help.
  • Take the tutorial all the way through publicly hosting a prototype. At least give an overview of the main hosting options such as a service like heroku, a VPS, or a dedicated server.
  • Conduct user testing. This does not need to be formal, but we can at least give people an easy way to share feedback. Referring people to IRC and mailing lists for help means we are not identifying what is working and what is not working.

Thank you to Karen for so clearly articulating the ways the tutorial can be improved. If you are interested in collaborating on improving the tutorials in a way that should generalize to other projects, please get in touch. If you have specific suggestions for how to improve the tutorial, please feel free to share them here.


About ehmatthes

Teacher, hacker, new dad, outdoor guy
This entry was posted in programming and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s