22
May 13

Celery, gunicorn, and supervisor

I use webfaction to host my web sites, and they are great!

Webfaction provides a “django” application type, which installs Apache + WSGI + Django for you, plus a Python instance. I really appreciate having had this setup, but now I understand what I’m doing well enough to throw it out and restart. Here’s what I have now:

1. django-supervisor with this config file:

[unix_http_server]
file={{ settings.SUPERVISORD_SOCKET_FILE|default:"/tmp/supervisor.sock" }}
[program:celeryd]
command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py celery worker -l info -B
[program:gunicorn]
command=gunicorn --bind 127.0.0.1:{{ settings.GUNICORN_PORT }} \
   --workers 3 heitmgt.wsgi

2. bunch o’ pip requirements:

django-supervisor
django-celery
gunicorn

3. RabbitMQ/Erlang. Yay?

4. Startup script that basically runs

python manage.py supervisor -d
python manage.py supervisor stop all
python manage.py supervisor start all

What the heck is going on here?

There is a pretty good reason I didn’t start out with this setup. I had no idea what the heck these things did or why you would need them. Why is everything so complicated??

Here’s the short of it:

  • Gunicorn: A python program that’s happy to serve up wsgi applications.
  • Celery + RabbitMQ: A queuing/processing system that lets you send tasks to be done later. For example, I just started using Django-celery’s database-stored “periodic tasks” to replace cron for some maintenance. (To handle periodic tasks you need a CeleryBeat worker; the “-B” flag to celery worker does this.)
  • Django-supervisor: A wrapper around the program “supervisor” that lets you generate a config file from Django settings files. Supervisor seems to be a guardian for processes–it daemonizes foreground processes. There are also some tricks to this e.g. if DEBUG=True it is supposed to restart the processes.

The move to Gunicorn was mainly because I wanted to get rid of the apache instance and consolidate web management into supervisor. The Celery part, though, is a really big deal. It is extremely helpful/a paradigm shift to have both synchronous requests and asynchronous tasks in the same application.

All that said, I migrated from Webfaction’s provided django apps to a new “custom app listening on port” that’s Gunicorn–and it was much less painful than I thought it would be. Yay!


01
May 13

Django environment setup

I have been doing some programming in “Django,” which is awesome. Django is a Python-based web programming framework.

Over the course of the last couple of months, I’ve worked on my Django development infrastructure. I’m kind of a stickler for doing things the “right” way, and this is what I’ve decided is the right way so far. Here’s how my Django infrastructure is set up:

Directory layout

A “bashmark” and an Emacs bookmark point to the root of my Django tree, which is checked out from git (and which I maintain with git-flow¬†and small time-duration feature branches).

  • Local (.gitignore’d) files:
    • .dir-locals.el: Sets pony-settings to use the virtualenv version of python, and virtualenv-workon and virtualenv-default-directory for Emacs virtualenv.
    • .pyflymakerc: Sets VIRTUALENV, PYLINT, TEST_RUNNER_COMMAND, IGNORE_CODES_PYLNT (‘E1101’), and TEST_RUNNER_FLAGS
    • .venv: Name of virtualenv
  • Makefile (see below), manage.py
  • app/ — where Django apps go, e.g. “app/hello/”. Each application may then have the following files (so far):
    • admin.py
    • ajax.py (used by dajaxice)
    • exceptions.py
    • forms.py
    • migrations/
    • models.py
    • templates/
    • tests.py
    • urls.py
    • views.py
  • bin/ — any command-line files that don’t cleanly fit into another directory
    • get_config.py — creepy program that spits out the identified Django config value, used to get Django database username et al for
    • pg_dump.sh — Shell script that creates a Postrgres dump file & can be added to cron w/o any output on success.
    • post.sh, post_local.sh — Shell scripts that are run after “make pulldown”. In prodcution, post_local.sh restarts the web server, uses curl to pull a test page, and reminds me to do other routine tasks associated with builds.
  • docs/ — “.text” files formatted via Markdown. A Makefile can compile these.
  • <<project>>/ — the Django project config directory
    • client_secrets.json, urls.py, wsgi.py
    • settings/ — Using this super-awesome __init__.py to let you have multiple settings files including a “common.py” plus an environment-specific one plus one called “local_settings.py” that doesn’t get checked in that has all your passwords in it
  • lib/ — (very simple) Pythony support libraries I’ve written. Currently includes:
    • json_shortcuts.py: to_json() function that correctly serializes datetime.datetime and datetime.date objects
    • middleware.py for custom middleware
    • nose_tweaks.py to make Nose shut up about South logging (plugin called “SilenceSouth“)
    • re_shortcuts.py for super common regexes
    • tests.py
    • time_shortcuts.py
    • unicode_csv.py for simple csv module wrapper that handles Unicode encodings
  • patch/ — diffs that I want to patch from that involve external stuff. Makefile can update these diffs. This is a crazy thing in and of itself and am not sure whether it would be helpful for others. Let’s just say that this helps my Django site look the same as my WordPress site.
  • requirements/ (see below for module list)
    • common.txt
    • dev.txt
    • test.txt
    • test-libs.txt (required by dev.txt and test.txt BUT NOT prod.txt)
    • prod.txt
    • unseen.py (hack to tell me what doesn’t show up in any requirements file that does show up when you run “pip freeze”)
  • static/
    • css/
    • js/
    • images/
  • templates/
    • base.html, {404,500}.html
  • uploads/
  • var/ — in .gitignore but this is where I keep database backups.

Makefiles

I tried using Fabric, and more power to you if you use it, but I didn’t need it. From my limited understanding, Fabric is good for helping you do stuff across systems. However, for someone with a background in cfengine Fabric is very frustrating to me. I wanted a way to automate parts of my build process but I didn’t need to do cross-machine deployments. So, I went back to Make. I don’t know why but I love Make.

Make targets:

  • pulldown — runs the “clean” target, then
 git pull
 git fetch --tags
 pip install -r requirements/${DJANGO_ENV}.txt
 python manage.py migrate
 # Needed for permissions to be created/updated:
 python manage.py syncdb --all
 python manage.py collectstatic --noinput
 cd bin && ./post.sh
 # (note: post.sh can then reboot the web server)
  • pushup — runs “git push –all”, “git push origin –tags”
  • clean — deletes .pyc files and runs “clean” for other Makefiles
  • virtualenv — builds the virtualenv you need for the project. (I just built this target and it’s pretty sweet.) This is totally a hack and not very robust. Just so you can get a feel for it (but FYI you can’t copy & paste this cleanly):
test ! -d ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 bash -i -c 'lsvirtualenv > /dev/null'
 curl -O http://pypi.python.org/packages/source/v/\
  virtualenv/virtualenv-${VIRTUALENV_VERSION}.tar.gz
 tar xvf virtualenv-${VIRTUALENV_VERSION}.tar.gz
 python virtualenv-${VIRTUALENV_VERSION}/virtualenv.py \
  --no-site-packages --setuptools \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 rm virtualenv-${VIRTUALENV_VERSION}.tar.gz
 rm -rf virtualenv-${VIRTUALENV_VERSION}
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && add2virtualenv .'
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && pip install -r requirements/${DJANGO_ENV}.txt'
 echo "export DJANGO_ENV='${DJANGO_ENV}'" >> \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "export DJANGO_SETTINGS_MODULE='${DJANGO_PROJECT}.settings'" \
  >> ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "${VIRTUALENV_ALIAS}" > .venv

Modules/requirements

  • common.txt (for all environments) includes:
    • Django
    • South (totally the best thing since sliced bread and extremely the type of thing you don’t know you need until 3 months after you need it)
    • django-extensions (even better than South. runserver_plus, shell_plus, and graph_models are my faves)
    • pytz (see my “Time” post)
    • django-dajaxice (still not sure how I feel about this but it has been pretty useful so far)
    • Scrapy (is awesome)
    • coverage (really cool but I am not using it enough yet–along with django-nose, this will tell you thoroughly you’re testing)
  • dev.txt:
    • Werkzeug (O. M. G.)
    • pep8
    • pyflakes
    • pygraphviz
    • pylint
    • jedi (this helps with Emacs autocompleter stuff)
  • test-libs.txt (non-production envs):
    • django-nose
    • factory-boy (really good for testing)

Emacs

Using Aquamacs. ~/.emacs.d is checked into its own git repository currently shared via dropbox.

  • pony-mode
  • virtualenv
  • yasnippet
  • markdown-mode
  • jedi
  • python (vanilla 23.3 ‘python python-mode)
  • json-mode
  • flymake
  • django-nxhtml-mumamo-mode (worth every letter)

Things I Now Try To Do

  • Use factories rather than fixtures for tests (I thought factories were dumb until I started changing models & had to mess with raw .json — this kills the programmer)