I have been doing some programming in “Django,” which is awesome. Django is a Python-based web programming framework.

Over the course of the last couple of months, I’ve worked on my Django development infrastructure. I’m kind of a stickler for doing things the “right” way, and this is what I’ve decided is the right way so far. Here’s how my Django infrastructure is set up:

Directory layout

A “bashmark” and an Emacs bookmark point to the root of my Django tree, which is checked out from git (and which I maintain with git-flow and small time-duration feature branches).

• Local (.gitignore’d) files:
  • .dir-locals.el: Sets pony-settings to use the virtualenv version of python, and virtualenv-workon and virtualenv-default-directory for Emacs virtualenv.
  • .pyflymakerc: Sets VIRTUALENV, PYLINT, TEST_RUNNER_COMMAND, IGNORE_CODES_PYLNT (‘E1101′), and TEST_RUNNER_FLAGS
  • .venv: Name of virtualenv
• Makefile (see below), manage.py
• app/ — where Django apps go, e.g. “app/hello/”. Each application may then have the following files (so far):
  • admin.py
  • ajax.py (used by dajaxice)
  • exceptions.py
  • forms.py
  • migrations/
  • models.py
  • templates/
  • tests.py
  • urls.py
  • views.py
• bin/ — any command-line files that don’t cleanly fit into another directory
  • get_config.py — creepy program that spits out the identified Django config value, used to get Django database username et al for
  • pg_dump.sh — Shell script that creates a Postrgres dump file & can be added to cron w/o any output on success.
  • post.sh, post_local.sh — Shell scripts that are run after “make pulldown”. In prodcution, post_local.sh restarts the web server, uses curl to pull a test page, and reminds me to do other routine tasks associated with builds.
• docs/ — “.text” files formatted via Markdown. A Makefile can compile these.
• <<project>>/ — the Django project config directory
  • client_secrets.json, urls.py, wsgi.py
  • settings/ — Using this super-awesome __init__.py to let you have multiple settings files including a “common.py” plus an environment-specific one plus one called “local_settings.py” that doesn’t get checked in that has all your passwords in it
• lib/ — (very simple) Pythony support libraries I’ve written. Currently includes:
  • json_shortcuts.py: to_json() function that correctly serializes datetime.datetime and datetime.date objects
  • middleware.py for custom middleware
  • nose_tweaks.py to make Nose shut up about South logging (plugin called “SilenceSouth“)
  • re_shortcuts.py for super common regexes
  • tests.py
  • time_shortcuts.py
  • unicode_csv.py for simple csv module wrapper that handles Unicode encodings
• patch/ — diffs that I want to patch from that involve external stuff. Makefile can update these diffs. This is a crazy thing in and of itself and am not sure whether it would be helpful for others. Let’s just say that this helps my Django site look the same as my WordPress site.
• requirements/ (see below for module list)
  • common.txt
  • dev.txt
  • test.txt
  • test-libs.txt (required by dev.txt and test.txt BUT NOT prod.txt)
  • prod.txt
  • unseen.py (hack to tell me what doesn’t show up in any requirements file that does show up when you run “pip freeze”)
• static/
  • css/
  • js/
  • images/
• templates/
  • base.html, {404,500}.html
• uploads/
• var/ — in .gitignore but this is where I keep database backups.

Makefiles

I tried using Fabric, and more power to you if you use it, but I didn’t need it. From my limited understanding, Fabric is good for helping you do stuff across systems. However, for someone with a background in cfengine Fabric is very frustrating to me. I wanted a way to automate parts of my build process but I didn’t need to do cross-machine deployments. So, I went back to Make. I don’t know why but I love Make.

Make targets:

• pulldown — runs the “clean” target, then

 git pull
 git fetch --tags
 pip install -r requirements/${DJANGO_ENV}.txt
 python manage.py migrate
 # Needed for permissions to be created/updated:
 python manage.py syncdb --all
 python manage.py collectstatic --noinput
 cd bin && ./post.sh
 # (note: post.sh can then reboot the web server)

• pushup — runs “git push –all”, “git push origin –tags”
• clean — deletes .pyc files and runs “clean” for other Makefiles
• virtualenv — builds the virtualenv you need for the project. (I just built this target and it’s pretty sweet.) This is totally a hack and not very robust. Just so you can get a feel for it (but FYI you can’t copy & paste this cleanly):

test ! -d ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 bash -i -c 'lsvirtualenv > /dev/null'
 curl -O http://pypi.python.org/packages/source/v/\
  virtualenv/virtualenv-${VIRTUALENV_VERSION}.tar.gz
 tar xvf virtualenv-${VIRTUALENV_VERSION}.tar.gz
 python virtualenv-${VIRTUALENV_VERSION}/virtualenv.py \
  --no-site-packages --setuptools \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 rm virtualenv-${VIRTUALENV_VERSION}.tar.gz
 rm -rf virtualenv-${VIRTUALENV_VERSION}
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && add2virtualenv .'
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && pip install -r requirements/${DJANGO_ENV}.txt'
 echo "export DJANGO_ENV='${DJANGO_ENV}'" >> \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "export DJANGO_SETTINGS_MODULE='${DJANGO_PROJECT}.settings'" \
  >> ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "${VIRTUALENV_ALIAS}" > .venv

Modules/requirements

• common.txt (for all environments) includes:
  • Django
  • South (totally the best thing since sliced bread and extremely the type of thing you don’t know you need until 3 months after you need it)
  • django-extensions (even better than South. runserver_plus, shell_plus, and graph_models are my faves)
  • pytz (see my “Time” post)
  • django-dajaxice (still not sure how I feel about this but it has been pretty useful so far)
  • Scrapy (is awesome)
  • coverage (really cool but I am not using it enough yet–along with django-nose, this will tell you thoroughly you’re testing)
• dev.txt:
  • Werkzeug (O. M. G.)
  • pep8
  • pyflakes
  • pygraphviz
  • pylint
  • jedi (this helps with Emacs autocompleter stuff)
• test-libs.txt (non-production envs):
  • django-nose
  • factory-boy (really good for testing)

Emacs

Using Aquamacs. ~/.emacs.d is checked into its own git repository currently shared via dropbox.

• pony-mode
• virtualenv
• yasnippet
• markdown-mode
• jedi
• python (vanilla 23.3 ‘python python-mode)
• json-mode
• flymake
• django-nxhtml-mumamo-mode (worth every letter)

Things I Now Try To Do

• Use factories rather than fixtures for tests (I thought factories were dumb until I started changing models & had to mess with raw .json — this kills the programmer)

Recently I built a scheduling application for my work web site. It’s pretty awesome.

Anywho, I had to learn about time, and how it’s handled. Going into this, I always thought that if I were a systems administrator all over again I would have my machines all run in GMT to eliminate timezone problems. Now that I’ve learned, I know it’s now called “UTC” and it’s still a really really really really really good idea to do as much as you can in UTC and only present localized times only when the user sees them.

I learned about how time works in four tools: Postgres, Python, Django, and Javascript.

Postgres

Totally sweet support for time. You can say

generate_series(date(X at time zone 'UTC' - interval '120 minutes'),
                date(Y at time zone 'UTC - interval '120 minutes'),
                '1 day'::interval)

and get all the dates between X and Y. Everything about datetimes is super duper good.

Python

datetime library is pretty good. It is a little unfortunate (although understandable) that you need a non-Python library (usually pytz) to localize times. (This is because Python does not want to handle the political problems (literally) of keeping timezone definitions up to date.)

Notably, I built a small library called “time_shortcuts” to help with common time stuff:

• utc_dt(str)–convert the string into a UTC-localized datetime object.
• utc_now()–right now in UTC
• xtimerange(start, end, step)–walk from start to end by step.
• minute_diff(start, end)–returns a diff between start and end in minutes, not days. makes a bad assumption about how many minutes are in a day.

This way I could write utc_dt(“2013-04-25 18:00:00″) and know I was getting back a datetime timezone-aware object. This is much more helpful than it should be.

Django

Django gets a little angry if you feed it DateTimeFields that aren’t timezone-aware. Hence the above helper library. Other than that, it makes the Python <-> Postgres interface transparent, which is all you can ask for. You deal in Datetime objects, and it handles Postgres.

I have one notable instance where I call straight SQL (using a cursor) so that I can do some of the Postgres goodness described above.

Javascript

Good stuff: Javascript’s Date() object.

Bad stuff: Javascript’s Date() object.

It is nice to call

new Date().getTimezoneOffset()

and get the timezone diff in minutes. It is also nice to be able to call .getUTCHours().

It is unfortunate that I don’t know of a way to change the timezone that Javascript uses for a given session. This resulted in me creating a clunky <select id=”timezone”> object that has all the timezones and their minute offset and this becomes a weird interactive variable that redraws the time-related parts of the page when it’s changed.

In the spirit of “basic things I do to make my life easier,” here’s how I have Chrome set up:

Extensions

• AdBlock Plus: Blocks many ads.
• Ghostery: Lets you control what JavaScript is executed as you browse, e.g. you can turn off some DoubleClick tracking.
• OneTab: You click the button and all your open tabs go into OneTab. (They stop taking CPU cycles/RAM.) You can then re-open them as you see fit. I open tabs constantly, and OneTab is an easy way to “save” those sets of tabs if I’m not ready to look at them.
• 1Password (installed via the 1Password application): super good application/extension to keep track of passwords and generate passwords. Note: once the extension’s installed, ⌘-\ will fill in passwords for you.

Bookmarks

The only bookmarks I have in Chrome are on the bookmarks bar, right below the location bar. I use Pinboard for all my other bookmarks.

• “fxns” folder:
  • pinboard bookmarklet to bookmark a page
  • “read later” bookmarklet to bookmark in pinboard with “read later”
  • “pinboard random” to read a random pinboard unread bookmark
• “tabs” folder: each of the tabs I want open whenever I’m in Chrome. (I have one window that’s my “work” window with my email, to-do list, etc as tabs.) Then you can right-click this folder and say “open all”. For my work account:
  • Gmail
  • Google Drive
  • Toodledo
  • Google Calendar
  • Google Voice (Note: this links directly to my “alt”/personal account, so I don’t have to click to change Google accounts. Each Google product handles this differently, but for Google Voice it’s https://www.google.com/voice/b/1#inbox where the number is a 0-based count of which account you want to open.)
  • Google Intranet for my business
  • HootSuite
• “temp” folder: folder for stuff I’m going to reference a lot, but just for a little while. For example, when I’m doing a creative commons image search for a presentation then I might bookmark that page in this folder.
• other web sites, bookmarked straight on the bookmarks bar, with NO TEXT. If you edit a bookmark and delete its text, Chrome will still show the icon. I do this for the Django documentation, for example–I just see the icon for Django (~¼” wide) rather than the icon + description (1+” wide). This way your bookmarks bar stays less cluttered.

Settings

I personally turn off form auto-fill and saved passwords, because I use 1Password for that. (1Password can store your “identities,” credit card info, etc, so you can have 1Password fill out these forms.)

Fancy things

• Hush: separate program that detects when you are seeing a Hulu commercial and mutes your volume until the commercial is over

I get fixated a little on how to be more “efficient.” Part of that includes setting up my environment so it works well. When I’m using bash (a UNIX shell), here are a couple of things that’ve helped save time:

• bash-completion: Totally Awesome. Use this or else. (Note that ssh ser<TAB> will complete the server name, for example.)
• Bash Emacs Editing Mode cheat sheet. (Note I grew up in vi but have been dedicating a LOT of time to learning Emacs, so I use set -o emacs.)
• bashmarks: gives you simple aliases you can use to move directories, e.g. “g alias” goes to the directory associated with “alias”.
• git-flow-bash-completion
• venv_cd: If you use virtualenvs, this will auto-”workon” them for you. Make sure you source this *before* bashmarks. I don’t know why, but my experience is that this needs to get sourced first so that “g alias” will also check for virtualenvs.
• Learn your basic bash shortcuts, e.g. the power of squiggles “mv x{,-old}”, variable manipulation (e.g. “${x%.html}” removes “.html” from the end) and re-use of previous arguments e.g. “ls -l !$”

Update 4/25: added a section about menus.

I am not a WordPress expert. That said, I have learned a little bit about WordPress, and Lauren encouraged me to write it down.

Users

I use two users: “admin” and “john”. If you’re going to mess with your blog and it’s brand new, consider creating an account for yourself and giving it WordPress administrator access. Then everything will at least be recorded as you. Then “admin” user becomes your “user of last resort.”

General Settings

Reading

1. Create two “pages”–”Home” and “Blog”. Home will be your home page. The content of “Blog” will be totally ignored; just create the page.
2. Go to Settings > Reading.
3. Change “Front page displays” to “A static page” and set Front page to “Home” and Posts page to “Blog” (the two pages you created in step #1).

Permalinks

1. Go to Settings > Permalinks
2. Select “Custom Structure”
3. Type in “/blog/%year%/%monthnum%/%postname%/”. This will make your blog posts start with “/blog/” and your URLs look like “/blog/2013/04/stuff-you-should-do-when-building-a-wordpress-site/”

Plugins

Install and set up these plugins:

• Akismet: pay for it if you need to (I did). Very important.
• Disqus comment system: set up an account and do this. Move the comments infrastructure (including managing identities) from being your problem to being their problem. If you already have a blog and switch to Disqus, you can have WordPress import your comments. I did this last week but it took like 5 days before the import completed.
• Google Analytics for WordPress: get an ID and do this.
  • Once installed, go to the settings. Click the “show advanced settings” checkbox. Set “Ignore Users” to “Subscriber (ignore all logged in users)”. Now when you’re logged in you won’t be added to the analytics.
• Google XML Sitemaps: install.
  • Once installed, set up Google Webmaster Tools and submit your sitemap.
• If needed, Open in New Window Plugin. This adds JavaScript that will convert all off-site links to open in a new window. I used to think this was horrible practice, but many people seem to be doing this nowadays and now I kind of expect it.
• Redirection, so you can manage your HTTP redirects within WordPress.
• Slick Social Share Buttons. This can put a bunch of different services’ share buttons on your page. You can configure the heck out of how they’re displayed
• W3 Total Cache, if you are concerned about potential spikes in traffic. This plug-in can be super complicated but it can also help you use really advanced capacity management services like CloudFlare.
• Widget Context. This allows you to say widgets only show on certain pages–for example, the blog categories list should only show up on blog pages.

Themes

If you are the kind of person who will make ANY changes to your theme, you need to create a “child theme” for it. Basically you create a new theme directory and create a file called “style.css” in it that has a special comment header in it. Part of that header (“Template”) tells WordPress it’s a child theme. After you’ve set this up, you can select your child theme from the Themes list and it will inherit everything from the parent theme.

You put your mods into the child theme (e.g. additions to functions.php, other PHP files, styles). When you update the parent theme, these mods “stick” and you don’t have to re-do them.

Stuff you could potentially do with your theme:

• Make an “author-<<username>>.php” page to have a fancier blog author page.
• Change how blog lists show up and are summarized
• editor-style.css: Change the stylesheets use in the editor to better mirror what visitors see.
• Change your footer.php
• Add JavaScript on top of your pages e.g. I made a “hide”/”show” thing for my work email sign-up widget.

Widgets

If you installed Widget Context, you can change where your widgets show up. I find this helpful especially to limit the blog-specific stuff from showing up on posts (i.e. non blog pages).

Menus

By default, WordPress will build a top-level menu for your site that includes all your pages that don’t have a parent. To change this,

1. Go to Appearance > Menus
2. Create a Menu (maybe called “Home menu”? The name is not important)
3. Add whatever you want to it
4. Save it
5. Under “Theme Locations” (upper left), hopefully you have an option for “Primary Menu”. Set this to the menu you created (“Home menu”).
6. Now only what you add manually to this Home menu will show up as your top-level menu.

Every couple of years I seem to update my blog. So, here goes.

I just moved my web hosting to webfaction. The shortest version of the story is, I started a company and have been doing some Django programming and webfaction is really good for Python-based hosting and I realized some of what I’ve been missing out on. In fact, I’d used them many years ago to host a Plone site for the UU Fellowship of Winston-Salem. (Anyone who advertises that they host Plone sites is pretty hardcore.)

Looking under the hood, I feel better about webfaction because…

• They let you run multiple “applications,” which you can tie to one or more “domains.” For example, ‘johnborwick.com’ (no ‘www’) is running a bare bones application that just forwards people to ‘www.johnborwick.com’.
• They have their own WordPress installer, which makes me feel better than the Fantastico/whatever the replacement for Fantastico is called. For my old host, if you wanted to keep your site up-to-date with the package manager you’d wait for a few weeks before they got the upgrade script. (I have also started to feel more comfortable giving WordPress the permissions needed to update itself.)
• If I want to start building simple services for people, I can easily set them up as new applications on my account. (Before, I’d used Google App Engine to build a few test apps, but you have to subtly change your apps to run in that environment and I’d read some FUD about the ability to scale/lack of control you have with GAE.)
• I feel like I have a lot of control, but the framework makes good sense.
• They are highly regarded.

I used to use FreeBSD a lot.  (FreeBSD is an alternative operating system, kind of like Linux.  It is the basis for OS X and many other things.)  However, ever since I stopped being a systems administrator I haven’t had a FreeBSD machine.  Well, last week our Apple Time Capsule (wireless base station + backup device) died, and I wanted to try to “roll my own” server that could be our wireless access point plus backup system and more.

The thing about FreeBSD is, you have to really learn a lot to use it.  To my knowledge, there aren’t many simple configuration tools beyond the “sysinstall” installation tool, and if you run into any problems you are on your own (to try and Google your issues and fix the problem).  In case it’s helpful for others, here is what I’ve learned about installing FreeBSD on a Lenovo H430 desktop computer.

Things I’ve learned

• FreeBSD 9.0-RELEASE does NOT recognize the built-in wired or wireless network devices.  You have no network connectivity.
• Use the amd64 version so you can use all 8 GB RAM.
• FreeBSD 8.3-RELEASE recognizes the built-in wired network device (yay!) but doesn’t recognize the wireless network card (boo).
• I scrapped the wireless network card and bought a TP-LINK TL-WN881ND wireless card.  HOWEVER, I then found this card doesn’t work with 8.3-RELEASE.  This FreeBSD forum thread recommends you  recompile your kernel to remove the 8.3-RELEASE drivers, pull the FreeBSD subversion HEAD or STABLE/9 versions, and load those modules.  This worked for STABLE/9 at the point this thread was written (April 2012) but doesn’t work today (September 2012).  John B’s solution?  Use ‘svn -r {2012-04-21}’ to pull STABLE/9 from April 21, 2012!
• Audio does not work automatically but it is simple: the snd_hda driver
• For my monitor, Xorg’s autoconfigure gave me a max resolution of 800×600 (yay!)  I once again had to find the horizonal sync and vertical refresh rates for my monitor.  Do you know how many times in my life I have had to search through technical specifications to find the hsync and vrefresh?  Hint: it’s a lot, and finding it is usually pretty hard because my monitor doesn’t work properly.

Current setup

For the TL-WN881ND card, I had to do a bunch:

Here’s how I built the if_ath and if_ath_pci drivers:

PREFIX=stable/9

svn co -r {2012-04-21} {http://svn.freebsd.org/base/,}$PREFIX/sys/dev/ath
svn co -r {2012-04-21} {http://svn.freebsd.org/base/,}$PREFIX/sys/modules/ath
svn co -r {2012-04-21} {http://svn.freebsd.org/base/,}$PREFIX/sys/modules/ath_pci

( cd $PREFIX/sys/modules/ath && env CFLAGS=-I../../ make )
( cd $PREFIX/sys/modules/ath_pci && env CFLAGS=-I../../ make )

Built a custom kernel; here’s my unified diff from the GENERIC kernconf:

--- GENERIC 2012-03-03 01:15:13.000000000 -0500
+++ JOHNB-NOATH 2012-09-23 03:38:18.000000000 -0400
@@ -258,10 +258,10 @@
 device wlan_tkip # 802.11 TKIP support
 device wlan_amrr # AMRR transmit rate control algorithm
 device an # Aironet 4500/4800 802.11 wireless NICs.
-device ath # Atheros pci/cardbus NIC's
-device ath_hal # pci/cardbus chip support
+#device ath # Atheros pci/cardbus NIC's
+#device ath_hal # pci/cardbus chip support
 options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
-device ath_rate_sample # SampleRate tx rate control for ath
+#device ath_rate_sample # SampleRate tx rate control for ath
 device ral # Ralink Technology RT2500 wireless NICs.
 device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.

/boot/loader.conf:

if_ath_load="YES"
if_ath_pci_load="YES"
snd_hda_load="YES"

I’ve done several other things to the computer since, such as setting it up as a gateway (dnsmasq is awesome!!) and AFP server, but I think these were the main showstoppers.