06
Jan 15

Kindle workflow

In December I acquired my first e-reader, a Kindle Paperwhite.

I’ve been reading very few books for the last few years, although in truth I have been reading: mainly reddit posts or pinboard popular bookmarks. Lauren’s been using an e-Reader for 4-5 years now, and she recently got a Paperwhite, so after seeing hers I decided I’d try it.

With the Kindle, I’m definitely reading more books. I’m also reading several books at once, which I’ve never done before, but with the Kindle there’s no hassle in trading multiple books and finding bookmarks. Lauren and I also enabled the Kindle Family Library, which lets you share Kindle books across accounts.

I also wanted to learn more about other content providers, and from that I found Calibre. Lauren had used Calibre a few years ago but it didn’t fit her needs well. After installing it, I learned that recipes are written in Python, and I became more interested.

My biggest “win” with the Kindle has been using a recipe called “Pinboard-Recipe”–I even contributed a bit of code to it! This recipe goes through your pinboard unread bookmarks, selects the 15 most recent bookmarks, and creates a magazine that Calibre can then send to the Kindle via its email address.

Over the last ~5 years I’ve bookmarked over 600 things with pinboard’s “read later” function, but I’ve never come back to read them. Now, I’m reading all kinds of good longer articles, and I can switch back and forth between them and traditional books.

Also Calibre makes it really easy (trivial, even) to load a Project Gutenberg book to the Kindle. I finally finished Typee.

Finally, I sorted out when I use Amazon wishlists vs. goodreads lists. I use goodreads for books I want to read, and wishlists for things I want to purchase. I added a couple of goodreads shelves: “read-soon” for a subset of “to-read” that I actually plan to read in the near future, and “owned” to indicate whether I own the book in one medium or another.


08
Oct 14

Homebrewer’s Clubhouse in Winston-Salem, NC

My friend Jon Eilbes has started a business, the Homebrewer’s Clubhouse in Winston-Salem, NC. They sell all the raw ingredients you need to brew as well as kits to help you get started!

If you know Winston-Salem, it’s on Hawthorne across from the Food Lion in that strip near where the coffee house used to be.


30
Aug 13

Using South to change a Django model’s parent class

I recently wanted to introduce a new parent model for some of my Django models. (The model is called “PayableModel” and is used for any model for which you might pay money.)

Like any reasonable person, I use South to manage my models. This is a Good Thing.

However, much like changing AUTH_USER_MODEL, there is not much support for how to go through this process and keep your existing data. So, in the hopes it’s useful, here’s what I did.

Note: this is a one-way migration. Your model’s ‘id’ field is going to be dropped, and it’s pretty hard to get that back (although now I have some ideas). In any case…

  1. Create your new parent model and run schemamigration as normal (python manage.py schemamigration your_app_name –auto)
  2. In the to-be child model(s), create a new nullable IntegerField called parentmodel_ptr. For example:
    paymentmodel_ptr = models.IntegerField(null=True)
  3. Run schemamigration again. If you’re into that sort of thing, add a “depends_on” clause to the migration to make it dependent on the migration from step #1.
  4. Create a datamigration. (Fancy!):
    python manage.py datamigration your_app_name \
      populate_me_pointers
  5. Edit the datamigration:
    def forwards(self, orm):
      for child in orm.ChildModel.objects.all():
        # in which we create the parent:
        parent = orm['parent.ParentModel']()
        parent.save()
        child.parentmodel_ptr = parent.pk
        child.save()

    This snazzily creates the parents and links it to the secret Django magic _ptr field that will tell Django where to find the parent. At this point, the field is nothing special.

  6. Edit your model to (1) remove the parentmodel_ptr and at the same time add the new parent class. Then run another schemamigration.

The brilliance of this model is I have idea how South converts the IntegerField to the foreign key. It will see the field as changing, rather than being deleted and re-added, because you made the change all at once.

I hope this helps!

If there was one post that made me want to switch to a Markdown-based blog…


15
Aug 13

The best way to protect your identity is to lie

“Assume rational actors.”

Part of Nixon’s strategy for the Cold War was to act crazy. International politics, along with economics, assumes rational actors. If someone is crazy, then you can’t use traditional game theory on them. Who knows what they’ll do?

If you assume we are in a surveillance society, where everything you do is monitored, pretty much the only thing I can think of that you can do to protect your identity is to lie.

By this I mean, monitoring etc assumes you are a rational actor–that everything you’re doing makes sense. If I call a Pizza Hut, then I probably like pizza. If I “like” the NRA, then I’m probably a conservative.

Arguably (given what’s been discussed this year so far), you can’t protect this information. If your phone calls are being logged, then people can see that you called the Pizza Hut. There are many steps you can take to try to keep people from getting this information in the first place, but I’d argue you really can’t keep this type of information safe.

On the other hand, you can affect how reliable this information is. Right now, if 100% of your likes on Facebook represent what you like, then advertisers/others can trust your likes to build a representation of you.

However, what if 20% of your likes were totally ridiculous? E.g. liking leather shoes along with PETA, or liking the NRA and the ACLU, or whatever is against your typical personality type. Or, what if some of your calls were to your local union as well as to your small business agency? Basically, what if some of your actions were irrational?

In other words, there is another dimension of security that people don’t necessarily think about. If you increase the cost of verifying that the collected data is accurate, you introduce a new dimension of security.

(Note: I am not a lawyer and you need to read the terms of service etc for any services you use to determine if this breaks any rules etc.)


15
Aug 13

Social media sites chosen like music, not like cell phones

Yesterday I read this article, “I’m 13 and None of My Friends Use Facebook.” There have been many of its ilk. I’m wondering though if people select social media sites less like how you choose a cell phone, and more like how you choose music.

Conversations about social network providers all seem to be about which One True Service is the correct service for all users, very much like cell phones: you like iPhones, Android phones, or a couple of other types–but you are going to be on a recent version of one of these types. Very few people stick with their old vendors–they migrate (&/or are forced to migrate) to the latest options. With this thinking, a new social media site could usurp Facebook immediately–its fate determined like Blackberry’s (sorry Blackberry).

But what if this is the wrong mode of thinking for social media? Think about how people choose the music they like. The music you like is an extremely personal decision, and people tend to be very caught up in identifying themselves by the music they like. Your friends are the other people who also like They Might Be Giants, for example (holla!). We may stick with the bands we like from a long time ago, regardless of what they’re doing now.

Social media selection, like music selection, could also be generational. Lauren was telling me about a study showing that people tend to listen for their whole life to the music they listened to in their teens. Maybe the people who identify with Facebook right now will always use it, but the next generation will use a different site (or set of sites). And their identity may be caught up in the service(s) they use being different than their parents’ site.

Is Facebook like the Beatles (and Google+ either like Elvis, or the Monkees, depending on your bias)? We are at such an early stage of social media, just as how Rock and Roll was at an early stage of development with the Beatles, that Facebook like the Beatles could have swept away a generation.

I have to think that as new generations take for granted the platforms we have, it will be more and more likely for them to use have a large number of social media profiles the same way that younger people tend to be on many chat services at the same time. Then services (a la Adium/Pidgin/HootSuite) will exist to let you bulk-manage profiles. Services will need to enable automated APIs to attract users, rather than disabling them to keep from losing users.


10
Jul 13

Everything ages

Recently we’ve been teaching our son that everything ages.

Here are two emotional videos that make the point:

0 -> 1

A Second a Day from Birth. from Sam Christopher Cornwell on Vimeo.

1 -> 100


22
May 13

Celery, gunicorn, and supervisor

I use webfaction to host my web sites, and they are great!

Webfaction provides a “django” application type, which installs Apache + WSGI + Django for you, plus a Python instance. I really appreciate having had this setup, but now I understand what I’m doing well enough to throw it out and restart. Here’s what I have now:

1. django-supervisor with this config file:

[unix_http_server]
file={{ settings.SUPERVISORD_SOCKET_FILE|default:"/tmp/supervisor.sock" }}
[program:celeryd]
command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py celery worker -l info -B
[program:gunicorn]
command=gunicorn --bind 127.0.0.1:{{ settings.GUNICORN_PORT }} \
   --workers 3 heitmgt.wsgi

2. bunch o’ pip requirements:

django-supervisor
django-celery
gunicorn

3. RabbitMQ/Erlang. Yay?

4. Startup script that basically runs

python manage.py supervisor -d
python manage.py supervisor stop all
python manage.py supervisor start all

What the heck is going on here?

There is a pretty good reason I didn’t start out with this setup. I had no idea what the heck these things did or why you would need them. Why is everything so complicated??

Here’s the short of it:

  • Gunicorn: A python program that’s happy to serve up wsgi applications.
  • Celery + RabbitMQ: A queuing/processing system that lets you send tasks to be done later. For example, I just started using Django-celery’s database-stored “periodic tasks” to replace cron for some maintenance. (To handle periodic tasks you need a CeleryBeat worker; the “-B” flag to celery worker does this.)
  • Django-supervisor: A wrapper around the program “supervisor” that lets you generate a config file from Django settings files. Supervisor seems to be a guardian for processes–it daemonizes foreground processes. There are also some tricks to this e.g. if DEBUG=True it is supposed to restart the processes.

The move to Gunicorn was mainly because I wanted to get rid of the apache instance and consolidate web management into supervisor. The Celery part, though, is a really big deal. It is extremely helpful/a paradigm shift to have both synchronous requests and asynchronous tasks in the same application.

All that said, I migrated from Webfaction’s provided django apps to a new “custom app listening on port” that’s Gunicorn–and it was much less painful than I thought it would be. Yay!


01
May 13

Django environment setup

I have been doing some programming in “Django,” which is awesome. Django is a Python-based web programming framework.

Over the course of the last couple of months, I’ve worked on my Django development infrastructure. I’m kind of a stickler for doing things the “right” way, and this is what I’ve decided is the right way so far. Here’s how my Django infrastructure is set up:

Directory layout

A “bashmark” and an Emacs bookmark point to the root of my Django tree, which is checked out from git (and which I maintain with git-flow and small time-duration feature branches).

  • Local (.gitignore’d) files:
    • .dir-locals.el: Sets pony-settings to use the virtualenv version of python, and virtualenv-workon and virtualenv-default-directory for Emacs virtualenv.
    • .pyflymakerc: Sets VIRTUALENV, PYLINT, TEST_RUNNER_COMMAND, IGNORE_CODES_PYLNT (‘E1101’), and TEST_RUNNER_FLAGS
    • .venv: Name of virtualenv
  • Makefile (see below), manage.py
  • app/ — where Django apps go, e.g. “app/hello/”. Each application may then have the following files (so far):
    • admin.py
    • ajax.py (used by dajaxice)
    • exceptions.py
    • forms.py
    • migrations/
    • models.py
    • templates/
    • tests.py
    • urls.py
    • views.py
  • bin/ — any command-line files that don’t cleanly fit into another directory
    • get_config.py — creepy program that spits out the identified Django config value, used to get Django database username et al for
    • pg_dump.sh — Shell script that creates a Postrgres dump file & can be added to cron w/o any output on success.
    • post.sh, post_local.sh — Shell scripts that are run after “make pulldown”. In prodcution, post_local.sh restarts the web server, uses curl to pull a test page, and reminds me to do other routine tasks associated with builds.
  • docs/ — “.text” files formatted via Markdown. A Makefile can compile these.
  • <<project>>/ — the Django project config directory
    • client_secrets.json, urls.py, wsgi.py
    • settings/ — Using this super-awesome __init__.py to let you have multiple settings files including a “common.py” plus an environment-specific one plus one called “local_settings.py” that doesn’t get checked in that has all your passwords in it
  • lib/ — (very simple) Pythony support libraries I’ve written. Currently includes:
    • json_shortcuts.py: to_json() function that correctly serializes datetime.datetime and datetime.date objects
    • middleware.py for custom middleware
    • nose_tweaks.py to make Nose shut up about South logging (plugin called “SilenceSouth“)
    • re_shortcuts.py for super common regexes
    • tests.py
    • time_shortcuts.py
    • unicode_csv.py for simple csv module wrapper that handles Unicode encodings
  • patch/ — diffs that I want to patch from that involve external stuff. Makefile can update these diffs. This is a crazy thing in and of itself and am not sure whether it would be helpful for others. Let’s just say that this helps my Django site look the same as my WordPress site.
  • requirements/ (see below for module list)
    • common.txt
    • dev.txt
    • test.txt
    • test-libs.txt (required by dev.txt and test.txt BUT NOT prod.txt)
    • prod.txt
    • unseen.py (hack to tell me what doesn’t show up in any requirements file that does show up when you run “pip freeze”)
  • static/
    • css/
    • js/
    • images/
  • templates/
    • base.html, {404,500}.html
  • uploads/
  • var/ — in .gitignore but this is where I keep database backups.

Makefiles

I tried using Fabric, and more power to you if you use it, but I didn’t need it. From my limited understanding, Fabric is good for helping you do stuff across systems. However, for someone with a background in cfengine Fabric is very frustrating to me. I wanted a way to automate parts of my build process but I didn’t need to do cross-machine deployments. So, I went back to Make. I don’t know why but I love Make.

Make targets:

  • pulldown — runs the “clean” target, then
 git pull
 git fetch --tags
 pip install -r requirements/${DJANGO_ENV}.txt
 python manage.py migrate
 # Needed for permissions to be created/updated:
 python manage.py syncdb --all
 python manage.py collectstatic --noinput
 cd bin && ./post.sh
 # (note: post.sh can then reboot the web server)
  • pushup — runs “git push –all”, “git push origin –tags”
  • clean — deletes .pyc files and runs “clean” for other Makefiles
  • virtualenv — builds the virtualenv you need for the project. (I just built this target and it’s pretty sweet.) This is totally a hack and not very robust. Just so you can get a feel for it (but FYI you can’t copy & paste this cleanly):
test ! -d ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 bash -i -c 'lsvirtualenv > /dev/null'
 curl -O http://pypi.python.org/packages/source/v/\
  virtualenv/virtualenv-${VIRTUALENV_VERSION}.tar.gz
 tar xvf virtualenv-${VIRTUALENV_VERSION}.tar.gz
 python virtualenv-${VIRTUALENV_VERSION}/virtualenv.py \
  --no-site-packages --setuptools \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}
 rm virtualenv-${VIRTUALENV_VERSION}.tar.gz
 rm -rf virtualenv-${VIRTUALENV_VERSION}
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && add2virtualenv .'
 bash -i -c 'workon ${VIRTUALENV_ALIAS} \
  && pip install -r requirements/${DJANGO_ENV}.txt'
 echo "export DJANGO_ENV='${DJANGO_ENV}'" >> \
  ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "export DJANGO_SETTINGS_MODULE='${DJANGO_PROJECT}.settings'" \
  >> ${VIRTUALENV_ROOTDIR}/${VIRTUALENV_ALIAS}/bin/postactivate
 echo "${VIRTUALENV_ALIAS}" > .venv

Modules/requirements

  • common.txt (for all environments) includes:
    • Django
    • South (totally the best thing since sliced bread and extremely the type of thing you don’t know you need until 3 months after you need it)
    • django-extensions (even better than South. runserver_plus, shell_plus, and graph_models are my faves)
    • pytz (see my “Time” post)
    • django-dajaxice (still not sure how I feel about this but it has been pretty useful so far)
    • Scrapy (is awesome)
    • coverage (really cool but I am not using it enough yet–along with django-nose, this will tell you thoroughly you’re testing)
  • dev.txt:
    • Werkzeug (O. M. G.)
    • pep8
    • pyflakes
    • pygraphviz
    • pylint
    • jedi (this helps with Emacs autocompleter stuff)
  • test-libs.txt (non-production envs):
    • django-nose
    • factory-boy (really good for testing)

Emacs

Using Aquamacs. ~/.emacs.d is checked into its own git repository currently shared via dropbox.

  • pony-mode
  • virtualenv
  • yasnippet
  • markdown-mode
  • jedi
  • python (vanilla 23.3 ‘python python-mode)
  • json-mode
  • flymake
  • django-nxhtml-mumamo-mode (worth every letter)

Things I Now Try To Do

  • Use factories rather than fixtures for tests (I thought factories were dumb until I started changing models & had to mess with raw .json — this kills the programmer)

25
Apr 13

Time

Recently I built a scheduling application for my work web site. It’s pretty awesome.

Anywho, I had to learn about time, and how it’s handled. Going into this, I always thought that if I were a systems administrator all over again I would have my machines all run in GMT to eliminate timezone problems. Now that I’ve learned, I know it’s now called “UTC” and it’s still a really really really really really good idea to do as much as you can in UTC and only present localized times only when the user sees them.

I learned about how time works in four tools: Postgres, Python, Django, and Javascript.

Postgres

Totally sweet support for time. You can say

generate_series(date(X at time zone 'UTC' - interval '120 minutes'),
                date(Y at time zone 'UTC - interval '120 minutes'),
                '1 day'::interval)

and get all the dates between X and Y. Everything about datetimes is super duper good.

Python

datetime library is pretty good. It is a little unfortunate (although understandable) that you need a non-Python library (usually pytz) to localize times. (This is because Python does not want to handle the political problems (literally) of keeping timezone definitions up to date.)

Notably, I built a small library called “time_shortcuts” to help with common time stuff:

  • utc_dt(str)–convert the string into a UTC-localized datetime object.
  • utc_now()–right now in UTC
  • xtimerange(start, end, step)–walk from start to end by step.
  • minute_diff(start, end)–returns a diff between start and end in minutes, not days. makes a bad assumption about how many minutes are in a day.

This way I could write utc_dt(“2013-04-25 18:00:00”) and know I was getting back a datetime timezone-aware object. This is much more helpful than it should be.

Django

Django gets a little angry if you feed it DateTimeFields that aren’t timezone-aware. Hence the above helper library. Other than that, it makes the Python <-> Postgres interface transparent, which is all you can ask for. You deal in Datetime objects, and it handles Postgres.

I have one notable instance where I call straight SQL (using a cursor) so that I can do some of the Postgres goodness described above.

Javascript

Good stuff: Javascript’s Date() object.

Bad stuff: Javascript’s Date() object.

It is nice to call

new Date().getTimezoneOffset()

and get the timezone diff in minutes. It is also nice to be able to call .getUTCHours().

It is unfortunate that I don’t know of a way to change the timezone that Javascript uses for a given session. This resulted in me creating a clunky <select id=”timezone”> object that has all the timezones and their minute offset and this becomes a weird interactive variable that redraws the time-related parts of the page when it’s changed.


25
Apr 13

Chrome setup

In the spirit of “basic things I do to make my life easier,” here’s how I have Chrome set up:

Extensions

  • AdBlock Plus: Blocks many ads.
  • Ghostery: Lets you control what JavaScript is executed as you browse, e.g. you can turn off some DoubleClick tracking.
  • OneTab: You click the button and all your open tabs go into OneTab. (They stop taking CPU cycles/RAM.) You can then re-open them as you see fit. I open tabs constantly, and OneTab is an easy way to “save” those sets of tabs if I’m not ready to look at them.
  • 1Password (installed via the 1Password application): super good application/extension to keep track of passwords and generate passwords. Note: once the extension’s installed, ⌘-\ will fill in passwords for you.

Bookmarks

The only bookmarks I have in Chrome are on the bookmarks bar, right below the location bar. I use Pinboard for all my other bookmarks.

  • “fxns” folder:
    • pinboard bookmarklet to bookmark a page
    • “read later” bookmarklet to bookmark in pinboard with “read later”
    • “pinboard random” to read a random pinboard unread bookmark
  • “tabs” folder: each of the tabs I want open whenever I’m in Chrome. (I have one window that’s my “work” window with my email, to-do list, etc as tabs.) Then you can right-click this folder and say “open all”. For my work account:
    • Gmail
    • Google Drive
    • Toodledo
    • Google Calendar
    • Google Voice (Note: this links directly to my “alt”/personal account, so I don’t have to click to change Google accounts. Each Google product handles this differently, but for Google Voice it’s https://www.google.com/voice/b/1#inbox where the number is a 0-based count of which account you want to open.)
    • Google Intranet for my business
    • HootSuite
  • “temp” folder: folder for stuff I’m going to reference a lot, but just for a little while. For example, when I’m doing a creative commons image search for a presentation then I might bookmark that page in this folder.
  • other web sites, bookmarked straight on the bookmarks bar, with NO TEXT. If you edit a bookmark and delete its text, Chrome will still show the icon. I do this for the Django documentation, for example–I just see the icon for Django (~¼” wide) rather than the icon + description (1+” wide). This way your bookmarks bar stays less cluttered.

Settings

I personally turn off form auto-fill and saved passwords, because I use 1Password for that. (1Password can store your “identities,” credit card info, etc, so you can have 1Password fill out these forms.)

Fancy things

  • Hush: separate program that detects when you are seeing a Hulu commercial and mutes your volume until the commercial is over