Tag Archives: tech

10 Jun
2010

Poor Man’s Caching

Describing something as a “poor man’s” version of the thing is to say its a bit of a cheap or short-cutting version. I thought I’d share some VERY poor man’s caching that I came up with the other day.

To set the scene – I’m hosting the Dutch PHP Conference this week and there’s a LOT of information to keep track of. Between tweets using the hashtag, people tweeting either my username or any of the conference twitter accounts, the schedule itself which will change as we’re having a no-prior-scheduling uncon, and the event and talk-related comments coming in via joind.in itself, its quite a bit of stuff to track. Add into the mix the fact that my iphone’s data package will be too expensive to use in NL and although I will have a phone with a data package it wont’ be mine and there’s no guarantee exactly what it will be. Oh and conference wireless, which last year wasn’t bad at DPC but you have to assume there’s a bottleneck.

So I figured that since I have a fast production server with lots of spare resource, it can do the hard work on processing the feeds, and just serve me a basic page with everything on and some internal hyperlinks to get around. So I wrote this monster, which runs a bunch of twitter searches and pulls in schedule and comments from joind.in, and that was good.

Except, it does take quite a long time to run (well, a few seconds, but that’s too long in my book) so I thought about caching each result set in memcache, and writing a cron job to repopulate those regularly, so every time I hit the script, the cache would be warm. Then I realised I was overcomplicating the matter, and simply wrote a cron job to run my php script and pipe the output to a static HTML file every minute! Whenever I hit the page, I’ll get the latest version. It doesn’t scale but it doesn’t need to, its only for me to use for a few days! Here’s the cron job:

* * * * * php -f /path/to/generate.php > /path/to/index.html

Job done! (the actual page I made is/ here if anyone is inquisitive)

12 May
2010

A “new” Netbook for Conferences

A couple of years ago, I attended ZendCon for the first time. At that time, the only laptop I had was a work machine weighing 5kg, and I was staying in a different hotel from the main conference – so I hastily bought pretty much the first non-ee netbook on the market (which was released about a week before my trip) and took that with me to the event. There were lots of limitations of it but I didn’t have a whole lot of choice … since then I’ve improved it significantly!

More than 512 MB RAM

I’ve now had the machine more than 18 months and I’ve been using it for events and keeping it by the bed pretty much ever since. However this year I’m speaking at TEK-X in Chicago and found myself wondering if I should get a new little machine to take with me. A few months back I upgraded the RAM in the machine (it had 512 MB originally!), and I came to the conclusion that actually, I didn’t need to splash out for a new netbook since with a bit more memory, its pretty usable.

Bigger Battery Life

Instead of replacing the machine, I bought the extended battery for the aspireone, taking me from about 2 hours to about 6 (I think, haven’t tried this battery life in anger yet). Its chunky, but not offensive:

DSCF1476

It is quite a bit heavier than the old one – my original post about getting the aspireone weighed it with its original battery at 971g and with the extended battery, its 1274g, which is still only half the weight of my current work laptop.

DSCF1480

Kubuntu Netbook Edition

Reading back to that original post about the netbook it was pretty obvious that I was hating that operating system. A couple of years ago, the support for aspireone in Ubuntu Netbook Remix became usable, and I installed that. Its been fabulous and I had no intention to do anything more than upgrade … until someone at OggCamp showed me they had the Kubuntu Netbook edition on their netbook.

DSCF1486

It is shiny and blue, and since I use KDE on my laptop I couldn’t resist and I upgraded it with a week to go before I take it as my only machine as a speaker at a conference!

In Summary

I haven’t spent a lot of money on the upgrades, the battery was about 50 GBP and the RAM came out of something else. I did have a large capacity SD card in the expansion slot since I only have 8GB hard drive. Then there was a problem with the machine suspending with the expansion slot in use and at around the same time I got a camera that takes SD … so I put my big card into there instead :) I don’t keep data on the netbook so its not an issue for me. I’ll be flying to Chicago for Tek in a few days, and presenting off my netbook as usual. For a budget machine with a little bit of love and attention added in, “Merry” (short for Merope) the netbook has been quite a fabulous investment!

10 May
2010

Podcast: How and Why to Become a Speaker

This is a podcast version of my rant-in-the-bar advice to anyone thinking about speaking, or wondering how to begin. Personally I think many more people could be sharing their expertise at events than actually do so, and I would really like anyone who wants to get involved to have a starting point. So if that’s you, and you have a spare ten minutes to listen to my thoughts on the subject, then the mp3 is here.

Let me know what you think, and if you have any other advice you’d give to someone who isn’t yet speaking.

06 May
2010

Thoughts on OggCamp in Liverpool

Last weekend I went to Liverpool (my first visit there!) to attend OggCamp for the second year in a row. This isn’t a part of the software community I normally interact with, but I accidentally went to a LUGRadio event once a couple of years ago and have been hooked ever since!! OggCamp went to 2 days for the first time this year and was a triumph of organisation, with a great venue and a wonderful feel to the whole event. I was particularly impressed (as an event organiser myself) that for an event with no registration, they had the right number of chairs, the venue was the right size, but in fact they didn’t have that information so a very well done to whoever made that call :)

I spoke again this year, it seemed to me like since there was voting on the talks, if my talk wouldn’t be a good fit then I wouldn’t get a slot! So I did put in a talk about source control; a topic that I’ll be speaking on in a couple of weeks at TEK-X in Chicago and one that I still feel a little bit wobbly about. Taking the main stage to give a hastily-reduced version of the talk to such a well qualified audience and coming off the stage to feel like it went OK was an excellent preparation for my next speaking event and I got chatting to all sorts of people while I was there. I’m not really a linux nut but I do have ubuntu or kubuntu installed on everything, and it was great to hear a bit more about so many aspects of technology that I use but don’t know much about.

The event ended in the traditional live podcast with all the presenters of Linux Outlaws and the Ubuntu UK Podcast all on stage at once. This is always good fun and there were some laughs from the crowd as always.

OggCamp Live Podcast

All in all, a great event, and I’m very much looking forward to next year’s!

26 Apr
2010

Bzr-svn: Preserving Commits and Rebasing

Recently I’ve been using bzr-svn when I’m working with subversion repositories; its a bridge between a local bzr repository and a remote subversion one. I travel fairly frequently and the pain of being disconnected from the server, unable to diff or log, and then either unpicking everything I’ve done or sending a monster commit with the message “Things Lorna did in the last 2 days” is just tedious and annoying. Possibly more tedious and annoying for the rest of the team than me even, so a better solution is more than welcome!

Some time ago I blogged about starting to use bzr-svn but its taken me a while to get past those first steps. Having the repo locally is so fast to use, and the speedy diff and log functionality had me completely sold straight away. I’ve been mildly annoyed by bzr squashing my commits though, meaning that I’m still doing a single monster commit to subversion when I come back online after being gone for a few days.

Today, with some help from the very lovely people in the channel on freenode, I found out how to preserve my commit history when sending the changes back to subversion from bzr-svn. The bzr-svn user guide recommends a particular set of commands, but this includes merging your changes into the mirror of SVN. Even in a standard bzr branch, this would show in the logs as a single, combined commit.

Push, Don’t Merge

The key to retaining the commit history is to push changes from the branch, rather than merging into the trunk mirror. To make this clearer, I’ve shown an example. The SVN is checked out into lorna-trunk and I’m making changes in the lorna-branch directory, including adding the db directory and the hello.php file.

$ bzr commit -m "initialising database files and hello.php"
Committing to: /home/lorna/data/personal/publish/tek10/svn_vs_distributed/bzr-svn-example/lorna-repo/lorna-branch/
added db
added hello.php
added db/setup.sql
Committed revision 2.
$ bzr push ../lorna-trunk/
All changes applied successfully.
Pushed up to revision 2.

No matter how many times you commit, when you push, all your changes will be sent to subversion as individual commits in the same way. This will really help me next time I’m offline for a little while!

When There Are Changes In Subversion

Charmingly, while answering my questions to get me this far, the channel inhabitants then immediately explained to me that to handle changes in subversion, I’d have to rebase my branch before I could push my changes. This is because you can only push to a branch that is in sync with where yours was when you started making changes. If you try to push to subversion when there are changes in it, you will see this error:

bzr: ERROR: These branches have diverged.  See "bzr help diverged-branches" for more information.

All I needed to do was to run bzr update in the mirror, and then rebase my branch onto that. At first my system didn’t recognise the rebase command, but this was because I needed to install the bzr-rebase package (called bzr-rewrite in newer versions but bzr-rebase in mine, Ubuntu 9.10 Karmic Koala). Rebasing was easy to do and brings the changes from the repo into your working branch. Here are the commands and how the output looked for me (with shortened paths to make this more readable):

lorna@taygete:~/.../lorna-branch$ cd ../lorna-trunk/
lorna@taygete:~/.../lorna-trunk$ bzr update
+N  readme
All changes applied successfully.
Updated to revision 3.
lorna@taygete:~/.../lorna-trunk$ cd ../lorna-branch/
lorna@taygete:~/.../lorna-branch$ bzr rebase ../lorna-trunk/
All changes applied successfully.
Committing to: /home/lorna/.../lorna-branch/
modified hello.php
Committed revision 4.
lorna@taygete:~/.../lorna-branch$ bzr push ../lorna-trunk/
All changes applied successfully.
Pushed up to revision 4.

I haven’t had a lot of experience with using this in difficult situations with complex branching or conflicts yet, if you have any tips to add though, a comment would be excellent :)

22 Apr
2010

svn+ssh with specific identity file

Here is a quick howto on using a config file to associate an SSH connection with a particular identity file. When I ssh directly from the command line, I can specify the port, identity file to use, and a whole raft of other options. However these options aren’t available when checking out from subversion using svn+ssh, and I needed to specify a particular private key to use. These instructions are for my kubuntu 9.10 installation but should work on every flavour of *nix as far as I know.

I was trying to check out using something like this:

svn co svn+ssh://[email protected]/desired/path

To do this, I created this file: ~/.ssh/config and in it I put the following:

host lornajane
HostName svn.example.com
IdentifyFile /path/to/private_key

Now I update my checkout syntax to make use of my alias rather than specifying the host directly, like this:

svn co svn+ssh://lorna@lornajane/desired/path

SSH realises that it should pull the host information from your local config file, and uses your HostName, IdentityFile and any other settings you specify (there are many options, see this reference for more information. For example I often use this for port numbers if I’m tunnelling my SSH between places, the possibilities are endless.

Hope this helps, if you’ve got anything to add then I’d be delighted to see a comment from you.

19 Apr
2010

Book Review: Making it Big in Software

Recently the kind folks at Prentice Hall contacted me to ask if I’d like to read a new publication of theirs , “Making it Big in Software“. It sounded like something I’d be interested in and the copy duly arrived. My first impressions were of a wordy book, very much aimed at large organisations, or those working in academia. And although I’m 20-something myself, the advice in the first sections of the book is aimed more at students or recent graduates than, I felt , at me.

Moving on into the book I found more that applied to me, including some great advice on communicating in different settings and with different types of people. I was also interested to find some thoughts on debugging as a skill to be used instead of, or before, launching a built-in debugger. This makes sense to me because my first programming languages weren’t the kind that have IDEs or other advanced tools.

The chapter on time management starts with the words “if you only read one chapter in this book, this should probably be the one” – and I must say I agree. I consider myself to be organised but there were some great strategies outlined in that particular chapter. Also running through the book like a thread is a series of interviews with software luminaries, each of whom were asked mostly the same questions, including how they manage their time and work-life balance. So there were some real-life stories in there too – I was encouraged that almost all of the people interviewed said they didn’t think they found balance, because personally I know I haven’t.

The interviews provided a great structure on which to hang a book full of good advice. One chapter advises how to identify and avoid “career killers” – those dead-end projects that will hold up your progression. Later on the focus moves away from the big organisation that the author and interviewees have most experience of, to look at starting software companies and the skills and attitude needed for this. Certainly once I got past the first third of the book I was much more drawn in; maybe because that dealt with an early stage of career that I felt I had already mudddled through?

Overall this is quite a dense book to read over a short period of time, but an excellent one to keep around and dip into or read another chapter every few weeks perhaps. Certainly its one that will be going on my bookshelf for me to look back at and refresh my thoughts on various topics covered for some time to come.

24 Mar
2010

Ada Lovelace Day 2010: Mrs Maginnis

This is a post in celebration of Ada Lovelace Day, recognising women in science and technology. Last year I wrote about Kathy Sierra for Ada Lovelace Day; this year my subject is someone much closer to my own life.

I’d like to write about someone who influenced me greatly – in my choice of study and in my attitude to the gender imbalance in that area and in the industry in which I now work. Mrs Maginnis taught me advanced maths when I was 16-18, and making decisions about my next move. I was educated at an all-girls grammar school, which in some ways was an advantage. The experience of those all-female maths classes taught me a lot about how different women are when they meet their individual challenges in the company of one another.

The classes were very tough academically (as they should be!), and I did struggle during the course. At the start of the summer holiday, Mrs Maginnis handed me a textbook with a list of questions to try, and her phone number. All summer I wrestled those questions, with the occassional phone call for help. I went back to school, completed the year, and left the following summer with two A grades in A-level maths – undoubtedly because of the help and support I had been given by this one teacher. Those two A grades landed me my first choice of university place – I thanked her and off I went.

However, the strong influence of this character has stayed with me long since the memorable day I opened the results envelope. I was headed for a course in Electronic Engineering and Mrs Maginnis had been a Mechanical Engineer herself and I think had a good idea of what lay in store for me. The thing is – I was educated entirely with women … I genuinely did not know that there were subjects that girls just don’t do. So when I arrived at university with my academic grades from the all-girls school, I was confronted by a course that was 95% male and contained mostly people with grades at least as good as mine and usually including more practical subjects such as Electronics, which I hadn’t had the opportunity to study. So it was a struggle from that point of view.

The teacher who had, on paper, taught me maths, had in fact taught me so much more. Even now I sometimes remember stories, anecdotes, and advice that all drifted across the classroom along with the hyperbolic functions and calculus (which is a much more distant memory now). She had been an engineer herself, and through her stories she showed us all how it would be done, something on what to expect and some tips on how to handle it. Even now I don’t think I’ve had a stronger female role model for industry – and she managed all this from academia, when I was a teenager. So – I raise my glass to Mrs Maginnis. Thankyou for all you taught me.

23 Mar
2010

GETting RESTful collections – may I filter?

At work at Ibuildings recently, I’ve been teaching some classes on web services, and its a topic that I’ve spoken about once or twice at conferences. But something has always bothered me, so I find myself in the unusual position of blogging a question.

RESTful collections

So, when you are retrieving information from a RESTful service. You have two options: retrieve a specific resource, whose URL you know; or retrieve a collection, which may contain a list of resources. I’ve also seen some nice ways of filtering collections, by creating kind of “sub collections” or “views”, similar to what twitter does with the URL of lists, for example http://twitter.com/dpcon/speakers10 which is like a filtered list of twitter users.

What if I want to search and sort?

Is it RESTful to add GET parameters to a collection in order to add functionality such as filtering, sorting, or pagination? What I have in mind is a URL that looks something like this:

  • http://example.com/users?orderby=firstname&start=0
  • http://example.com/users?start=0&limit=25
  • http://example.com/users?active=1&orderby=join_date&limit=12

This is what I would do with a search results page in a web application, and I use the same approach to web services which works really well and I recommend it to everyone! But is it RESTful?

I am also wondering where OpenSearch would fit into the answer for all this, I only noticed it recently but the more I look at it the more I think it could be an interesting addition!

Thoughts, links for me to RTFM, and all other additions are welcome in the comments box :)

01 Mar
2010

Simple Database Patching Strategy

One problem that seems to need solving too often is how to keep databases in sync across different platforms, different developers working on a project, and deploying database changes along with code changes. There are lots of ways of approaching this, none of them are really excellent however and personally I tend to err on the side of simple being better. Fewer dependencies means a solution more likely to work on every platform (and no additional complications for the live platform). Usually this means patch files of some kind – here’s an outline of my usual approach. For the impatient:

  • add a table for meta data to the database, set a database version parameter to 0
  • export structure and any essential data (lookup tables, etc) into an sql file
  • for any changes, create a numbered patch file e.g. patch-1.sql, including the change and an update statement to bring the version meta data to match the patch file
  • store all of the above in source control
  • for bonus points, create another sql file with some nice sample data

Give the Database Some Version Awareness

Naming is completely changeable but mine looks something like this:

CREATE TABLE `meta_data` (entry varchar(255) primary key, value varchar(255));

INSERT INTO `meta_data` SET entry="version", value="0";

This new row will hold information about what patch version the database is up to. Every patch that is created will update this number so that it is possible to tell which patches have and have not been applied.

Create an Initial Data Structure

Firstly create a database directory for all these files to live in. This should be outside your web root but inside your source control project.

Take your database and dump just the structure of the tables using the –no-data switch to mysqldump like this:

mysqldump -u  -p  --no-data > create.sql

You will also want to populate tables which hold things like lookup values, country lists, that sort of thing, so that these are set up. People starting to develop with this project, or if the application needs to be deployed to somewhere new, can use this file as a starting point.

Create Patches for Changes

This is where the discipline element comes in – no direct edits on the database are allowed! Instead, write the SQL for your change and place it in the patch file, then run it against the database. If that sounds too much like hard work then copy/paste the SQL you use to make changes, or the SQL generated by whatever SQL tool you use, and place it in the file.

Every file must have its own incrementing version number in its file name, e.g. patch-1.sql, patch-2.sql etc. Within the file the version must also be updated to match, with a statement like:

UPDATE `meta_data` SET value="1" WHERE entry = "version";

Recommended Practice

Here are a few pointers on getting the most out of something like this:

  • Under no circumstances is it acceptable to edit a patch file that has been committed to source control. Someone might have run it already and you’ll confuse everyone completely.
  • Having matching undo scripts alongside the patches can be really useful in case a change gets deployed and needs to be undone.
  • Make a script to look up the database settings in the config file of your application, query the database for what version it is at, and run any outstanding scripts. This makes life much easier especially if you have large numbers of scripts (I’ve seen the patch numbers hit the hundreds)
  • A possible adaptation of this approach is to create patch files for use for updating a database, but to also update the install.sql file to make it correct at any point in time, this means a much shorter and easier setup time for new deployements/developers. The version awareness works in the same way regardless
  • Creating a sample database which creates a few records in each table can really help for development purposes – its quicker for people to get set up and attain a working system that they can make progress with.

I’m sure there are many improvements and variations on this theme of simple database patching, leave a comment and let me know what works for you and why!