Top Tips from Linux-Fu

I have recently been giving a talk entitled “Linux-Fu for PHP Developers” (slides are here) and although it includes some really basic commands, even some quite experienced people have thanked me for particular bits that they picked up. I’m considering dropping some of the sections from this talk so here are the things people most tell me they learned:

cd –

If you pass a single minus as the argument to cd, you will change back into the directory you just came from. Ideal for those switching-between-two-deep-links situations.

pushd and popd

I didn’t include these in my talk formally but I mentioned them almost every time I gave it (or one of the attendees did!). I don’t actually use these commands, but they are like an extension to “cd -” – you can build up stacks of directories and then jump around between them. They sound pretty handy so do take a look.

ls -lrt

I got a lot of negative comments for including “cd” and “ls” in my talks – but I had quite a few people say that the -t (to sort by time) and the -r (to reserve the sort) switches were new to . I use these together for viewing which files are new in a directory, great for logfiles or upload directories.

Hope these are helpful to someone, let me know if you use these or have others you’d like to share!

Architecting Web Services – FOWA Tour

I spoke at the FOWA Tour event in Leeds yesterday, as their local speaker (they set aside a slot for each place they go). It was a great event, good atmosphere and I really enjoyed the both crowd and content. I can’t imagine what the logistics for something like this are like but it seemed like everything was going pretty smoothly.

My talk was “Architecting Web Services” – just a half-hour slot to give an overview of what services are and take a look at some things to bear in mind when designing them. The slides are on slideshare so feel free to take a look and let me know if you have any comments or questions.

I had a great time meeting up with so many friends – old and new – yesterday. Hope that I’ll see some of those again at future events or online. Anyone for Bar Camp Leeds?

Error Feedback for Web Services

I have been thinking, writing and speaking a little bit about web services recently, starting with a few thoughts on authorisation mechanisms for services. Today we’ll look at another really important aspect of authoring web services, and one feature that will definitely get used – error handling and feedback! Having clear and robust error handling in your service will help those trying to consume it immeasurably. Nothing is more annoying that impenetrable errors, unclear responses, or a service which accepts your input but then turns out not to have done what you expected. And I say that from experience.

Stacks of Errors

What’s more annoying than getting an error from a web service? Getting another error from the service every time you fix the first one!

Most services have a few steps of checking incoming variables. Checking that the user has supplied all required fields, and that all incoming fields are of the required format, and that the data they refer to does actually exist – there’s a lot going on here. Too many systems take fright at the first sight of an error, and return straight to the user like a child reporting a sibling’s misdeeds to a parent. I mean something along these lines:

if(!isset($_POST['username'])) {
  return 'username is missing!';
}
if(!isset($_POST['password'])) {
  return 'password is missing!';
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    return "unexpected field ".$key;
  }
}

What’s more useful is if the user can have a better overall view of everything that is going wrong, since often they might be caused by the same misunderstanding or perhaps be related to one another. So I’m considering code that looks more like this:

$messages = array();
$can_proceed = true;

if(!isset($_POST['username'])) {
  $messages[] = 'username is missing!';
  $can_proceed = false;
}
if(!isset($_POST['password'])) {
  $messages[] = 'password is missing!';
  $can_proceed = false;
}
foreach($_POST as $key => $value) {
  $expected_fields = array(
    "username",
    "password"
  );
  if(!in_array($key,$expected_fields)) {
    $messages[] = "unexpected field ".$key;
    $can_proceed = false;
  }
}

if(!$can_proceed) {
  return $messages;
}

The nice thing about something like this is you’ll see a series of messages where there are problems – so when you mis-spell a field name, you’ll see the “missing field” message for a field you know you are sending, but you’ll also see the “unexpected field” message and hopefully that will make it easier to spot what’s gone amiss.

Error format

Its tempting to return error information in a completely different format, after all it is quite different from the request that probably would have been returned from a correct request. Some web service protocols dictate how errors should be sent – SOAP has the soap-error response, for example. But for something where we have more control, such as an RPC style or REST service, we can choose. Usually I think its appropriate to return an appropriate status code (for REST) or wrapper (for RPC) and then include the error information in the same format as the response would have been. This is mostly for ease of consuming the response, saving clients from having to parse an additional format!

Approaching Errors

Having malformed input to services is inevitable – through misunderstandings, typos, and of course rubbish input by users. Making sure that all these eventualities are gracefully caught and information returned to the user means that the user stands a much better chance of being able to interact successfully. If only the success case works, but the service either doesn’t respond, returns nonsense (or worse, misleading information!), or appears to work but actually hasn’t, your users won’t hang out for long to work out why.

I’ve covered some really basic ideas here but I’m sure there are plenty of other nice ways to help guide users – feel free to add comments for the features you implement in your systems to help users when things aren’t going quite right!

By giving more information to users, it becomes much easier for them to develop against your service quickly and easily – and its not much more to add on the service side.

Speaking at FOWA Tour, Leeds, May 28th

I’m surprised and delighted to announce that I will be one of the speakers at the rather excellent FOWA Tour even in Leeds on 28th May (two weeks today!). The day is like a roadshow version of the famous FOWA conferences, with a morning of (free!) cloud workshops and an afternoon of excellent speakers. I had been hoping to attend this event anyway so to get the invite to speak is very exciting indeed!

My talk is entitled “Architecting Web Services” and will take a look at the types of service available, and identifying functions which would be useful as a service. I’ll run through how decisions about service and format can be taken and we’ll look at some examples of these while we’re at it. I’ve been building a few services lately for one thing and another and I’m pretty excited about the whole topic! Luckily I only have a half-hour on stage but rest assured I’ll probably still be talking about it over drinks later in the day!

Anyone going? Thinking of going? Let me know and make sure to come and say hi on the day :)

Leeds GirlGeek Dinner 27th May 2009

I’m delighted to hear that the date has been set for the next Leeds GirlGeek Dinner – Wednesday May 27th 2009! I’ve been really enjoying this series of events and always meet some really interesting ladies when I attend these events. The idea is to get geek girls together for some talks (and there have been some really good ones in Leeds), some food, some drinks and some networking – men are allowed but only as the invited guest of one of the girlgeeks. This enables us to keep the balance and actually makes for a brilliant atmosphere. Its not often the men in the room are horribly outnumbered :)

This month we have Reshma Sohoni from SeedCamp and Jennifer O’Grady from Democracy PR – and I can’t wait to hear both these women speak and hear their stories. The details are all on the website but in short the event starts at 6pm, its at The Loft Leeds, and I hope I’ll see you there!

Presenting Under Linux: xrandr

This is a mini-primer on using xrandr to configure a second screen under Linux, but first I’ll start with some background.

I’ve been a linux user for a few years now but when I started working for Ibuildings I started giving presentations, either as a speaker or when delivering training – and for a long time I used to boot into Windows for those, using a VM if I needed to code.

The main reasons I wasn’t comfortable using Linux to present were:

  • I have Powerpoint and I use the presenter view a lot
  • Linux doesn’t automatically find and configure monitors like Windows can.

Since Ubuntu Intrepid Ibex (8.10), its been possible to reconfigure screen in Linux without restarting X. But I was kind of scared of xrandr and I still liked the Powerpoint presenter view so I just carried on with windows. Then I realised that Open Office had also released a presenter view and I gave it a try. With one obstacle removed it was time to learn to work xrandr, which means I can also work external monitors without fiddling with xorg.conf, very useful. I’m now pretty confident with it so here’s the benefit of my wisdom*

xrandr

The first thing to do is figure out what you have and what your options are. So plug in the second screen and run xrandr with no arguments. On my system I get something like this:

Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2560 x 1024
VGA connected 1024x768+0+0 (normal left inverted right x axis y axis) 338mm x 270mm
   1280x1024      60.0 +   75.0
   1024x768       75.0*    60.0
   800x600        75.0     60.3
   640x480        75.0     59.9
   720x400        70.1
LVDS connected 1024x768+0+0 (normal left inverted right x axis y axis) 304mm x 190mm
   1280x800       60.0 +   50.0
   1024x768       85.0     75.0*    70.1     60.0
   832x624        74.6
   800x600        85.1     72.2     75.0     60.3     56.2
   640x480        85.0     72.8     75.0     59.9
   720x400        85.0
   640x400        85.1
   640x350        85.1
HDMI-1 disconnected (normal left inverted right x axis y axis)
HDMI-2 disconnected (normal left inverted right x axis y axis)

This shows VGA (your external output) and LVDS (the laptop screen) and all the various modes they support. I sometimes see output under HDMI-1, and this seems to cause problems so I always turn off this output by running:

xrandr --output HDMI-1 --off

Next I send a command to set the resolutions on both screens. For projectors I tend to go with 800×600 as this almost always works and it means the text on my laptop screen is then so large I can see it from several feet away!

xrandr --output LVDS --mode 800x600 --output VGA --mode 800x600

Its perfectly fine to have different resolutions on both machines but bear in mind that X has one desktop and will show you sections of it on each monitor so if they’re different sizes you sometimes get either bits missing off one monitor or one of the outputs looks stretched. I don’t really have any general advice with this other than get the projector in the right ratio and then fiddle with the available modes to see what works for you.

To set the second screen to be different from the first (i.e. tiled rather than mirrored), just tell xrandr where you want it.

xrandr --output LVDS --mode 1280x800 --output VGA --mode 1024x768 --right-of LVDS

Again you’ll get sections of a single desktop, not two separate areas, so things might look odd if they are on both screens. For me this command has worked really well since I got to grips with it in the last few months, and I’ve plugged various machines in to various external monitors and given a few different talks. I’m sure there is a lot more to know about xrandr but I haven’t needed it yet – hope this is helpful! If you have any experiences with this or anything to add, just drop a line into the comments form.

* I’m not an expert by any means, but the above works for me on Ubuntu with my Thinkpad T400 and my Acer Aspire One.

Screen-Profiles – Improved Screen in Ubuntu Jaunty

I recently upgraded my kubuntu installation to 9.04 “Jaunty Jackalope”. It turns out that they have added some very cool features to screen in this version. If you don’t use screen its its worth finding out more, certainly its a tool I now can’t live without.

The first thing to say is that if you have an existing .screenrc file, you don’t get any notification or prompt about the new features of screen, I was lucky and fell over the information. To get to the features:

  • if you have a .screenrc, rename it for a moment
  • aptitude install screen-profiles screen-profiles-extras
  • run screen

EDIT: From Karmic Koala (9.10) and later these packages are now byobu and byobu-extras – they work as outlined here however, just the package names changed.

The first thing you’ll notice is that there’s a whole lot more stuff going on at the bottom of the screen. I have had a custom .screenrc file which gave me tabs for a while, but now they’re part of the standard setup.

If you look in the bottom right corner you’ll see there’s an “F9 Menu” – yep, screen responds to a whole bunch of function keys! I’ll keep my traditional ctrl+a mapping I think, but you can reconfigure this to your hearts content. And look at the menu itself:

You can change the colour themes (more on that another day, I’m still playing with the settings) and you can also turn on and off a load of different widgets, basically lots of different system information. I’ve got most of it turned off on my laptop (since I already have a battery meter etc) but for a server this could be really useful. Examples:

I’m really happy to see Canonical improving this particular tool, and I wanted to mention it purely because existing users of screen won’t see the new functionality by default which seems like a bit of an oversight. If you’re a screen user, upgrade to Jaunty, try it out, and leave a comment on how it works out for you.

Kubuntu Freezing on Splash Screen after Jaunty Upgrade

Today I upgraded a bunch of machines to the latest Kubuntu offering, version 9.04, or “Jaunty Jackalope”. Well, it was all going a bit too well and when I came to upgrade my main work machine, a thinkpad T400, I ran into problems. There are a couple more kinks but I had one particular error which I wanted to blog about. When I upgraded and then restarted Kubuntu, it froze on the splash screen after showing three icons (up to the globe).

The problem was that the files in the ~/.ICEauthority directory were owned by root so couldn’t be written to by my user. I followed the instructions I found on this post where someone had exactly the same problem, and just chowning the whole directory to my user fixed the problem and allowed me to boot Kubuntu.

Hopefully if anyone else has this problem they’ll find either this post or the other one and also find the easy fix! If this works for you, leave a comment.

Auth Mechanisms for Web Services

Having been involved in quite a few service-related activities in the last year or so, I’ve been having a few thoughts about what I’ve learned from this and what decisions I make when designing a service. Hopefully there will be a few of these posts – but to start with, I’m considering the options for authorising users.

Quite a lot of services don’t require any authentication at all, similar to quite a lot of the web. In either setting, the information is there for users to consume when they want. However the difference comes when services start doing more than making data available. If changes can be effected by the service, then we need to identify who is requesting the change.

Traditional websites use a username and password, and we can do exactly the same thing here. Services work on a series of discrete requests and its common to require that the username and password be supplied with every request. However for high-load services or where a particularly fast response time is needed, we can use something similar to sessions, where the user first has to authenticate and is given a token. On subsequent requests they supply the token and we wave them through without requiring their credentials again.

There are a number of considerations involved in deciding whether this setup can work for a particular application:

  • Does it take time to authenticate? For example is there an external system to wait for or lots of user information to retrieve?
  • How guessable is the token? Any kind of reasonable length hashing will help you here. I tend to use salted md5 tokens*.
  • How long will the token be valid for? If interaction with the service is likely to be a burst of related requests, you might allow validity for 30 minutes for example.
  • Will you require other identifying information as well as the token? For example you might require that the user also supply their username, which would have to match the token. I’ve also seen systems which only accept tokens from the same user ip address as the user’s original authentication call came from.

Also think about storing these tokens. They can go in their own table along with the information you want to use frequently – this is the same idea as storing information about a user in a session, for example. So user id, maybe display name plus the token itself, some information about when it was created or when it expires, and anything else that will be needed to check the token’s validity. With this information being independent and just used to verify the user, there is also the option of storing this in an alternative, faster, mechanism such as memcache.

This isn’t by any means everything there is to think of, but just some ideas of things to consider when designing a service.

* I blogged about salting md5s in PHP recently, if you are interested

Quiet Diff

I recently saw a problem that we were having difficulty replicating, despite assurances that both the code base we were replicating on and the one that exhibited the error were identical. They are large codebases and when I got copies of them both I tried to check for differences:

diff -ur dirA dirB

The result was large and messy and included a lot of .svn files (long story). So to get an idea of how many files had differences I ran diff with -q for Quiet. This just outputs one line per changed file and also a line for if a file only exists in one or other directory. I then used grep to ignore any lines with .svn in them, and finally passed the whole lot to wc (for Word Count) to tell me how many lines there are.

diff -urq dirA dirB | grep -v .svn | wc -l

If you get a number greater than zero, your codebases are not identical and you have discovered why your fault is “intermittent”.