Silencing Curl’s Progress Output

I seem to have been writing a lot of shell scripts to do things with curl lately, and the main difference with piping curl’s output to somewhere rather than just looking at it on the screen is that curl will then start outputting a whole bunch of progress information that you don’t usually see. This can be frustrating if you wanted the same output as it viewed on the command line, or (as in my case) just wanted to grep for something in the header output.

To silence the additional output, use the -s switch – but be aware that this will also silence any error messages as well! Something like this:

curl -s http://beta.lornajane.net | grep PHP

This will enable you pass into grep just the output you would usually see on your terminal. Hope this is useful to someone!

Cookies and Curl

curl is the C URL library – its a command-line tool for making web requests, with libraries available in many languages. Personally I prefer to use it from the command line and recently I have been using it with cookies for a web services which set a cookie at login and then needed this to be supplied on all subsequent requests. This turned out to be really simple so I thought I’d put some notes down on how to do it.

Storing Cookies in a Jar

Quite enchantingly, its traditional to call a cookie storage a “cookie jar” which makes a lot of sense when you think about it! To do this with curl, use the -c switch:

curl -c lj.txt http://www.lornajane.net

If you now examine the resulting file, lj.txt, you’ll see that it contains all the details of the cookie. You can edit this if you want to (health warning: only do this if you know what you are doing! If you need to test something though, its useful), and then submit the next request with that cookie attached – exactly as your browser would.

Making Requests with Cookies

To make the next request with the cookie, simple replace the -c with a -b to dip into the cookie jar and sent all relevant cookies for this domain with your request, like this:

curl -b lj.txt http://www.lornajane.net

As I say, I was using this with a web service, where it made no sense to use a browser as I also needed to pass data with the requests. You might also like to refer to my curl cheat sheet previous post.

New Camera: Fuji Finepix F70

Recently I’ve had a new gadget in my life – a Fuji Finepix F72, which is really an F70 but exclusive to Jessop’s in a different colour (I have no idea why retailers do this but hey, it means mine is shiny black).

The camera is a birthday/christmas gift from my parents, since I’m taking increasing numbers of photos these days and our 3-year-old Fuji Finepix F650 is on its last legs! I have been looking around and although there are a few other things in the same kind of market, I read an astonishingly detailed review of the F70 and, considering how much I had liked the previous fuji, decided that this would be a good bet.

Its smaller than the last one while having a lot more resolution, I’m not really a camera geek but it does some funky things that I quite like. Even left entirely in auto mode, the photos are much better quality – especially in low light which was always an issue for the old camera! The scene presets are there and useful as ever. It has some strange settings where it takes multiple photos of the same thing to provide more detail – for example the mode which is pretending to be shallow depth of field takes a bunch of photos and then sticks them together in software, with interesting results:

Profocus Seagulls

I’m still getting to grips with the different settings, and am attempting to join in with the daily shoot which is an exercise in taking a new photographic assignment every day. I’m running at less than 50% success rate but I am taking more photos than I would have otherwise so its good in any event I think! One thing I really want to crack is taking recognisable photos of people … so far the only person I seem to do well with is my toddler niece (perhaps because she doesn’t fear the camera the way the adults do?)

Natalie in the crate

So, thanks mum and dad – and if you’re interested in following my photographic adventures, my photos are on flickr.

Sound Issues with Kubuntu Karmic Koala

Since upgrading my work machine to karmic koala, I’ve noticed that my sound had stopped working. There were some broken packages in aptitude and Skype knew there was a problem as it notified me when I tried to make a call. I saw some issues reported with karmic, notably this one, so I uninstalled pulseaudio

sudo aptitude remove pulseaudio

When I restarted Skype, everything seemed to work as expected – for reference I have a thinkpad T400, if you’re having the same issues, then hopefully this will help!

Email As a Means of Social Communication

Recently I’ve been spending a lot of time away from home, and busy with clients within office hours, which means I’m not available via my usual real-time communication mechanisms. This has given me the opportunity to enjoy email as a means of staying in touch with people that I would more usually chat with on IRC or some form of instant messaging.

When I write an email, I’m usually (mostly) doing that one thing, and the result is quite joined-up. Anyone who has had the experience of trying to carry on a conversation with me over messaging when I’m doing multiple things knows that I can easily lose the thread of an explanation and I often find I have to read back the logs of a conversation to be able to keep up.

When someone takes the time to write an email, the reader can’t interrupt or divert the writer and so the writer is able to express their points at their own pace. Email is also easier to re-read and reformat, if you are the kind of writer that doesn’t necessarily think of everything that needs to be said in quite the order it ought to be said.

When someone is emailing asking me to do something, I won’t lose an email (you’d be surprised how often I lose things people ask me to do on IRC, its on a par with asking me in the pub). Admittedly, my inbox can be several months deep but when I have time to attend to your query, I will – and that goes for social correspondence as well.

Finally, and most importantly, it enables me to stay in touch with people whose committments don’t fit around mine in terms of online availability due to work, life, or timezones.

I’m quite enjoying a slightly more formal exchange of the written word with those friends who are emailing me while I’m on these little jaunts – you know who you are and I don’t enjoy being away from home so those little messages are welcome, thanks :) And for everyone else – next time you are lurking online looking out for someone, take the time to write an email instead – and let me know how it works for you.

Burndown Charts

Disclaimer: I am not an expert on this topic and have never studied it formally. This is purely an explanation of how I’ve seen these charts used within Ibuildings and included in my talk Passing the Joel Test in the PHP World which triggered lots of questions and tweeting.

Burndown charts are a way of tracking how much work was estimated to be remaining at a given point in time, and we track this over time to give us some idea of the rate of change. We start with some estimates, then we track how much time is left at regular intervals, I do this daily, because its a sensible unit but there are no rules. Its much easier to show this rather than tell it so here is an example – we’re making a dolls house.


(yes, this is a photo of me. This was my fourth birthday and the dolls house was a home-made gift from my parents – I still have it)

Task List and Estimates

We need some numbers to start with, and I don’t mean looking at a task and going “a week and a half … ish?”! Tasks need to be in small enough increments that we can safely estimate them. By “safely” I mean realistically and hopefully accurately enough that you won’t find your project manager following you to your car at night with a pitchfork in his hand. For our dolls house, the table looks something like this:

Task Estimate
Assemble pre-built frame 2
Paint exterior 2
Paint windowframes 1
Decorate rooms 27
Fit carpets 5
Add furniture 1

As you can see, we’ve got a large element listed in the middle of the table – 27 hours is too big to be a single estimate, so we’ll break it down further.

Task Estimate
Assemble pre-built frame 2
Paint exterior 2
Paint windowframes 1
Decorate bedrooms 8
Decorate kitchen 7
Decorate bathroom 5
Decorate reception rooms 7
Fit carpets 5
Add furniture 1

Charting Estimate Remaining Over Time

The aim of this is to chart how the estimated outstanding work changes over time. So we start with the total estimates in place – I’ve added a week’s worth of days, just because its easy to fit onto the page, but your chart might have dates rather than days and go on for longer. Each day copies the number of hours of the previous day as its initial value – we’ll edit them when they change.

On Monday, all the work is still to be done, so we start with a chart that looks like this:

Task Estimate Mon Tue Wed Thu Fri
Assemble pre-built frame 2 2 2 2 2 2
Paint exterior 2 2 2 2 2 2
Paint windowframes 1 1 1 1 1 1
Decorate bedrooms 8 8 8 8 8 8
Decorate kitchen 7 7 7 7 7 7
Decorate bathroom 5 5 5 5 5 5
Decorate reception rooms 7 7 7 7 7 7
Fit carpets 5 5 5 5 5 5
Add furniture 1 1 1 1 1 1

To make the chart is as simple as adding a “total” field to each of the day columns and turning that into a bar chart:

Let’s say on Monday we get the first two tasks done – we set the “Monday” column for these tasks to zero because there is nothing outstanding. Now our table and chart look like this:

Task Estimate Mon Tue Wed Thu Fri
Assemble pre-built frame 2 0 0 0 0 0
Paint exterior 2 0 0 0 0 0
Paint windowframes 1 1 1 1 1 1
Decorate bedrooms 8 8 8 8 8 8
Decorate kitchen 7 7 7 7 7 7
Decorate bathroom 5 5 5 5 5 5
Decorate reception rooms 7 7 7 7 7 7
Fit carpets 5 5 5 5 5 5
Add furniture 1 1 1 1 1 1

So each time some work is done which causes the estimated remaining work to be reduced, we update the figures in the table.

During the Main Phase

Imagine we’re partway through and its Thursday, let’s check in on our dolls house project:

Task Estimate Mon Tue Wed Thu Fri
Assemble pre-built frame 2 0 0 0 0 0
Paint exterior 2 0 0 0 0 0
Paint windowframes 1 1 1 1 1 1
Decorate bedrooms 8 8 8 6 1 1
Decorate kitchen 7 7 7 7 7 7
Decorate bathroom 5 5 5 5 2 2
Decorate reception rooms 7 7 3 3 3 3
Fit carpets 5 5 5 5 5 5
Add furniture 1 1 1 1 1 1

At this point, its pretty obvious that the project isn’t going well. The curve on the chart isn’t steep enough to be able to reach zero by Friday and without having to interrogate developers (who of course are always 90% finished), we can see that overall, things aren’t great. Project managers may know that there is one blocker which is causing the problem but if you didn’t know … then you do now :)

A closer look at the numbers against the tasks show that we started the reception rooms and the bathroom but neither are finished, the bedroom we’ve worked on for the last couple of days and its almost there … you get the picture.

The Aim of the Game

The ideal burndown chart tends towards zero at the point at which the project needs to be finished … but working this way gives a nice graphical overview of whether thing are heading in that direction or not. Flat lines are generally the sign of something bad, and progress is really visible even when its spread across many tasks or many developers. The chart could have a column for who owns each task, and for extra points add an “actual” column so you can compare how the team performs against its estimates (this is “project velocity”).

This post was really only intended to show things as I see and use them – if this helps you then great. If you want the “official” version then I can only suggest you get a book on scrum and find out how to do it from someone much more qualified than me :) If you’re using this or something similar, then I’d love to hear how it works out for you and what you do differently – leave comments and tell me!

Speaking at Bradford LUG

Next week I’m taking the plunge and attending a LUG (Linux User Group) meeting for the first time when I attend the Bradford LUG meeting on Wednesday as a speaker. I’ll be giving my talk “Working With Web Services” which I’ll give at PHP Barcelona a mere 36 hours later (Wednesday is the dry-run, let’s hope it goes well!). I’m excited about this topic and looking forward to meeting a new group of geeks – if you are in the area then I hope you’ll pop in and join us.

Portugal and The Internet Cafe

Things may have seemed a bit quiet around here lately – that’s because I spent last week in the Algarve in southern Portugal. My sister is working in a hotel there as head nanny, I think she’s spent 8 days of 2009 in the UK so this is about the only way to get to visit her, a beach holiday!

Despite the holiday I still had a lot on my to-do list so I took my netbook with me, and got my sister to show me the local internet cafe – it was a great place and worthy of mention, so thanks very much to Hugo Beaty in Praia Da Luz for a stable connection and good coffee :) It was good enough to work from, I almost wished I’d gone for longer and not taken holiday (I telecommute, nobody would notice!)

If anyone is interested, there are a few photos in a flickr set of the trip.

Working at The Hub, Bristol

A couple of weeks ago I had some friends coming to the UK for a little while, but they were staying in Bristol and I live in Leeds. My weekends were pretty full so I wasn’t sure if I would get to see them – but since I telecommute I can work from anywhere with a decent connection and some quiet space. A friend recommended The Hub in Bristol, so I worked from there for the day.

Its a great place, I haven’t been to Bristol before and the views are rather good since the space is on the top floor. There was plenty of desk space and both wired and wireless connections available and they also have meeting rooms which were getting some good use the day I was there. The atmosphere was friendly and everyone had a smile for me, even though I was a stranger and only there for the day. The office is part-shared by sustrans so there were other nice ethical touches, like fairtrade tea and coffee and a good selection of recycling options.

I had a great day and will be looking out for opportunities to use The Hub again – either in Bristol or elsewhere. Thanks!

Lame Excuses for Avoiding Conferences

At the moment I am getting to quite a few conferences, as a speaker, as an organiser, and sometimes as a plain old attendee. I get so much from these opportunities to learn from experts in their various fields, meet people in the flesh whose blogs I read or whom I know from IRC. I also hugely value the opportunity to socialse and build personal connections, and to be a bystander for technical conversations between leaders where I understand most of the words but can barely follow the flow. I can quite appreciate that different people come to conferences for different reasons, but I cannot accept that people actively avoid conferences because they think its not for them – and the reasons for this, from people who have never been to a conference, are wild and varied. Most are based on misconceptions and I’d like to take the time to examine some of these.

I won’t know anyone

This situation will persist until you go to a conference and meet some people! Then, you’ll know some people at the next event. When I went to IPC in Frankfurt in 2007, I knew nobody but while I was there I met Derick, Sebastian and Zoe … and these three are now conference friends wherever I go! Lots of people at conferences are there on their own and will be happy to chat to you and find out who you are.

Its too expensive

While I’m lucky enough to have the support of my employers, Ibuildings, to send me to at least a couple of conferences per year, I’ve worked for plenty of other organisations that didn’t invest in their people. I think this is unforgivable but the reality is there are plenty of us in this situation. If you are paying your own way to these events I can appreciate that $1100 (~700 GBP) plus international flights plus a week in a hotel in Silicon Valley can seem pretty expensive if you want to get to ZendCon. But there are cheaper and closer conferences are available for most of us – so do your homework and get to something you can afford, even if you don’t do it every year. I’ve yet to get to a conference where the cost outweighed the benefit so in my view this excuse is invalid.

My employer won’t pay

No, well, see previous point. This is true for plenty of people and while I don’t have any numbers on people paying their own way – they do exist and they almost invariably move on to work for employers that do invest in their future. Do this for you, not for them.

I might have to talk to people/strangers

This is the excuse I hear the most often, or a variation on this. Actually you don’t have to talk to anyone if you don’t want to. I went to an event last year and introduced myself to a guy who said “Hi, I’m James. I don’t have any social skills” and proceeded to say nothing further (his name may or may not have been James, I can’t really remember). To be honest I didn’t really think anything of it. Conferences are firstly about the technical content so if you want to come and get the technical sessions and then disappear again – that is your call. I can’t agree this is a good idea but there is absolutely no pressure to be the life and soul of the party, and in fact if you want to sit in the corner and mutter to yourself that is also fine … we’re all geeks after all! Nobody will judge you, in fact if you don’t talk to anyone probably nobody will notice you – just COME and you might be surprised :)

I haven’t been to a conference

Why not? Pick an event you like the sound of, join in the preconference hype (more about this in my post about making friends at a conference) and see how it goes. If you hate it, then don’t go to another. But don’t stay home immobilised by lack of experience, you’re missing out :)

Avoiding Conferences

Getting to a conference costs time, money and effort and if you don’t want to invest any of those things in your professional development then I respect your decision. However if you think you’d like to attend something, but you don’t know what to expect or you have concerns about what is expected of you, then try to put those fears aside and dive in! I think I’ve covered the things I hear most often – what excuses do you hear from conference-avoiders?