The worst thing you can do is find some random, underqualified person who represents the demographic you want to include, and put them on the stage. Although gender is often the issue we hear most about, the same applies to anyone who isn't a young, white male; it's just that gender is easier to see and talk about than either age, ethnicity, sexual orientation, or anything else, and also since I'm a young, white female, it's the only aspect I can comment on.Women are in such a minority that they are, almost by definition, representative (see http://xkcd.com/385). Anyone who sees your randomly-selected woman speak will simply go away thinking that women aren't really good at speaking.If you do ignore the above advice and pick a woman speaker for your event for the wrong reasons, try to avoid telling her that her gender is the only reason she is there. This is quite distinct from other people assuming her gender is the only reason she is there; that comes with the territory and is unavoidable. Last year I received an invitation to speak at a conference and when I talked with the organiser on the phone he immediately said "Oh, we were so pleased to hear about you! It is so hard to find a woman to put on the technical track for this conference". Previously I had thought perhaps the organisers had heard of my reputation as a good speaker at other events in the area!
As an aside, I accepted that speaking invitation and went on to feel decidedly out of place and find my talk just didn't really fly. Next time I hope I'll heed the warning signs and decline the invitation to speak. As a newly freelance consultant and a speaker aspiring to find a wider range of opportunities, I ignored my instincts and paid the price on that occasion.
I am a conference organiser, I know how few submissions there are from women to CfPs in this industry. Two female speakers submitted to each of DPC (with 240 submissions in total) and PHPNW (with 150 submissions in total). Obviously with those kinds of numbers, the odds of having any women speaking at all at those events pretty much disappear into nothing. Personally I think the CfP stage is the point in the process where we need to encourage women to step up if we want to increase the number of women speaking, and I know this is an area that I haven't done enough towards this year.
There are some great resources about the issues of women speakers at technical events. I strongly recommend the Geek Feminism Wiki page about women speakers, which presents causes and solutions in an eloquent and constructive way, and this thought-provoking post from Sarah Milstein on the O'Reilly Radar site. However you approach this problem, if it is one, please try to avoid the token woman approach; in my view having no women speakers is better than having one that doesn't fit.
You won't find much "women-in-tech" related content on this blog, because I'm much more interested in technology than women, although I cannot overstate how much the support groups PHPWomen, Systers, DevChix and LinuxChix have done for my career and sanity. My friend Emma Jane Hogbin coined the Unicorn Law - and I guess today it caught up with me.