I then used the following steps to compile PHP:
./configure --prefix=/usr/local/php5.3 --with-apxs2=/usr/bin/apxs2 make make test make install
The configure arguments are to get the resulting code put into /usr/local/php5.3/ and to specify where to find apxs2 itself. Strictly speaking the "make test" step is optional, however as this is a development version of php I consider it polite to run this automated test and then press the button to submit the results at the end, it runs quickly and if it helps the QA people, then great.
It was at this point that I realised that I had overwritten a pre-existing file, critically it was /usr/lib/apache/modules/libphp5.so, the shared object file for my existing PHP installation! I fiddled with the PHP configuration at length but could not figure out how to get PHP to spit a differently named or located file out of the other end of the process.
This evening I mentioned my problem in the #phpc channel on freenode (warning, its not a help channel, don't go there and ask questions, you won't get a warm welcome) and after some prodding from the guys there I worked around my problem by skipping the make install step - its this step which uses apxs to move the file and causes my problem. The makefile shows the list of install targets, and I needed to exclude install-sapi. My steps now look like this:
./configure --prefix=/usr/local/php5.3 --with-apxs2=/usr/bin/apxs2 make make test make install-cli install-build install-headers install-programs install-pear cp libs/libphp5.so /usr/lib/apache2/modules/libphp5-dev.so
The new make install line just installs the bits I'll want, it seems that any combination of these can be used, depending on which bits of the language will be needed. The final line copies the apache module into the correct location for my system (Ubuntu Gutsy Gibbon).
I just wanted to add a note here to say that I would probably never have got to the bottom of this problem if I hadn't been collectively rescued from it by some of the seriously leading lights in the PHP community - people who shouldn't really know my name never mind stop to help me with a fairly basic problem. So thanks and hugs to Derick, Johannes, Davey and Ben. Thanks guys - for solving my problem *and* reminding me why the PHP Community is a cool place to be!