Script for Database Patching at Deploy Time

I’ve written before about a simple way of patching database versions and there’s a much more comprehensive article from Harrie on TechPortal as well. I often find though that projects with patching strategies are missing the scripts to apply these automatically when the code is deployed, so I thought I’d share mine.

My current project (BiteStats, a simple report of your google analytics data) uses a basic system where there are numbered patches, and a patch_history table with a row for every patch that was run, showing the version number and a timestamp. When I deploy the code to production, I have a script that runs automatically to apply the patches.

Continue reading

Dealing with MySQL Gone Away in Zend Framework

I wrote recently about having gearman in my application, however I have been seeing problems with the long-running PHP worker scripts. My logs had entries like this:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

The worker is a Zend Framework application, run from the CLI, and it seemed like the Zend_Db_Adapter had no way of knowing when MySQL had let go of its end of the connection. I tried a few different things, including Zend_Db_Adapter::getConnection(), but without success – until I dug through the source code (with some help from a friend) and realised that ZF was not reconnecting at all if it thought it already had a connection. So instead, I expressly disconnected and reconnected the database handler. At bootstrap time, I place my database handle into the registry, so I simply added this at the start of the actual function that the gearman worker calls:

$db = Zend_Registry::get('db');
$db->getConnection();

At the end of my script, before it returns to the loop waiting for another gearman job, I just disconnect my database:

$db->closeConnection();

Now Zend_Db_Adapter knows that when I ask it to connect, it needs to go off and make a new connection, and everything works really well! I was seeing the errors because I’m still only testing the system so it can go days between getting any new jobs, and the timeout on MySQL is shorter than that.

Using Gearman from PHP

I’ve introduced Gearman into a project I’m working on, and since a few people have asked I thought I’d share my experiences. Basically, this application generates some PDFs from a variety of data sources, makes images, and emails it. Since the whole data processing, image handling, PDF generation process is fairly heavy, I’m putting the requests to generate these onto a gearman queue and having some workers process the jobs. The eventual aim is to bring up EC2 instances (or php-specific cloud hosting perhaps? Recommendations gratefully received!) to do this work but right now I have one worker and it’s all on one server.

Continue reading

Adding PUT variables to Request Object in Zend Framework

When I wrote recently about testing web services within Zend Framework, I missed out a really key piece of information! I didn’t explain how I was reading the PUT vars in my controller code in the first place – so I’ll rectify that omission now.

Its very simple: I have extended Zend_Controller_Action with my own, and all controllers inherit from here. This has a routeAction() which grabs the incoming variables from a PUT request and sets them as parameters within the usual $this->getRequest() scope, then forwards on the request. Here is my class:

class My_Controller_Action extends Zend_Controller_Action
{
    public function routeAction()    
    {
        // handle grabbing PUT vars
        if($this->getRequest()->isPut()) {
            parse_str($this->getRequest()->getRawBody(), $params);
            foreach($params as $key => $value) {
                $this->getRequest()->setParam($key, $value);
            }
        }
        $this->_forward(strtolower($this->getRequest()->getMethod()));
    }
}

So in my controller code, I simply call out to $this->getRequest()->getParam(‘name’) or whatever, and in my tests I can set those parameters as I showed in my other article.

I hope this makes sense, its one of those things I set up once and use a lot (and now I’ll be able to refer to how I did it!), if you have any queries, comments, improvements or if this helps you then please leave a comment – I haven’t come across anyone else doing anything similar but I know there must be, so let me know!

Using Zend_Test for Web Services

Recently I had cause to develop a web service and so I wrote some tests to go along with it – or I was about to. When I looked at the asserts available in Zend_Test, they were all geared towards HTML/CSS output – but we can test just as effectively on another output.

Using Zend_Test, I set up the request object I wanted to send, and dispatched it. Then I retrieved the body and, since this service returns JSON, I json_decoded it. This gives me an object – and I can go ahead and use all the functionality of PHPUnit, with or without Zend_Test’s additions, to test my service. Its perhaps easiest to show this in a few steps.

Setting up the request object

The idea here is that you set up any parameters you need to including the HTTP verb to use and cookies if needed, then despatch the request. Here’s a few examples, first a simple GET method, with a cookie.

        $request = $this->getRequest();
        $request->setMethod('GET');
        $request->setCookie('token','xxxx');
        $this->dispatch('/user/24');

Including data with a POST request:

        $request = $this->getRequest();
        $request->setMethod('POST');
        $request->setPost(array(
            'name' => 'new user',
            'organisation' => 49
            ));
        $this->dispatch('/user');

This is a REST service, so I also tested PUT and DELETE methods. DELETE just needs the setMethod() call since it doesn’t have any data with it, but PUT was a bit trickier – here’s an example of what I used:

        $request = $this->getRequest();
        $request->setMethod('PUT');
        $params = array('name' => 'Harry Potter');
        $request->setRawBody(http_build_query($params));
        $this->dispatch('/user/48');

Decoding the Response

This is the easy part, all I do is check the status code is what was expected, and then decode the response. My web service returns JSON so this part of each test looks something like:

        $this->assertResponseCode('200');

        $response = json_decode($this->getResponse()->getBody());

Testing the content of the response

Here we get into classic PHPUnit territory and simply use the assertTrue and assertEquals calls we’d use when testing anything else, an example is included for completeness:

        $this->assertEquals($response->contentType, 'user');
        $this->assertTrue(is_numeric($response->id));
        $this->assertEquals($response->name, 'new user');
        $this->assertEquals($response->organisation, 49);

In Conclusion

By combining the request/response awareness of Zend_Test with standard PHPUnit testing strategies, its easy to test web services as well as web pages. I hope the examples are helpful – if they help you or if you have anything to add, then leave a comment!

PHPUnit with Zend_Controller_Action_Helper

I’m currently working on a REST service built in Zend Framework and I ran into problems very quickly – when I tried to write the first unit test for the first action in fact! PHPUnit was just dying when I asked it to dispatch() any URL which didn’t return HTML, it wasn’t even giving its usual output.

What was actually happening was I was making use of Zend_Controller_Action_Helper_Json, my service returns JSON and this action helper takes input, transforms it into JSON, sets the content-type correctly and tells ZF not to look for a view since we don’t need one. I thought this was pretty neat.

But look at the start of the declaration for the action helper:

class Zend_Controller_Action_Helper_Json extends Zend_Controller_Action_Helper_Abstract
{
    /**
     * Suppress exit when sendJson() called
     * @var boolean
     */
    public $suppressExit = false;
...

By default the JSON action helper will exit() – which of course when run from phpunit causes that to exit as well! There is the class variable there so it was simple to turn off – I just extended my class and changed the value of that $suppressExit variable.

class Zend_Controller_Action_Helper_ServiceJson extends Zend_Controller_Action_Helper_Json {
    public $suppressExit = true;
}

My tests now run successfully and I can build my application, hopefully next time I’ll realise what I’m doing wrong faster!

Introduction to Zend_Db

I recently worked on a project which was based on Zend Framework – I haven’t worked with it before and I was temporarily confused by the existing implementation of some of the database-level stuff. After much reading and untangling of code, I’m now pretty clear how this should look, so here’s my overview. I’m not going to go into setting up a whole application, but this is a quick primer on how data models go together.

Modelling Tables

Zend_Db_Table is a class that represents a table. You instantiate one of these and call query functions against it. The actual code for my classes is really minimal, here’s an example:

class UserTable extends Zend_Db_Table
{
    protected $_name = 'users';
    protected $_rowClass = 'User';
}

The two properties that I’m setting here are all I use for basic tables. $_name tells Zend Framework which database table to use. $_rowClass tells it which class it should instantiate for the results from your select query. By default you’ll get Zend_Db_Table_Row objects but you can have it use any object which extends this class instead.

Getting Results

To get results from a table, you instantiate the relevant table class, and then fetch some results. You can restrict the results you get but Zend_Db_Select is a whole other kettle of fish that perhaps I’ll post about some other day. So a controller action might look something like this:

function viewAction()
{
    $users = new UserTable();
    $user_list = $users->fetchAll();
}

The $user_list variable will then hold an instance of Zend_Db_Rowset.

Working with Rowsets

The rowsets are iterators, you can loop over them and they will return objects which represent the rows in the resultset of the query. By default these are objects of type Zend_Db_Table_Row but if you set the table object’s rowClass, either in the object declaration or by calling setRowClass() before you fetch the results, then you can have any class which extends Zend_Db_Table_Row used. Its pretty common to pass the rowset straight to the view and iterate over it there – you can put any functionality you need for this object into the class you use. Let’s look at my user class for this example.

The Row Class

    class User extends Zend_Db_Table_Row
    {
        public function getName()
        {
            return $this->first_name . ' '.$this->last_name;
        }

        public function getProfileUrl()
        {
            return '/users/profile/id/' . $this->id;
        }
    }

These are overly simple examples, but giving this kind of functionality to the user model allows it to be re-used any place you need it. I’ve also found it useful to have an intermediate table row class that contains functionality that will be used by data from more than one table – for example for formatting dates in a consistent manner.

Zend Framework and Models

This is a pretty simple overview but it took me a while to get to this point – the framework does have documentation but its quite case-specific and doesn’t really show how it ties together. Now I understand these classes and their relationships, my development project is going a lot more smoothly.