HTTP Toolbox

As Web Developers, we need to know how to work with HTTP from every angle. I gave a 2-hour tutorial at PHP UK that included some of my most trusted tools – but it was sold out and a bunch of people asked me if there was video (there wasn’t, tutorials make little sense when videoed). Instead, I thought I’d try to set out a self-study version of the workshop (I rarely teach these days so I’m unlikely to deliver it anywhere else).

There’s a slide deck, some exercises and a sample repo on GitHub … let’s dive in!

Step 0: Get the Materials

You’re more than welcome to grab the materials I used for the workshop, they are all in a GitHub repo here: https://github.com/lornajane/fun-with-http – look in the workshop/ folder for the slide deck and all the exercises. The slides are probably hard to follow by themselves so I’ll try to put some basic outline and links into this post.

Step 1: Understand HTTP

You think you understand HTTP already, I know :) And I’m not going to write an essay here – I already wrote the book so if you need words, they’re available.

You should set up the sample app (check the README in the project repo) at this point.

Then, try an HTTP client of your choice (maybe try something new?) and make simple requests to the app. These are the suggestions I offered in the workshop:

curl https://curl.haxx.se/
Postman https://www.getpostman.com/
HTTPie https://httpie.org/
http-console https://github.com/cloudhead/http-console
Paw (Mac only) https://paw.cloud/client
Insomnia https://insomnia.rest/
probably your IDE?

I’ve also blogged about this before and there are some excellent recommendations on that post so check that out also.

Step 2: Get comfortable with data formats

I kept it pretty simple and focussed on form-encoded and JSON formats in the workshop but you may also come across XML and there’s a trend for binary formats too so you may need more tools than I’m covering here, depending what you are doing.

Recommended tools for JSON:
– jq https://stedolan.github.io/jq/
– fx https://github.com/antonmedv/fx

Step 3: Making simple API calls with PHP – and debugging them

For this section I showed two options for making simple POST requests from PHP: firstly using streams, which is very powerful and does not need dependencies – and secondly using Guzzle which is fabulous, also very powerful, and a bit more user-friendly. Here’s the two sets of example code so you can compare and contrast:

$url = "https://httpbin.org/post";
$data = ["name" => "lornajane", "message" => "Hi there"];

$context = stream_context_create(
    ["http" => [
        "method" => "POST",
        "content" => http_build_query($data)
    ]
]);

$response = file_get_contents($url, false, $context);

And with Guzzle:

require "vendor/autoload.php";

$url = "https://httpbin.org/post";
$data = ["name" => "lornajane", "message" => "Hi there"];

$client = new GuzzleHttpClient();
$response = $client->request("POST", $url, [
    "form_params" => $data
]);

At this point it would be good to mention a couple of tools that I use when I need to diagnose if my PHP application is making the HTTP request I hope it is! Sometimes when integrating multiple applications it isn’t clear at which end the problem is, and these tools can help.

First up: HTTPBin is a hosted service which offers a series of endpoints that return known data, headers, status codes, whatever you need for testing. This is very useful for testing error cases that are hard to replicate on the real API you are using.

My preferred tool in this space is RequestBin – please note that this link points to my own GitHub fork since the project itself seems to have been abandoned and there were a couple of patches in the pull request queue that are needed to run this on Heroku. It’s a brilliant tool for testing your outgoing web requests and can run in docker on your laptop or on the cloud very easily.

I challenged the workshop attendees to complete a half-made route in the sample application for this part of the workshop. We also stopped for coffee, so you might like to do that too :)

Step 4: Ngrok for development platform access

If you’re not already familiar with ngrok, you might like to be! It is an application that runs on your development platform, and makes your local webserver accessible from the open internet via a secure tunnel. I use it a LOT, mostly for testing incoming webhooks but also sometimes to see how something will work on mobile, or to let someone in another location take a look at something I’m working on.

Ngrok also allows you to inspect the full request and response of the traffic going over the tunnel, which is fantastic especially at the development stage. And that isn’t even the coolest part! The best bit is the “Replay” button available alongside the details of each incoming request: if your code doesn’t quite do the right thing, you can edit it and then run the request again, see what happens the next time! This is such a gift especially when the event that triggers an incoming requests is expensive or annoying to reproduce – again, I use this mostly for webhooks and you can probably see why!

Take the time to try the feature you made in the previous step via Ngrok.

Step 5: Proxies for Debugging

OK so one of these tools is not a proxy (Wireshark actually copies your traffic rather than it passing through the appplication) but I use them all for the same sorts of problems! Knowing how to use proxy tools gives you a very easy way to inspect all the requests and responses and this can be really valuable for allowing detailed debugging of your application without having to make a lot of diagnostic additions to your actual project.

My recommended tools in this category:

Charles Proxy (paid product, free trial available) https://www.charlesproxy.com/
Mitmproxy (open source) https://mitmproxy.org/
Wireshark (slightly lower level tool but brilliant for buggy HTTP) https://www.wireshark.org/

I have used all three of these and especially Charles Proxy has been a huge boost. It enables you to play tricks such as rewriting URLs (because the images are hardcoded to the wrong path and don’t work on test, perhaps) and simulating slower network speeds so you get a sense of what the experience will be for a real user. Another big feature with Charles is that you can replay requests, and also store and export them – very handy for attaching to an open issue in a bug tracker that shows exactly what happens, and will allow someone else to replay that request again later!

You had to be there

Hopefully some of the links and suggestions here are useful, the content wasn’t designed for a blog post but if you’re interested and motivated to work through the exercises for each step as a “self study” exercise then you’re really welcome to do so. I’m also happy to take questions if you have them – just leave me a comment!

LornaJane Blog