Tag: api

mitmproxy is my preferred proxy for API testing these days. I'm using it mostly in reverse mode though, so I thought I'd capture my setup in case it's useful for anyone else. My main use case is to inspect the traffic coming into the API server and being returned, mostly when running tests against it. Where mitmproxy is typically used as a proxy to monitor traffic passing into/out of a client application, for this I'm concerned with the server's handling. Reverse mode lets me inspect and adjust the traffic as I go along. I start up the proxy, configure the tests to point to it instead of the actual server, and I'm all set.

Today I'm sharing a quick-and-dirty script to take an OpenAPI description, spin up a docs server locally, and copy the URL into your clipboard. I also use a bit of glob expansion in my script to find the right folder, because I have a lot of APIs with long and formulaic directory names (TM Forum members know this story). I'm spinning up Scalar here as it's my current favourite just-works local API docs platform.

If you publish a RESTful API and there are situations when you need less (or more!) detail in the API responses from your API - then this post is for you. When I run into this with the teams I advise, the initial problem statement usually arrives more like "We need to move to GraphQL", but this is a solution, not a problem to solve. The problem usually turns out to be either or both of:

• the API responses are simply too large for some of the clients or uses cases, users need to select just the fields they want

• the API doesn't have enough information in the response payload for this client or use case, users need to choose to include nested data

GraphQL is one way to achieve these things, but this post is about the RESTful way to provide the right level of detail in an API response.

I've been using Optic's CLI, an OpenAPI tool that does a bunch of things including diffing OpenAPI descriptions and comparing HTTP traffic with OpenAPI. My use case was an established API that didn't have an OpenAPI file yet - using Optic we could create one as a starting point, and then move to a design-first workflow to make the changes that I was there to help with. For this blog post, I've used the example of https://api.joind.in as an excellent representation of an API still in use, but without an OpenAPI file and not built with code that a code generator would recognise.

In a recent project around open source contributors, I wanted to take a look at which projects a particular user (actually a few of them, but I wrote a wrapper to repeat the process for each handle) maintains. GitHub doesn't show this maintainer relationship, so instead I used the v4 GraphQL API and looked at pull request comments on repositories that the user has access to. I'm sharing my query and the Python script I used to make the API calls to.

For teams that generate OpenAPI from their codebase, there's a tough choice between maintaining rich and extensive content such as Markdown descriptions and examples in codebase annotations, or in making those changes to the generated file - and then losing them when the code changes and the file is regenerated. The new OpenAPI Overlay Specification defines a format for storing updates to an OpenAPI document, and there's a new generation of tools to make it easy to do, so let's take a look.

Most APIs aren't perfect. By design they are long-lived and may serve many audiences. I see a lot of organisations maintaining multiple versions of APIs, or simply not documenting some of the endpoints to avoid them being included in the published documentation and confusing users. The problem of API redaction is a real one in most organisations, and it goes unsolved or is solved in a way that causes ongoing friction. I've helped a few API projects to solve this problem with the new OpenAPI Overlay specification so I thought I'd share!

I work or have worked with APIs of all sorts of standards, sometimes many standards at once - or none. Each standard exists for a purpose and might be a great fit, or a terrible fit for other purposes. Today's post is a recap of some of the common API standards around and what you need to know about each one.

Working on API tools, I get to see inside lots of different organisations' API projects and processes. Every scenario is different, but a common theme is that many companies use a more complicated API description workflow than you see in conference slide decks! This article shares my typical workflow, steps and chosen tools that might show up in an API description pipeline.

I've been thinking about API linting lately, partly because I work on a tool for API linting in my day job, and partly because I get quite a lot of questions from teams wanting to improve their API quality. The "best" ruleset depends entirely on your context, where you are in your API practice journey, and how much you want to invest in the API in question. I typically use a 4-levels model of API linting readiness, and I'm sharing them so that you can find yourself on the map and see where you might go from here.

With OpenAI's new Actions feature allowing you to get a GPT that uses your own API, many new doors are open. However giving an AI an access token the keys to your API and telling it to have fun may lead you to realise that one of the doors available leads to the empty lift shaft of overwritten data, or the bottomless well of cloud bill shock. To reduce the risks, OpenAI supports an extension x-openai-isConsequential that you can add to your OpenAPI description, to indicate which endpoints should not be called without a human confirmation step.

I'm not doing much with OpenAI right now, but I do plenty with OpenAPI and the question "how do I add this field to my existing API description?" is one that I can answer! What's more, you can use the advice in this post to add other extensions or additions to your OpenAPI descriptions using Overlays, this advice isn't OpenAI-specific, but it's used in the examples.

I've heard a variety of bad advice (and some untruths!) regarding how to version your OpenAPI description lately. Rather than dig into those, instead I'll share my opinion on how I like to version API descriptions in my own projects. Every API project is different, but perhaps there's something here that can help your project too.

I talk to a lot of people about their API lifecycles and processes, and one thing that has value in almost every case is immediate documentation preview for all updates to an OpenAPI description. The time it takes to set it up (about ten minutes now I know what to do) is an investment that makes every future change easier to propose, review, and accept. Here's my recipe for open source OpenAPI documentation using Redoc (disclaimer: I work for Redocly, who publish Redoc) with the files pushed to Netlify for hosting (but it's HTML, so really any hosting will do and you should feel free to adapt the instructions for your preferred platform, and then share them in the comments).

Giving talks about OpenAPI often means lots and lots of slides of YAML. In fact, one of my talks even contains a "warning, contains yaml" slide! If you aren't a professional yaml developer, it can be difficult to digest and learn from, and so I wanted to do better. The various elements are nested inside one another, and so I had this idea to build a representation of OpenAPI using Blockly (the tooling also used for Scratch, if you've seen that).

I love documentation, which is supposed to be rare for software engineers. I am pretty sure that loving documentation is actually very common, but not very cool and so few people admit to it! I love reading good docs, and so I love writing them too. Between hanging out in the Write the Docs community, and recently being at the APItheDocs conference, I've answered a bunch of questions about getting good documentation from an OpenAPI document, so I thought I'd round them up in case it's useful.

I've been working with SDKs quite a bit this year, simple ones that wrap a bunch of API endpoints. The best bits of these SDKs isn't just the delightful developer experience of using the clunky endpoints with the elegant SDK wrapping them. I mean, it is, but I already wrote about consistent and Delightful SDKs for Nordic APIs; today I want to cover the some of the "extra" things that make a big difference in the APIs.

Having API descriptions in your workflow is a game-changer, but how do you get from nothing to a functioning spec in the first place? I recently made a very simple spec for a very simple API, and thought it might make a good example for anyone looking to create a spec for the first time. OpenAPI specs are very verbose so this very tiny API makes quite a manageable example!

I'm using Postman more lately and showing off the Environments feature really often. I really like this feature for giving easy ways to handle values I use frequently, but it seems to be a bit of a hidden gem, so I thought I'd write about when I find it useful, in case you do too!

As a Developer Advocate for an API company, I spend a lot of time talking about APIs and webhooks and HTTP in general. Recently I've been focussing on HTTP tools, but I really wanted a very simple example API that I could use that would return JSON but really let me focus on the tools, not the API. So I created a "Planets and Webhooks" API with a couple of GET endpoints to return JSON data, and another endpoint to receive and log incoming webhook data.

I already wrote about using Prism as an API test platform but I didn't include an example with callbacks so this post is to fill that gap! If you didn't read the previous post, the tl;dr is that Prism is a nodejs open source tool that can take your OpenAPI specification and then do an impression of your live API, validating the API calls sent to it and responding with the example data described in the spec. It can also follow up the API response and send an HTTP request of its own to mock the callbacks described in your OpenAPI spec.

I'm a huge fan of Ngrok, a tool to create a secure tunnel to your laptop. It is brilliant for testing, well, anything running locally really. Someone asked me about my setup recently and I'm using a couple of config files to keep things quick and consistent, so I thought I'd share here what I shared with them.

You can use a central config file for ngrok, and/or specific config files for each project. I use both, so I'll show you around my setup.

I work quite a bit with serverless tech, particularly on IBM Cloud since I work there. At the moment I'm building a microservice using web actions. When a user creates data with a POST request, I want to redirect them to the URL of the new thing they created - but for that I need to know the URL that it was accessed with. This is a relatively new feature on API Gateway so here's a quick howto for grabbing that URL in both JavaScript and PHP.

After answering a bunch of questions about the Bluemix CLI tool for IBM Cloud, I thought I'd share my personal cheatsheet in case it helps anyone looking for examples!

I'm working on a CouchDB library for PHP, and so I needed to write some tests for it. CouchDB has an HTTP API so I'm basically making web requests and while I could certainly set up a test database and run full-on integration tests, there are a few limitations with that approach. Firstly: it means I'm testing the database as well, which isn't what I want and brings extra dependencies that make the tests harder to run. Also: I want to be able to test error cases, rate limiting and so on, that would be difficult to recreate reliably.

For some years I've been creating a bundle of links covering articles I recommend for various topics in the Zend Certified Engineer Exam (if you can here looking for the bundle itself, it's at {filename}/zce-links-collection.rst). This was done using bitly's bundles feature, which I thought was a great way to share links. In fact I had 10-15 bundles that I had created from collating all the links in a particular training course or talk, so that people didn't have to try to write down URLs as I went along. Unfortunately they sunset their bundles and then removed them completely, and I missed the announcement (it all happened quite quickly, they'd seen some abuse of the feature, it's free, all totally reasonable) BUT they also didn't respond to my support questions about how to recover the data.

It turns out, it's an undocumented feature on their API, so here is everything I know about recovering your bundle data, including the script I used to rescue my own data.

I get a lot of complaints about an API that I maintain (http://api.joind.in) which is "missing" the ID field. This ID field is the database's primary key; if the user doesn't have access to the database (they don't), then it seems to me that the primary key …

I'm working on a project that uses a search engine to show images on a particular topic ... but I need my search to be localised since I'm in the UK and so "football" doesn't mean what a generic search engine thinks it means. Getting this working was MUCH harder than I expected, so here's a quick post on what I did so that I can remember for next time - and if this helps you as well, then great :)

It's that time of year again, the PHP North West conference is almost upon us, and this year they are once again running a hackathon. These events are a great way either to carve out some time to get your head down and hack on an idea that's been in the back of your mind for a while, but they're also a fabulous way to get involved in collaborating on projects. At PHPNW, you'll find there are quite a lot of open source projects at the hackathon, standing by to take on anyone interested in getting involved, either just for the evening or beyond. I'll be there, representing joind.in, a tool which is used by the conference itself. So what kinds of things will there be to do and how can you get involved?

In case you missed it, Twitter updated their APIs recently, so that you have to authenticate to use even their search APIs to return publicly-available results. This is an increasing trend for API providers, to provide either very limited or nonexistent access for unauthenticated users, I think so they can rate limit consumers that swamp them. To cut a long story short, that meant I needed to update my dashboards that keep an eye on twitter searches to do more than just call file_get_contents in the general direction of the right URL.

Twitter is great for one-liners, but it's very difficult to carry on any kind of advanced conversation there. Therefore when I saw this tweet yesterday, I knew I'd be picking a different medium to reply:

@lornajane sory for getting on your nervs with #rest,but are subquerys (like couchDB does) restfull to? is there a rule? like .../?type=bla

— Maximilian 'Berghoff (@ElectricMaxxx) June 12, 2013

While working on a book ("PHP Web Services" from O'Reilly, not out yet but soon!) recently, I was looking for some place I could make HTTP requests to, to show off how to make different kinds of requests with different tools. On my own machine, I have a couple of scripts that chatter back giving debug information about the requests that were made, but I wanted to get the tools examples going without any additional dependencies at all. I hadn't used anything like these tools before, but I found quite a few alternatives, so I thought I'd share what I came up with.

I'm working on a little hobby project which needs to know what the weather is going to be. I had a look around and noticed that the MetOffice had released a new API called DataPoint. They have a selection of APIs, including some map overlays and some actual weather data (more on that another day) but I was especially charmed by their text APIs - this is basically the basis of weather forecasts used everywhere :)

I'm working on some demos for a tutorial I'm giving next month and since I'd like to show off Github's API, I needed an access token for it. They have the usual web flow but I'm cutting as many corners as I can to keep the demos nice and quick, so I looked into the support Github has for generating an API key programmatically.

I write a lot of APIs, and I also preach that your API isn't finished until it has excellent documentation. Which is great, but that means I therefore have to lead by example and document my APIs :) Enter iodocs from the talented folk at Mashery.

Iodocs is a node.js application (which is fun for a PHP developer. Most developers write a bit of JS, but this one hasn't). You describe your API and all its methods in JSON, and then iodocs presents an interface for you to enter API keys, add parameters to each request and press the "try it!" button. This makes your API call and shows you the results on screen, which seems like a great way to demonstrate what all the various parameters do!

This week I've been working on a feature which handles an incoming API call from an external source (actually Paypal's IPN service). Making external calls into my development platform isn't how I usually work, so instead I captured the body of the request that was being sent, and replayed it …

I am a huge fan of bit.ly and use their tools for a wide variety of different things. They recently did a big relaunch with some lovely new features, which are for the most part pretty good, but which are inaccessible in places. In particular, it seems that there aren't any short URLs for the bundles - which is annoying for me as I use that feature a lot!

To get around this, I used their API to make a page which lists my bit.ly bundles, and creates shortlinks for each of them (once you've created a shortlink for a given URL once, bit.ly just re-uses the same ones the next time you ask to shorten the same URL, so this is less silly than it sounds).

In case the code is helpful, I thought I'd share.

I'm a regular speaker at a variety of (okay, mostly technical, so not really that varied!) events, and I submit talks to many CfPs (calls for papers). Whenever this happens, I tend to look back at whether I have any existing talks that I gave and liked and which would be a good fit. I use my joind.in speaker page for this: http://joind.in/user/view/110 as it's simpler than dredging through my directory of talks/articles on my hard drive (this is now rather large and unmanageable!).

I've recently been thinking that I should also do a better job of linking through to the various talks I'm giving/have given - and at around the same time I was contacted by the good folk at mojoLive about integrating against joind.in. To cut a long story short, the joind.in API now has the functionality for users to retrieve their list of talks!

I've been working on something recently where I'm pulling information from lots of places onto a dashboard. Each API has its own little quirks so I'm trying to write up the ones that weren't idiot-proof, mostly so I can refer back to them later when I need to maintain my system!

I've written about Google and OAuth before, but that was OAuth v1.0, and they are introducing OAuth2 for their newer APIs; in this example I was identifying myself in order to use the Google Plus API (which turns out not to do anything you'd expect it to do, but that's a whole separate blog post!).

If you read this blog often, you'll know that I am:

• crazy about APIs

• living with some accessibility issues

Put these two things together and what do you get? Actually don't answer that! Today what you get is an example of integrating with JIRA's REST API, because their recent "upgrade" locked me out of the issue listings pages completely and I really do need to be able to see a list of bugs! Their bug editing screen is quite usable, so it's just the list that I need here, but you could easily call their other API methods as you need to.

I had an issue this week on a system which has been working fine for a while, but stopped fetching some data from google's user account API. I was getting a 403 response from the API, which seemed odd. Luckily I was logging OAuth::getLastResponse() to my error logs (this …

This is the third installment in my series about writing a RESTful web service in PHP (the previous entries are about understanding the request and routing it. It is probably the last one but there are a few other things I'd like to cover such as error handling, so I might keep adding to it, especially if I get any particular requests or interesting questions in the comments. So far we've covered parsing requests to determine exactly what the user is asking for, and also looked at routing to a controller to obtain the data or perform the action required. This post gives examples of how to return the data to the client in a good way.

disclaimer: I am not a client-side developer, and I don't write javascript. However I am committed to supplying useful APIs of all kinds, and JSONP falls into this category

Early in the development of the new Joind.In API, someone else started consuming the service to populate the javascript widgets they were making*. Since these scripts are intended to be used on many external pages, and they retrieve data from the joind.in API, cross-domain issues were a problem.

I'm a big fan of the google charts API - it draws much better-looking graphs than I would ever manage and all I have to do is assemble the right URL to make it work. I recently got a feature request to add QR codes to joind.in, so that speakers and event admins could easily allow people to link in to a particular talk page.

When I wrote about launching a prototype of a new joind.in API, quite a few people started to try it out. My friend David Soria Parra emailed me to point out that many of the numbers in the API were being returned as strings. He said:

It's just a …

Following the principle of "release early, release often", I put live a very early version of the v2 API for joind.in today (so that I can use it in another project!). I haven't updated the documentation yet but in case anyone was thinking of consuming data from joind.in …

If you are attending my Web Services tutorial at PHP Community Conference (if not, probably nothing for you to see here) later this week then you might like to download the sample code. I'll be referring to this and inviting you to "play along" as I go creating services during …

I had a funny (funny weird, not funny haha) problem the other day when working with pecl_oauth in PHP to talk to a service. I'd gone through all the handshaking steps, got the acces token and was ready to start talking to the service itself. However when I tried to …

Recently the nice folks at Addison Wesley sent me a review copy of "The Developer's Guide to Social Programming" by Mark Hawker (you can find it here on amazon). This is right up my street as I'm doing so much with with APIs, social and otherwise, at the moment and the book covers things I'm using such as OAuth and google's services.

Google Charts API is a nice, freely available tool for creating really good-looking graphs very painlessly. Don't be fooled by the "API" bit though, there is no need for advanced understanding of HTTP here - you generate most graphs just by adding parameters to a URL! In this post we'll take a look at a few different ways to generate charts.

I'm deeply involved with an open source development project (joind.in, the event feedback site) and we recently made the decision to move our issue tracking away from GitHub (because it's awful and meets none of the requirements for a bug tracker, but that's a post for another day). In order to make this happen more smoothly, we wanted to migrate our open issues from github (to JIRA Studio, since Atlassian generously provides accounts here for worthy open source projects - thanks Atlassian!).

I looked around for some export functionality for github but I got a lot of posts complaining it wasn't there. Since I hate applications that take your data and refuse to let you remove it, I was disappointed by this news but further inspection showed that although there might be no "export from github" button, there's an API that more than has it covered. The API returns JSON which is easy to work with from many programming languages, and is perhaps even more powerful and flexible than the simple export I initially expected, so here are some examples.

Recently I've been working on a project using Google Charts API and I absolutely love it. It makes pretty graphs and shows data really nicely - and it's very painless. I did trip over a little trick today though which took me a little while to work out, so I thought …

It is pretty common when using OAuth for there to be a relationship between the provider and consumer; as a consumer you usually register with the provider to obtain a consumer key and consumer secret. Google's APIs however do not require this. It is recommended that you register your application …

Recently I started playing with the google analytics API, looking at ways to bring analytics onto dashboards and generate simple reports from the data in there. Very shortly after I started to look at the API, I had working data retrieval, so I thought I'd share my experiences (and code!).

I'm working with Google Analytics at the moment, to pull information about web traffic from analytics into another system. Google have excellent APIs and that makes this job much easier. I'm using pecl_oauth to authenticate users against their google accounts (see my post about using OAuth from PHP), but even after I have a valid google user, working out which analytics accounts they have access to and how to refer to them is a puzzle in itself, so I thought I'd share what I learned.These examples use pecl_http, since I have control of my platform and I find it easy to work with. I've tried to write this with explanations of the overall process in between the code snippets so hopefully this makes the process clear whether or not you will use exactly the same implementation.

Recently I was working with some google APIs and needed to retrieve some namespaced elements from the result set. This confused me more than I expected it to so here's my code for the next time I need it (and if you use it too, then great!)

I was reading from their analytics data feed API, this returns a few key fields and then multiple <entry> tags, each with namespaced children. The entry tags look something like:

Earlier in the year I gave a talk at PHP UK in London entitled "Best Practice for API Design". I really enjoyed giving this talk, since I work so much with APIs and enjoy sharing my ideas. The audio is now online so if you missed the talk, feel free …

I've been looking into OAuth recently and really like what I see, so I started looking at actually starting to play with something that uses it (and isn't twitter). In the pursuit of this, I spent some time walking through the process of how to actually authenticate using OAuth, as a client. I chose Yahoo!'s service, because they have some fabulous developer documentation and have a standard OAuth implementation. Although you don't strictly need any special libraries to handle OAuth, that would be a bit like decoding XML with a regex, so I used the OAuth Package from PECL. For others (including me after I've slept), here's an outline of the process.