Tag: tips

mitmproxy is my preferred proxy for API testing these days. I'm using it mostly in reverse mode though, so I thought I'd capture my setup in case it's useful for anyone else. My main use case is to inspect the traffic coming into the API server and being returned, mostly when running tests against it. Where mitmproxy is typically used as a proxy to monitor traffic passing into/out of a client application, for this I'm concerned with the server's handling. Reverse mode lets me inspect and adjust the traffic as I go along. I start up the proxy, configure the tests to point to it instead of the actual server, and I'm all set.

I have a things-as-code project that outputs mostly text-based formats, but a lot of them. To keep an eye on consistency, I rebuild all the outputs and dump them into a local git repository so I can very easily diff to spot any changes - which was fine until we added a build timestamp, so every file looks changed on every run! This post is about ignoring the matching line with git diff -I.

I consider myself pretty git-confident, I've worked with it a lot, taught it, been a git consultant, run engineering and various things-as-code teams. This week I had a spectactular git problem where merging one branch into another produced changes that didn't exist on either branch. Turns out, renaming directories in a monorepo with multiple almost-identical boilerplate documentation files comes with surprises...

I use a lot of asciidoc these days for work documentation (and I love it) and I've been so happy that GitHub renders it when you view a repository in the web browser, just like it does for Markdown and ReStructuredText. BUT what GitHub does not do is render the image types that asciidoc does so even though I'm working with asciidoc and PlantUML, and the asciidoc tools render those diagrams nicely in PDF and HTML output - GitHub's rendering doesn't. So here's a quick overview of how I handle those repositories.

OpenAPI tags have always been annoying: user-supplied arbitrary data for endpoints should be a fabulous feature - but the documentation tools seem to think that tags are only for them so it becomes more difficult to use tags for other purposes. In fact it is very useful to be able to tag endpoints with lots of different categories of data and so in OpenAPI 3.2, tags were enhanced to include an additional "kind" field so that different kinds of tag could be used.

OpenAPI has always had support for simple tags, but the OpenAPI 3.2 release brought in some serious tag upgrades including a summary field, a "kind" field with registry, - and the ability to nest tags which is the focus of today's post. If one level of organisation isn't enough for your API (and on bigger APIs I'd argue it shouldn't be) then the ability to indicate which tag is the parent of this tag will be a good feature to adopt when you upgrade your OpenAPI descriptions.

Today I'm sharing a quick-and-dirty script to take an OpenAPI description, spin up a docs server locally, and copy the URL into your clipboard. I also use a bit of glob expansion in my script to find the right folder, because I have a lot of APIs with long and formulaic directory names (TM Forum members know this story). I'm spinning up Scalar here as it's my current favourite just-works local API docs platform.

Too many open source projects suffer from inadequate documentation, and that hurts their adoption, their communities, and puts more burden on maintainers. Many people who create open source software don't see see themselves as writers, but today I'm sharing tips for documenting open source projects without it needing to take up a huge amount of time to get something published and ready for (docs) contributions.

For teams that generate OpenAPI from their codebase, there's a tough choice between maintaining rich and extensive content such as Markdown descriptions and examples in codebase annotations, or in making those changes to the generated file - and then losing them when the code changes and the file is regenerated. The new OpenAPI Overlay Specification defines a format for storing updates to an OpenAPI document, and there's a new generation of tools to make it easy to do, so let's take a look.

Reviewdog is a tool to use with GitHub actions for applying review tools in your CI. I use it with Vale, and it's really good. One thing that has tripped me up multiple times is that by default, it only applies the checks to the changes in the pull request, not to the whole project. So when you add something like Vale, which is a prose linter, it only checks ... the workflow file you just added! I always forget that it works this way, and how to change it, so I'm pasting some examples for future reference.

With OpenAI's new Actions feature allowing you to get a GPT that uses your own API, many new doors are open. However giving an AI an access token the keys to your API and telling it to have fun may lead you to realise that one of the doors available leads to the empty lift shaft of overwritten data, or the bottomless well of cloud bill shock. To reduce the risks, OpenAI supports an extension x-openai-isConsequential that you can add to your OpenAPI description, to indicate which endpoints should not be called without a human confirmation step.

I'm not doing much with OpenAI right now, but I do plenty with OpenAPI and the question "how do I add this field to my existing API description?" is one that I can answer! What's more, you can use the advice in this post to add other extensions or additions to your OpenAPI descriptions using Overlays, this advice isn't OpenAI-specific, but it's used in the examples.

I've heard a variety of bad advice (and some untruths!) regarding how to version your OpenAPI description lately. Rather than dig into those, instead I'll share my opinion on how I like to version API descriptions in my own projects. Every API project is different, but perhaps there's something here that can help your project too.

Employees expect a lot from their managers, and as managers we do our best to meet those expectations. However most managers are only human (I've met a few that I had doubts about), and so it can be difficult to get absolutely everything right absolutely all the time. I'd like to share a tactic that I learned from another manager around the time that I took my first line manager role, that I think really helped me to at least give the impression that I knew what was going on: a second calendar, visible only to you, with key dates in.

I read a lot of commit messages that make me wonder who the committer had in mind when they wrote it. If you don't read commit messages yourself, I think that can make it even more difficult to think about who the audience is, or when someone would be reading …

I'm working almost entirely with docs-as-code setups for prose at work these days and while I do use vim for most of what I do, VSCode is increasingly in the mix. It's just about accessible enough with keyboard shortcuts and the command palette for me, and I thought I'd share the extensions I'm using and find helpful.

I've seen quite a bit of chatter later about the GitHub Codespaces feature lately, and I have beta access to it so I thought I'd share my first impressions and make some notes of what I've tried that might be useful to others.

One of the major problems I run into with video conferencing tools is how many of them are not at all accessible. I'm a keyboard-only user so if I need to hover a pointer in a particular area of the screen and then click an icon to mute ... then I can't do that. And if you mute me when I join because there are a lot of people here, then I can't participate at all. To work around this, I need a one-liner to mute (and unmute) my mic ... so here it is.

I work a lot with JSON APIs in my Go projects, so almost every struct has some field tags on it to translate AccountID into account_id and that sort of thing. Then one day I needed to load data from config using mapstructure and it took a few attempts with the search engine to find the syntax I needed, so it's here for next time I need it (or in case you need it too).

Once any document gets over a certain length, a table of contents can be useful. I have a favourite markdown table-of-contents tool but it usually takes me a trip to the search engine to remember what it is! (tl;dr: github-markdown-toc )

I'm using Postman more lately and showing off the Environments feature really often. I really like this feature for giving easy ways to handle values I use frequently, but it seems to be a bit of a hidden gem, so I thought I'd write about when I find it useful, in case you do too!

I'm writing a Go application that glues together a bunch of other things, so it has a whole bunch of basically unrelated config, dumped in a yaml file. I was struggling a little with this non-standard use of Viper but actually, it does everything I needed. And, presumaby, more besides. I thought I would put some examples here to show how to handle this.

When I teach git I try to show loads of good practice about how to inspect what's staged before commit, etc etc. Good practice is good of course, but knowing how to undo a mess you created is better - and mistakes will happen. For example, accidentally including a node_modules directory in your otherwise excellent and useful commit. This post will walk you through how to fix the problem without losing any of your work.

When chatting about source control good practice the other day, I got a question about repeated git commit messages. In general, I would always advise that the same commit messages appearing multiple times in a project's history is a definite red flag - and if I'm responsible for that repository I will probably make fun of you for doing it.

I've been a long-time fan of RequestBin, but it's no longer active since it suffered so much bad traffic. It's never been too difficult to set up locally and when I tried to do that last week, I realised it has got even easier because it now has a docker-compose configuration.

Now I work with more programming languages, I start to miss features from other languages when I come "home" to PHP. One that I hadn't seen in PHP before I saw it in other languages such as NodeJS (I think Ruby had the original implementation) was: a way to easily control setting your environment variables, particularly in development. In NodeJS the dotenv library is great for this; handily in PHP vlucas has already created phpdotenv so we are all set to apply these tricks to PHP applications!

I'm a regular and happy user of Apache CouchDB, so much so in fact that I'm writing a library to talk to it from PHP. While working on tweaking a feature, I realised that the laptop I use for development didn't have the right/enough data on it to test …

At a conference a few days ago, I put up a slide with a few of my favourite tools on it. I got some brilliant additional recommendations in return from twitter so I thought I'd collect them all in one place in case anyone is interested - all these tools are excellent for anyone working APIs (so that's everyone!). First, my original slide:

I mostly use OpenWhisk with NodeJS (which is lucky for me, it's the best supported of the languages and default for the documentation examples!) and while there are a bunch of npm modules already installed on OpenWhisk, sometimes there will be others that you also want to include. Alternatively or additionally, you might also want to deploy your package.json since this can specify the entry point if it's not index.js which is the default.

Working with openwhisk, it's easy to create many isolated actions and build them up into sequences; the output of one action is passed to the next action in the sequence. In my case, I wanted one action to spawn potentially many other actions. I had to look up how to do it and here it is so I can look it up more quickly next time!

As a developer-of-another-discipline who is now transitioning into a bunch of NodeJS projects, I grep a LOT. However I am usually only interested in the code in the project at hand, and not the dependencies - of course grep has a switch for that! To grep your project and not the …

I have a new job (Developer Advocate with IBM Cloud Data Services) and subsequently a new work laptop. It's a mac and after almost 10 years as a linux-only user, that's rather a shock! Due to some nasty RSI issues, I don't use a mouse or any other pointing device*, so as well as learning a whole new operating system I also needed to learn its accessibility tools. I'm still at the swearing stage but mostly past the tears so I thought I would share what I've found - using a computer from the keyboard is fast and productive for everyone as well as less painful for me, so you may find some tools in here that you want to try yourself.

I collected all my tools, my own notes and scribbled cheatsheets and put them into this gist so that I could refer to them later; I also intend to keep updating this as a reference.

Composer is dependency management for PHP, and it consists of two main files:

• composer.json where you specify your dependencies

• composer.lock where composer itself records exactly which precise version of every library and every dependency of every library it picked, so all installs will be identical

Crucially, the composer.lock also includes a hash of the current composer.json when it updates, so you can always tell if you've added a requirement to the composer.json file and forgotten to install it.

One of my git repositories has developed a tendency to refuse to checkout a feature branch locally that exists on the remote repo. My git bash completion works, but then strange things happen! It turned out to be that I had two remotes with the same refspec, so I thought I'd write down the behaviour I saw and hopefully help someone else to fix this problem faster if they see it.

If you type git pull and expect a fast-forward update, but get a merge instead, don't panic! This usually happens when we're collaborating on a branch with other people, and we've made changes on our local version of a branch, and someone else (or the other you, if you use git to sync between multiple dev platforms) has made changes to the remote version of a branch in the meantime. It also happens really frequently in teams where all commits are to the master branch ... yet another reason to have a decent branching strategy.

All that's happened is something like this:

I create my slide decks from ReStructuredText, which is a text markup format. Working this way makes it easy to add into source control, fast to work with, and also accessible since I don't use a mouse or other pointing device so traditional slide deck creation programs are kind of difficult. Text-based wins every time for me. While working on a new slide template, I ran into some difficulties and had to figure out how to inspect what was going on. I seem to struggle with this every time so I am writing my troubleshooting guide here for when I need it next.

I do a lot of code reviewing, both in my day job as principal developer and also as an open source maintainer. Sometimes it seems like I read more code than I write! Is that a problem? I'm tempted to say that it isn't. To be a good writer, you must be well-read; I believe that to be a good developer, you need to be code-omnivorous and read as much of other people's code as possible. Code reviews are like little chapters of someone else's code to dip into.

Over time I've developed some particular processes that I find helpful when reviewing code. In particular, I often surprise people at how much review I do before I run the code. Sometimes I grab the branch so that I can use my local diff tools, but I don't actually execute code until I've established some basic facts. This post is a little insight into what's happening in this not-running-the-code-yet zone.

I have a favourite set of switches to git log, but today I wanted to answer the question "You deleted how much code today?" so I thought I'd share how I did that

git log --numstat will show you how many lines were added (first column) and removed (next column …

I find myself needing to take a PDF, output it at a specific size, and have the result offset to the top right hand side of the screen. To achieve it, I needed a few new switches to my good friend PDFJam, so I thought I'd share my command!

pdfjam …

At the moment I'm working on a tricky project where two similar projects diverged. Very similar things happened to them both, but not quite the same things - and now we're merging the codebases to give us as much common code as possible. All this simply serves to set the scene of exactly what I was doing spending a whole day with large code diffs - I had to look up a few things so I thought I'd capture them while I can remember.

I'm a big fan of good tools, however struggle to find products that fit me because there are some key constraints:

• I use Linux (specifically Ubuntu 14.10)

• I don't use a pointing device. At all. If I can't use a tool from the keyboard, I can't use it at all (as a side effect, I use keyboard enablers in my browser so if your website has "helper" keyboard shortcuts, I probably can't use that either)

• My other devices (phone, tablet, work phone) are all android

Taken together, this makes finding tools a challenge - but I've had good experiences with todotxt and the ecosystem around it.

Much of the development I do these days uses vagrant machines to make sure that my code is running in the correct environment. This is great, but spinning up too many machines at once can rather stretch the resources of the computer you're running them on - and I keep starting …

Submodules are one of the most powerful and most mistrusted features in git, at least in the web development part of the internet where I spend my time. I've seen them go horribly wrong, but I've also had teams adopt submodules and have their development process run much more smoothly as a result - so I thought I'd take a moment out of my day to write down the process (and the gotchas) of development with submodules.

When working with curl, it can give lots of excellent and detailed information, but sometimes it is too much! Today I'm fiddling with the caching headers on a client's application, so I'm only interested in seeing the headers and not the body of the response. Making a HEAD request changes …

I teach git and often have issues with bad projectors where you can't see the colours. Recently I had a setup where even white on black was more or less invisible, but using black text on a white background worked okay. There's lots of documentation on how to turn on …

I'm a vim user and I somehow completely missed this excellent feature until much more recently than I care to admit! Usually vim has its own clipboard, but it doesn't share with the operating system. You will need a vim-gtk install, this isn't available in really basic vim (I'm a …

Today's little-known git feature (or maybe everyone knows but me? I only found this a few months ago) is for quickly switching between branches. Usually I would switch branches with:

git checkout [branchname]

However if you switch from one branch to another and want to switch back again (this happens …

I work with PDF a lot, and it bothers me that I can see an outline view when I open the document, but I don't seem to be able to grab just that view. My presentations are mostly PDF and the titles and section headings show up nicely. Today I …