The bugs we agreed to have

In every software product, there’s going to be bugs. A big difference between a well maintained codebase and a badly maintained one (and everything in between) is how hard it is to learn about, debug, and fix those bugs.

It also happens pretty much everywhere that a “bug fixing sprint” is scheduled at some point, and people rush through trying to solve all the problems. Here, I want to talk about hard/obvious bugs vs designed bugs.

Hard bugs are your software failing to do something that you tried to make it do. Hard bugs are your error 500 and failures to sign up. Hard bugs are obvious, you know that they are bugs even if you barely know the product. There’s no doubt this was a mistake.

Designed bugs are a different beast. They are errors disguised as features, or features thought to be errors. Someone wrote a filter to behave exactly the way it behaves and fully knew how it worked. They might even have tests for it. They probably wrote a guide for testing it that included success scenarios that ...aha!... match your newly reported bug. 

This kind of bug surfaces most often when a new member joins the team. They are motivated to learn about the product and they are poking at everything. They keep asking questions and suddenly they receive a different answer to “how does this work” than what the software itself does.They’ll get all giddy and report it, with their hopeful new person faces : “@here: I FOUND A BUG! It looks like a high priority issue.”  (Bugs are always subjectively perceived as high priority, ask anyone).

The product manager will read the bug, and they’ll go “yep, that’s a problem folks.” The dev who wrote the feature (bug!) will start to freak out a bit. “Ehhh nope. No, no, you are wrong. No. This works as expected, look how nice my tests are and look everyone reviewed this and agreed it was ok.” And that’s how you realize that yes, there’s a big problem to solve.

There’s many reasons these kind of issue happens. An unclear issue in Github, a one on one Slack conversation, a product lead on vacation, a developer not asking for their specs to be reviewed, the dev not writing a spec, not having a dedicated QA person who knows the product inside out, a lack of good documentation, a bug-fixing sprint with little to no validation of the actual fix needed for a bug, rushing through things, tech debt, and probably many others.

Sometimes, the dev has sent their feature for code review, someone has tested it, but they all followed the instructions provided by the dev, so of course, it passed. They might have shared a Spec that got reviewed as well, but somewhere in the middle, the communication broke.

This happened a few times in the last 6 months. Every time I get annoyed, and every time I have to remind myself that it’s only a symptom of something I (or someone else) did wrong, a symptom of a process that needs to change or an extra “check” that needs to happen. In this story, the annoying developer is me, and everyone else = my awesome coworkers.

There’s a feature in our platform that I’ve come to love and hate, just because nobody seems to agree on how it should work. I think I fixed the same thing about 10 times, just because I didn’t validate assumptions with everyone that might have an important opinion on it. I’d ask someone I perceived to “know the product” and they’d go “yep, that’s correct, go do it”. And there I went, coded my precious shiny bugfix (AKA new bug) and asked for a review. Some poor dev reviewed and helped test it, and there it went, on to our unsuspecting product manager to be tested. They tested it, and they probably were tired and didn’t notice that something wasn’t quite right. Or maybe they actually thought “oh yea that looks better than it was”, which is mostly the case. So the new bug goes to production and everyone is happy. And we annoyed a lot of people inside the organization that depended on, wait for it… the previous bug.

When you read this, you might think “Wow these Ushahidi people are really disorganized”. Let me tell you (from my biased point of view!) , they, we, generally aren’t. I’ve seen this happen in every company I’ve worked, and I’ve experienced the same thing being reported as “should work like A” and “should work like B” by the same person, in the same week, in the past.

A lot of people aren’t that good at keeping track of what they say, and in the case of product managers, they spend their days answering questions, so if you don’t have a very clear Spec that multiple people can be referred to, you are doomed to go through this. The main difference I’ve seen in Ushahidi so far is that people are willing to take risks. So when you ask about X to someone who may not have authority over X, they have an opinion, they give it to you, and you should understand that “hey I should also ask our PM about this” and “I probably should coordinate to know if someone has run user tests on it” are important steps in the process.

A while back our platform team implemented a new process/methodology for writing our software. It’s been quite good, not because of the specific process but because there are more tools and documents now to track how we work in detail, and we can check-in every few weeks (retros) and decide, if things aren’t working out, to change it. This is different to what we did before, where we mostly just decided ad-hoc “this isn’t working” and moved on, leaving half the team using the last version of the process and half using the new one.

How to avoid this situation, mostly:

Great documentation that gets updated, on time [we will forever still be working on this!]

Well written, reviewed Specs. With Spec reviews you want someone who knows the product well and someone who knows the codebase well. The person who knows the product will hopefully block any attempts to create a new bug. The person who knows the codebase will yell at you before you go and create a new kind of bug just because you didn’t know there was an obscure part of the project that depended on the buggy behavior.

Write your (human friendly) tests scripts before you write a single line of code. This should be part of the Spec. A test script that describes all the scenarios makes it easy to see if you understood the requirement. It also makes you think of everything that can go wrong and even forces you to look at the software closely to understand potential complications.

If possible (and generally, it’s possible) you want to do demos as soon as something can be used as a demo. You want dumb demos, html only demos, totally fake data demos. Whatever, you want something that you can show in a meeting with your product manager  and verify you understand the Spec correctly.

As soon as you have a working solution, record a demo and send it to the relevant people. Attach a link in the ticket, and beg people to watch it with a critical eye. If possible, do a voice over demo (recorded) so you can explain what’s happening.

most of the time you will find an issue when you try to demo it. Use this opportunity to polish the behavior.

If at all possible, ask the people involved with the feature to review your work. The PM might know exactly what they are looking for, but if an actual user can tell you “yep that’s good”, you are golden. In Ushahidi, we have users within our staff, so we are lucky that we can ask people to check something out and get a reaction with very little friction, instead of having to release an alpha/feature flagged version to actual external users.

Finally, use your product. In Ushahidi, everyone uses the product. I was lucky that in my first few months here I got to participate in Uchaguzi (second round) and learn how people actually benefit from what we do. It helped me realize how important some features were, but it also made me have a bit more awareness of what “feels wrong” for certain types of users. If the filters in the data mode don’t work while you select a post, that might seem like just an annoyance, but it’s a huge deal if you are structuring data. While we can’t always fix all the issues we are aware of, we can avoid creating new ones and be more aware of what the things we build will be good at.