Effective Bug Reporting
“It doesn’t work”, and other ineffective bug reporting techniques in an age of agile development.
When I started my software career in the late 1990s, the ideas of bug reporting weren’t new, but in the 20 years since, much has changed with agile development, newer software systems and ideas about tracking software and tasks.
In particular, there was the following seminal classic by Simon Tatham back in 1999:
Even now, (or perhaps, especially now), Simon’s comments remain relevant, and this has always been the article that I’ve referred people to. However, after years of software development, and working closely with QA engineers, and with reference to newer reporting systems, it’s time to make my own version, especially since I haven’t been able to find anything that is as comprehensive as what I want to articulate here.
What is a bug?
First, I’d like to turn the paradigm on its head and explain that when I say “bug”, I’m definitely not limiting the meaning to a software fault of the kind we have traditionally thought about.
In some reporting systems, it’s called a “story”. This is deliberate, and in keeping with what I’m about to assert. To this end, you may interchange “bug” with “story” or “task” or whichever nomenclature your systems or development style uses.
Here are some examples of what a bug or story might be:
- A report of a fault
- A task to be completed or to-do list
- A change request
- A group of sub-tasks under another story — sometimes called an “epic”
- A group of task to be completed in a given timeframe — sometimes called a “sprint”
This is list is naturally incomplete.
Here’s my point — a bug or story is something to be done — it describes the current situation and what the outcome might be. Put differently, it could describe every single step of software development — the specifications to write, testing to be done, documentation to write, faults to fix, etc etc. Now, to seasoned software engineers, this is probably obvious, but for many engineers, I know it’s not necessarily so, so I’d like to repeat it one more time:
A bug/story is a description of a task to be done and tracks its life cycle.
Clear on that? Good. Then the rest should make sense.
What a bug isn’t (In my opinion)
- An observation without conclusion or task. “The sky is blue”.
- A forum for speculation (at least not initially, and not without follow up)
- Multiple issues — unless they are closely related, they should be broken up and treated separately.
But they can be…
- To Do lists (as long as there is completion or follow on)
- Repeating tasks — such as release processes
- Wishlist features which may or may not be possible
- QA/QC test procedures orthogonal to developer tasks
Bug Reporting Systems
Here are some of the common bug reporting systems around:
- Bugzilla — the old standby, and much in evidence when Simon wrote his piece, but not so much used now.
- Trello — Task based, “Sticky note” system. Not really a full featured system, but also appealing in its simplicity.
- Jira — In some case, the industry standard. Free for small use, but they charge for more users/features. But very fully featured — perhaps too many features.
- Assembla — Similar, but not quite as flexible.
- Trac — The bug system built into github.
- CRMs — Customer issue systems are definitely bug report systems too. What’s reported in them usually has a lot of steps to go before it can be turned into a credible issue, but they are definitely part of the process.
Why bug reporting systems can be bad
There are lots of reasons bug reporting systems can be intrusive or irritating. Like much, it’s a matter of balance.
- Excessive emails — systems like Jira can produce extraordinary numbers of emails if you look at them the wrong way. If you subscribe to everything (as I’m personally wont to do), then it’s a lot. Most engineers simply want to see the stuff they’re doing right now, and find the rest distracting.
- Engineers forget or don’t like to update — in many cases, writing down what was actually done falls by the wayside — or worse, what they *intend* to do — you know that person. Using systems effectively is a matter of habit and sometimes badgering.
- Taken to extreme, systems like Jira and Assembla become entities unto themselves, and you can end up managing the bug reporting system — “Feed me Seymour”
- Managers sometimes end up managing titles rather than issues (“why a good title is important”) and promoting “urgent” things at the cost of all else.
The Importance of Titles
“A good bug title states something which is true, but you want to be false”.
Not my quote, but a good rule of thumb. Of course, not all bugs fall into this category, but an effort here pays dividends upon review of a bug much later.
Good:
- “The application shows the measurement to too many decimal places”
- “My widget can’t connect to the internet via a specific adapter”
Bad:
- “Network issues” (Really?)
- “When I went outside and tried to use my tablet, it was too bright and I couldn’t see the screen and I had to go home and I was sad” (Shouldn’t be a narrative)
- “Bug in user interface” (Of course it’s a bug, but what is the issue)
The point here is that poor titles are misleading, cause miscategorization and misunderstanding. So, your aim for a title should be as terse as possible, but also as accurate as possible.
What a good report might have
- Short but descriptive title — As above
- Software versions — To allow tracking of when it was discovered, fix and regressed.
- Repeatability — Obviously important. A bug that’s not repeatable is confounding. However, in my considerable experience, many times the poor repeatability is down to assumptions made by the reporter.
- No assumptions — just the facts — Sometimes it’s up to customer support to tease these out.
- Should not have (or be!) any follow-on problems — There’s a strong temptation that once an issue is discovered, to work around it — of course, this leads to yet more issues. There are exceptions to this of course, but it almost all case, it’s really not worth it — coming across an issue should mean a halt to testing of that system.
- Have a checklist— that is, there might be list of steps that all bugs go through when they are entered. This is because the software is expected to work in certain ways before even the new issue can be considered — or it might immediately shed light on the issue.
- Be able to evolve — descriptions change as the issue is better understood. Titles and even the whole body can be changed. The bug can be split up or made obsolete.
- Might follow a template — sometimes companies enforce certain information to be entered. This can be setup in the reporting system, depending upon features, but it might just be a list of attachments, and the results of the checklist above.
Other guidelines
- Pictures are good! Unless it’s text, then cut and paste is much better, since that means search ability.
- I discourage videos — pictures are better. I personally find videos distracting — they sometimes seem useful to the reporter, but they have a strong tendency to allow laziness — before and after pictures are better
- Avoid contractions — readability is important, and sometimes shortcuts outlive the owners of bugs.
- Good spelling and grammar is always important. Remember — the next person that reads the report is likely to be yourself
- Proofread; make sure it makes sense.
Exercise
For practice; come up with some bugs relevant to your situation. Make sure it follows the guidelines I’ve outlined.
Have a peer review it, who’s familiar with the problem:
- How does the title read?
- Does it try to avoid assumptions?
- Do my guidelines work for your situation? What would you change?
Finally
Not everything here works for everyone. But I think most of it does. But good guidelines adapt like good bugs. I expect what I wrote here won’t be as correct as it was in a few years, just as Simon’s is no longer complete.
Good luck!
