Saturday, November 14, 2009

Knitting & Testing

This afternoon, I dropped by while James Bach was signing copies of his latest book, The Buccaneer Scholar. Somehow I mentioned that I knit, and he explained one of his new teaching ideas, that test script writers could learn a lot from the way knitters write patterns. It's common to write test scripts so that anyone could read and execute them. But a knitting pattern is written for the (more-or-less) knowledgeable knitter, who is already familiar with a number of abbreviations and other conventions in knitting patterns. Just as it would be incredibly tedious to re-explain how to ssk (slip-slip-knit) every time that stitch is required, we could perhaps save a lot of tedium in our test scripts by developing a common jargon within a testing community[1].


[Edit: In the following paragraph, I completely failed to mention the contributions of @superpuppy, who shares with me an interest both in knitting and in breaking software.]

Then James sent me on a mission to find 2 books -- one that just assumed you knew knitting and used its jargon, and another that provided the introduction needed to understand and implement the patterns in the more advanced book. I found some books for him (5, since I'm an over-achiever), but the process sent me in 3 other directions as well:


1) The books on that shelf were not the intro ones I would have recommended if I was picking from all knitting books ever. While the Borders' collection of knitting books was respectable for a mall-sized chain bookstore, the selection felt limited compared to a yarn store or the internet. The main book I referenced while learning to knit was the Vogue Knitting Quick Reference. For crochet, I used 10-20-30 Minutes To Learn To Crochet.[2] The books I did offer James were still reasonable intro guides, covering the jargon and formatting of patterns. The crochet intro explained not only the cryptic terminology, but also the standard way of diagramming crochet. (The knitting may have taught how to read knitting charts as well, though I didn't notice it.)


2) There are a lot of parallels between knitting books and software testing texts. There's the plain old pattern collections, like test scripts. Then there's books that talk about what to do when things go wrong. And just as a tester may turn to hardware documentation while testing software that runs on it, so a knitter might find a book exclusively about various types of fibers instructive. At first I thought this was because the two crafts are similar, but perhaps it's more because human beings learn different skills in similar ways.


3) It feels like there's a lot more connections between knitting (and/or crochet) and testing than just the idea of how to write up scripts that James identified. I'm going to start a list here, and I hope you'll join in by commenting below, or tweeting me.


- A pattern is a test case/script. There's a set of steps, and an expected output.


- Keeping track of exactly what you've done so far is important. Count your rows, log your actions.


- Following the pattern/script exactly is a reasonable first step. But variation is good too! Try a different yarn and finishing stitch. Try it using the mouse instead of the keyboard, or with a bigger input file.


- When you diverge from the beaten path, keep track of what you did, so you or someone else can do it again later.


- There are books that can teach you how to do this stuff. But a workshop (at Weaving Works, or PNSQC) can be so much more effective. There are also videos (how to knit a moëbius, how to test an Easy button).


- There are heuristics. The Knitting Stitch Bible is nothing but stitch pattern heuristics -- no patterns for whole garments, no 'how to knit and purl' at the beginning, but 39 ribbing variations and 37 cables to choose from. I have yet to find such a concise book of heuristics for software testing, though I do find individual heuristics scattered all over the internet, and James gives great detail about a few in The Buccaneer Scholar. (Why isn't there an encyclopedia of testing heuristics?)


- There are holy wars, based on over-generalizations, imprecise terms, and uninspected biases. Knitting is more refined than crochet. Automated testing is more rigorous than exploratory testing. Well, what do you mean by 'refined', and is that necessarily a good thing? What do you mean by rigor? Or automated?




[1] A community could be a project team, department, company, formal or informal intellectual group, etc.

[2] This book deserves special note as the only intro book for crochet I've ever found that respects left-handed students. Most books show the first diagram or two redrawn for a left-handed context, and then expect you to just reverse the right-handed diagram for all the other techniques they teach. 10-20-30 includes diagrams for many (all?) of the techniques where hand positioning is important.


Monday, August 24, 2009

Upstairs, they call that a Liz

This is the best analogy I've seen yet for how I became a software tester:
http://www.youtube.com/watch?v=avSLMAKaBxg

I didn't accidentally kill anyone while getting my CS degree, but I sure wrote a lot of bugs and crashes into my homework assignments. Spending all that time chasing down segfaults seems to have trained me to look for similar mistakes in other people's applications.

Thursday, September 18, 2008

You Don't Want to Hire a Tester

Testers can be a real pain. As a developer, you are hiring someone to complain. And not about life in general, or The Establishment, but about the application you have spent long hours lovingly crafting out of bits. You spelled a word wrong, that text box isn't quite aligned with its label, and the whole thing takes just a little too long to load. The documentation's rather terse, and shouldn't the icon be just a little less shiny? 

Testers ask questions. Annoying questions. Where are the release notes? Does that change affect this other thing? If the app can do this, why can't it do that? Which third-party apps do we need to be compatible with? Where did this half-baked, unplanned, crazy new feature come from? 

Testers check the quality of things besides the software you told them to work on. They clog the network stress-testing an app. They accidentally delete your source control repository if the permissions aren't set right. They critique the bug-tracking database. They suggest changes to the lunch-ordering process. 

Testers delay releases. They always seem to find a serious bug when you're just about to push a new version. And now you're elbow-deep in backtraces, rather than basking in the warm fuzzy feeling of a release. 

Testers cost money. Besides their paycheck, they want a Mac to test the app. And maybe a second one, to use while they're re-installing the OS on the first one. And then a Wacom tablet, because it sends different events than a mouse. 

Testers are trouble. You don't want to hire a tester. 

Friday, September 12, 2008

Summary and Questions

So far, it seems to me that there are a few different broad topics this presentation needs to cover: 
  • The difference and trade-offs between beta testing and internal testing.*
  • How to utilize beta testing most effectively.
  • What internal testers can and should be doing to test your applications. 
  • What skills, experience, and other traits you should be looking for when hiring a tester.
  • What tools and resources are available for testers and their bosses. 
  • How to get along with your tester and make them more efficient.
A few of these topics will require further thought and research on my part, but I think they're all things you need to know. And please let me know if there's something I've missed that you'd like to see addressed. In other words, let's do this presentation backwards, and start with the questions. 
-----
*By "internal" testers, I mean both direct employees and contractors/consultants–people who are using your software because you pay them to, not because it helps them get other things done.

Tuesday, September 9, 2008

Inspired by C4

I went to C4[2]*† this weekend. I've been interacting with Mac developers all wrong. I should not be apologizing for being a tester, and participating in spite of my profession. It is because of my talents and experience that I can and should attend these events and discuss software, and everything related to it, with Mac developers. 

Several people at C4 suggested there should be a talk on testing next year. But as @arclite tweeted, C4 is a state of mind. So I'm not going to wait until next year. I'm not going to make you wait until next year. You, Indie Developer, need some advice on testing your Mac apps now. As I develop my talk for next year (which may or may not actually occur), I'll post the content here. Expect the first installment within a week. 



*Is there a better link for people wondering what C4 is?
†Some people have trains of thoughts. Mine seem to be closer to trees. Footnotes seem less distracting than parenthetical commentary.