lauantai 20. huhtikuuta 2013

Using unix shell scripts for testing RESTful web apis

Death by a thousand cuts

What I like about in unix/linux operating systems is that much of the operating system consists of small programs, which can be piped together to make something more complex. Similar approach to test tools can also be quite beneficial as you can quickly create even complex tools just by knitting various commandline utilities together.

Here are my notes about using bash shell scripts and various small utilities for testing restful  apis. I am using and demonstrating these tools under windows with cygwin, but similary you can use them on unix, linux or mac.

1. HTTP Get and Posts with curl:

Curl is a nice nifty little command-line tool for interacting various protocols (inc, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP).

When testing web-based software, I prefer to use curl for automating little things like filling and submitting a form or automatically posting and getting xml responses from SOAP-apis.

Below are commands for getting and posting of xml files to REST service.

curl -H Accept: application/xml -o response.xml

What happens here, curl sends HTTP GET to URL:
with "HTTP HEADER: Accept: Application/xml" (meaning, send me xml back please). The last "-o response.xml" means that it will write the returning response (hopefully xml formated to file response.xml). If you wish to see the HTTP codes (eg. HTTP 200 OK) which the server returns append "-i" to the curls parameters.

Similary you can also "POST" (eg. send) xml files to the server. All you need to do is to define the curl to use post-method with parameter "-X post". With "-d @MyRequest.xml" you can define that read file which has filename MyRequest.xml and add that as payload for you HTTP post request.

Example for "posting" xml-file called MyRequest.xml to
curl -i -X post  -d @MyRequest.xml -H "Content-Type: application/xml" 

2. Formating and parsing xml input with xmlindent and xmlstarlet.

Now when you receive xml response from the server, it might be a bit messy or the whole xml might be wrapped in to a single line. Thankfully there are utilities like xmlindent which can straighten it for you.

Example, xml often comes from server in format like this:
<?xml version="1.0" ?> <note><to>readers><to><from>samuli</from> <body>Hello!</body></note>

With xmlindent, you can format it a little bit:

$ xmlindent.exe -f note.xml
<?xml version="1.0" ?>

Or you can do the same thing with xmlstarlet and using the option "fo" (format).

$ xmlstarlet.exe fo note.xml 

<?xml version="1.0"?>
  <to> readers </to>

3. Grepping values from XML-files with xmlstarlet

"XMLStarlet is a set of command line utilities (tools) which can be used to transform, query, validate, and edit XML documents and files using simple set of shell commands in similar way it is done for plain text files using UNIX grep, sed, awk, diff, patch, join, etc commands."

In the example below I am using it for parsing values from xml file and then using these values to create a new xml file which is then posted to webservice.

Example. lets pretend we have xml response (named response.xml) we received from webservice:

<?xml version="1.0"?>
  <query id=101> OK </status>

For this we wish to grep the query id -number attribute and the amount value from the amount-element,
so we will write a bash script to grep the values and then print them out or send them again via http-get

Var1=`xmlstarlet sel -t -v //amount`
Var2=`xmlstarlet sel -t -m "//Response/@id" -v .`
echo $Var1

The first line asks the xmlstarlet to go through all fields in the xml-file and print out values of the amount-elements. In this case I only have one amount-element but as I was too lazy to write full path to the amount-element I just ask it to select "all" amount-elements and print out the value to variable Var1.
The second line selects the value of the id-attribute inside the query element and then stores it variable Var2.

We can now use these variables as we wish. In my example, they printout the var1 to console and use var2 as parameter for http-get request.

4. Modifying xml files with xmlstarlet.

xmlstarlet can also modify existing xml-files and create a new xml-files from them like this:

xmlstarlet ed -u "//amount" -v 42 -u "//description" -v "test purchase" request.xml > newRequest.xml

This would read the request.xml file, replace all values of amount-tag with value 42, and replace all values of description with value "test purchase" and print out modified request.xml  in to a new file called newRequest.xml and leave the old one as it was. When using inside shell-script you could write in in format like this: xmlstarlet ed -u "//amount" -v $1 -u "//description" -v $2 request.xml > newRequest.xml
You could then call the script with parameters like: 42 "test purchase" which would then add those to the newRequest.xml file. This is particulary useful when you want to use a value from previous requests and then send them to another service. You could also hook this up to text/number generator and create multiple testdata files.

5. Parsing json requests with resty and jsawk.

For parsing JSON request there is a nice little tool called resty.

You can grab it from following github url.
Below a short description on how to use resty for get, post, put, delete json requests.

Resty is provided as shell-scripts and aliases. To activate/setup resty you need to source the resty-file:

$  . resty
 To define target host for the resty client, you need to do this:

Then we can proceed to execute get, put, delete, post request to json services like this

  $ GET /blogs.json
  $ PUT /blogs/2.json '{"id" : 2, "title" : "updated post", "body" : "This is the new."}'
  $ DELETE /blogs/2
  $ POST  -v -u user:test /blogs.json '{"title" : "new post", "body" : "This is the new new."}'

For modifying json requests there is a tool called jsawk  (from
jsawk uses javascript for its command language. In the example below we http-get a json-formatted response and then change the favoriteColor to blue and then http-Put it back. (copied from jsawk readme file).

 GET /people/47 | jsawk 'this.favoriteColor = "blue"' | PUT /people/47

Because jsawk uses javascript you can also do tricks like these:

GET /people/47 | jsawk 'this.age++'

Thanks for reading. If you have any questions or improvement suggestions. Please add comment or send me a tweet @samuliel. Happy testing :-)

maanantai 3. joulukuuta 2012

What does it take to become a better software tester?

Lately I have been thinking on the subject about how to become a better software tester. Actually what defines a good software tester? What are the skills that good software tester should have?
So I decided to make a list on three most important skills which I should have in order to be a good software tester (in addition to the actual skill of software testing):

1. Computer science skills.
I think these are must for anyone working at things related to computers and computer software. These skills include things such as programming languages, understanding inner workings of computers and common operating systems, understanding things such as algorithms and computing theory, understanding computer networks,  etc. To test software effectively you need to understand how software actually works and how can you ease your workload (automation, visualization, alerting, etc.) with the help of the computer. If you are looking for points where to start studying check out MIT Open courseware video lectures on computer science ( or the just awesome Khan Academy ( where you can study lots of different subjects.  For programming languages my favorite language of choice is python (,, )

2. Systems Thinking and modeling skills.
To quote famous late Mr. Russell Ackoff:

"System is not a sum of the behavior of its parts; it’s the product of their interactions". Software can and should be thought as complex system. In order to understand those systems I need to be able to see the whole picture (eg. To recognize the forest from the trees).  To test such system I need to understand how the different parts interact together and I need to be able to model this. For this, the Systems Thinking is an important tool. Recommend this video for quick 12min description on Systems Thinking:  , for more I recommend this book

3. People and communication skills.
Ultimately all software is made by people for the people. I need to be able to understand and communicate with the people in order to test the software effectively and present my findings in clear and professional manner. For this I need to study psychology, social sciences, rhetorics etc.

So how do I work on these skills?
One way is to read books, lots of books. Here is a nice mind map from my fellow tester colleague Aleksis Tulonen. Funny thing is that even if you read just one book a year related to software testing, you are in the top 10% of testers who study their craft. (Ok, maybe this is an overstatement but sadly I find that less than 1 out of 10 testers in Finland, whom I have met, actually read books related to software testing others don’t read even the single book). Then again I would say that 1 out of 20 books related to software testing actually deserve to be read others are just waste of paper. And remember “all work and no play makes a jack a dull boy” so don’t just read IT-related books but read books on psychology, read books on social sciences, read books on philosophy, read books related to observing, read books on how to conduct scientific experiments,  etc. All these things add up and raise your general knowledge on things which in turn make you better prepared for future testing related problems.  Knowledge attracts knowledge!

Second way (and I think the most important way) to study testing is to actually practice it. Like they say about musicians, a musician who does not play or compose music is not a musician. Same goes for testing. In order to be a good software tester, you actually need to test. Everyday.  So after reading those books on software testing, systems thinking, computer theory, observation, you need to apply those ideas to actual testing. So go out, think, apply and test. Ok, I hear you saying “I cannot test anything as I am not currently part of any software project or I don’t have any time”. I say that is just an excuse. Do you have a smartphone? Do you have a personal computer? Do you browse the internet? You can practice testing with all of those.  If you have a time to browse or play with you phone, then you have time to test. Try mini-sessions  (10-15min of testing). Participate in open source projects, test websites and give feedback to webmasters, try and install a new game or software for you smartphone.  You can even test the software inside of your “smart-TV” these days!  Just test it.

Third way to practice your craft is to learn how to talk about it. Go out and attend to conferences, talk with the testers you meet, ask for coaching from other testers. You can also do parts of these things via internet. Just start a testing related blog and blog about your findings. Or if you don’t have time for blog, then surely you have time to write max. 140 characters on your twitter account. You can also ask for coaching via skype from people like James Bach or Pekka Marjamäki.  I have also founded a channel in IRCnet called #softwaretesting if you like to hangout in IRC, feel free to join in.

 - Samuli

tiistai 30. lokakuuta 2012

Stranded on Rapid Software Testing course with James Bach

My review of the Rapid Software Testing course by James Bach (Helsinki, October 2012)

So last week Monday I found myself stranded on a Hotel Marski at Helsinki together with about 20 other people and James Bach.

I was attending course called Rapid Software Testing (RST). RST is a class or maybe more like an editorial opinion on testing by James Bach and Michael Bolton. The class consists of three 7 hour days, which consist of testing lectures and exercises. James tells that he invented the class because after reading dozens and dozens of books, talking to other testers, attending conferences. He noticed that nobody actually teaches or talks on the actual testing. Often the testing is treated as it was black magic and that it is the tooth fairy who finds the bugs. Thus he and Michael Bolton decided to make a class out of their study on "what testing is actually all about".

The Exercises. (exorcism of testing fallacies?)

Photo by Heidi Elomaa
Every exercise in this class has number of traps in it. Ofcourse I stumbled to almost all of them :-)

The first exercise we had was the famous wordpad exercise. In this exercise you are asked to open Microsoft wordpad and test a specific feature in it. In this case it was the font size option. You are not given any spec or test case, but just told to "test it". So at first I thought being clever and tried to ask the specs from the oracle.

Me: "James, how should the font-size feature work?"
James: Looks me like I am retarded, "Haven't you used computer before?!"
Me: I try to squeal, "But how should I know how you want it to work!? You're the oracle."
James: "Its a wordpad! you know how it works. Samuli, there really are stupid questions."
And proceeds to tell how me he got fired at some job in the first day after 90minutes.

James: "You want me to test it for you?" "You want me to test it with you?"
This was the catch! You should test it together with the oracle, ask him to sit down with you and to show you how it works, while you feed him with chocolate cookies. Same principle is surprisingly effective also in real world (when possible).

The class consisted of many other similar type of exercises which main purpose was to inform students about good testing heuristics and get rid of common testing fallacies such as "the amount of testcases"-problem. Actually there was a good exercise on this where the students were asked to find bugs from ip-address calculator with as little testcases as possible. The funny thing was that the bug in the ip-address calculator appeared randomly to each student. So that the amount of testcases was actually not the issue as some people got lucky and got it with only one or two tests. My number was 3. Then there were some people who tried as many as 48 times before the bug appeared which was after they were able to break out of their testcase patterns. Thus it also acted as defocus-technique learning point.

Yes, we also did the dice game. Which I had never done before and I am planning to do it at next competence development-evening at my company.

The Lectures

Photo by Heidi Elomaa
 The lectures&exercises are based on James (and his peers) previous experiences on testing software. James uses this socratic-teching method coupled with his awesome improvised acting skills. James drives his points through by putting the students to hot seat of pressure where they are trying to solve (often impossible) testing dilemmas in front of the class. When I told my wife after what happened on the first day to my collegue Aleksis . (James basically spit on him, see this post from Pekka Marjamäki) My wife was shocked and asked why do I want to be on class where I am almost humiliated in front of everybody :-)

Ofcourse this is all just acting, thus I was not really taking James attacks seriously but the method was quite effective on driving the ideas to students heads. Many of the exercises can also double as interviewing questions to new hires or to dispose illusions on things like Boundary value analysis-technique.

Actually, I finally understood why explotory testing is approach not technique.James was telling the class that one of the things he is actually author of is exploratory testing, as he invented the term. I interrupted him and asked that didnt Cem Kaner invent the term before him. He told me that Cem was actually the first one to mention the term in his book (Kaner, Falk, and Nguyen, Testing Computer Software). But Cem forgot about the term, and James picked it up later in order to rename his "Organic testing"-approach which he earlier called AdHoc-testing approach. Later Cem re-read his book and noticed that James had been feeding him his own words as new words :-) So Cem gave James the ownership to exploratory term. Cem and James later studied what the explatory testing is actually made off and came to the current explanation which is: Concurrent test design and execution. You can read more on exploratory testing here. James, If I have misunderstood, please ping me.

Sadly we ran out of time, thus we did not cover through all the topics. The one I would have liked to hear more was the test documentation. Luckily I can also study it by my own from the slides (which are available at and James has promised that whenever he his on skype and has time, he can provide mentoring on testing matters. Maybe one of these days, I'll catch him through skype or email and ask more on the subject. 

After thoughts

Photo by Heidi Elomaa
So what did I get from the class? The most immediate thing was this warm "feel good" -feeling. I felt like I gained more power and if I keep trying I can gain these IronMan-style testing superpowers. I got more motivation to study testing which actually consist of many disciplines such as social sciences. Also I learned that testers are like scientists. Their testing is applying logical and critical thinking coupled with empirical methodologies to solve testing problems. Actually James said in the class, "Apply the skills of scientist to the field of testing and you get Rapid Software testing". This is what I also need to become, in order to do my Job as a tester. Thank you James.

 Here are some great quotes, (ripped out of context :-) from the class:
"Always know your mission, and understand the context where you are",
"Bug is something that bugs people who matter",
"Testers are not here to recreate the spec, testers are here to find the actual behaviour",
"Tests are performance, test is about DOING!"

Lastly,  "Curiosity matters!".

Stay curious my readers and be ready to strike upon uncertainties.

Buccaneers motto

What is buccaneer testing?

Buccaneer is someone who asks no permission to go to adventure.
Buccaneer tester is someone who asks no permissions on his methods of testing and is always ready to defend his work. Buccaneer tester is someone who tests freely but still stays accountable for his actions. Buccaneer tester is someone who has passion for curiosity and asking questions.
Also Buccaneer tester is not a leech, he shares his knowledge with others for free.

Buccaneer tester is always on voyage to hunt new information from the stranded&isolated islands of knowledge. Rise the sails!

Source: Wikimedia Commons