| STAQS |
Software Testing and Quality Services Ubi dubium, ibi occasio. |
| Home | Software Testing Primer | [ Starting Points ] | Presentations | Blog | Distractions |
"I keep six honest serving-men (They taught me all I knew);
Their names are
What and Why and When and How and Where and Who."
- Rudyard Kipling
What I know about Software Testing comes from the School of Hard Knocks: stuff I've read in books and on the internet, things I've tried (and failed and tried again), conferences I've attended, workshops and seminars that I've participated in, and from discussing ideas with some of the many smart people out there (consultants, authors, professors, fellow testers, my wife, and so on).
After a while you begin to see the patterns of what's likely to work and what won't in certain situations. This page captures some of the things I've learned about Software Testing over the years and some of the links that I've found useful on past testing projects.
| Quick Section Jump: | Ruby and Watir, Exploratory Testing and SBTM, Internationalization Testing, Performance Testing, Security Testing, Community Resources (Job Hunting Tips). |
I have been using Ruby and Watir (Web Application Testing In Ruby) for a few years now, and I am totally hooked on Ruby! Never in my life have I been so productive in developing test automation scripts as I have been with Watir! (Mercury who?)
The foundation of testing web apps with Watir is made up of 3 parts:
Download the Ruby and Watir one-click installers from: http://wtr.rubyforge.org/
The thing to note here is that many people test with Ruby all the time but don't use Watir (i.e. using just 1 & 3 above), and you can use the Watir library to talk to web apps without necessarily testing them. There's flexibility in how those 3 elements above can be used together, separately, or interchanged with other components. The above combination for the purpose of testing web apps from a user's perspective is the purpose I'm interested in here.
Depending on your particular testing project's needs, you might also need to use other Ruby libraries. For example, to help you read/write to Excel spreadsheets, parse XML, or use an additional scripting framework - e.g. like RSPEC.
If you are interested in Ruby, there are many sites and resources available to help you learn more. Some include:
Some helpful Watir resources include:
So now that you have a free, powerful test framework in place to help you test your web apps. Where do you go from here?
The answer to that question is tricky. Many people think that the answer lies in the programming side, but I think it's more important to understand the testing aspect first. That is, do you really understand what it is you are asking the computer to automate? Is the test really worth automating? How will the computer detect errors or bugs? How will it recover from unexpected events or errors? What value will you get from running the exact same test repeatedly? and so on.
Final thought: Break free of the testing norms of specifying the exact steps and test data required to drive your test scripts. Teach your automated scripts to perform exploratory scripting -- i.e. learn as it goes, and vary the inputs for each run!
>> Follow this link for a page dedicated to the SBTM and ET lessons that I've learned over the years and to download my Ruby SBTM tools <<
Lots to say here. Where to begin? In a nutshell, ET is a form of unscripted testing that requires skill, creativity and discipline to effectively go where no tester has gone before. The productivity and effectiveness of an Exploratory Tester will greatly surpass that of any generic scripted testing approach any day. The reason for this is simple - the focus of this approach is on the learning and iterative nature of testing to leverage the system feedback to maximum effect. Traditional scripted tests put the focus on writing test documentation and test cases - which can be an incredibly huge waste of time, money and brain power.
There are few ET articles currently written that I really like. I recommend starting with the basics and moving on from there. It will likely be very difficult to learn to do this by reading alone, so I suggest you network and find people to share experiences and knowledge to help you pick it up quicker. Try conferences, workshops, or presentations in your local Testing community.
Start with an introductory article: "What is Exploratory Testing?" Exploratory Testing is defined here simply as the concurrent Learning, Test Design and Test Execution.
Once you accept the idea that unscripted testing can be more effective than the documentation-based alternative, you will need to find a Test Framework to help you manage your testing. I've used the "Session-Based Test Management" (SBTM) framework that James and Jonathan Bach developed. I've met some people who have used WIKI's as an alternative to SBTM. The idea is the same, just the metrics and visibility to the test notes are different.
I find that this style of exploratory testing is a perfect complement to Agile Development projects where changing requirements are common and rapid feedback is desired. ET is often my primary approach to testing new projects because it allows me the freedom to explore features and investigate ideas more rapidly than by other means.
Internationalisation (I18N) and Localisation (L10N) testing are a lot of fun! I wrote an article titled "Testing with an Accent" which appeared in the February 2005 edition of Better Software magazine.
There are a lot of useful links on the web. Here are a few:
Here are some ideas for when you're actually testing:
Every now and then I like to dabble in some Performance Testing. There are many different types of Performance Tests, tools, and techniques.
One book that I would highly recommend if you are thinking about doing Performance Testing is: "The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling", by Raj Jain. 1991, Wiley. Microsoft also has an excellent introduction on their MSDN Library site that you should read. Follow the link for "Chapter 16 - Testing .NET Application Performance" and read the whole page - several times.
In the Fall of 2004, I was asked to get some Benchmark performance numbers on a web application. I evaluated several free tools (MS Web Application Stress Tool (WAST), MS Application Center Test (ACT), and WAPT 3.0), but OpenSTA was the only one that came close to working with the Microsoft .Net app and simulating the desired workload.
Lesson Learned - Using OpenSTA: Dammit Jim, I'm a tester not a programmer!
One of the problems I had with the tools listed above is that they are geared towards programmers. Even the tools that appear simple enough for non-programmers to use will inevitably require help from programmers. It really is unavoidable. At some point you will need help understanding the web application architecture so you know what your tests are measuring.
My recommendation for similar types of performance testing projects in the future? Team up a programmer and a tester for the design and data collection phase of the testing project -- i.e. until you have the desired observations and measurements. You can then let the programmer off the hook to return to their regular duties while you complete the analysis and presentation of results.
Lesson Learned - Using Watir:
My first exposure to Ruby and Watir came when I needed to walk through a .Net web application to collect some page size and load time performance measurements. Turns out that scripting the application with Watir was so easy that it was like reading and writing pseudo code! Some of the commands I used to collect data were simply:
ie.down_load_time - how long (in seconds) the page took to load in IEie.html.length - what is the size of the loaded HTML page?Through the power of Ruby and its libraries, I have since expanded my scripts to collect many different kinds of performance metrics as required. There are some good examples available on the web. For instance, check out HttpWatch for some sample scripts that show you how this cool tool can be scripted with Ruby and Watir.
One thing I have not yet tried with my Watir scripts is Load Testing. I have written scripts to simulate a single-user load, but just haven't gotten around to playing with multi-user load yet. I will update this page when I have more info here.
Fun, fun, fun! When all is said and done, how safe and secure is your application or system?
For me, much of Security testing is like I18N testing - an integrated part of how I just regularly test software. My focus lately has been on testing web applications. (Just the web application itself though, since the whole system is covered by a myriad of security protocols, audits, firewalls, and other barriers that are well beyond the scope of your average test team.)
Here are some helpful links:
Some starter books that might be of interest include:
I still remember the hard time I had trying to get a full-time job when I graduated from school back in the 90's. The High-Tech sector was there but the internet wasn't as well-organised as it is now, so I knocked on a lot of doors until I got lucky. This section offers some ideas on resources to help you get into Software Testing and Quality Assurance in our local community.. which happens to be Kitchener-Waterloo, Ontario, Canada, by the way. (That's about 100 km West of Toronto.)
STEP 1. Get to know the Area:
STEP 2. Finding the Jobs in town:
FYI: While the links in this section may not be as useful to internet visitors from afar, I encourage (nay, challenge!) you all to ask yourselves the same questions: What information and resources do you think would be useful to someone trying to get a job in your local community? How can you help?
| Home | Software Testing Primer | [ Starting Points ] | Presentations | Blog | Distractions |
©2008 Paul Carvalho. Contact me at: paul [at] staqs [dot] c o m