It hasn’t been easy to organize our first test competition. We’ve had to look for collaboration tools, make sure they scale, plan the scoring, find a way to record the bugs, figure out what to test …
We have also had five registered teams, and, in that time, a lot of similar questions. Today, I will take a stab at the answers.
1. What is it about (in a nutshell)?
Sure, we test all day, but often under pressure. We often try the same thing we did last time. We don’t feel the freedom to experiment, or may not own the process. A test competition takes all those worries away, and allows us to try new things and learn – while flexing our test muscles, maybe even creating a credential we can talk about in public.
2. Why do we (as testers) / I (the individual tester) need it?
You don’t need the test competition. If you don’t enter, your life will go along just fine. What we have created here is an opportunity. By working in teams, you’ll get to know your teammates and how they think. You may come up with cool test ideas. You’ll have the freedom to experiment with something you’ve never done before without the pain of a buggy project if something goes wrong. By having a compressed timetable, when the competition is over, you can step back, examine what you did, and how it worked.
This brings the idea of deliberate practice to software testing, something we often talk about but don’t do nearly enough.
3. What is the timeline?
A week before the competition, all registered teams will receive invitations to the bug tracking tool, to explore it, along with details about our performance test tool, AppLoader. They will also receive a google calendar invite with details on our kickoff meeting, the 19th of April at 10:00AM Eastern time.
At 10:00AM Eastern you log in, go to the meeting, and try to figure out the ‘customers’ expectations. The customers are the judges; you’ll be pulling your test mission out of us. (We’ll have a blog post up at the same time that details what sites to hit.)
We will also probably have a chat room setup for you to ask the judges questions during the day. The only tools you should need for this are Skype and a Browser.
The functional competition runs for three hours. At the end of three hours, you will have filed some bugs (we hope) and will turn in a written test report.
The second portion of the competition is the performance test, which will happen over the weekend. Each team will reserve a three-hour block of time on a test server to run load tests (we will provide a load testing tool), and turn in a test report when they are complete. Teams can reserve time on a first-come, first-served basis.
4. Can I use it to enhance my CV / impress my boss?
We hope that is not your primary goal, but if it is on of your goals, well, yes. The test competition is an objective evaluation of your skills from an external party. If you even compete, at all, that speaks of how much you care about testing. To win in a category would say even more.
5. How do I participate; where can I apply?
Email Matt@xndev.com to register your team, along with your team name, players name, company (if a company team), and skype accounts/emails. Teams must register by March 10th. You will not need a skype account until early April, so if you don’t have one now, don’t worry.
6. Do I need a team?
It is possible to test on your own. Then again, it is also possible to not register at all, just go to the blog at the right time and play along. For the best value and best learning experience, we request competitors create teams of at least two people.
7. What is a valid team size?
In an earlier post Matt wrote teams of 2 to 5, but if you really want a larger team, we will allow up to seven.
8. How do we communicate?
That’s up to you. If your team is in the same room, you can just talk to each other or draw on a whiteboard. For geographically dispersed teams, we recommend Google Hangout.
9. What tools will be used?
You can use whatever tools you would like. Your functional test and load test reports should come back in MS Word Format, and we expect to use Telerik TeamPulse for bug tracking.
For performance, we will provide you with copies of Apploader and support that tool.
10. Can I use my own special tool?
Sure.
11. How is the scoring done; what is it based on?
Judges will grade each team on the importance of the bugs they find, the quality of the bug reports, the quality of the test report, the accuracy of the test report for each category. It is possible to earn bonus points by finding performance or load weaknesses and pointing the customer to improve in those areas.
12. When will the results be given?
We will announce the scores the following week by a blog post and email.
13. Are there team prizes or only individual winnings?
All of the prizes are team prizes. It is tempting for us to give an individual prize, but how do we know that the person who filed the bug found it, or even decided what to write? We can only grade the team effort.
14. What are the prizes by the way?
You’ll have to play to find out, but you’ll like them. Really.
15. I am new to some of these tools; is there a way to have support during the competition?
We’ll try to organize a skype group chat prior to the competition so you can test that. Our event kickoff tool should run in a browser. We’ll get every team member a TeamPulse account and login before the event, and if you are using AppLoader, we’ll get you support before and during the event. If you want training, this is a free way to learn a performance test tool, with a company that is motivated to see you be successful.
Whew. That’s a lot of tools. That says something about the work to set up distributed software projects. See that? I’m learning, and we haven’t even started yet!
16. Is there a way to communicate with helpers, judges or others before or during the competition?
You can leave your questions in a comment or email Matt@xndev.com. We’ll get to them as soon as we can.
17. Will (bug) reports be visible to all?
Not initially. Each team will have three hours to find bugs and create a report. After the competition, everyone will get read access to all the databases, so you can see how the other teams think. Each team will have the option of sharing reports or keeping them private.
20. Are there any rules in testing the software chosen?
We call this problem in software testing “the mission problem”. Sometimes, especially early on in projects, the mission is pure bug-stomping — “find all the bugs you can as fast as possible.” Sometimes it is to focus on a specific risk, or to make an assessment. You’ll come to the kickoff meeting at 10:00 to discover that mission, then spend the next two and a half hours executing it.
21. What is this ‘test report’ you speak of?
We’ve set up a simulation with a customer. The customer isn’t going to dig into every little bug report, but he would like a summary of the status, a list of the critical bugs, some understanding of what that means. If he has time, the customer might also be interested in the test strategy you chose — and why you chose it. That is where a test report comes in.
… Now.
Did I answer your questions?