It was a quiet Friday Morning in the United States. Or, I mean, Friday afternoon in India and Singapore. Later in New Zealand, and in Europe ...
The point is, there were a lot of people playing. Seventeen teams registered four different continents for three hours of aggressive testing, followed by a weekend where teams rotated through performance testing.
Now, the winners in each category.
It’s here! It’s here! If you are reading this, then the Test Competition is finally here!
No, you don’t need to have registered in advance to play, just to win the big prizes and have your name in lights.
Either way, here’re the rules of play.
On March 11th, 2013, registration for the test competition closed.
We have sixteen teams registered, from India to Europe, Southern Asia, and the United States. We have a team that has people from two continents.
And that's just the folks who gave me a street address.
One of our contributors, Pete Walen, recently put out a blog post on leadership and power. It is a good post; if you enjoy reading long-form articles that are more conversation than bullet-points, I suspect you'll get a kick out of it.
Pete's article did a lot for me, but most importantly, it got me to think, to finally publish that piece on power that had been niggling in the back of my mind for nearly a year.
Instead of a rambling conversation, I'd like to start out with an assertion, then defend it, then suggest a thing or two to do about it.
My claim is very simple: Most of what you've been taught about Power is probably wrong.
If you've been following me talk about Method-R, you might wonder how I learned it. In my case, I learned it the "right way", the "official way": My employer brought in a consultant to teach a three-day class in performance tuning. Method-R was the method he taught.
It also got me very, very interested in code profiling, which is a way to measure performance at a much more detailed level -- not just is it fast enough, but, instead, how long does each piece take?
Let's consider, for a moment, the poor IT manager, told by end users that the application is "too slow."
I've been that guy. You may be that person some day ... you may be that person now. Either way, it is safe to sat that those in that position would prefer to be out of it.
The manager wants to fix the problem, so he goes downstairs to the DBAs and tells them. The DBA's do some research and find out that the memory percentage is way high, so the database is going to disk (a lot), and disk is slow.They suggest buying a great deal more memory , which is the cheap and easy solution.
In most cases, the solution is a lot more complex, and involves tuning, news boxes, creating new indexes, maybe upgrading the network switch. After two days, two weeks, two months or two quarters, the DBAs say they are done, and show some dashboards like the one above, with all the data trending in the right direction. Average CPU use is down, so is memory, disk I/O requests are down, response times are down - everything looks good.
Allow me to ask, just for the sake of argument. What happened to average customer response time?
We don't know.
It is this problem, our love affair with data that is often irrelevant, that Method-R solves.
It hasn’t been easy to organize our first test competition. We’ve had to look for collaboration tools, make sure they scale, plan the scoring, find a way to record the bugs, figure out what to test ...
We have also had five registered teams, and, in that time, a lot of similar questions. Today, I will take a stab at the answers.
1. What is it about (in a nutshell)?
Every week on "The Apprentice" may be different, but we know one thing: Donald Trump is going to walk in, ask questions, fire someone, and drive on.
You might want to work like "The Donald" on the show, but if you measure software performance, you probably feel more like a contestant on the show. After all, you start with a problem too broad and vague to adequately solve in the timefame, are involved too late to impact quality, and assigned too little staff, software, and hardware resources to do the job right.
If you've ever felt like that, well, I have too, and I survived. Today I'm going to offer a way out.
It has been two weeks since we announced the details and how to register for the NRGGlobal test competition. Since that time, we've had three teams register.
No, I'm not worried, here's why: At the same time I am organizing this, I am also co-chair of the test and quality track of the Agile Conference. That conference had almost no submissions, I was near despair, until the submission deadline of February first approached. Now I have dozens of submissions to evaluate, which puts me in a different kind of despair. :-)
No, seriously, it is alright, we have a review team that is slicing up the reviews right now. I will be fine.
It is called the basketball exercise, or sometimes inattention blindness, and it demonstrates a real problem in many kinds of automated testing.
What is this thing, you ask? It can be hard to explain, but I can show you in about a minute. Look, there's video:
Today I’ll cover the story of project number two, "just a little code" to integrate two systems. But first, let’s talk about my motivations.
About the Killer App Series
"The Killer App" is not a work of fiction, or a work of historical fiction. It is not a made-for-TV movie, "inspired by" true events but with the drama cranked up a notch or two; it is history as I recall it.
While it has been a pleasure to get this off my chest, I have no axe to grind. The players have moved on. Enough time and space now separates us that even the “bad guys”, as much as there were bad guys, would probably not recognize themselves in the writing.
Nor is this a case study, all wrapped up in a bow with a key learning takeaway.
The Killer app is ... a story.
My story, about a time in my life before I was a writer or consultant, back when Matt was an employee, and, sometimes, felt more than a little trapped by his situation. The story points out that there is more to life than process -- that often, it is the sticky blobs of flesh called humans that can make a difference. I didn’t always make the right choices. Back then I was more than a bit scared of conflict. But if a person or two can learn something from the story, and do it with a smile on their face ... I hope you’d agree with me to call that success.
Now it’s time for round two.
The Next Killer App
At the end of the killer app, our hero had transferred off the project, to go work on project number two. The transfer was no brilliant duck-and-groove out of the matrix.
The reality was that I failed a code review, so they transferred me to fix the code.
At least, I tried to fail the code review.
You see, the code came to me late in December, between Christmas and New Year's, with a bow on the end that said it "had to go into production by the end of the year."
The code itself pushed data from one system into another and provided
Over the past two weeks I have spent a great deal of time doing research, gathering resources, and judging interest in an on-line test competition in software testing. My conclusion is kind of like Nike.
Let's do it!
It's time to talk about details.
What: The first NRGGlobal Online Test Competition will be April 19, 2013, starting at 10:00AM Eastern with a blog post. The blog post will describe what software to test, how to get it, how to submit bugs, and how to submit a final test report.
We called it the Killer App, and, for the insurance industry, it was magic.
At least, it was supposed to be.
Forget waiting two weeks to get an explanation of benefits from the insurance company, two more to get a bill, then two hours on the phone to figure it out. We were going to take care of the whole thing while you were at the doctors office. To do that, we were going to adjudicate (that’s a fancy word for ‘process’) the claim in real time, so the customer could pay direct with an HSA Debit card -- and know he was paying the right amount. That meant taking the claim from the doctor, adjusting to the negotiated fee schedule, figuring who owed what, and reporting that back so the customer could pay.
After that, if the customer used cash and had a flex spending account, we would know they qualified to spend out of their FSA, and cut them a check back, from the account, automatically.
Like I said, it was the killer app. I was working for the insurance company, and we hired a vendor to build us a system, to talk to the bank to calculate the balance on the claim. My job was to write the software to pull from the claims database, compare to the amount in the Flex Spending Account, generate the flat file, and transmit it to the print vendor to make checks.
At least, that’s what I was supposed to do.
You see, the project was nine months late when I started. The data tables I was supposed to pull from were not set up, nor did I have any written requirements. All I had was conversations with a supervisor.
It was time for me to talk to people.
Imagine for a moment two companies have an identical problem - the website is slow. Perhaps this is a huge problem; the company bet it's future on a new product, that customers sign up for on the web, and performance is bad enough that people are abandoning the site and going to the competition. In a few more months, the competition will 'own' the market, and the percentage of customers that are even worth fighting over will be a small percentage of the total - more like 9% than the 90% of market share hoped for in the initial offering.
Like I said, there are two companies. Both appoint a point man on performance improvement and give him the speech that the future of the company is in his hands. Both companies find a war-room, an easel to write on, some markers and stickynotes. Management asks for daily reports, and steps away. Knowing the PMI playbook backwards and forwards, plus a half-dozen other techniques, both newly appointed 'performance project managers' are competent, smart people, and they have a plan: Figure out what component(s) are the bottleneck, and speed them up. It's not rocket science, right?
Then differences show up.
Last week I put out the Black Swan In The Enterprise, which argued that most uptime planning comes from the ability to predict the future -- and sometimes, the future is wildly different that you would have predicted. Things fall apart; the centre does not hold -- and they fall apart very differently than what the team planned for up front. If that's the case, what's the company to do, and how can Operating Systems promise those 99.995% uptimes and take themselves seriously?
The incredibly positive uptimes we see in sales and marketing material all come from the same place -- looking at one component of the project in isolation. For example, let's look at that Amazon Outage again. The severs lost power. This has nothing to do with the operating system, it's not the fault of the operating system! You can almost hear the executives arguing "look, man, I said 99.995% uptime, but you can't get that if you yank the power cord out."
Or yank the internet connection.
Or a bridge or router fails.
Or a sysadmin makes a mistake an mis-configures a router table.
Or a sysadmin bingos a firewall rule, and suddenly your webserver is prevented to talking to the internet.
Or the loadbalancer gets bingoes, pointing to an old server that is no longer there.
The problem isn't an individual component -- it is all the components, acting as a system. Just like the old cliche, the delivery chain of software is only as strong as its weakest link. My suggestion on what to do with this is more than a little bit counter-intuitive.
When the staff at NRGGlobal asked me to start blogging for them, they asked what we had in common, and what they could do for the test community.
Over the past few years, I've gotten to know the test community pretty well, and it seems to me there are really two test communities.
First there's the folks I see a few times at a year at conferences, who work in big cities and have a reasonably large amount of personal influence and career mobility.
Then there is everybody else, the folks who say "that must be nice", with maybe a touch of honest disappointment in the words.
Well, everybody else, this one is for you.