Test Competition Results!
May 8, 2013
Another Way to Monitor Performance
December 19, 2013
Show all
Performance Matters

By Matthew Heusser
SpeedometerHello; I am Matt Heusser, a risk management consultant and sometimes writer.  You can learn more about me at xndev.com.

The folks at NRG Global have asked me to start blogging for them because their place in the community aligns with mine.  We are both interested in minimizing software quality risk by inquiry, assessment, and evaluation.

Given the time of year, I thought it might be appropriate to start … with a story.

A Performance Story

It was more than a few years ago.

I was a bright, young software engineer, working for a large insurance company.  Y2K was finally over, the internet crash had begun, companies were sinking millions into ERP upgrades … all the while under pressure to “do more with less.”

How we ever thought we would do more while executing on a multi-year, hundreds-of-person ERP conversion was beyond me.  One thing that did come up was offshore work, which we could have done for 50 cents on the dollar.

So we offshored some straightforward work — pulling data out of the database and creating reports.

They came back slow.

I mean really slow.  “Finish in twenty-four hours if you are lucky” slow — “never finish at all if you are unlucky” slow.

This was a problem.

The Requirements Problem

We took the issue to the vendor.  Clearly, they had created the reports in an empty, or extremely small, ‘test’ database.  We had tens of millions of claims to process in production.  What were they going to do to fix it?

The vendor explained, very politely, that we had failed to list performance requirements in the software.  This was our fault as customers; not the vendors.  After all, how could they know?

That answer was not good enough for me.

The vendor claimed publicly to be “Enterprise CMM Level 5”, where CMM was Carnegie Mellon’s Capability Maturity Model, a standard for process maturity. They knew that requirements review was a core practice, required to reach level 3.

If the requirements were defective, the vendor should have caught the problem in review and told us, right?

You think managing vendors is hard? Try managing vendors on other continents.

We took that question back to the vendor. Their reply boiled down to this: “Since CMM adds entire sets of extra practices, it adds extra cost.  So we only do CMM five work on request.”

Now think about that. More mature processes are too expensive so we only do them when asked.  (Don’t think too hard; your head will hurt.)

We never did get good performance out of those reports; we just rewrote them and tried to focus more on staff augmentation, where we had more oversight, than on offshore work.

That vendor was a large, publicly traded company.  A few years later they merged with another large company.  After spending some time on google several times after the merger, I am unable to find reference to that outsource center, or its level five certificate.

The Moral of the Story

Performance Matters.  Poor performance kills careers, it kills business relationships … sometimes it kills companies.

It is hard to predict how many users you’ll have.  If you can do that, you have to predict what they will do, and how often.

Then there is running the actual test, and figuring out what it means.

There’s certainly plenty of work to do.

Where We Come In

Over the next few months, I hope to make this blog (and the greater NRG Global website) a safe place to talk about performance, among friends, and a resource for the community.

Our first event will be a testing competition (functional, security, and load and perf) scheduled for late April. — we will probably need a few volunteers.  I also hope to bring in a few guest bloggers.  If you’d like to be involved, or have something you’d like to hear about, please, leave me a comment below.

This is going to be fun.

You might want to buckle up, though.

It’s going to be quite a ride.