Testing Blog

Interesting insights on testing from NRG Global.

5 Reasons to Consider an Image Recognition Testing Tool

There's an ongoing debate in the applications development world on whether front end or back end testing is the most effective method. However, the discussion shouldn't be just about where the testing takes place, but rather about the ease of use and effectiveness of the tool. Traditional testing solutions must tunnel into existing application databases to deliver an effective result. Newer technology now enables testing based on image recognition and is giving us a glimpse of what may be the wave of the future. Here are five reasons you should think about using an image recognition testing tool. 1. Simple deployment Most traditional testing tools are protocol dependent and must integrate with application infrastructure, database, and back end processes. This makes the testing process more complex, expensive, and time consuming. An image recognition tool is application neutral and doesn't need to understand the underlying technology of the target software. With screen...
Continue reading
910 Hits

Installing Citrix XenApp 7.5 on Windows Server 2012

Installing Citrix XenApp 7.5 on Windows Server 2012 This blog will demonstrate my process of installing XenApp 7.5 on a Windows Server 2012. It is important that the account being used to install XenApp is part of a domain, this is necessary in order to manage Citrix Studio. In this article the installation process will be explained for the Delivery Controller and the VDA (Virtual Data Agent) on the same server by running the XenApp installer twice. It will also show how to manage Citrix Studio, Storefront, and how to activate the 30 day trial license for this guide. I. Installing the Delivery controller 1) Start the Installer and click on XenApp. 2) Click on the Delivery Controller box. 3) Accept the License agreement then click next. 4) Since we are installing on the same server go ahead and install all 5 components. 5) Once you get to the Features...
Continue reading
2347 Hits

Test Competition Results!

It was a quiet Friday Morning in the United States.  Or, I mean, Friday afternoon in India and Singapore.  Later in New Zealand, and in Europe ...

The point is, there were a lot of people playing. Seventeen teams registered four different continents for three hours of aggressive testing, followed by a weekend where teams rotated through performance testing.

Now, the winners in each category.

Continue reading
20212 Hits

The Moment We've Been Waiting For

b2ap3_thumbnail_race.JPGIt’s here!  It’s here!  If you are reading this, then the Test Competition is finally here!

No, you don’t need to have registered in advance to play, just to win the big prizes and have your name in lights.

Either way, here’re the rules of play.

Continue reading
10680 Hits

The Consultant's Gambit

b2ap3_thumbnail_wallpapers-house-of-lies-tv-show-33268245-1024-640.jpgThe class where I learned method-R performance tuning also taught me something else, something I call the consultant's gambit. This is that story.

Continue reading
13491 Hits

Update: One Month To Test Competition!

Quitters quit, but winners win.On March 11th, 2013, registration for the test competition closed.

We have sixteen teams registered, from India to Europe, Southern Asia, and the United States. We have a team that has people from two continents.

And that's just the folks who gave me a street address.

Continue reading
22711 Hits

More On Power

One of our contributors, Pete Walen, recently put out a blog post on leadership and power.  It is a good post; if you enjoy reading long-form articles that are more conversation than bullet-points, I suspect you'll get a kick out of it.

Pete's article did a lot for me, but most importantly, it got me to think, to finally publish that piece on power that had been niggling in the back of my mind for  nearly a year.

Instead of a rambling conversation, I'd like to start out with an assertion, then defend it, then suggest a thing or two to do about it.

My claim is very simple: Most of what you've been taught about Power is probably wrong.

Continue reading
11762 Hits

Method R Meets Profiling

b2ap3_thumbnail_classroom_training_cloud.jpgIf you've been following me talk about Method-R, you might wonder how I learned it.  In my case, I learned it the "right way", the "official way":  My employer brought in a consultant to teach a three-day class in performance tuning.  Method-R was the method he taught.

It also got me very, very interested in code profiling, which is a way to measure performance at a much more detailed level -- not just is it fast enough, but, instead, how long does each piece take?

Continue reading
8603 Hits

Tuning Performance with Method-R

b2ap3_thumbnail_Improvement.pngLet's consider, for a moment, the poor IT manager, told by end users that the application is "too slow."

I've been that guy. You may be that person some day ... you may be that person now. Either way, it is safe to sat that those in that position would prefer to be out of it.

The manager wants to fix the problem, so he goes downstairs to the DBAs and tells them. The DBA's do some research and find out that the memory percentage is way high, so the database is going to disk (a lot), and disk is slow.They suggest buying a great deal more memory , which is the cheap and easy solution. 

In most cases, the solution is a lot more complex, and involves tuning, news boxes, creating new indexes, maybe upgrading the network switch. After two days, two weeks, two months or two quarters, the DBAs say they are done, and show some dashboards like the one above, with all the data trending in the right direction. Average CPU use is down, so is memory, disk I/O requests are down, response times are down - everything looks good.

Allow me to ask, just for the sake of argument. What happened to average customer response time?

We don't know.

It is this problem, our love affair with data that is often irrelevant, that Method-R solves. 

Continue reading
9398 Hits

Test Competition Frequently Asked Questions

b2ap3_thumbnail_comp.jpgIt hasn’t been easy to organize our first test competition. We’ve had to look for collaboration tools, make sure they scale, plan the scoring, find a way to record the bugs, figure out what to test ...

We have also had five registered teams, and, in that time, a lot of similar questions. Today, I will take a stab at the answers.

1.   What is it about (in a nutshell)?

Continue reading
14379 Hits

On Power

b2ap3_thumbnail_power.jpgEvery week on "The Apprentice" may be different, but we know one thing:  Donald Trump is going to walk in, ask questions, fire someone, and drive on.

You might want to work like "The Donald" on the show, but if you measure software performance, you probably feel more like a contestant on the show. After all, you start with a problem too broad and vague to adequately solve in the timefame, are involved too late to impact quality, and assigned too little staff, software, and hardware resources to do the job right.

If you've ever felt like that, well, I have too, and I survived. Today I'm going to offer a way out.

Continue reading
10120 Hits

Test Competition Update

Test Competition Update

b2ap3_thumbnail_climb_the_ladder_competition1.jpgIt has been two weeks since we announced the details and how to register for the NRGGlobal test competition.  Since that time, we've had three teams register.  

No, I'm not worried, here's why: At the same time I am organizing this, I am also co-chair of the test and quality track of the Agile Conference. That conference had almost no submissions, I was near despair, until the submission deadline of February first approached. Now I have dozens of submissions to evaluate, which puts me in a different kind of despair. :-)

No, seriously, it is alright, we have a review team that is slicing up the reviews right now.  I will be fine.

Continue reading
12378 Hits

Serious Web Load Testing

It is called the basketball exercise, or sometimes inattention blindness, and it demonstrates a real problem in many kinds of automated testing.

What is this thing, you ask?  It can be hard to explain, but I can show you in about a minute.  Look, there's video:

Continue reading
11033 Hits

The Next Killer App

The Next Killer App

Today I’ll cover the story of project number two, "just a little code" to integrate two systems.  But first, let’s talk about my motivations. 

About the Killer App Series

"The Killer Appis not a work of fiction, or a work of historical fiction. It is not a made-for-TV movie, "inspired by" true events but with the drama cranked up a notch or two; it is history as I recall it.  

While it has been a pleasure to get this off my chest, I have no axe to grind.  The players have moved on.  Enough time and space now separates us that even the “bad guys”, as much as there were bad guys, would probably not recognize themselves in the writing.

Nor is this a case study, all wrapped up in a bow with a key learning takeaway.

The Killer app is ... a story.  

My story, about a time in my life before I was a writer or consultant, back when Matt was an employee, and, sometimes, felt more than a little trapped by his situation.  The story points out that there is more to life than process -- that often, it is the sticky blobs of flesh called humans that can make a difference.  I didn’t always make the right choices. Back then I was more than a bit scared of conflict.  But if a person or two can learn something from the story, and do it with a smile on their face ... I hope you’d agree with me to call that success.

Now it’s time for round two.

The Next Killer App

b2ap3_thumbnail_neo-dodging-bullets.jpgAt the end of the killer app, our hero had transferred off the project, to go work on project number two.  The transfer was no brilliant duck-and-groove out of the matrix. 

The reality was that I failed a code review, so they transferred me to fix the code.

At least, I tried to fail the code review. 

You see, the code came to me late in December, between Christmas and New Year's, with a bow on the end that said it "had to go into production by the end of the year."

The code itself pushed data from one system into another and provided

Continue reading
11376 Hits

The Killer App Finale

Last week I posted The Killer App, the story of a real software project with a cliff-hanger, pick-a-path ending.  It's time to close the loop. "The Killer App" was a project that was not just late, but hopeless; a claims processing system that could not process a single claim end-to-end.  When I suggested that we should get both technical teams in a room, the PM insisted that we communicate on paper only.  After all, without a paper trail, the vendor would do whatever they would do, and claim they "heard it in the room." The project was a train wreck.  The software didn't work.  It wouldn't run.  If, by some miracle, we got the system to process a single claim, there was no way the vendor really understood all the combinations. For the first time in the history of the company, it was about to experience a multi-million dollar failure. It's a...
Continue reading
8869 Hits

More On The Test Competition

b2ap3_thumbnail_test_comp_2.jpgOver the past two weeks I have spent a great deal of time doing research, gathering resources, and judging interest in an on-line test competition in software testing.  My conclusion is kind of like Nike.

Let's do it!

 It's time to talk about details.

 What: The first NRGGlobal Online Test Competition will be April 19, 2013, starting at 10:00AM Eastern with a blog post.  The blog post will describe what software to test, how to get it, how to submit bugs, and how to submit a final test report.

Continue reading
37110 Hits

The Killer App

b2ap3_thumbnail_killer_app_20130119-180614_1.jpgWe called it the Killer App, and, for the insurance industry, it was magic.

At least, it was supposed to be.

Forget waiting two weeks to get an explanation of benefits from the insurance company, two more to get a bill, then two hours on the phone to figure it out.  We were going to take care of the whole thing while you were at the doctors office.  To do that, we were going to adjudicate (that’s a fancy word for ‘process’) the claim in real time, so the customer could pay direct with an HSA Debit card -- and know he was paying the right amount.   That meant taking the claim from the doctor, adjusting to the negotiated fee schedule, figuring who owed what, and reporting that back so the customer could pay.

After that, if the customer used cash and had a flex spending account, we would know they qualified to spend out of their FSA, and cut them a check back, from the account, automatically.


Like I said, it was the killer app.  I was working for the insurance company, and we hired a vendor to build us a system, to talk to the bank to calculate the balance on the claim.  My job was to write the software to pull from the claims database, compare to the amount in the Flex Spending Account, generate the flat file, and transmit it to the print vendor to make checks.

At least, that’s what I was supposed to do.

You see, the project was nine months late when I started.  The data tables I was supposed to pull from were not set up, nor did I have any written requirements.  All I had was conversations with a supervisor.

It was time for me to talk to people.

Continue reading
13783 Hits

A Tale of Two Performance Projects

A Tale of Two Performance Projects

b2ap3_thumbnail_twoface.jpgImagine for a moment two companies have an identical problem - the website is slow.  Perhaps this is a huge problem; the company bet it's future on a new product, that customers sign up for on the web, and performance is bad enough that people are abandoning the site and going to the competition.  In a few more months, the competition will 'own' the market, and the percentage of customers that are even worth fighting over will be a small percentage of the total - more like 9% than the 90% of market share hoped for in the initial offering.

Like I said, there are two companies.  Both appoint a point man on performance improvement and give him the speech that the future of the company is in his hands.  Both companies find a war-room, an easel to write on, some markers and stickynotes.  Management asks for daily reports, and steps away. Knowing the PMI playbook backwards and forwards, plus a half-dozen other techniques, both newly appointed 'performance project managers' are competent, smart people, and they have a plan:  Figure out what component(s) are the bottleneck, and speed them up.  It's not rocket science, right?

Then differences show up.

Continue reading
13496 Hits

System Shenanigans

b2ap3_thumbnail_systemantics.jpgLast week I put out the Black Swan In The Enterprise, which argued that most uptime planning comes from the ability to predict the future -- and sometimes, the future is wildly different that you would have predicted.  Things fall apart; the centre does not hold -- and they fall apart very differently than what the team planned for up front.  If that's the case, what's the company to do, and how can Operating Systems promise those 99.995% uptimes and take themselves seriously?

System Effects

The incredibly positive uptimes we see in sales and marketing material all come from the same place -- looking at one component of the project in isolation.  For example, let's look at that Amazon Outage again.  The severs lost power.  This has nothing to do with the operating system, it's not the fault of the operating system!  You can almost hear the executives arguing "look, man, I said 99.995% uptime, but you can't get that if you yank the power cord out."

Or yank the internet connection.

Or a bridge or router fails.

Or a sysadmin makes a mistake an mis-configures a router table.

Or a sysadmin bingos a firewall rule, and suddenly your webserver is prevented to talking to the internet.

Or the loadbalancer gets bingoes, pointing to an old server that is no longer there.

Or ...

The problem isn't an individual component -- it is all the components, acting as a system.   Just like the old cliche, the delivery chain of software is only as strong as its weakest link.  My suggestion on what to do with this is more than a little bit counter-intuitive.

Continue reading
14352 Hits

An Online Test Competition

b2ap3_thumbnail_competition.gifWhen the staff at NRGGlobal asked me to start blogging for them, they asked what we had in common, and what they could do for the test community.

Over the past few years, I've gotten to know the test community pretty well, and it seems to me there are really two test communities.

First there's the folks I see a few times at a year at conferences, who work in big cities and have a reasonably large amount of personal influence and career mobility.

Then there is everybody else, the folks who say "that must be nice", with maybe a touch of honest disappointment in the words.

Well, everybody else, this one is for you.

Continue reading
14080 Hits

Yugo Testing

Recently, I was invited to observe a colleague’s monthly stress test exercise. There is a large room filled with mangers and technicians, experts in various aspects of the system, general network specialists, application specialists, customer support representatives and a collection of people with backgrounds in various aspects of testing. Once a month, they get a group of people together and observe how the systems behave under massive load levels. They make observations and notes and examine what possible causes for various behaviors can be. Then they compare the results with what happens in their test environment.   Questions and Answers This caught me by surprise so I asked the question: “You don’t run this exercise in your test environment? You do this live?”The person I was sitting next to seemed baffled at the idea. “Of course we can’t run this exercise in the test environment. It is far too small and...
Continue reading
6587 Hits

The Black Swan In The Enterprise

Amazon’s Service level agreement for it’s cloud computer service is 99.95% uptime for a service region (that’s a data center) during a one year period. That means that the site will be down five one-hundredth’s of a percent of the time. That means that EC2 should be down for less than four and a half hours per year.  Being  tester, I set out to evaluate this claim, starting with a google search for “Amazon Ec2 Down 2012.”   I found a half-dozen issues in the twenty-minute to half hour range plus a Seven Hour Outage in April.   Instead of tearing into Amazon, I’d like to be fair.  They did a lot right.  They have a great deal of technical depth, they have multiple redundant switches, internet connections, backup power, and servers.  They have technical depth, 24/7 monitoring, and keep most outages to twenty minutes by acting swiftly.  When they have...
Continue reading
12587 Hits

Another Way to Monitor Performance

A few years ago, one of the companies I worked with created a "Production Application Support Team", or PAST.  These were the sysadmins and programmers intended as the next line of support after the helpdesk, who could triage some problems and perhaps actually fix them. The team had a problem - vague complaints that the website was "slow" for login.  Yet anytime a leader got the phone call, they would give the site a try, and see reasonable response times.  PAST had a motivated programmer on it called Doug and a free copy of a functional test tool. They gave the tool to Doug and asked him to figure it out. Doug was a sharp guy, and he had an idea.  He used the tool to record the login operation and write the timing of the full page download to a database, then set it to login every five minutes, all...
Continue reading
14615 Hits

Deviant Deviation

A few years ago, I worked a performance project.  You probably know the type:  Customers were upset, executives were upset, technical staff were asking for specific direction (and not getting much) ... nobody was happy. The code was already in production, and there was no obvious ‘roll it back to the previous version that is just fine’ trick for us to pull.  It was pretty bad. There are worse projects.  When the technical staff starts pointing fingers at each other, then you know nothing is going to get done.  At least we had hope. Then came the fateful day.  The CEO was on the conference call, and he said, very sincerely, “We’ve got problems with the website ... we really need some more testing.” At this, the director of product quality perked up. “I have to politely disagree.  Testing will tell us if the product is slow or not.  We already...
Continue reading
12686 Hits

(Just Starting to) Analyze Performance Data

If you've done any performance testing, or even seen a demo, you are probably familiar with the “big line graph” that is created as the test runs. A Typical Response Time Graph The typical scenario is simple:  Have the load tool start with one simultaneous user running a script.  Then add a new user, say, every thirty seconds plus a small random pause.  The script consists of a set of pre-defined operations - login, search, click on some item, and logout. As more users are added we expect to see response times get higher.  We might see response times go through the roof at some magic number as the system hits a bottleneck, or, perhaps it goes up gracefully.  With some cloud-based systems, ideally, performance stays within a range over time. It’s a simple way to do testing.  It is straightforward, even easy to implement. And it has a pile of problems. Let’s...
Continue reading
9731 Hits

Performance Matters

By Matthew Heusser Hello; I am Matt Heusser, a risk management consultant and sometimes writer.  You can learn more about me at The folks at NRG Global have asked me to start blogging for them because their place in the community aligns with mine.  We are both interested in minimizing software quality risk by inquiry, assessment, and evaluation. Given the time of year, I thought it might be appropriate to start ... with a story. A Performance Story It was more than a few years ago. I was a bright, young software engineer, working for a large insurance company.  Y2K was finally over, the internet crash had begun, companies were sinking millions into ERP upgrades ... all the while under pressure to “do more with less.” How we ever thought we would do more while executing on a multi-year, hundreds-of-person ERP conversion was beyond me.  One thing that did come...
Continue reading
9099 Hits

Understanding the difference between total users and concurrent users.

Many of our clients have enterprise applications that are accessed 24 hours a day, seven days a week.  Their applications may be accessed remotely from all over the world.  In such cases, total users could number in the thousands. When configuring a test plan for our clients we want to establish a realistic number of concurrent users, in other words, the number of users actually in the system, actively using the application at a given time. Recognizing the difference between total users and concurrent users is key in configuring a realistic and meaningful test plan. Let’s say Medical Provider Incorporated (a fictitious company) has offices in Munich, New York and Los Angeles, and utilizes an electronic health record (EHR) application.  Each office has 2,000 employees.  First, determine the number of employees that actually access the EHR application under test.  For our example, we’ll say 1,000 in each office.  Of the 1,000, let’s say...
Continue reading
13666 Hits

Testing the Upgrade from Eclipsys 5.0 to Eclipsys 5.5:

If you are starting or thinking of starting your Eclipsys 5.5 upgrade, you are probably wondering what approach to take when testing the upgrade and what issues others have uncovered during testing. Eclipsys 5.5 features a suite of modules wrapped by a new gateway user interface to allow for faster work and better flow and integration of data access. This intuitive new gateway interface allows users to access one or two or many modules at one time. This major change in Eclipsys introduces many new questions about the performance of the application. Some are: Will it require same amount of memory/CPU resources as previous version? Will the current security settings be intact or will it be modified after the upgrade? Will accessing multiple modules from the Gateway cause any integration problems? Will the database and MSMQ storage requirements increase? How about using Eclipsys through a portal like Citrix? How about session endurance or printing reports? And many more… Testing and finding...
Continue reading
10707 Hits

Blog Select List