Why testing doesn’t really happen until after D-Day

It happened to all of you. You work on a project and keep delivering features at a steady pace. You test your own work, and get constant confirmation from your project managers that each feature has been tested, and you have a green light to move on from the customer.

The release date is closing in, you manage to wrap everything up, and launch. You lean back in your chair and think to yourself “ahh, another job well done”.

Then, out of nowhere, after just a few hours in, your task list / inbox gets filled with things like “How can I do X?”, “We have an issue with Y” and the dreaded “Z isn’t working”. You open the list, scan the first few items, and it hits you:

“Wait a minute, we’ve been over these, this is the way this feature was supposed to work, it’s been tested.”.

Ahh, but you see, it wasn’t.

But how could this happen?

Here’s the thing: your customers don’t really test. Your project managers don’t really test. You don’t really test. Not until the lauch date that is. That’s when everyone really starts testing.

“But, we had at least 6 people involved in this project at all times. Surely between them they tested everything”.

Well, not really, no. You see, every person involved in your project has a certain way of  looking at it.

You have a list of features that needs implemented. While you churn out feature after feature, you only tend to test the specific behavior you built into each one. “Right, so the user clicks this, fills out this form, and ends up here.”.

You don’t really try to break your own application, so you only interact with it in the way it’s meant to work. And of course, it works … “on your machine”.

Your project managers don’t test because they don’t have the time for it, because they have that meeting about that other project with that other client in 10 minutes, just pop them an email and they’ll look it up later … They won’t. They’ll be too busy.
They’ll rephrase parts of your email (because really, you suck at writing to corporate clients), and forward it to the customer. And they will not get a chance to test it again … until after the lauch date, when …

… the customer will start testing. The customer, the person you build the entire project for, doesn’t even remotely start testing until after the project has set sail. Why? Because until then, to them, the project is not real yet. It’s “under construction”.

They will open it up, have a peek, and unless something is downright wrong, they will not notice it. They will not follow the individual interaction flows, they will not start using the project like they will use it for real. “Oh, I can see some products, that probably means the entire administering, checkout and billing processes must be working.”.

Most of the problems that will rise from the customer’s lack of testing are those when he expects a feature to work a certain way, but he described it to your project manager at first in a totally different way. And she described it to you in yet another way. And you developed it “your way”.

Who’s fault is it?

No one’s really. All of yours.

Who should test?

First of all, nothing beats professional testing. Ideally, you should have dedicated testers, but odds are you will have to make do with what you have available.

So there is good news, and bad news:

The good news is that, when searching for bugs, it’s easy! Grab a few members of your team that don’t work on this particular project. Best to do this with all types of colleagues: other developers, account managers, even the office secretary.

Sit down with them, and politely ask them if they can spare any amount of time to test your product. Then assign them some user roles, and let them loose.

The bad news is that application bugs are the easiest to find and fix. The biggest issues come from what I said earlier: miscommunication.

My bet is that shorter release cycles, more customer involvement (the project managers really have to step up here), and enhanced communication can at least lessen the impact of these problems on the typical project. Also, the User Stories and Acceptance Tests agile development concepts fit perfectly.

June 8, 2010 | Filed Under Development, www | 2 Comments 

Google Analytics goal steps recorded as exit pages

We encountered a problem with the Google Analytics goal steps on one of our websites, where some of the steps were recorded as exit pages. In the following image, the first exit page is actually the second step in the goal:

Funnel visualization

After a bit of investigating, it seems that the first step matches all the subsequent ones, so proceeding to the next step in the goal is actually recorded as a page refresh, hence exit page. Our steps where:

1
2

/apartamente-finalizate-in-greenfield
/apartamente-finalizate-in-greenfield/cere-detalii

The solution in this case was to explicitly specify that the step URL matches the ending of the actual URL:

1
2

/apartamente-finalizate-in-greenfield$
/apartamente-finalizate-in-greenfield/cere-detalii$

Note that i encountered this problem with both head match and regular expression match on two different websites.

July 28, 2009 | Filed Under Analytics, www | 1 Comment 

Update de toamna – 2008

A trecut concediul, ne-am intors la munca, vine si toamna, totul pare ca intra in ritmul normal.

In ultima perioada am lansat cateva proiecte si lucram la inca multe altele. Imi place sa observ ca proiectele (atat cele pe rol cat si cele lansate deja) devin din ce in ce mai mari, clientii din ce in ce mai cunoscuti si echipa din ce in ce mai solida si unita.

Am lansat de curand site-ul Farmaciile Dona, campania Aventura Koleos 4×4 si site-ul ACMS.

Primele doua au fost lansate in cadrul echipei HyperActive, iar cel de-al treilea in colaborare cu Ovidiu Boc. Rolul meu a fost realizarea backend + frontend pentru toate.

Mi-a facut placere sa lucrez la toate 3, in mare parte datorita caracteristicilor si scopurilor diferite ale fiecaruia. Totusi, dintre toate, site-ul lantului de farmacii Dona mi s-a parut cel mai provocator, atat prin anvergura cat si prin domeniul cu care a trebuit sa ma familiarizez (spre exemplu clasificarea medicamentelor la nivel mondial pe o structura ierarhica cu 5 nivele de adancime, administrabila si editabila evident :) ).

Pe parcursul dezvoltarii am compus si un mind map al facilitatilor, pentru a-mi fi mai usor sa tin minte sectiunile care mai aveau nevoie de lucru (ca paralela, recomand aplicatia MindMeister atat pentru organizare de genul celui mentionat, cat mai ales pentru sesiuni de brainstorming solo).

September 11, 2008 | Filed Under Personal, www | 3 Comments