 |
| Photo: https://www.youtube.com/watch?v=ahtZayhxb5U |
This is my second
entry* in a series on the software procurement process. My examples are
going to be governmental administrative systems because that’s what I
know, but it applies to any kind of software demonstration. (* The first entry was about writing software requirements.)
******************************************************************
Disclaimer: All of the ideas expressed in this article are my personal statements and opinions, and do not reflect the opinions/statements of the City of Urbana.
******************************************************************
******************************************************************
Disclaimer: All of the ideas expressed in this article are my personal statements and opinions, and do not reflect the opinions/statements of the City of Urbana.
******************************************************************
Two opponents prepare for battle. On one side is a group of novices unfamiliar with the terrain. Squared off against them are veteran warriors with dozens of notches on their belt.
The point is not who will win, but how did the mismatch get to be so bad?
And because the contest is a software demonstration (hereafter: “demo”), the mismatch is magnified. Consider the stakes:
- For the people watching the demo, they are choosing the everyday software they’ll be using for a decade.
- For the software demonstrator, this is one of dozens of presentations they may deliver this year. Soon it will be over and they’ll be eating dinner, considering their steaks.
OK, so the process is flawed. What can be done to improve it?
Write better demo scripts
First of all, let’s clarify that there are two types of software demos.- In an Overview demo, the demonstrator decides what to show you, and in what order, with little or no input from the audience. The purpose is to give you a sense of the software – or at least the pieces they want you to see.
- The Scripted demo allows the audience to dictate exactly what they want to see, by telling the software vendors what processes and features to show.
Go back to the battle analogy at the start. The point of writing better scripts is not to “gotcha” the demonstrator and even the odds – that’s not possible, and it doesn’t help anyone. The point is to accurately judge the vendor based on a demo, AND for the vendor to learn about the requirements before signing a contract to deliver them. So both sides want the same thing: a clear vision of how the software meets the needs.
Here are some suggestions for better demo scripts:
Give specific business scenarios, but let the vendor set it up ahead of time
For example, include details such as the code values, names, and specific dollar amounts you want to see – and don’t make it optional to use them! The point here is to make the vendor set up your example from scratch, like a micro-implementation. It lets you SEE how your setup will look. Most importantly, it prevents the vendor from using their canned examples.Note the latter part of the suggestion, however … the last thing you want is to waste valuable time watching the vendor do data entry – so give them details AND the time to get it set up. Even a micro-implementation is still an implementation. I’d say that sending the script up to 4 weeks in advance is best – and make yourself available to answer questions about the script. (Handle it like any other questions about the RFP – post responses for all vendors to see.)
Focus on a few key user processes, instead of a laundry list of features
This is where most demo scripts make a huge mistake. Too many times, a script asks for SO MANY features to be shown that the demonstrator must rush through items. (This is absolutely true – I had many demos where we had 60 minutes to get through 40 requirements, some of which are complex enough to require a five-minute discussion to address.)So don’t have a script asking to see individual fields on the screen… if your requirements already stated the need for field X or field Y, then why ask them to show it to you? The demo is not a good way to verify their answers. Only ask to see data entry fields if they are related to a user process. (“Show how someone can change the workflow rules so that the Department Head approval of Requisitions increases from $5,000 to $10,000.”)
Focus instead on a FEW processes, and give yourself time to absorb their demonstration - and for you to ask questions. It’s similar to my suggestion about open-ended requirements (in this post). So ask them to “Show your features to manage purchase agreements spent by multiple Departments.” And then just watch. (See below for suggestions on how to watch a demo!)
Build your scenarios on top of each other
For example, a single demo could show hiring, modifying, paying, and terminating the same employee. Scripts like this are both more interesting and make certain that the demonstrator must show all of the pieces. Otherwise, a demonstrator might show setting up employee A, then paying employee B, who’s already set up. What they might be hiding by doing this are extra steps that they don’t want to show you.But please note that not all pre-entry is malicious. Some processes require external actors and can’t be real-time. For example, you can’t show bank reconciliation without having prepared an incoming bank file to use with the right check numbers and amounts. We used to call that the Julia Child method of demonstrating. If you’re showing how to cook a turkey on live TV, you need a pre-baked turkey in the oven to show what the end result looks like!
Don’t forget queries and reporting
Include scenarios in your demo script that show how users look up information. Focus on common and important processes, such as looking up the status of an invoice or accessing the details for an employee’s pay from a specific time period.Ask for ad hoc queries and give the demonstrator the exact fields you want to see on a report they must make from scratch right in front of you. When you do this, you’re not just checking out the reporting tool. More importantly, you’re looking at the reporting “meta data”. When someone makes a query are they seeing understandable field names? (Or are you seeing ugly database names?) Does the meta data pre-join the information from different data sources, so that query users don’t need to know the details?
By the way, don’t be surprised if an ad hoc report doesn’t include entries from earlier in the demonstration. Many large systems have a data warehouse load process; it may not be a real-time read of the database.
How to Watch a Demo
This section applies to both Scripted and Overview demos.Focus on the user experience
So the demo will not tell you whether the software will meet all of your desires. (Spoiler alert… it won’t.) But the demo will show you this: what’s it like to USE the software? Is it intuitive? Does it take a lot of steps?Direct your scripts around processes that you want to understand visually. Don’t focus on tasks that require a lot of data entry (setting up an employee, registering a new vendor). You don’t want to watch that, which is why you should give demonstrators that setup to do ahead of time.
IMPORTANT NOTE: This is a good topic to discuss with people who don’t know the requirements in detail, but will be attending the demo. Coach them that they can help judge the feel of the software regardless of their prior knowledge of the scripts or RFP.
There should be a “right” answer to some parts
In the guidance above, I suggested writing scripts with your actual code values and dollar amounts, so that vendors must set up your data. This will help you follow the responses – and provides a checkpoint to see how things are done.If possible, craft scenarios so that there actually is a correct number to be calculated. (Footnote 1) Maybe you’re modeling union contracts, and you’re asking them to pay someone with a suitably wacky payroll scenario (an EMT-trained firefighter is acting Division Chief for 2 hours on a holiday…)
Another clearly wrong answer is any time the demonstrator must re-enter data that was entered in a prior step. If that happens during the demo, then imagine the parts they’re not showing you! Information should always be brought forward from one entry to another.
Pay attention to how many steps it takes to do an action
Because there’s a lot of narration during a demonstration, it’s hard to gauge how fast an action can be. So pay attention to the number of screens, fields, and clicks someone needs to make to complete an entry. Look for extra steps that the demonstrator must go through that seem unnecessary. (And watch closely – any good demonstrator will go very quickly through these parts!)And demonstrators – a hint to you: all of the narration during data entry makes it look worse than it is. (You don’t want an AP clerk telling people “It took ten minutes for them to enter a vendor invoice!”) When something is easy in your system, show the entry as quickly as possible, then go back and talk about it.
Ask them to SHOW you tools that make data entry easy
All vendors say they have easy data entry, but you can only judge it when you see it. Ask them show you common navigation scenarios like looking up a code by description during data entry. (Again there’s a right answer here: it should be part of the entry screen – you should NOT need to leave the data entry screen.)Another feature to look for – how easy is it to create an entry directly from another? For example, how do you correct a timesheet, reverse a journal entry, or copy a user’s security setup to create another user? All of these cry out for simple and intuitive data entry. If it’s not easy, then that’s a bad sign.
Weighing the Demo’s Impact
This might be a strange statement from a person who just wrote a whole post on this topic… but how important should the demo be, anyway?The demo does NOT define the scope of the project
The proposal (and written responses) define scope. It’s difficult to hold a vendor responsible for something they showed at the demo unless it’s recorded. (Only a handful of my demos were recorded, but I assume it is far more common now.)But even a recorded demo does not provide conclusive answers. A demonstrator is taught how to answer questions during a demo without providing details. So you might ask “Can you notify an employee to refresh their data two weeks before their anniversary date?” They will say “YES”, but the unspoken part of the answer is “Yes… you can run a report every day to find those people, then e-mail them.” They didn’t lie – they just didn’t provide details.
So the recommendation is to focus on the demo to show the look and feel of the software, and to let the written responses define scope.
There’s a whole implementation ahead, don’t expect them to nail it
This one is the counter-balance to my guidance earlier about giving a vendor steps and scenarios that you want to see. There’s still a whole implementation project ahead, so don’t expect that the vendor is going to be able to address all of your needs based on four weeks’ notice and a few scenarios.This is a good place to look for agreement between how they responded in the written proposal to what they're showing in the demo. If they said it's not "Out of the Box" on the written responses, the demonstrator's answers should match.
So, there’s still value in giving them detailed scenarios, but keep your expectations grounded.
It’s subject to random events
I’ve delivered demos on very little sleep, seen (other!) people oversleep then, and once did one without my equipment (footnote 2). I had to give a demonstration on the afternoon of September 11, 2001. Anything can happen: there could be a blackout, a network crash, a fire drill...Or maybe the audience is bored – this is the 3rd out of 5 vendors they’re watching, and it’s 2 PM on Tuesday, with demos going for the full week. (Just like any performer, the demonstrator’s performance is definitely impacted by the audience’s attitude.)
And pity the poor demonstrator for a minute. They are standing in front of a full room, with their own VPs and peers observing, subject to the whims of computers, projectors, and buggy software. (You’re always demonstrating the newest version you can, which means you’re always finding new issues.) Having a demo fail is one of those excruciating moments for everyone involved, but especially for the person who needs to continue talking for 30 more minutes. (And trying to avoid those situations was how I spent my demo career.) Here’s a hint for demonstrators… learn from me and never say this phrase: “I’m not sure why that didn’t work.” THAT didn't go over well.
So a good product can look bad for many reasons – some that aren’t even the fault of the demonstrator.
But if they can’t address your needs, run away!
They’re putting on their best face at the demo – the software will never look better than this. But if you know what you’re looking for, and you don’t see it, then it probably ain’t there. (Unless there’s a big discrepancy from their fit according to their written requirements, in which case you may need to improve your demo script and maybe look at them again.)**************************************************************************
Footnote 1: The best example of this I ever had was the Hawaii Department of Transportation. Due to their complex federal billing needs, they wrote a project costing scenario with an indisputably “right” answer. They even offered a walkthrough of the scenario to assist with demonstrations, which led to the strange experience of flying back and forth to Honolulu to attend a 3-hour meeting. But the walkthrough was essential, because their scenario was that wacky – it depended on layers of custom calculations. In case you’re curious, our software couldn’t do it without custom work either, so I tried to convince them that taking an alternate approach to billing (one that we DID provide) would be better, even if it got different answers. That approach didn’t work; they wanted it the way they'd always had it. I believe they canceled the procurement, because no one could do it like they wanted.
And that was 2003. Here’s a status update in 2015: they STILL can’t find someone to replicate what they used to do: http://www.govtech.com/transportation/Hawaii-Abandons-139-Million-Financial-Accounting-System-Transportation-System-.html
Moral: they should have been more open-minded to change. (Maybe there’s the topic of the next blog post!)
Footnote 2: This story encapsulates what my life was like for a while. To save on airfare, I had flown a commuter flight from Chicago to Milwaukee, then I was switching airlines for a multi-legged flight (to the middle part of California) and finally back to Chicago. The plan was SUPPOSED to be to fly there, give the demo, sleep, and then fly back. But because all the legs were within a 24-hour span, the computer routed my baggage directly from Milwaukee to Chicago.
Bottom line – I got to California, but without my equipment or my clothes. I was wearing a suit already, so we went to the site and explained. We used dial-up (this was around 2002!) and I did a web demo from a development environment. The demo sucked and I flew home the next day in the same clothes.