Predictive evaluation, Cognitive Walkthroughs & Heuristics

Predictive evaluation was a tool introduced in Kate Devlins HCI lectures in 2011. I am planning on applying it to the project as a form of prototype evaluation prior to its introduction to a focus group. This should allow me to identify any glaring issues and tweak the prototype accordingly before going to do a pilot test.


I am referring to a few resources whilst preparing for it: the book 'User Interface Design And Evaluation by Deborah L. Stone, Debbie Stone' as well as the following online resources http://courses.csail.mit.edu/6.831/archive/2008/lectures/L18-predictive-evaluation/L18-predictive-evaluation.pdf http://www.sigchi.org/chi96/proceedings/shortpap/Hamilton/hf_txt.htm. The importance of it as a n early developmental evaluation methodology is emphasised in the book (Chapter 19, page 406) 'Alyson: Heuristic or predictive evaluation and trial-based (user based) evaluation, complement each other. If we'd only used predictive evaluation, we wouldn't have had the response from the real individuals who are going to be affected by the system; as a usability assessor, I could have sat there and said "Yes, it's not going to affect memory load, yes the design is consistent,.... yes the design is simple." But I'm not a user. I wouldn't have known what it was like to use it for real.'


There are positive and negative aspects to said evaluation. In the first link (a PDF from a User Interfact Design lecture at MIT. 


'Today’s lecture is about predictive evaluation – the holy grail of usability engineering.  If we had an accurate model for the way a human used a computer interface, we would be able to predict the usability of a design, without having to actually build it, test it against real people, and measure their behaviour.  User interface design would then become more like other fields of engineering.  Civil engineers can use models (of material stress and strain) to predict the load that can be carried by a bridge; they don’t have to build it and test it to destruction first.  As user interface designers, we’d like to do the same thing. '

It points out the necessity of having a model for how a user interacts with an interface. This link http://www.usabilityfirst.com/usability-methods/hci-design-approaches/ gives a list of approaches to HCI and explains the GOMS method: Goals, Operators, Methods and Selection Rules. It was developed in 1983 by Stuart Card, Thomas P Moran and Allen Newell.

'Goals are defined as what the user desires to accomplish on the website. Operators are the atomic-level actions that the user performs to reach a goal, such as motor actions, perceptions, and cognitive processes. Methods are procedures that include a series of operators and sub-goals that the user employs to accomplish a goal. Selection Rules refer to a user’s personal decision about which method will work best in a particular situation in order to reach a goal.

The GOMS model is based on human information processing theory, and certain measurements of human performance are used to calculate the time it takes to complete a goal. For example, the average time it takes a human to visually fixate on a web page, move eye fixation to another part of the web page, cognitively process information, and make a decision of what to do next can be measured in milliseconds. The times it takes for each of these operators can be added up to produce the total time for a particular method. Multiple methods can be compared based on the total time to complete a task in order to determine which is the most efficient method for accomplishing the task.'

Advantages of doing predictive evaluation is that it doesn't involve users, or in some cases, require a prototype. It allows for comparison between different ideas and can identify problems before implementation is complete. 'predictive evaluation not only identifies usability problems, but actually provides an explanation of them based on the theoretical model underlying the evaluation... User testing might show that design A is
25% slower than design B at a doing a particular task, but it won’t explain why.  Predictive evaluation breaks down the user’s behaviour into little pieces, so that you can actually point at the part of the task that was slower, and see why it was slower.' - http://courses.csail.mit.edu/6.831/archive/2008/lectures/L18-predictive-evaluation/L18-predictive-evaluation.pdf

KLM is also useful wih predictive evaluation (Keystroke-Level Monitoring as it evaluates the time required by users to execute tasks. 'The first predictive model was the keystroke level model (proposed by Card, Moran & Newell, “The Keystroke Level Model for User Performance Time with Interactive Systems”, CACM, v23 n7, July 1978).. This model seeks to predict efficiency (time taken by expert users doing routine tasks) by breaking down the user’s behavior into a sequence of the five primitive operators shown here.

'Human Information Processing Model

Human Information Processing (HIP) Theory describes the flow of information from the world, into the human mind, and back into the world. When a human pays attention to something, the information first gets encoded based on the sensory system that channeled the information (visual, auditory, haptic, etc.). Next, the information moves into Working Memory, formerly known as Short-Term memory. Working Memory can hold a limited amount of information for up to approximately 30 seconds. Repeating or rehearsing information may increase this duration. After Working Memory, the information may go into Long-Term Memory or simply be forgotten. Long-Term Memory is believed to be unlimited, relatively permanent memory storage. After information has been stored in long-term memory, humans can retrieve that information via recall or recognition. The accuracy of information recall is based on the environmental conditions and the way that information was initially encoded by the senses. If a human is in a similar sensory experience at the time of memory recall as he was during the encoding of a prior experience, his recall of that experience will be more accurate and complete.'

Predictive evaluation involves task analysis, and it was not only this that i wanted to analyse. I also wanted to analyse how easy the prototype was to use, how easily the instructions are to follow and to generally analyse the interface. It is for this reaso that i have decided to do a 'cognitive walkthrough instead' which is a step by step analysis of the processes undertaken to interact with a system. Also as it is such a simple interface the tasks executed will be done quite quickly and taking timings will not be massively beneficial. My 'cognitive walkthrough' seeks to evaluate the presentation and execution of the system as a whole, not just timings of individual tasks. I am going to tweak this 'predictive evaluation' into more of a predictive walk-through with a novice user. I am going to use Schneiderman's 8 Golden Rules to assess whether my interface is successful with its accompanying manual http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html

I am going to do a step-by-step evaluation of:

1. Reading the manual
2. Manipulating the bear
3. Closing communication

If any problems arise i will say them aloud, and will do a 'cognitive walkthrough' to identify every step that is being made. I will record the audio of evaluation and analyse the results. I will need an end user, and then will swap roles with that person to evaluate their responses. There will be two 'predictive evaluations. I will probably need two manuals as the two bears will be different. I will need to consider the possibility of proximity sensing 'how do i know they are there and received my message?'.

I will use this evaluation to create a version 3.1 prototype (and manual) which i will then do a pilot user test with and then a focus group.

I am referring to these guidelines on cognitive walkthroughs: http://www.usabilityfirst.com/usability-methods/cognitive-walkthroughs/ http://ics.colorado.edu/techpubs/pdf/93-07.pdf. I will also take timings on how long it takes to execute  individual tasks.