More Analysts can mean Less Effectiveness in Usability Evaluations

Research has revealed evidence that heuristic evaluation by multiple analysts actually reduces the effectiveness of the method, contrary to previous beliefs. Only by studying analyst decision-making can anyone begin to understand why Usability Inspection Methods (UIMs) succeed and fail in particular contexts, argue Alan Woolrych and Gilbert Cockton of the University of Sunderland.

Their work, to be reported at HCI2002, shows that existing means of combining analysts' responses often leads to a reduction in the accuracy of heuristic analysis as a predictor of user problems.

'The use of multiple analysts is recommended for improving the thoroughness of UIMs,' say Woolrych and Cockton, citing the work of Jakob Nielsen on heuristics. 'As more diverse more analysts are added, then hits increase as confirmed by Nielsen's studies. Nielsen's recommendations do not consider how analysts' predictions are merged. No further confirmation or elimination resources are proposed, and thus we must assume that when one analyst confirms a problem, it will be preserved, even when additional analysts eliminate some problems.'

Their formula for analysing the effectiveness of multiple analysts shows that more bodies must increase the thoroughness of a UIM: Thoroughness = Number of real usability problems found / Number of real problems that exist. But if their formula for validity is juxtaposed with this, it can be shown that multiple analysts also reduce UIM validity unless account is taken of false alarms: Validity = Number of real usability problems found / Number of problems predicted by UIM.

'Where several analysts discover the same improbable problem (false alarm), all analysts must have adequate analysis resources to eliminate it. If only one analyst fails here, the false alarm will remain in the problem set. Multiple analysts are thus highly likely to inflate false alarms. As a result, the overall effectiveness of the UIM will drop, where: Effectiveness = Thoroughness x Validity.'

The Difference Between Usability and User Experience
Source: CMS Wire, 22 March 2010
 
Never hurts to be reminded: semantics, or genuine conceptual distinction?

Use Your Users’ Language - And Use It Well!
Source: Multilingual Search, 20 March 2010
 
British English is very different from (US ) American English and Australian English. It is so very important to use your users’ language.

Cyber-criminals don't need Technical Skills
Source: SFGate, 19 March 2010
 
The unexpected downside of great usability.

Advice on Designing Mobile Sites and Apps
Source: Poynter Online, 18 March 2010
 
Series of useful tips from a recent panel discussion.

Website Usability and Conversion
Source: UsabilityBlog, 17 March 2010
 
Useful presentation on the relationship between website usability and conversion.

Tips for Usability Testing with Children
Source: Econsultancy, 16 March 2010
 
Not even chocolate will help you here.

The Hygiene Factor of Usability
Source: inspireUX, 15 March 2010
 
Is it true that usability can no longer take us 'beyond lack of dissatisfaction'?

The Business Benefits of building Accessible websites
Source: Econsultancy, 13 March 2010
 
There’s a good business case for making your website more accessible to the UK’s disabled community.

Internet access is 'a fundamental right'
Source: BBC, 12 March 2010
 
Almost four in five people around the world believe that access to the internet is a fundamental right, a poll for the BBC World Service suggests.

The Net generation, Unplugged
Source: The Economist, 11 March 2010
 
Is it really helpful to talk about a new generation of "digital natives" who have grown up with the internet?