User Testing Is Largely Pointless

August 18th, 2010

This post earlier reminded me of something I have been thinking about for a long time.  For years usability languished in the FOSS world due to a combination of elitism and the belief that it was just a theme-pack away.  As a result most FOSS projects are riddled with massive usability problems.  It appears now that this problem has been noticed and is starting to get addressed, which is good.  The problem is how it is getting addressed.

The solution, unfortunately, appears to be user testing and the creation of all sorts of fancy monitoring programs, testing suites, eye tracking and other such nonsense.  As with the earlier link to the Rhythmbox user testing results it is good data and good points but the reality is that it is pretty shocking if it requires user testing to point out these problems in the first place.  When I use a piece of software I instinctively notice things that are bad usability, undiscoverable or do not behave in the ways that you would expect them to.  Sure some points are valid and useful, especially in regards to wording and language but the majority of the points are obvious to anyone who understands usability – and large amounts of the issues are not even related to the designated target audience.  Just because experts are more capable of dealing with crap usability it does not mean that it is ‘for experts’.

To me user testing is the final stage.  It is what you do when you yourself are happy with what you have done and want to see how the users react to your ideas.  If the thing was made without real regard to a target market and usability then using user testing is going to be a spectacularly long and expensive slog as you’ll require weeks of focus groups to identify issues that should be obvious in the first place.

What’s more user testing in this fashion will, at best, simply produce something mediocre.  You’ll have all the rough edges sanded off but you will never produce something truly great.  As the famous Henry Ford quote goes, “If I had asked people what they wanted, they would have said faster horses.”  At best user testing will tell you what is wrong (and these comments are valid, don’t get me wrong) but virtually all the time the users suggested fixes are entirely wrong.  You will not get any good ideas out of it you will only instead find out what it bad.

All too often the FOSS version of an application will simply be a loose reinterpretation of a previous closed source title.  Rhythmbox, for example, is layout wise pretty much the same as iTunes and by and large FOSS titles rarely bring anything innovative in terms of UI development to the table.  People love to hate on Microsoft but the Ribbon interface is innovative, the way Aero Peek + Preview works is innovative.  People love to hate on the iPhone and iPad but a multitouch based gesture system is actually incredibly innovative.  And none of these things were the result of user testing, they were the result of some people what cared about usability sitting down and saying ‘How can we make something better‘.  Sometimes it fails, sometimes it succeeds.

You user test to find out how your ideas are working, not to find ideas.  If your program was uninspiring and unimaginative before user testing it will be uninspiring and unimaginative afterwards.