Skip to main content

Website Usability Testing - A Comprehensive Guide

Website Usability Testing


Usability testing can mark the difference between success and utter failure for a firm, a manufacturer or a service provider, between standing out in the market or sinking into obscurity.

Below, we have prepared a comprehensive guide regarding website usability testing, with all you need to know about it.

Usability testing - what exactly is it and what’s its goal?

Usability testing is the process of evaluating a product or service through testing by representative users.

The whole essence of this testing is to spot and assess any usability issues, collect qualitative and quantitative data and measure the user's satisfaction regarding the performance and effectiveness of the product/ service, and the extent to which it fulfilled the participant’s expectations.

Naturally, usability testing is often used by product designers and service providers that wish to improve their products or services. Nowadays, anything, from everyday appliances, like smartphones, ovens, and cables to complex machines, like cars or drones, is repeatedly tested by representative users. Experience designers use them frequently to improve the feeling a customer gets while unboxes its newly purchased machine and software designers to identify problems or areas requiring improvement in their solution packages, be it ERP systems or PC programs.

If there is a product or service out there in need of constant usability testing is the online experience. Nowadays, web designers always pass their websites, e-shops, and applications through usability testing to make sure everything works correctly.

What is a "Representative User" and why is his/her role so important?”

Well, a Representative User is the person assessing and defining the quality of a given service or product, chosen for meeting all or most of the basic characteristics of the future users – consumers of the product/service. Such a person or persons are in the best position to identify possible issues and give a chance to its creators to rectify it before launching it to the market.

Remember: usability testing is all about “specified users with specified goals in a specified context of use.” Of course, any kind of testing is better than no testing, but having one’s product examined by people who belong to your market audience can save a company much embarrassment – or even more serious trouble.

Picking out representative users for a usability testing is critical to its overall success. Factors such as age, gender, educational level and occupation, location, and nationality, financial and social status, as well as prior exposure to the product/ service or some of its earlier versions and editions must be taken into account and used as selection criteria.

For instance, having a member of the team that created a website to evaluate its functionality wouldn’t exactly serve the usability testing’s purpose since the person would know in advance the fastest or easiest way to do things.

People with much prior knowledge and exposure to the tested product or service are not the best picks for representative users. You cannot have a father judge his child, can you?

Hence, using targeted users – especially when testing the functionality of a specific feature on a website – is of the essence. Using randomly selected users could lead to seriously distorted or flawed results and conclusions and ultimately to a launching disaster.

In fact, we can lay a hand on that. Use our specially designed feature that allows you to pick targeted users who meet your demographic criteria, thus ensuring your participants we yield truly valuable feedback.

Which are the main types of usability tests?

There are two basic types of usability testing, namely formal and informal. They can both prove valuable at certain stages of the development process. Let us see how.

Informal Usability Testing

This is one usually involves early stage prototypes – that is, prototypes that are actually far from being realistic and sophisticated. Evidently, at such an early stage, the web designers wish to test their central concept and see which ideas and features might do the job and meet their client’s needs and which are better off – thus saving a tremendous amount of time and energy.

To wrap it up, an informal usability testing verifies the viability of your main ideas.

Formal Usability Testing

After repeated informal usability testing, the time will come for the real thing. Remember, at this stage, you have long decided on which ideas to work on and now wish to see whether and to what extent they fulfill their purpose.

A formal usability testing should aim at adding the finishes touches, not changing the main concept of your team’s work. So, before taking a formal usability testing, make sure your work has successfully undergone sever informal ones and has integrated all the procuring feedback.

Now, let us turn to the formal usability testing. The prototype should already seem quite ready to launch, and just needs some… refining, or polishing! It will unveil all flaws and oversights and help all team members concentrate their efforts on areas that could disappoint or even annoy your future clients.

Needless to point out how important is for a company or team to use both types of usability testing repeatedly before launching since each of them offers valuable data on different stages of your work.

Sadly, numerous firms have found themselves in the awful position to almost starting from scratch all over again, just because they neglected to perform informal usability testing, while others ruined their reputation after launching their products without a formal one, and saw thousands of their loyal customers to bid them farewell.

Do take our word for it – both types of usability testing are critical to your success.

How is a usability testing conducted?

As a norm, one more representative users are asked to complete certain tasks while being observed by a supervisor and a few watchers, who listen, make questions and take notes, but never take part or offer help in any way whatsoever.

However, things are much less spontaneous than they may seem. Usability testing requires careful planning and preparation in advance, including

  • identifying the future users – audience and specifying their needs;

  • selecting proper representative users;

  • a usability expert or agency designs a tailored study script usually referred to as the “experiment protocol.” It’s of the essence to assign that job to a skilled person since this document is the core of the whole testing session. It should combine profound knowledge on qualitative research and domain expertise.

Website Usability testing usually involves a live one on one session between a moderator and a study participant. The moderator asks the participant to complete specific tasks that correspond to the ones the actual users will or are expected to perform in the future. Quite often, the user is also required to answer specific questions, both before and after the testing session.

The moderator’s role, aside from guiding the participant and naming the tasks to be performed, is to collect as much usability data and feedback during the session, thus enabling the development team to get a deep insight into user behavior and consequently draw a clearer picture of the user’s needs. Such extensive understanding can only improve the product and boost its market success.

In most cases, think-out-loud protocols are preferred in website usability testing, since it enables the participant to share its thoughts and observations in real time while completing the requested tasks.

What is remote usability testing?

Well, remote usability testing utilizes an online software program that records the user’s actions while he or she tests the product. This type of usability testing allows the participant to undergo the testing session from the comfort of their office.

The first and most important benefit of remote usability testing is the vast widening of the pool of potential representative users since location and traveling are no longer an issue. Also, the moderator can squeeze more testing sessions in a shorter amount of time, saving time, energy and staff expenses.

Another significant advantage is the user’s overall state of mind. Being in their environment, they are more relaxed, and their reactions, thoughts, and observations are much more spontaneous and honest – hence more all the more genuine!

What is the difference between moderated and unmoderated remote usability testing?

In turn, remote usability testing can be performed in two different ways - the moderated and unmoderated testing. Which is the most suitable for you depends on your goals and development stage. Let’s take a closer look.

Remote moderated usability testing

In remote moderated testing, the moderator and user are in different locations, but they do share screens and communicate directly through an online conference tool. In other words, they are in the same “virtual” working space, interacting in real time.

Moderated usability testing offers two main advantages.

First, making the most of real-time interaction, the moderator can ask the participant to amplify or better explain his/her answers, or even make additional questions.

This is often necessary since some user comments are far too vague or generic to serve the test’s purpose. To illustrate: when the participant comments such as “that’s not what I was expecting” or “this feature is a bit disappointing,” the moderator can ask him to explain what exactly he means by that.

Second, remote moderated usability testing is far more flexible and easily controlled since the moderator can offer guidance or help the user better understand what is expected of him/her during a task, move tasks around, add some or eliminate others, depending on the way the session goes.

There are some cons though. First, the moderator is obliged to stay through the whole session, which is more time-consuming and tiring and can often result in a smaller sample size and less feedback.

Second, the testing session’s quality greatly depends on the skill and experience of the moderator. Too much or too little guidance can have a negative impact on the results of the session, or even ruin it.

Also, moderated usability testing requires excellent conference tools, impeccable connection and a strict schedule – which translates into more time-consuming preparations. Such demands considerably increase the risk of technical problems. For it to work, everything in a remote moderated usability testing must run like a clock.

What is remote unmoderated usability testing?

In a remote unmoderated usability test, the moderator uses a software program that records the user’s actions to automate the testing session. The user receives the list of tasks that must be performed while their screen (and, sometimes, their voice) are recorded.

The place and time of the testing session are decided by the user, and all observations, comments, and feedback are sent to the moderator afterward, in the form of an audiovisual recording. In turn, the moderator can review and work on the results of the testing session at his/her leisure, whenever and wherever it suits him/her best.

Naturally, such a session does not include any live interaction, and follow-up questions must be asked and answered sometime after the session.

The remote unmoderated usability testing makes preparations for testing sessions far easier and allows for a higher number of participants to join in and provide their valuable feedback. Everyone, from moderates to representative users, works when and where it suits them best, which can only affect the quality of their work positively.

On the other hand, the moderator has much less control over the session, and live interaction and following-up are evidently out of the question. Users can easily get confused and have their attention diverted to the wrong features. This also means that the tasks and overall test plan cannot be too complicated and challenging since the user won’t have any prompt help to complete the tasks.

When to use remote moderated and unmoderated usability testing

It is true that moderated usability testing is a far preferable option in testing sessions that include more intricate and thorny tasks. In such cases, the actual online presence of a moderator simplifies things and ensures that the obtained feedback has the depth and quality necessary to be of any help.

Additionally, testing sessions that require a chained sequence of task completion can only be properly performed with a moderator present, since failure to conclude a task immediately terminates the session.

Generally speaking, the more experienced and skilled the user must be to conclude the session, the smaller the sample size is – hence, moderated usability testing turns its con into an advantage when used in this case.

Finally, when safety, security or technical issues are at hand, moderated usability testing is also the sole option for the company.

Then again, unmoderated usability testing is better when the session includes very straightforward and specific tasks, which hardly require a profound knowledge of a subject and rich experience. Any product or service in need of a large sample size in a relatively small amount of time can only turn to unmoderated usability testing.

The key benefits of website usability testing - Why it is Important for your Business

We are sure most of the benefits are already quite clear from what we have already said, but summing it up a bit will help you understand better how important usability testing is for your business.  

Time, energy, and money saving

Usability testing does require some expenses on your part, but the last thing you want to do is see them just as an unnecessary cost. Usability testing is an investment as well as an insurance against failure. Let us explain.

Usability testing enables your firm to discover and foresee timely pain points, possible complaints, and embarrassment and tackle them way before launching. Hence, apart from saving you tons of time, energy, operational cost, and frustration, it lays the foundation of further expanding your business, extending your reach, amplifying your brand awareness and reliability repute, and getting new customers – which automatically means higher profits for the future.

In the meantime, you have the chance to pinpoint problematic procedures or practices within the firm, weak links in your staff chain, and unnecessary expenses, thus allowing you to ameliorate your business’ efficiency from top to bottom.

You gain valuable insight

Usability testing affords invaluable insight into your users’ way of thinking, their needs and wishes, their demands and ever-changing whims and fads.

Such live interaction simplifies things and stimulates the mind, making ideas and solutions flow faster – a genuine treasure for any development team.

Note also that that usability testing has the advantage of highlighting user behavior, not preferences.  The majority of users hardly know what exactly they want – much less what they need! Still, by observing and studying their behavior, you will be able to understand which solutions best suit their real goals and desires.

Undoubtedly, the better you get to know your users, the more effective the solutions you offer them will be.

Pacifies the spirits within your team

Web development is an art, and artists are often obstinate and ready to fight for their ideas to the end! Such moods seldom contribute to a serene and calm working spirit within the team.

Well, usability testing can help quell the passions before becoming a problem, for a very simple reason: it moves decision-making from the developer’s mindset and opinions to the client’s feedback and real needs. Ideas that do not meet their desires are quickly dispensed without any fuss and complaining from their creators. All their energy and passion can now be funneled into making your product better instead of having heated debates with their colleagues.

Guarantees the best possible final product

Finally, having your users’ feedback and guidance straight from the beginning means you will have sooner or later the best possible outcome, first for your customers and then for the reputation of your business.

All in all, usability testing can do something truly wonderful for your firm: it can guarantee your success. Without it, you just work and wish for the best, like looking for a place just by picking upturns. You may find, you may not. Having a map, however, will make sure you will reach your destination, no matter how many mistakes you may do in between. This map is usability testing.  

The truth is that the creator can never be truly objective with its creation. Being too close to it blinds the web developer, making it difficult to perceive what matters to the user. Usability testing affords a fresh, genuinely objective look at things.

Also, of all professionals, web developers should never underestimate the importance minor flaws and nuances may have for your user’s online experience. Modern-day online users are more demanding and less patient than ever, getting mad and being ready to reject an elaborate project that took months of hard work on a whim, just because something did not run as smoothly as they may have expected.

It maximizes your conversion rate and minimizes support costs

it is official: based on a survey of the Nielsen Norman Group study, website usability testing has doubled conversion rates for all participating agencies and companies, and skyrocketed user satisfaction. Considering the benefits mentioned above, this is hardly a surprise.

Usability testing also minimizes support costs, since it cuts off unnecessary work on web developing and redesigning.

You get the upper hand against your competitors  

Closing, usability testing, after benefiting your business in so many ways, it gives you the upper hand against your competitors and helps stand out in your industry. Suffice it to say that brand awareness and visibility, as well as reviews, are your best advertisers out there. Usability testing guarantees both.

How to set up a Website Usability Test

Now, we are ready to get to the essence of things. Setting up a usability test can be divided into certain stages or steps.

Stage 1 - The Ground Work

Naturally, there is a lot of ground laying before scheduling the testing sessions. First of all, you must determine your goals. The questions below, or similar ones, can help a lot. All department managers and key team members must sit down and ask themselves:  

  • What are we trying to achieve here? Test our basic concept, a few core elements, or identify weak points?

  • Which are the specific questions we need to answer through this usability testing? Is there any chance we already know the answers to some of them?

  • At which stage of the development are we exactly?

  • When is the best time to have the usability testing?

  • Who is our audience? Can we write down the criteria we are going to use to select our users?

As soon as you know the basics, you can start shaping the program and character of the testing sessions themselves. Ask yourselves:

  • Which type of usability testing best suits our purpose?

  • How many representative users do we need?

  • How long will the sessions have to be?

  • How much will the participant be compensated?

Naturally, it wouldn’t have any point to suggest possible answers to such questions, since everything depends on your goals and type of product. True enough, most website usability testing sessions involve around 5 – 6 participants, but the higher their number, the surer you can be of spotting all the pain points.

As for the length of the session, usually 45 minutes to an hour does the trick, but again, your needs may prolong it further.

Step 2 – Assembling your Moderators and Representative Users

Again, much of this stage rests upon your goals and intentions, but you must exert yourselves in finding the proper user sample, both regarding its size and qualities.

You need to specify the level of experience of the participants, as well as their degree of exposure to prior versions or editions of your product. Other factors, such as their age, gender, educational level, etc. are determined by your audience.

Finding proper participants, who will meet all or at least most of your criteria can prove challenging. Possible pools of users are always friends, family, and colleagues (provided they do fit your profile), but always consider the danger of allowing politeness or fear of displeasing you to affect their responses!

Other possible sources are your social networks, primarily LinkedIn, Facebook, and Twitter, as well as forums and chat rooms of colleagues and people of your industry.  

As soon as you have a list of the selection criteria for your participants, you must assign roles to the members of your team that will prepare the list of tasks, the conference tools and questionnaires that will be used, the contact details of the participants, the session schedule, etc.  

Stage 3 - Preparing the Testing Session

Only when we are positive we have covered all the above can we start framing testing sessions.

The first thing to do is to structure the session. Most studies include three parts:

Pre Session questions

The first part consists of some questions the participants must answer before getting down to the actual tasks. Such questions usually aim at identifying the participant’s opinion of your brand, his/her expectations of your product and his/her view of previous versions or editions of your work.

Selecting the tasks

This is the most important part and will require much time and effort to be properly prepped. You will need to come up with a list of tasks, usually with escalating difficulty. Each of them will have to be written in an unbiased, simple language and described in a clear-cut way.  

Selecting the tasks shouldn’t prove too difficult. Just pinpoint the features and ideas you wish to check and are critical to your product’s success.

While it is vital to use simple terms and wording, you don’t want to be too “helpful” or shepherding. For example, when possible, avoid using terms explicitly used on your interface.

One thing you must be during the session is consistent: you must make sure all the participants are asked to perform the same number and types of tasks, in the same wording and order.

Post Session questions

At the end of the task, ask the participant to express his opinion on the overall quality and feel of their experience, maybe even rate some of its parameters, like its satisfaction, ease of use, effectiveness, efficiency, value, etc.

Ask them to state plainly what they liked or hated during the testing session, what they would replace or add to make the experience easier or more effective.

Try your best to make your script and questions as unbiased as possible. This is not as easy as it sounds. For example, while asking the participant to rate your product, do not include a “no problem at all option”. It would best to opt for a numbered scale, from 1 to 10, with 1 denoting a terrible performance and ten an excellent performance.

Another example: “How many times you used the online tutoring feature last year? ______ Number of times.” Such wording takes for granted that the participant indeed used this feature, and may encourage him/her to be inaccurate if he or she actually didn’t.

Best use this format:

“Have you ever used the online tutoring feature last year?”

“If yes, how many times you used it last year? Pick one of the following options

  • 1-3 times
  • 4-10 times
  • 11-20 times
  • 21 or more
  • I don’t remember

While choosing the words, expressions, and questions of the testing session, try not just to spot a possible weak point or problem with a product, but also measure its frequency and impact. It is one thing to have half of your participants complaining about an issue, and another to have 4 of them being unable to perform a task altogether because of it. Noting the frequency gives you just half of the story and does not stress its priority.

Therefore, try to dig up all the aspects of a given problem, frequency, severity, extent, impact etc.     

How to Conduct a Usability Testing Session


Introducing the session

First of all, greet and thank the participant for his role and cooperation, and stress upon him or her the importance of following your directions carefully throughout the session. Explain the way things are going to unfold during the testing session and make the participant feel as relaxed as possible. Make clear to them that the session aims at evaluating the product, not them!  

State the session’s length, ask them to be thorough, frank and as more elaborative in their answers and observations as possible, and help them understand that you will not take personally any negative comment or criticism they may express during their evaluation of the product.

Again, be direct and state clearly what they wish them to do; for example, “Please think and describe aloud your actions and steps.”

Remind them which of their actions are recorded and how, and explain to them the way you plan to use their feedback and comments, especially if you wish to post some key comments or phrases of the session online.  Ask them directly whether they give you their consent to record the session and possibly publish online some of their thoughts.

Moderating a usability study

Moderation is an art, and while it requires skill and knowledge, how to perform one is something that can be learned and sharpened through repetition. Just remember the following principles:

  • Be polite, patient and considerate.

  • Be ready to lay a hand when possible.

  • Remain unbiased and objective. Never comment, verbally or with your body language, on the observations the participant makes.

  • Resist the temptation to tell the user how to perform a task best.  

  • Speak as less as possible, minimizing the risk of inadvertently revealing information you shouldn’t.

  • Make sure they feel relaxed.

  • Check their reactions, facial expressions, and body language. Some participants may exhibit signs of frustration through their body instead of their comments.  

  • Urge them to speak. Encourage them to express themselves freely. Listen carefully and take good notes.

This last one – taking notes – is crucial. Note everything the user does – his/ her navigation around the interface, his/her comments, quotes and observations. You should also write down how much time it took the user to perform each task.

Again, facial expressions and body reactions are as important as their comments.

How to Analyze your Usability Testing Data

Now that the testing sessions are done, your team must get down to analyze all the data you have assembled. Your goal here is to take all the details a shape a precise picture of the strong and weak points of your product. Analyzing this information properly will allow you to make the most of your usability testing and use it to improve your product.  

Actually, there are several ways of analyzing usability test data. The first thing to do is check your notes, maybe even gathering them in a single spreadsheet and grouping them into different categories, such as pros, cons, positive and negative comments, miscellaneous, etc.

Next, isolate all your date for a specific aspect, feature or angle of your product. When possible, translate things into numbers. How many times has there been an issue with this particular feature? Which type of problem was the more frequent?

Look for patterns, group the participants’ comments when possible. For instance, how many of them characterized this feature as ineffective or frustrating?

Try to be accurate but, at the same time, categorize your data. You must link each given problem with a particular frequency, impact, severity etc.

Reporting and Sharing the Results with Your Team

Last, you need to report the findings and share them with your team members in a way that will help everyone understand them and rectify the problems that popped up.

First of all, concentrate your attention on presenting all issues and associated findings categorized by the level of severity. Make sure you have attached all the related data from the test plan and provide additional information to present a clear picture of the nature, severity, and impact of each issue.

Do your best to visualize the results and findings, make them as transparent and clear-cut as possible. Use concise sections, tables, screenshots, video clips, video annotations, highlights videos and graphics for your presentation.  

Let us break down these points into stages:

Write up a background summary 

The summary should include a short description of the features that were tested, details about the type of the usability testing and the way it was conducted, the number of participants, the tools used, the names of the moderators and testing team members, as well as a brief account of all the positive points and the issues and weaknesses that came up.

Explain the Followed Methodology

Sharing the key points of the method you used to obtain and analyze the data will help the rest of the team to understand the gravity and reliability of your findings. Make sure you mention:

  • the number of test sessions,

  • the number of participants,

  • the criteria used for their selection,

  • the tasks the participants were asked to perform

Test Results

In this section, you present the results of the testing. Don’t rush yet into conclusions. Just state the facts in a straightforward way, using the visual aids we mentioned above (charts, video annotations, highlights videos, graphics etc.), Make sure to mention

  • the number of participants that completed and failed to complete each task;

  • the average time required for the completion of each task;

  • the encountered issues with all the pertaining information (frequency, variations, impact, severity);

  • the comments, observations, and recommendations of the users;

  • other useful information the moderators noted down from the overall attitude and behavior of the participants;

Findings and Recommendations

Now is the proper time for stating your conclusions and general findings, as well as making some initial recommendations.

Each of your recommendations should be firmly based on certain data you have presented in the previous section. 

Remember to mention the positive findings of the usability testing. Strong points and encouraging comments by the users should definitely be communicated to the team members, thus strengthening their resolve to continue their efforts for the best.

Highlights videos

Creating a "highlights" video where you summarize the most critical findings is a great way to motivate your team and share the notion that your website should be fixed without having them watch all the usability videos or read a full study. Userfeel helps you create such a video in a matter of minutes. Here is an example:


Back up your findings and conclusions with a SUS

A great way to measure the perceived usability of a website or app is to use the System Usability Scale, invented by J. Brooke in 1986. SUS has gained wide recognition and repute for delivering highly reliable results by using a relatively small user sample.

SUS is a consistent way to measure the perceived usability of a website or application, as well as mobile phones, computing systems, software, call centers etc. All the user has to do is state his/her opinion about 10 different statements, with the answers scaling from 1 to 5, depending on how much he/she agrees with it: 1 stands for “I completely disagree” and 5 for “I fully agree.”

The SUS rating questions are:

  • I think that I would like to use this system frequently.

  • I found the system unnecessarily complex.

  • I thought the system was easy to use.

  • I think that I would need the support of a technical person to be able to use this system.

  • I found the various functions in this system were well integrated.

  • I thought there was too much inconsistency in this system.

  • I would imagine that most people would learn to use this system very quickly.

  • I found the system very cumbersome to use.

  • I felt very confident using the system.

  • I needed to learn a lot of things before I could get with this system.

The overall score runs from 0 to 100, depicting how friendly the feature or website really seems to the users. SUS score’s reliability has proven accurate and dependable over the decades by several disinterested parties, despite being much simpler and more economical than other similar evaluating methods.

Making the Most of your Usability Test

Needless to say that none of all these has any point if you fail or neglect to integrate these valuable insights into your development efforts. Set your priorities and try to implement all, if possible, the recommendations your users and team member came up with after studying the data of the usability testing.

Accurate usability testing requires time, effort and certain expenses – but, when done correctly, it can pave the way for the success of your project and further growth of your business.


Be the first to comment