We recently ran our first large-scale usability study for the Hotel industry.
We benchmarked the ten biggest international hotel chains by running a large-scale usability test involving more than 1,000 users.
Here follow the desktop results. We will also publish the results for mobile in the next months.
If you own or run a hotel or any other website that involves availability and booking, or you are just a fan of UX and customer experience, you will find the following data extremely useful and impressive. Enjoy!
We asked users to perform a typical scenario that includes three tasks:
1. Search for availability and pricing for a room for two adults and two kids (five and six years old) for four nights in Barcelona (or any other relevant to the hotel chain city).
2. Go through the booking process until you reach the final confirmation button.
3. Find out what is the address of the hotel.
For each task, we measured the completion rate, which means how many users succeeded to complete it along with the completion time. We also asked users to rate their experience for each task on a 1-7 scale from "Very difficult" to "Very easy".
We also included overall questions to get their rating for the aesthetics of the website and the delight to use.
Finally, we added the SUS (System Usability Scale) questionnaire and the NPS (Net Promoter Score) question.
Task 1: Search for pricing and availability
The first task involved searching for prices and availability for a room of 4 people (2 adults and 2 kids). The absolute winner was choicehotels.com with an average completion time of 2 minutes and 20 seconds. Bestwestern.com, ihg.com, and novotel.com were pretty good as well with an average time below 3 minutes. Sol-hotels.com, hyatt.com, hilton.com, achotels.marriott.com, and radissonblu.com follow with timings from 3 to 4 minutes. Finally tryphotels.com has an average completion time of 5 minutes and 42 seconds, and apparently, there is a serious issue with their search process.
Test completion rates show that ihg.com has done the process fail proof with a 97,12% completion rate and the rest follow nearby, while hyatt.com and radissonblu.com have about 60% completion rate. This means that 40% of the people who tried to accomplish the task, did not succeed. The problem is even more prominent at tryphotels.com, where only 33% of the users managed to complete the task!
While time to complete and completion ratio are objective metrics, Ease of use as rated by the users is subjective and influenced by factors like the aesthetics and the overall functionality of the website. In fact, since users don't measure their timings, this metric is more user-centric as it shows how users feel about this process.
Task 2: Checkout
After finding room prices and availability, next task is the booking (checkout) process.
Again, choicehotels.com with an average time of 1 minute and 53 seconds has the fastest booking process. Very close to that is hyatt.com which was 8th at time completion of the 1st task, along with tryphotels.com (3rd) which was 10th on the previous task!
But, if you look at the next graph, it seems that hyatt.com and tryphotels.com also had a very low completion rate, along with radissonblu.com. That means that it's quick to checkout if you find out how.
Again, the final judge is the user. And users like a lot the checkout process of choicehotels.com, don't like at all the ones of hyatt.com, radissonblu.com, tryphotels.com.
Task 3: Find the hotel address
The third task we asked users to complete was to find the hotel address, which is a typical user action after booking.
Bestwestern.com had an average time of 25 seconds, and is the winner on this, with radissonblu.com following at 29 seconds. At this task, sol-hotels.com and novotel.com seem to have the most problems because it took about one minute and 15 seconds for users to find their address.
The above ranking is similarly reflected in the task completion rates. While 99% of users managed to find the hotel address of ihg.com, only 83% could do the same for hyatt.com.
Users rightly voted for bestwestern.com and achotels.marriott.com on this task.
After completing the above tasks, users were asked to rate their overall experience based on the delight to use and the website attractiveness (aesthetics).
We also asked them the NPS (Net Promoter Score) question (how likely it is to recommend the website to friends). The following graph shows the average of the NPS scores; it's not the formal way of calculating NPS but gives a clear picture as to how these hotels websites rank on this aspect.
Finally, we asked users the 10-question SUS questionnaire which measures the perceived usability of the website. What's interesting here is that the SUS ranking is strongly correlated to the NPS rankings. In addition to that, the SUS scores reflect the overall ratings and timings we got in the previous tasks.
If you want this report in PDF format, or if you want us to measure your hotel's website, please contact us.