Some Notes on Monitoring User Experience for Consumers

In a previous blog on user experience monitoring, “User Experience Monitoring in Modern Data Networks”, we looked at some common services for business users and how operators can monitor them: voice, web browsing, and video conferencing. In this blog, we will further look at a couple of critical consumer use cases: gaming and video streaming. These two use cases are examples of services that require a seamless network to have an optimal user experience, and monitoring goes beyond traditional network metrics.

Gaming

Gaming is the arena for real-time performance, gamers demand an uninterrupted and responsive experience. It is a matter of life and death. When it comes to gaming, the most critical Key Performance Indicator (KPI) to monitor is latency. Latency here would mean the time it takes for a data packet to travel between the user and the game server.

And there is no lower limit, the less latency, the better you will score in the game. Take a look at the charts below from a study by the Worchester Polytechnic Institute: “On Latency and Player Actions in Online Games”

Figure 1 Worchester Polytechnic Institute: “On Latency and Player Actions in Online Games”

Here we study two different game types. First-person shooter games and racing games. The first chart (a) considers the hit fraction and the impact of delays on the hit fraction. As can be seen from the chart, the hit fraction is 60% smaller when network delays exceed 300 milliseconds compared to a network delay of 50 milliseconds or less. Similar results can be seen for racing games, a nearly doubling of the lap time when delays excel 250 milliseconds as opposed to having a latency near zero milliseconds.

A similar study from the same institute: “The Effects of Network Latency on Competitive First-Person Shooter Game Players” shows similar behavior for one of the most popular First-Person Shooter games Counter-Strike. The less latency we have in the network, the better accuracy, and scores for the game.

Figure 2 Worchester Polytechnic Institute: “The Effects of Network Latency on Competitive First-Person Shooter Game Players”

In general, you can consider 50 millisecond one-way delay as sufficient for providing a smooth gaming experience.

Video Streaming

Video streaming differs a lot from gaming. When it comes to network latency, it really is not that important for video streaming. An important KPI for video streaming that is close to latency is Time-to-Content. How long does it take for the video to start playing from the time that we click the Play button? But network latency affects Time-to-Content very little, the throughput of the network will determine how quickly the first video segment is downloaded and the video starts playing. Other factors that affect the user experience are rebuffering events: the number of times a video playback pauses to buffer more video data, as well as the video resolution. Streaming video will by nature downscale the video to lower resolutions if the network is congested and lower resolutions naturally mean a lower user experience of the video.

Video Mean Opinion Scores (MOS) is a KPI that is very good at summarizing the user experience of a video service. Standardized in the International Telecommunication Union as the P.1203 standard, it will summarize the quality into a simple scale from one to five, similar to the Mean Opinion Scores for voice. As long as the Mean Opinion Score is above four, you know that you are good for the video part.

To calculate the Mean Opinion Score, you need KPIs such as the rebuffering events, video resolution, and time-to-content. These KPIs cannot be measured with an active measurement such as TWAMP (Two-Way Active Measurement Protocol). The best way to obtain these KPIs is to simulate a user watching a video stream and analyze the events from the player.

Summary

In two blogs, we have looked at analyzing some of the services that are important for business users and consumers: voice, web browsing, video conferencing, gaming, and video streaming. We talked about the fact that the nature of each of these services is different, so there are different requirements for network performance and hence on how to monitor the user experience of these services. The table above summarizes for each of the services what the main impacting factors are, recommended requirements and thresholds for your monitoring solution, and the measurements that can provide the data.

Conclusions

Actively monitoring the key metrics for each service type allows the service providers to identify potential issues and proactively address them before they affect their customers. A proactive approach ensures a smooth user experience leading to happier customers and increased loyalty and is vital today as loyal customers and minimized churn are key for communication service providers to build a successful business.

About Creanord

Creanord offers technology solutions for network performance measurements and quality assurance. Driven to turn data into outperforming networks, Creanord helps mobile operators, managed service, and wholesale providers as well as critical communications providers to design, offer, and deliver services with confidence.

With PULSure, service providers can track network performance and user experience in order to meet the most demanding service needs of their customers.

Since 2000 Creanord has served customers who offer the most demanding networks and applications making societies smarter, businesses more competitive and our lives safer.

Creanord is headquartered in Helsinki, Finland with a global operations and partner network.