Site administrators are often torn between using real user or synthetic monitoring. There’s a lot of overlap in the tools that both types use, but the philosophy and approach of these two main ways for measuring a website’s efficiency are completely different. For example, both the RUM (real-user monitor) and the STM (synthetic transaction monitor) are able to calculate page load times and use a Pingdom outage checker to see where current problems exist on a given site.
But when you use synthetic diagnostics, you’ll have to create an outage artificially in order to see if the checker is actually working correctly. With RUM, you need to wait until a real-world outage first takes place before you can find out whether the checking function operates correctly. That’s just one example of the ways that RUM and STM deal with performance measurement. Here are other important differences between the two techniques.
Limitations and Advantages of RUM
There’s an analogy that helps to clarify the divergent techniques behind both real user and synthetic performance measurement: professional football. Sports teams hold regular practice sessions during the week but also play in regular season games. The real user approach is like using only the weekly games as a way to learn the game. Synthetic diagnostics are closer to the artificial environment of practice sessions, in which coaches try out hypothetical plays and game situations that may or may not actually happen.
The key limitation of only using actual site visitors to help diagnose problems is that you need a decent amount of traffic to test the system. You won’t be able to find out about all the possible problems that are likely to happen over the course of a year or more. The main benefit of this approach is that it does deliver measurable, real-world data based on what your site’s visitors are doing. If there’s a problem with a shopping cart, for example, you’ll find out about it when enough customers try to use the cart and face a difficulty checking out.
Limitations and Advantages of STM
Using artificial traffic to simulate visitors is at the heart of STM and leads to its greatest strength and its biggest weakness. Simulated traffic can’t create authentic scenarios, only fake ones. Continuing with the example above, it is possible to test shopping cart functionality without using actual customers. You can do it thousands of times and find lots of potential glitches in the cart. But not one of those glitches will be based on a real-life user’s experience, so the validity of the test is questionable.
The key advantage of this approach is that administrators can run checks on every conceivable part of a site, whether there has been traffic on it or not. Its similarity to the football coach who holds two practices each day the week before a big game. No, the plays during those sessions might never occur, but at least the team will be ready for them if they happen in a meaningful official game.