Metrics-concurrent users versus rates

I frequently see confusion regarding concurrent VU’s versus VU’s per hour, or what should be called sessions per hour or transactions per hour. When modeling web traffic on the open Internet, the rate-based metrics are better suited to finding out what will really happen. If a site slows down, do your users know that *before* they arrive at your site or after they get there. The concurrent user model assumes that a new user doesn’t arrive until the previous user leaves. If the last user in the queue isn’t leaving, that is the users is stuck on the system trying to perform a task, then no new user arrives. This simply doesn’t happen. A user doesn’t know and frankly doesn’t care how many other users are on your site until the user gets to the site and discovers that it is about to die–or that the user would rather die than use this site.

Transaction rate and the number of virtual users concurrently on the system affect the application server differently.  Transaction rate primarily takes CPU to process the delivered pages, while the number of concurrent users primarily affects memory.  Both are important, but they are independent variables.  If the site performs well and the scripts are modelled correctly, then the transaction rate and total number of concurrent users will match your web analytics.  If the site degrades, CPU is still maxed out, but memory may not immediately be maxed out.  However, as the number of concurrent users increases, memory utilization will also increase, as well as database connections, etc.

This means that applying a rate-based metric to drive the load combined with the right scripts and use cases will drive the best load to allow you to see the behavior of your application under high loads.

Does Geographic Distribution of load really matter?

The question of geo-distributed testing is really 3 questions. The first question is where you should be generating the load, second how many locations are required to generate load and third, can I use sample agents in some locations instead of having load generators everywhere.

The reasons for geographic distribution of load is both simple and complex. On one hand, you are testing outside of the firewall for 2 primary reasons: 1) that is where your users are located and 2) to do end-to-end testing.

If you are only testing externally to have an end-to-end test, then you could just as easily do the load test in a loop-back scenario, i.e. generate the load on a circuit that sends the traffic out on one interface/circuit and it comes back in through the primary ingress point(s). If you have enough bandwidth and load generation, this is pretty simple and you can even use NISTnet to try to emulate latency. However, it is really only half of the reason for performing external tests. A loop-back test doesn’t really tell you about latency, even if you try to emulate it.  Moreover, you assumed that your users were sitting in your data center or lab, which is pretty unlikely.

If you wish to discover the customer’s experience of the site under load, you need to have real geo-distribution. For SUTs where there is only 1 bandwidth provider and the volume of the test is relatively small, 2 locations will probably suffice. This is especially true for situations where the customer base is centrally located in a small number of locations, for example a local retail chain that is only present in a few states. If you have customers coming to you nationally or internationally, then you need more. Given the demographics of North America, I recommend either of the following options: 2-3 load generation sites distributed across the time zones and on different ISPs with 5-6 sample locations spread out among the rest of the high-traffic areas, or my standard practice of 9 load generation locations domestically—3 east, 3 central and 3 west. If you are international, then you’d need to think about whether your traffic is European or APAC. This also lets me avoid crashing individual Content Distribution Network POPs, although it still happens. For some reason, they get annoyed at me for this.

So think about the reasons you’re even testing outside of the firewall. If it is only to do a simple end-to-end test, then don’t bother paying a provider or anyone else and just loop the traffic. If you want a good representation of your traffic, plan properly and distribute the load as well as you can.  Professional load test service providers do more than just deliver some hits.