Does Geographic Distribution of load really matter?

The question of geo-distributed testing is really 3 questions. The first question is where you should be generating the load, second how many locations are required to generate load and third, can I use sample agents in some locations instead of having load generators everywhere.

The reasons for geographic distribution of load is both simple and complex. On one hand, you are testing outside of the firewall for 2 primary reasons: 1) that is where your users are located and 2) to do end-to-end testing.

If you are only testing externally to have an end-to-end test, then you could just as easily do the load test in a loop-back scenario, i.e. generate the load on a circuit that sends the traffic out on one interface/circuit and it comes back in through the primary ingress point(s). If you have enough bandwidth and load generation, this is pretty simple and you can even use NISTnet to try to emulate latency. However, it is really only half of the reason for performing external tests. A loop-back test doesn’t really tell you about latency, even if you try to emulate it.  Moreover, you assumed that your users were sitting in your data center or lab, which is pretty unlikely.

If you wish to discover the customer’s experience of the site under load, you need to have real geo-distribution. For SUTs where there is only 1 bandwidth provider and the volume of the test is relatively small, 2 locations will probably suffice. This is especially true for situations where the customer base is centrally located in a small number of locations, for example a local retail chain that is only present in a few states. If you have customers coming to you nationally or internationally, then you need more. Given the demographics of North America, I recommend either of the following options: 2-3 load generation sites distributed across the time zones and on different ISPs with 5-6 sample locations spread out among the rest of the high-traffic areas, or my standard practice of 9 load generation locations domestically—3 east, 3 central and 3 west. If you are international, then you’d need to think about whether your traffic is European or APAC. This also lets me avoid crashing individual Content Distribution Network POPs, although it still happens. For some reason, they get annoyed at me for this.

So think about the reasons you’re even testing outside of the firewall. If it is only to do a simple end-to-end test, then don’t bother paying a provider or anyone else and just loop the traffic. If you want a good representation of your traffic, plan properly and distribute the load as well as you can.  Professional load test service providers do more than just deliver some hits.