We did some EXTREME Internet speed testing

[KelAuth]

We are often asked this one question. Is speed testing a real measure of Internet performance if it isn’t accompanied with additional monitoring and details? The answer is always no, not at all. Speed testing alone will not reveal problems, especially ongoing intermittent issues. More data is needed to analyze issues.

That said, this article (copied from an earlier web site we had) came to life after spending a considerable amount of time developing the OutagesIO speed testing tool which is included in our members control panel soon. Even as we pride ourselves for constantly trying to think outside the box, we found ourselves asking more questions than we could answer.

Here we share some interesting notes we kept while doing some general testing that may be of interest to anyone that is genuinely interested in gaining a better understanding of what is being provided by their Internet service company.

The irony of testing

The main irony about testing speeds is that you are using up your bandwidth testing it. The more you test, the more you are using up your bandwidth. You can basically use up all of your bandwidth simply testing it. In fact, you are also using up your neighbors bandwidth when you are testing it. Oh, and don't forget, you're using up your data plan too. That is irony.

While working on speed test ideas we did some very basic testing which we share in this article. The results are somewhat daunting.

First, we used a 50Mbps (6.25MBps download) Internet connection for the test. We started a file transfer of a 2.4GB file that we regularly back up and allowed the transfer to settle in, giving it a little while to get up to speed. We started averaging around the 2000KiB/s area which is about 16.4Mbps. Of a 50Mbps, that leaves around 33.6Mbps. The transfer did hit higher speeds at times, up to around 2700KiB/s.

We need to point out that when a provider sells you a certain speed, that speed can and usually does range. Sometimes a little higher, sometimes a little lower and on average, that changes. So, we’ll just use the number we were sold on this connection which is 50Mbps.

We started testing the speeds using both speedtest.net ™ and fast.com ™ and found their results fairly comparable but fast.com showing slower speeds than speedtest.net quite regularly. One such example is as follows. We tested over and over again and fast.com generally showed less speed than speedtest did.

Of course, while testing, we were using up the bandwidth so it only makes sense that speeds would be lower on both sites if we tested at the same time. However, we tested only one at a time, waiting for the transfer speed to build up each time.

Interesting and odd results

The interesting part was that while both were showing a reasonable result, it was odd that our transfer speed ended up going as low as only 202KiB/s which is only around 206.8KB/s (kilobytes) or 1.65Mbps. Why is this very interesting?

First, we are downloading from one of our own servers which has a 1000Mbps connection on the Internet. The server was pretty much idle at the time of the test and testing from the server to another showed over 900Mbps available to us. While transferring, the speeds never hit what we are supposed to be getting which is 50Mbps yet there was bandwidth left which the transfer should have been able to use. Where did the extra bandwidth go?

Second, while testing, the transfer slowed down because we were using up the bandwidth of course. However, after we stopped testing but continued transferring the file, the 50Mbps connection was very slow, very sluggish, trying to pull up a web site on a very fast remote site that we often use was not fast, it was very slow to load the page, taking nearly 30 seconds.

That is odd because we have a 50Mbps which was using only around 16.4Mbps while transferring the file. There was no other heavy traffic on the connection, at least not at our testing location. So, where is the rest of the bandwidth?

What is limiting the transfer?

It is interesting to see this limitation because the transfer never hit 3000KiB/s or 24.6Mbps yet we have a 50Mbps connection to one of our 1000Mbps servers which we know absolutely is not being limited in speed in any way shape or form. Why were we getting an average of only 16.4Mbps then?

One explanation is that when you visit sites over the Internet, your connection (or packets) is being routed through various networks so even if your provider sells you a 50Mbps connection, it doesn’t mean you’ll be able to use it. Your provider doesn’t control networks not owned by them.

When we use speedtest.net, it always seems to default to the same data center that we are using. Our provider may have their own speedtest.net host at the same location but we have not determined this. We do see hops heading into other states before getting to the data center which is only some 100 miles away however so, not quite sure how they have their networking set up.

Why our test is maxing out at 16.4Mbps remains a mystery and trying to ask the provider about it would be crazy frustrating as you can never reach anyone that actually knows or tells anything. They seem to think these things should be kept secret while these details should in fact be fully transparent.

Honest speed testing

Finally, for months, we’ve been working on a speed test that would be as honest as possible but there are so many variables that this does not seem possible. The only way to know would be to run a test non stop using up all our own bandwidth and neighbors as well.

We’ve done it all, we tested in every way possible. We actually did try testing non stop, 24/7 to our own servers. We also tried testing every 5 minutes, 15, 30 and 60 minutes and while there are some averages, speeds often changed the more we tested, which makes sense.

Mostly, there were speed variables between just a few Mbps, up to 10Mbps and other times much higher for short periods of time. At one point, speeds looked perfect yet we could see very high losses in our connection, only to come back moments later. The trick was definitely to know the exact best times to test but in the long run, speed testing wasn’t showing us all that much.

The image graph above was one of the more interesting testing that we did. In this case, a 100Mbps service tested regularly around the 50Mbps range as average and the next week, started getting closer toward the 100Mbps speed sold by the Internet provider. This one stumped us because this is a massive difference in speeds. We will continue to monitor this service to see what happens.

In another case, a 50Mbps connection (above) that was tested showed an average of over 50Mbps yet this connection was awful, constantly disconnecting real time communications such as ssh, VoIP. Speed testing alone showed very little but once we added the other metrics that OutagesIO gathers, things started making sense.

We could see that a network path our provider was taking was acting up. Contacting both our provider and the upstream would have been extremely frustrating without additional details. The lesson in all of this was that without other details, speed testing alone is a hit and miss test and worse, often misleading when optimized and commercialized.

Shared bandwidth

This is the nature of shared bandwidth of course and tends to explain why there is no proper way of speed testing. It is simply not possible. Your connection is sold at a certain speed, you pay a monthly cost and you get a shared connection which can vary constantly as others in your neighborhood and other parts of the provider network use it.

You can test day and night and most times the results will show that you are getting what you are paying unless things are really broken but always keep in mind, that test is based on shared bandwidth.

The only type of connection you could test reliably is a dedicated connection but even then, you can only test from your location to your provider, everything else is an unknown. Even with dedicated connections, the provider is sharing their overall bandwidth with their customers.

The bottom line is that speed testing alone is not very useful, unless you do it at the exact moment that your service begins acting poorly. What we really need is an intelligent, fully automated speed test based on algorithms that gather up much more information than only Internet speeds.

This is an ongoing struggle for us and we continue trying ideas to see what will better work.