|
|
Summary
These
eValid one-page benchmarks
capture very accurate timing data
that reflects actual user experience
in first-download (empty cache download) of each domain's "top page."
This is the page at which most
users arrive when going to the website.
eValid data show accurate "last mile" timings that differ
remarkably from what is measured & reported by Keynote or Gomez.
Keynote Comparisons
The Keynote websites presents,
within each major category,
just one set of detailed timings.
Here are the comparative results from the current "top six" sites arranged
in order of increasing web page download time.
On average, the eValid figures suggest that the
Keynote values
underestimate actual download times by a factor of around 8.6 to 1.
»»Conclusion: The Keynote benchmark data do not appear to represent "last mile" measurements.
| WebSite Analyzed | Keynote Data | eValid Data | ||||
| 4-Week Average | 1st Week | 2nd Week | 3rd Week | 4th Week | ||
| 0.41 sec | 1.451 sec | 1.621 sec | 1.970 sec | 1.211 sec | 1.003 sec | |
| FedEx | 0.53 sec | 6.466 sec | 6.377 sec | 7.914 sec | 6.045 sec | 5.528. sec |
| Yahoo | 0.57 sec | 8.178 sec | 5.796 sec | 12.451 sec | 7.807 sec | 6.655. sec |
| Hewlett-Packard | 0.63 sec | 4.942 sec | 5.666 sec | 6.171 sec | 4.333 sec | 3.597 sec |
| Symantec | 0.66 sec | 4.442 sec | 4.502 sec | 4.878 sec | 5.002 sec | 3.387 sec |
| Sprint | 0.70 sec | 4.894 sec | 5.071 sec | 5.028 sec | 5.588 sec | 3.890 sec |
Gomez Comparisons
The Gomez download time numbers for six selected websites
were interpolated from the Gomez charts
(they do not give precise values).
On average the eValid measurements for top-page download are about
2.6 times longer than what Gomez reports.
»»Conclusion: The Gomez benchmark data do not fairly represent "last mile" measurements.
| WebSite Analyzed | Gomez Data | eValid Data | ||||
| 4-Week Average | 1st Week | 2nd Week | 3rd Week | 4th Week | ||
| America West | 1.7 sec | 14.01 sec | 11.6 sec | 13.61 sec | 13.49 sec | 16.79 sec |
| BizRate | 1.9 sec | 4.25 sec | 4.27 sec | 4.16 sec | 4.33 sec | 4.24 sec |
| Fidelity | 2.1 sec | 4.39 sec | 4.70 sec | 3.77 sec | 4.28 sec | 4.80sec |
| TD Waterhouse | 2.3 sec | 3.93 sec | 3.94 sec | 4.35 sec | 3.34 sec | 4.07 sec |
| NorthWest Airlines | 4.1 sec | 10.25 sec | 8.43 sec | 13.74 sec | 10.02 sec | 10.25 sec |
| MSN Shop | 4.5 sec | 7.47 sec | 11.70 sec | 6.98 sec | 5.42 sec | 5.76 sec |
Possible Conclusions From This Data
The eValid numbers show the kind of variation we would expect, but
the average of four weeks' data is an average of 10 * 4 = 40 measurements
so these data are very accurate.
Given that the timings were done with the most accurate web measurement
possible -- from within the timer-instrumented eValid browser -- it
is clear that the other values are measured either:
Our analysis of all of the data reflects this same pattern, for some 113 Gomez G1 websites and all 40 of the Keynote KB40 websites.
Laboratory Setup
All of the benchmarks were created by using an eValid
playback script.
The environment settings for eValid were as follows.
Caveats
Websites and benchmarks change all the time and the data on which these
experiments were based may be "out of date" by the time you read this page.
