eValid™ -- Automated Web Quality Solution Browser-Based, Client-Side, Functional Testing & Validation, Load & Performance Tuning, Page Timing, Website Analysis, and Rich Internet Application Monitoring. © Copyright 2000-2011 by Software Research, Inc.

Summary
The benchmarks described here illustrate eValid's ability to capture very accurate timing data that reflects actual user experience in first-download of the domain's "top page." This is the page at which most users arrive when going to the website.

Background
Timings of the "top page" of a website is important because these times affect how users perceive the quality of a site. Consider your response when you navigate to two commonly accessed top pages:

• Google is the most commonly used search engine. Google has has a very fast toppage; typically it has < 300 bytes of visible text, < 5 images, < 15 hyperlinks, and a base page size < 5 Kbytes. It typically downloads in around 1.5 secs.
• Yahoo! is a commonly used multi-purpose web portal. The Yahoo! site typically takes > 7.5 secs to download. Typically this site's top page usually has > 200 hyper links, and > 3500 bytes of visible text, involves around 20 images of various kinds, and shows typically nearly 4 Kbytes of visible text. Most of the time it has a fairly large base page of > 40 Kbytes.

Both of these sites are very popular. Ask yourself if you don't automatically think of Google as the better performer in terms of top page download speed?

Companies often spend a great deal of effort to achieve "top page" or "landing page" speed, and they spend a great deal of money trying to confirm the speed obtained. Two very widely used benchmarks -- those measured by Keynote and Gomez -- claim to measure download times and rank-order websites' "top page" speed. The eValid user-experience benchmarks -- all made from within the eValid browser -- suggest rather different results, as a comparative study of the data will show.

Benchmark Descriptions
The benchmarks all involve multiple URLs from a variety of companies. All of the benchmarks are scripted playbacks timed from within the eValid browser (see below for laboratory setup).

• KB40. Keynote Systems' KB40 Benchmark is a list of 40 selected companies. Keynote data differs from the eValid data in two important ways: (1) we are measuring full page download and resolution time for the indicated page; and, (2) we are using a "last mile" measurement very much like that experienced by users, whereas Keynote data is measured from its on-the-web network.

• G1. The Gomez 1-Page Benchmark involves 11 groups of similar firms' WebSites, with about 10 URLs per group. The total number of WebSites analyzed is 113. The G1 data differs from the eValid data in these ways: (1) the Gomez benchmark measures page download speed from its on-the-web network; and, (2) we are using full-browser download and rendering times, which will be longer than the Gomez values.

• FT-150. The Financial Times FT-150 Benchmark measures the 150 largest companies in the world. This is the set of WebSites we used in our recent FT-150 composite comparison.

Laboratory Setup
All of the benchmarks were created by using an eValid playback script (the script is shown with each benchmark) The environment settings for eValid were as follows.

• No cache, no cookies.
• eValid running in full-browser mode.
• Top page redirections [if any] are noted and corrected for in advance.
• Dedicated high-speed DSL line.
• 10-times repetition of the playback at 1.0 hr intervals to eliminate any hour-by-hour differences. Runs start at 0800, 0900, ..., 1700.
• Measured during the working day to capture data the way customers see it.
• Very short (1 msec) delay between commands to assure that all downloads are accomplished in as short a time as possible to prevent normal web latency fluctuations to mask server speed variations.
• eValid's detailed timing data mode is used so the fast and slow parts of each page will be visible in the accompanying performance charts.
• The baselines are updated weekly (with retrospective data for three prior weeks). The current schedule is KB40 is done on Tuesday; G1 is done on Wednesday; and, FT-150 is done on Thursday.