Your e-Business Quality Partner eValid™ -- Automated Web Quality Solution
Browser-Based, Client-Side, Functional Testing & Validation,
Load & Performance Tuning, Page Timing, Website Analysis,
and Rich Internet Application Monitoring.

eValid -- Financial Reporting System 1,000 User Case Study
eValid Home

Key Ideas: A very complex AJAX application was tested with nine distinct playback scripts that involve 600+ URLS and requires playback times from 110 to 584 seconds, and which in total download 101+ MBytes of data. All of the scripts required SyncOnText or SyncOnElementProperty commands to assure reliable operation in a LoadTest run that involved 1,000 Browser Users (BUs).

Tested Application Details
The application under test (AUT) in this case study was a very complex, AJAX-based, financial analysis and reporting system.

There is an HTTPS login (we were supplied with 1,000 login names/passwords), followed by a variety of operations that generally involve interrogating a database to produce a report. In some tests as many as four (4) sub-windows are open below the main window. All test sequences ended by logging out.

The test scripts -- a total of nine of them -- tended to be rather long, both in number of commands used and total playback time. Some scripts were over 200 lines in length, before addition of DOM synchronization commands. You can get a sense of the complexity if you estimate that 1-2 synchronization commands were needed for each newly arrived-at page.

Some tests require up to ~360 seconds (~6 minutes) total playback time even with the natural recorded Wait times all reduced to zero, i.e. with the tests run at a 99+% duty cycle.

In a few cases we added Delay 100 commands to allow adequate time for the DOM to stabilize when the page was particularly complex. For example, some pages had over 3,500 DOM elements.

The total test data volume -- due to the large size of some of the pages being downloaded -- was comparatively high, ranging from ~5 MBytes up to ~23 MBytes for a single script playback.

Script and Scenario Statistics
Here are the characteristics of each test script in the overall Loadtest scenario. These results were achieved with a Playback Multiplier of 1.0, and were run in "singleton mode".

#Test
Script
Name
LoadTest
Scenario
Percentage
Allocation
Original
Script
Lines
Original
Script
Actions
DOM-Based
Synchronization
Commands
Added
URLs
Visited
Playback
Duty
Cycle
Downloaded
MBytes
Total
Playback
Time
1Quote 5% 102 59 19 55 77% 8.02 323 secs
2 Calculator 8% 146 96 29 74 55% 11.11 364 secs
3 Charts 5% 114 6522 72 70% 10.48 284 secs
4Import 1% 81 57 9 66 71% 10.57 310 secs
5 Performance 4% 85 45 13 80 74% 9.01 167 secs
6Profiles 54% 265 200 63 88 59% 20.54 584 secs
7Publisher 8% 98 60 15 68 84% 13.97 579 secs
8 Queue 7% 63 39 11 72 71% 8.36 110 secs
9 Reporter 8% 98 45 17 49 59% 9.57 269 secs
Σ  100% 1052 666 198 624   101.63 2,990 secs

Script Conversion
The original scripts were recorded from the application "from life" and all of them worked out of the box using natural Wait times to assure synchronization. To optimize their use in a degraded server loading environment, in which we expected these scripts to encounter de-synchronizations, we augmented each long-running download in each script with one or more eValid SyncOn... commands.

Synchronization Methods Used
The Sync commands that were added to the scripts were either SyncOnText or SyncOnElementProperty commands:

Ramp Up Schedule
The load on the server was ramped up at the rate of ~10 users per minute, and an additional machine was added every 10 minutes. This process went on until a total of 10 machines or 1,000 BUs was achieved.

Cloud Computing Resource
The cloud machine images we used for these tests was an 8-core virtual CPU running at 2.67 GHz, and having 68 GBytes of RAM. The operating system on all images was Windows Server Data 2008 Center 64-bit [SP2].

eValid THIN Footprint
During the run we saw the footprint of eValid THIN LoadTest playback engine grow from a starting value of ~12 MBytes to as high 900 MBytes in some cases. The average eValid "THIN" footprint growth was to about 100 MBytes.

Resource Usage
Near the end of the test run, when all 1,000 eValid browsers were emulating users, the typical CPU usage rates were near 100%, and typical memory usage for each 100 eValid browsers was 38%-40% of the available RAM.