Your e-Business Quality Partner eValid™ -- Automated Web Quality Solution
Browser-Based, Client-Side, Functional Testing & Validation,
Load & Performance Tuning, Page Timing, Website Analysis,
and Rich Internet Application Monitoring.

eValid -- Mobile Agent Loading Experiment #1
eValid Home

In this demonstration experiment we ran 1[1]100 BUs in parallel against an AJAX site that is heavily used by smartphone clients. The site provides schedule access to a public transportation system in a metropolitan area. The results show a degradation in performance from the server side at around 85 Browser Users (BUs) in parallel.

Playback Script
In this experiment we ran the following eValid script against the target website, in which eValid is set up to behave like an iPhone 4:

# Recording by eValid V9 Build #305, Build Date: (Feb  2 2011) 
# Copyright (c) 2011 by Software Research, Inc. 
# Recording made on: Microsoft Windows Vista (IE 7.0.6001.18000)

ProjectID "Project"
GroupID "work"
TestID "setuseragentE"

ScreenSize 1280 1024
FontSize 2

InitLink ""
Wait 3000

# ------------------------------------------------------------------------------
# Apple iPhone 4 ...
SetUserAgent "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) " \
	"AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3"
Wait 2236
FollowLink 0 44 "Mobile XXXX" "http://www.XXXXX.XXX/" ""
SubFrameSet 1 "XXXXX - Mobile Web"
Wait 5192

# ------------------------------------------------------------------------------
# Reset the clock before making the request...

FollowLink 1 24 "Real Time Departures" "http://www.XXXXX.XXX/schedules/eta/" \
	"index.aspx" ""
SyncOnText 1 "XXXXX departures as of" ""
SyncOnText 1 "min" ""
SyncOnText 1 " car)" ""

# ------------------------------------------------------------------------------
# Read the time it took for the data to download and sync to the device...
ElapsedTime "Train Schedule"

# ------------------------------------------------------------------------------
# Write the value to a local file for processing...
SaveRecord DownloadTime.txt "%T \tInstance: $NUMB\tTime: %E" 

This script uses the internal time for each BU process to measure how long the information request to the site took, including the time for the AJAX delivery of the data to stabilize in the browser face. The phone has GPS capability, so the data reported depends on the location of the phone at the time the request was made. (It apparents defaults to a central station absent GPS details, which were not being sent in this test.) The script then records that time in a local file for later processing (see description of data chart below).

Scenario Description
The scenario used in this test was based on the idea that each playback takes about 15 seconds of elapsed time.

The launch sequence added a new BU to the composite load at the rate of one new BU every 3 seconds, so that after ~300 seconds all 100 BUs would be running in parallel. This is reflected in the eValid LoadTest Script. The repetition count was set to 50 -- each BU would replay the test script 50 times in a row -- so that the scenario could run for a short length of time at a full 100-BU wide level.

LoadTest Summary
Here is the LoadTest #1 Summary Report that shows the results of the test at the end. The earliest-launched BUs ran ~30 times, and the latest launched BUs (at the bottom of the table) ran up to 3 times. The script completed 1,419 times in the 10:06 minute long scenario. The data volume on the AJAX part of the playback (the section where the timing was done) was 329.1 KBytes per cycle.

In effect, as you look at the data from User Number 1 through User Number 100, the load is increasing linearly, and the average playback time increases as well, due to the increased load.

The data volume to the mobile device is very small so saturation of the I/O channel due to overload from all of the activity is not a factor in the slowdowns.

LoadTest Chart
Here is the LoadTest #1 Chart that shows the range of overall playback time values for all of the BUs running in parallel. Note that the number of data records involved for each BU running decreases as you move to the right of the chart.

Chart of Playback Times
Here is a chart of the specific timed sub-sequence in the script as a function time of data collection.

The scenario started at 13:30 and required almost 6 minutes to build up to 100 simultaneous BUs. After the driver machine driver stabilized we kept it running for an additional 5 minutes and then stopped the scenario manually. As the reports above show, we collected data from the test executions for a specific section of the playback script, namely the passage that involves collection and display of schedule information. (Click for the full-size image).

If as few as 100 simultaneous requests for this mobile service are made concurrently, the average delivery rate of data decreases by 3X or more.

The data in the chart also suggests that the simultaneous capacity of this server setup to keep performance within 2X from the base rate is about 50 users.