eValid™ -- Automated Web Quality Solution Browser-Based, Client-Side, Functional Testing & Validation, Load & Performance Tuning, Page Timing, Website Analysis, and Rich Internet Application Monitoring.

Introduction
This page analyzes certain eValid website analysis performance factors, but the clearly dominant factor is the total download size of the website. Hints on usages and expectations on performance are included. However, you should always remember: "your results may vary depending on your hardware (your machine and your internet connection)."

• Connection Speed. During a website analysis eValid downloads every page in the website (sub-website) that it examines and maps. Accordingly, the download transfer rate is the governing factor based on our internal estimates that < 10% of the time is spent parsing and/or mapping assuming it is not RAM-limited.

• Web Latency. Obviously if the web is running slowly then the scan will take longer, so web latency can be an important factor in how long a website scan can take.

• Sanning Mode. The Background (Quick) is faster than the Background (Normal) mode, and both are significantly faster than the Full Browser mode. In addition, both background modes do NOT involve rendering page, whereas Full Browser mode does. This is noteworthy because IE (and, because eValid uses the IE rendering DLLs, eValid also) is known to have memory leaks and other performance problems in long site analysis runs. We have noted that runs of 1000's of pages often produce anomalous behavior when done in Full Browser mode. In this case, it is more effective to switch to Background (Normal) mode.

• Available RAM. Parsing is done once for each full page downloaded. Mapping (updating the internal hash-coded table of links) is performed once for each link found in each page. It probably is a good idea to expect ~25% of the available RAM to be used by the OS and by eValid itself, leaving ~75% available for in-memory link/structure mapping. It appears that at/above about ~500K links the is never enough room (on a 512 MB machine) to keep the mapping in RAM but that varies with the total volume of URL strings.

• Machine I/O Rates. In many cases the machine on which you are performing the analysis has internal limits, e.g. maximum I/O transfer rate to/from the LAN, actual transfer rate to/form the modem (if the machine is direct connected), etc. In most cases, even with a high-spped LAN the machine doing the analysis usually only supports a fraction of the available LAN bandwidth. This results in a slower analysis.

• Number of Pages (URLs). The practical limit on the number of pages analyzed is determined by the combination of the acceptable total runtime, the transfer rate, and the average links per page.

• Average Links/Page. We generally see values in the 10-50 links/page average. Some websites are much more; some are very sparse. The more links per page the lower the total eValid capacity in one run.

• Link Length. During an eValid run links between URLs are mapped using the full names of both the parent URL and the child URL (the link data itself is hash coded). This means that if a site has longer URLs it will take more space than a site with smaller, shorter URLs.

• Run Efficiency. The measure we recommend for measuring run efficiency is the ratio of total time spend downloaded to the total elapsed time. This ratio is a good indicator of paging (see below) when it drops below ~85%.

• System Paging. The symptom that signals the upper bound has been reached is when, for any of the above reasons, the system begins paging. Once it has begun to page -- usually when the available RAM is less than 10-20 MB -- performance drops drastically. Basically, when the system begins paging the time it takes to map a site is so long that even with hashed lookups in the link table the run will have to be suspended. After paging, you pretty much have to split the site.

• Avoiding Infinite Depth Searches. Of course, you can set the depth limit to something realistic like 50 or 75, but it is wiser to understand, when an apparent infinite depth search occurs, why it occurs. The most common reason is that the scan includes a search engine which actually produces results when you click "GO" without asking to search for anything. The resulting synthesized results page will have a lot of links, including and additional search button. It's easy to see how this makes for an apparently deep site -- even though this depth is "artificial."

• Splitting The Site. The best solution to handling a very-large site is to split the evalid site analysis run into two or three or more parts. Partitioning a website is easy using combinations of the Include SubDomains feature and/or the Excluded URL Strings feature.

• Other Factors. There are other factors that affect the length of a run, including whether or not you are using the machine for "something else" during the run.

• Solutions. A reasonable goal is to have each separate eValid website analysis complete in ~1 hour. In most cases, a combination of more RAM and/or a faster connection to the website server will solve eValid site mapping performance problems. Analyzing very large websites on smaller machines probably will require splitting the work into parts.

Estimates
Here are some estimated results with typical values of some the above-mentioned parameters.

Actual Performance Examples
Here are some results of successful [ran to normal completion] eValid scans/searches.