In December 1999, we benchmarked IPWorX WC-100, a caching proxy being developed at Lucent. The tests were executed in PolyLab using DataComm-1 workload. We have followed the DataComm rules and methodology to ease comparison of the results.

DataComm workload was originally developed for Data Communications magazine in June 1999. A detailed description of DataComm-1 workload and June tests is available elsewhere. In June competition, vendors were given an opportunity to test their products at two different request rates. For these stand-alone tests, we present three datapoints.

As always, we do not give purchasing advise. We provide performance measurements and expect the reader to use this information, along with other important factors, in their decision making process.

The ``Executive Summary'' table below summarizes the performance results. Complete Polygraph logs are also available for those who want to reproduce the results or extract measurements not presented here.

The ``Total Price'' column is a sum of the list price of cache(s) and the cost of network gear in the test setup. In our price/performance analysis, we use total price rather than list price of a cache to adjust for expensive equipment that may be required to achieve high performance numbers and/or aggregate individual caches into clusters. If the reader already has the required network components in place, the price/performance ratios should be adjusted.

Mean response time is reported separately for hits and misses to show the proxy savings and overheads on the two most important request paths. The ``All'' column depicts response time for all request classes.

Price/Performance column shows throughput (in req/sec) normalized by Box price (in thousand dollars). In other words, ``how much throughput one thousand dollars can buy?'' To be precise, the column should have been named ``performance/price'', but we decided to use the traditional wording.

The ``Persistent Connections'' column shows if a box supports HTTP/1.1 persistent connections on server and client side of a proxy.

All runs finished with at most 0.01% of failed transactions. Note that the rules disqualify a run with more than 3.0% of errors.

This section gives a detailed analysis of major performance measurements. Since we have tested a single Lucent product, bar chart format used in our past DataComm-1 reports is not very useful. All major measurements are better expressed in a tabular format (see the ``Executive Summary'' table).

The DataComm-1 workload offers a hit ratio of 55% so a cache cannot achieve a higher hit ratio. IPWorX has demonstrated close-to-ideal hit ratio for all request rates.

To simulate real-world conditions, the DataComm-1 workload introduces an artificial delay on the server side. The delays are normally distributed with a 3 sec mean and 1.5 sec deviation. They play crucial role in creating a reasonable number of concurrent ``sessions'' in the cache. Ideal response time for this test would be 1.35 sec (45% of 3 sec mean), corresponding to zero-delay hits and 3 sec delay misses.

IPWorX average response time of 1.7 sec is 100-150 msec (or 7-10%) higher than most other products we have tested. Hit response times are also on the high side, especially under 690/sec load. Since miss response time does not change with load, we suspect that disk read channel may be a bottleneck. Finally, raw traces presented in the next section imply that 4 hour averages underestimate sustained response times.

The graphs below show document hit ratio and response time traces averaged at 5 minute intervals. The red lines correspond to a single Polygraph client, while blue lines are the averages across both client machines used for the tests. When the variance is small, blue lines may be hard to see because averages are very close to individual traces.

The traces are plotted with the same scale to ease the comparison. Both warm-up (first 24 minutes) and measurement phases are shown.

While hit ratio stays at 55%, response time is getting worse during the first 3 hours of the tests and seems to level off at the final hour. Constant hit ratio implies that response time problems are related to internal cache performance rather than disk capacity shortage or skipped hits.

Here are the configuration details for tested product. Please note that we have tested a beta version of IPWorX. Lucent expects to make the first production release in April 2000.

All numbers are totals for the tested configuration unless noted otherwise. IPWorX WC-100 is a single CPU unit.

It is a Polyteam tradition to give vendors a chance to comment on the tests after they have seen the review draft. The comments below is a verbatim vendor submission. Polyteam has not verified any of the claims, promises, or speculations that these comments may contain.

Lucent wishes to thank the Polyteam for running the DataComm tests on our IPWorX WC-100 product.

We are especially pleased with a performance / price ratio of 33.8 requests per second per $1000 total system list price. This places the Lucent product among the top performing caches tested by the Polyteam using the DataComm Tests.

We believe the results of this testing provide important information we can use to continue to improve the performance of our equipment.