In June 1999, we benchmarked five Dell caches running Novell Internet Cache Software. The caches were tested in our Lab using PolyMix-1 workload. We have followed the first IRCache Bake-Off rules, methodology, and presentation format to ease comparison of the results.

As always, we do not give purchasing advise. We provide performance measurements and expect the reader to use this information, along with other important factors, in their decision making process.

The table below summarizes the results and provides links to detailed specifications and performance graphs for each Box. Complete Polygraph logs are also available for those who want to reproduce the results or extract measurements not presented here.

The ``Cache with Network'' price adds the cost of network gear in the test setup to the list price of a cache.

The graphs below compare major performance measurements -- throughput, document hit ratio, and mean response time -- for all five tested products. We briefly explain each graph. For a comprehensive discussion about PolyMix-1 workload and measurements, please refer to the first IRCache Bake-Off report. In fact, comparing bake-off results with these tests is an interesting and educational activity.

The Document Hit Ratio graph (above) shows how a cache maintains cache hit ratio under increasing load. The PolyMix-1 workload offers a hit ratio of 55% so a cache cannot achieve a higher hit ratio. However, due to various overload conditions, insufficient disk space, deficiencies of object replacement policy, and other reasons, the actual or measured cache hit ratio may be smaller than offered 55%.

All of the Dell boxes maintained cache hit ratios higher than 51%. A close look at the results shows that for most request rates the hit ratios stayed around 55%. Each of the systems' cache hit ratios decrease as the request rate increases, with the exception of the peak request rate point. This variation in hit ratio can be attributed to the load level and the order of the runs. The highest request rate runs were executed immediately after the fill. Those runs had an advantage of ``fresh'' disk storage, yet undisturbed by random nature of PolyMix workload. This advantage led to hit ratios close to optimal at peak throughput points. Other high throughput runs have to cope with sub-optimal placement of objects on the disks and usually have somewhat lower hit ratios. Finally, at lower request rates, a cache has no trouble keeping up with the traffic, and hit ratios are close to optimal.

We cannot explain lower hit ratios for ICS-635B at two low request rate points. The rest of data points prove that ICS-635B can keep hit ratios around 55% regardless of the load.

The Mean Response Time graph (above) illustrates response times degradation with increased load on a cache. To add realism to the test, the PolyMix-1 workload introduces an artificial delay on the server side. The delays are normally distributed with a 3 sec mean and 1.5 sec deviation. They play crucial role in creating a reasonable number of concurrent ``sessions'' in the cache. The delays along with 55% hit ratio also affect transaction response time. Ideal response time for this test would be 1.35 sec (45% of 3 sec mean). An increase of response times above ~1.6 sec indicates loss of hit ratio and/or increased delays introduced by a cache.

Detailed graphs show that ICS-130A cache adds negligible delay even at highest request rates while the -B models cannot maintain optimal response times as they approach 100% utilization. The increase in response times cannot be attributed solely to lower hit ratios (see the hit ratios graphs and discussion above). Differences in hardware configurations (see the configuration table below) are more likely to explain the phenomenon. As far as performance components are concerned, the two ICS-B systems differ only in the number of disks. Adding extra disks allows to achieve higher request rates, but at the expense of noticeable response time degradation. Apparently, CPU or some other resources become bottleneck and slow down the systems.

Finally, the high-end system (ICS-635B) shows close to optimal mean response time under load with no signs of performance degradation throughout the reported request rate range.

Though unfortunate, the order-dependency has marginal effect on the results. We are improving the run rules and workload to eliminate the dependency.

The two graphs in this section are of interest to price-aware readers. Following the first bake-off model, we used the total price of the system in our calculations (i.e., the sum of the list price and the cost of the network gear).

The graph above identifies the most cost effective system for a given request rate. The same graph can also be used to calculate the most cost-effective clustering solution. Please refer to the bake-off report for detailed examples of calculations using this graph.

The last graph is similar to the ``Throughput Cost'' graph except cache utilization (i.e., percent of the peak request rate) is now used as the x-axis. This information is not very useful for making buying decisions because the latter are usually based on desired request rate. However, we believe that the graph depicts a certain efficiency of a product. Here, ``efficiency'' means how well a proxy is spending money at a given utilization level (rather than at a given request rate).

For example, ICS-635A and ICS-635B show the same efficiency despite different request rates. The latter may indicate that the price of ICS-635B was scaled to match the increase in request rate compared to ICS-635A.

All runs finished with less than 0.01% of failed transactions, with the exception of ICS-130B peak run that showed error ratio of 0.98%. Note that the rules disqualify a run with more than 3.00% of errors.

As during the first IRCache Bake-Off, a few systems did not meet participant's expectations. The table below shows the attempted and achieved rates.

We finish our summary with a table that illustrates configuration differences among tested caches. See Box-specific pages for details.

[Note: The text below was submitted by Novell/Dell team.]

The Novell/Dell results in the First IRCache Web Cache Bake-Off Official Report were produced with beta versions of Dell's ICS solutions. These new results, produced with Dell's shipping solutions, demonstrate significant performance improvements since the 1st IRCache Bake-Off.

• Peak Performance: The Dell ICS-635B, originally called the Novell/Dell-L in the 1st bake-off, is now rated at 1800 PolyMix-1 requests per second. This result is the highest single-cache result as well as the highest bake-off result overall.

• Cache Hit Rate: All of the Dell solutions maintained cache hit ratios higher than 51%. A closer look at the results shows that for most request rates the cache hit ratios remain closer to Web Polygraph's offered 55% cache hit rate.

• Mean Response Time: Two of the caches (ICS-130A and ICS-635B) handle load increase without response time degradation up to their maximum request rates. The other three caches (ICS-130B, ICS-435B, and ICS-635A) exhibit a theoretical exponential growth of response time as they approach 100% utilization.

To learn more about Dell's ICS cache appliances, see http://www.dell.com/products/poweredge/serversolutions/caching.htm.