This page describes the facts related to the ``CacheFlow Response'' document that was released by CacheFlow in October 1999 as a part of DataComm tests report and caused many complaints and questions from our readers.

In June 1999, Polyteam benchmarked seven caching products for a feature story in the Data Communications magazine. The caches were tested in our Lab using Web Polygraph and DataComm-1 workload. The results of the tests were made public in October 1999.

CacheFlow participated in June tests with a beta release of their CF-5000 proxy cache. As all other vendors participating in the tests, CacheFlow submitted ``vendor comments'' to be included as a part of the report. The comments contained a hyper-link to a document titled ``CacheFlow Response''. The ``Response'' was written by CacheFlow without Polyteam knowledge or participation.

Soon after the release of the report, CacheFlow competitors and others complained to Polyteam that the ``Response'' document contained strange statements and performance data that, in their opinion, were not true.

There were two items in ``CacheFlow Response'' that we have received complaints about. First, the ``Response'' contained a table where June performance results of CacheFlow competitors were compared with ``updated'' CacheFlow results quoted below.

The ``updated'' numbers were not only significantly better than performance demonstrated during June tests (which would be quite fine), but the measurements were absolutely identical across all configurations. The latter is technically impossible given the complexity of the test and the differences among products. Thus, the corresponding DataComm-1 tests were either not run or the results were altered. Unfortunately, our attempts to get Polygraph raw logs for these tests failed.

The second problem with the report was the implication that the ``updated'' results were produced by Polyteam. They were not.

After talking to CacheFlow and other interested parties, we came to the following agreement:

• Polyteam will remove the link to the ``Response'' document from the DataComm report,
• CacheFlow will change the wording that implied Polyteam participation in producing ``Response'' numbers, and
• CacheFlow will come to the PolyLab to produce real performance numbers using the same DataComm-1 workload.

In November 1999, we benchmarked two CacheFlow products: a single CF-5000 unit and a cluster of five CF-5000s. The caches were tested in our Lab using DataComm-1 workload. The description of these tests is available elsewhere, but here is the summary:

We want to once again discourage anybody from publishing Polygraph results without providing raw logs and detailed description of the tests. We suggest that readers ignore such results unless they are absolutely sure in their validity.

Please contact Polyteam if you are planning to use Polygraph for public benchmarking. If nothing else, we usually can provide free confidential consultation that will prevent you from making major mistakes.