In May 1999, Data Communications magazine invited caching vendors to participate magazine's performance review. Polyteam was hired to execute the tests in June - July. The review is published in October 7, 1999 issue of DataComm. Our local version of the review is also available. This page provides some preliminary information about the event.

Participants and Outsiders

Data Communications have invited CacheFlow, Cisco, Cobalt, Entera, Eolian, IBM, InfoLibria, Inktomi, Microsoft, Netscape, Network Appliance, Novell, Packetstorm, and Peregrine.

The following companies accepted the invitation: CacheFlow, Cobalt, Entera, Eolian, IBM, Network Appliance, Novell, and Peregrine. Novell has also asked for Compaq and Dell to be present as independent OEMs running Novell's ICS proxy software, and Dell agreed to take part.

IBM decided to bail-out before the tests begin. Entera simply did not show up for their test slot.

The Rules and Workload

The following subsections give an overview of Data Communication tests.

Experiments

We will be using the DataComm-1 workload for the performance tests. DataComm-1 is similar to the PolyMix-1 workload used during the first IRCache cache-off. However there are important differences that prohibits direct comparison of the results. Among those differences are longer experiments and presence of persistent connections.

The details of the workload are available on our Workloads page. Here is a brief description of experiments executed during the tests:

No-proxy:

A no-proxy test will be executed to verify the test cluster equipment. The no-proxy test will be one hour in duration. Results of the no-proxy test will not be included in the article.

Filling-the-cache:

The cache must be in a full state before the DataComm-1 experiments may be executed. A cache is "full" when its disk utilization stops increasing. Results of filling-the-cache will not be included in the article.

DataComm-1:

Two runs of the DataComm-1 workload, at two different request rates. Failed runs may be repeated, if time allows.

Box requirements

Here are some of requirements that a cache must meet:

• A cache must be fillable within 8 hours.

• The cache under test must meet minimum requirements for actually caching responses. The DataComm-1 workload offers a maximum document hit ratio of 55%. A valid run must achieve at least a 25% document hit ratio, as computed over the entire measurement phase of the run. A run which does not achieve 25% is disqualified, and the vendor may select another (presumably lower) request rate if time allows.

Reporting

The following metrics may be reported in the Data Comm article:

• request rates specified by the vendor,
• success or failure of a run;
  if a run initially fails and later succeeds (at the same request rate), the failed run is not reported,
• response time mean and median values, averaged at no less than one-hour intervals,
• document-hit-ratio and byte-hit-ratio values, averaged at no less than one-hour intervals,
• percentage of transaction errors.

Regulations

The following rules are enforced:

• Once testing has begun (i.e., the no-proxy test), the vendor is not allowed to make any changes in the Box configuration or software versions.

• In the event of hardware failures, vendors will be allowed to swap in new equipment and continue testing, strictly on a time-permitting basis. In the case of a hardware failure where time constraints prevent re-testing, this will be noted in the article. Vendors are strongly encouraged to bring spare interfaces, chassis, disks, etc., as a contingency for any hardware failure.

• Participants will receive all test results and have a chance to comment on their results before publication.

• Participants are not permitted to withdraw or suppress results once testing begins.

Schedule

Each participant gets a three day slot. The first day is intended for setup, preliminary tests, and filling the cache. The third day is for re-runs and packing. The end of one slot may overlap with the beginning of the next slot.

Working hours at the PolyLab are from 9am to 6pm.

P.S. This is not not a cache-off. The second IRCache cache-off will be held in January 2000.