2.5.4 -> 2.5.5 20010501 - updated copyright/license statements - added licensing terms acknowledgment to ./configure script - resurrected support for user-defined or "table-based" distributions; e.g., ``SimpleContent.size = table("/tmp/t.distr", "size");'' where /tmp/t.distr contains the distribution of type size_distr - added webaxe-3.pg configuration file for WebAxe workload; this file was distributed with Polygraph version 2.5.4 under the name webaxe-2.5.pg - Red Hat 7.0 compatibility fixes - AIX compatibility fixes - fixes for platforms with unsigned chars (e.g. AIX); PGL parser would quit on those platforms with error location pointing to the end of a file being parsed - fixed an old ltrace bug: sampling windows were sometimes getting huge, leading to excessive memory consumption and possibly somewhat incorrect trace plots - fixed local time detection bug on Solaris; the bug caused Polygraph to complain about client/server clocks being out of sync even if both processes were running on the same host - replaced "MB/sec" column heading produced by make_report with less confusing Mbit/sec 2.5.3 -> 2.5.4 20000724 - added "special_req_factor_beg" and "special_req_factor_end" fields to Phase specs to control the presence of special requests (IMS, Reload, etc.) in the generated stream (PGL) - identify cases when HTTP race conditions lead to connectivity errors on used-to-idle persistent connections; retry transactions when such errors occur; log the number of retried transactions (that otherwise do not affect statistics) - eliminated most of the special (non-Basic) requests during the fill to shorten the fill phase (PolyMix-3) - when a phase meets its goal, wait until all remote phases are ready before proceeding - added the "--sync_phases " option to disable auto phase synchronization if needed - log statistics about oid generation algorithm - distinguish "hot" requests in stats samples - polished stats samples and made them smaller - report WSS in terms of bytes (in addition to object count); be careful when interpreting the values -- adjust for embedded objects and other hosts in the bench - complain if server agent port is not specified - upgraded current and required log versions from 2 to 3; log versions were not upgraded for a while despite format changes, unfortunately - warn about log extractor <--> log producer versions mismatch when reading a log - fixed typo in X-Srv-Sets header parsing code that led to growing WSS for public (shared) objects, resulting in lower-than-expected HR - fixed ./CdbBodyIter.cc:43: assertion failed: '!theInjector' - fixed coredumps when all configured phases are ended and Polygraph adds a "cold" phase for the last transactions to finish and coredumpts with segmentation fault - made reportgen.cfg dependent on ./configure parameters 2.5.2 -> 2.5.3 20000719 - ProxyCacheSize in polymix-3.pg should now be set to the *total* proxy cache size, without adjusting for the number of polyclt processes; all adjustments are now done in polymix-3-guts.pg, without user participation (the patch with the above functionality was posted shortly after 2.5.2 was released) - added the clientHostCount() function to compute the number of client hosts based on the PolyMix-3 rules (PGL) - added client_host_count to benchPolyMix3; if not defined by the user, its value will be computed on-the-fly in polymix-3-guts.pg: TheBench.client_host_count will be set to clientHostCount(TheBench) - added the "fexit" phase to PolyMix-3; fexit increases hit ratio to peak level while decreasing request rate; this phase may be required for proxies that get overwhelmed by high loads with low hit ratio; the latter were possible during "inc1" when hit ratio was increased along with request rate, but not "fast enough". - adjust Robot's recurrence ratio for Reload requests (we are still unsure what to do with IMSes) - collect small transaction sample during the fill phase (PolyMix-3) - polypxy can now handle misses (not yet robust in error-rich environments) - a Robot that did not request any private objects from a server before WSS was frozen should try not to request new private oids from that server; doing otherwise populates the working set with oids that cannot be requested again in the future (because the corresponding WSS is zero) but may consume cache space, etc. Generating a new private oid is always the last resort, but it should not be the second best choice. - hot set objects may have been sometimes chosen outside of the working set, leading to false false hits and lower hit ratio; other objects may have been subjected to this as well - request embedded objects during the initial server scan (not requesting them makes Polygraph think that an embedded object was requested before when it was not, decreasing HR) - made polymon report Network Bandwidth usage again (untested) - moved foreign request error reporting so that we still have request headers to dump - marking a freshly accepted socket as a non-blocking or not-naggling socket may fail when poly* is overloaded; do not exit the program but rather continue after reporting the error - fixed two coredump-inducing bugs in the polypxy module - aka was exiting silently when no "anonymous" options where specified - support the "log_dir" option to label_results 2.5.1 -> 2.5.2 20000713 - accept more than one alias range (aka); useful for running robots and servers on the same machine (but do not do that for production tests!) - do not attempt to extract the broadcast address of the interface, we do not need it and some interfaces (e.g., lo0 on FreeBSD) may not have one (aka) - added tools/pmix3-ips.pl script which is identical to pmix2-ips.pl except that it also prints IP ranges for each host in the bench (in default, verbose mode) - added IMS/reload statistics collection and reporting IMS and reload transactions are now treated separately from "basic" transactions and do not affect hit ratios, fill rates and other statistics that is hard to define for IMS and reloads (e.g., what IMS response can be count as a hit?); The "Contribution" table in generated reports needs more work as it is not 100% in sync with the above change - added working_set_cap() function to freeze WSS based on the number of transactions rather than run time (PGL) - added min() and max() functions to PGL - split polymix-3.pg into [modifiable] polymix-3.pg and [read-only] include/polymix-3-guts.pg - specify WSS in terms of fill responses rather than time (the calculation is still time-based, just like in Polymix-2); WSS is meant to be equal to 4 hours of fill traffic during a "top" phase, approximately (polymix-3.pg) - specify ramp durations as a fraction of plateau durations to ease scaling of a test (polymix-3.pg) - added "wait_wss_freeze" boolean field to PGL's Phase; if set, the phase will finish only if the working set size is frozen (and all other goals are satisfied, of course) - make sure that fill phase stops only if working set is frozen - do not request embedded objects while in the initial server-scan mode - shortened launch window because we no longer request embedded objects - decreased warmup phase goal to a single launch window - detect malformed HTTP request or response lines and complain - check that embedded object type makes sense and complain if it does not (instead of dumping core) - Polygraph was erroneously reporting "hit on reload requests" for some embedded objects - removed "http." prefix from some lx object names - report the number of local private objects in the working set size - obey RFC 2616: do not send any extension-headers with 304s - Zip files should have ".zip" extension, not "zip" (contents.pg) - yet another attempt to make oid exchange algorithm more robust (needs more work) - increased new_oids_history default to from 1024 to 2*1024 - increased new_oids_prefetch default to from 128 to 256 2.5.0 -> 2.5.1 20000626 - added support for realistic HTML simulation - added aka support for IP alias manipulation on systems that use virtual interfaces such as eth0:1, including Solaris (untested) and Linux (tested) - polished the code that searches for local interface addresses in hope to make it work on more OSes - added robotAddrs() and serverAddrs() functions: PGL now knows how to compute PolyMix-2,3 IP addresses, given the peak request rate (and other parameters); make use of that feature so that people do not have to spell out IP addresses in polymix-3.pg; explicit addresses are still supported, of course; IP aliases still need to be created before the test - added "Bench" type to describe parameters for the IP allocation schemes and include/benches.pg to describe standard IP allocation schemes - support variable recurrence ratio: replaced "recur_factor" in Robot with recur_factor_beg / recur_factor_end pair for smooth change in recurrence ratio (PGL) - polished definitions of fill-related phases to make use of variable recurrence in PolyMix-3 - polished launch window calculations in PolyMix-3 (needs more work) - added the "--ign_bad_cont_tags" option to ignore content tags that a Robot cannot parse; useful with realistic HTML simulation - added the "--ign_urls" option to generate content regardless of the URL being requested; useful when running polysrv in a "demo" mode - fixed logging of ICP statistics - added reporting of ICP measurements to the Report Generator; the above reporting is enabled only when ICP stats are present in the logs - make sure IMS timestamp is always at least 00:00:00 UTC, January 1, 1970 - fixed type unification compile-time bugs - Solaris portability fixes (IP aliasing improvements are untested due to the lack of root access to a Solaris box) - "everything expires when modified" should use "nmt" not "lmt" in olcStatic in contents.pg - replaced pop_model settings to make old workloads "compile" with new PopModel type (PGL) - support number/rate operation (returns time) (PGL) 2.2.8 -> 2.2.9 20000113 - added downtime-2.pg workload specs for the "power-off" test - when transaction fails, it may take a while for polysrv to stop waiting on the corresponding object id; the latter may lead to a situation when hits are generated only using oids that are smaller than the lost oid, preventing WSS algorithm from "sliding"; we now attempt to report "lost" oids back to the server so it does not wait for them; the latter may improve hit ratio for "small" caches in the presence of errors - freeze WSS based on last object id sent to the server rather than last object id confirmed by the server (to prevent funny tiny working sets in the presence of errors when oids may get lost) - groups with working set size of zero were not handled correctly; a group has zero size if it was never used prior to the working set freezing time - on the client side, distinguish closed connection with no headers sent from a closed connection with some headers sent; that is, distinguish "connection closed before sending headers" from "premature end of msg headers"; the former may by due to HTTP race condition, the latter is definitely a bug. - added "Errors Total Count x 1000" screen to polymon - assume BB/*.pm files are in /usr/local/polygraph/BB or in the current dir - polished bb.pl output - added tools/pmix2-ips.pl and tool/BB/forsome scripts that help in configuring and running [large scale] experiments - removed most defaults values for options of Report Generator tools; reportgen.cfg should be used instead; needs more work - allow up to 0.1% difference between fill count measured and computed based on hit and cachability ratios (Report Generator) - do not complain about too many waiting requests because it is not clear what "too many" is (Report Generator) - plot cachability ratio along with hit ratios (Report Generator) - ported Report Generator to Solaris (the patch was available for 2.2.8) - do not dump core if a requested phase is not found in the log (lx) - do not quit on logs with [very] different absolute times (ltrace) - made log merging algorithm less susceptible to minor local time differences between machines where the logs where generated (lx, ltrace); "level" traces (e.g., those produced by Report Generator) no longer have weird slight downward slopes - statistics about "waiting" (queued for connections) transactions was not collected appropriately resulting in zero readings - fixed "request rate for robot X is not positive" configuration time bug (the patch was available for 2.2.8) - compile IpSocket.cc even when IPFW is not supported (the patch was available for 2.2.8) - prevent "invalid type X for default argument to Y" compilation errors 2.2.7 -> 2.2.8 19991227 - added "req_inter_arrival" field to Robot to specify other than Poisson inter-arrival time distributions; for example, downtime test will use constant distribution (PGL) - redesigned lx to concentrate on phase-related statistics and support multiple log files; phases with equal names are merged, and phases with different names are concatenated to produce correct "totals" - added `ltrace', a tool to generate traces based on binary logs; this functionality was moved from `lx' that no longer supports trace extraction - changed naming scheme for log objects to be more consistent and scalable, added a few new objects - log statistics about transactions waiting for resources to become available (e.g., requests waiting for a robot connection pool to allocate a connection); this changed log format, unfortunately - dump more information on "foreign content " and "malformed content " errors - added a cool "summary" screen to polymon - do not quit when all phases met their goals when there is a positive idle timeout; wait for that timeout to expire (so that servers do not quit while clients are running) - do not log ascii representation of addresses when storing current state - "level" statistics was not maintained properly between phases - "fill" stream was measured as "miss" stream instead of "cachable miss"; polyfill-2.pg was not affected because all polyfill objects were cachable - gettimeofday(2) system call is broken on FreeBSD; make sure internal Polygraph clocks never go backwards and show sane values - avoid infinite recursion coredumps when we fail to write a log entry; report and try to salvage logging errors instead - do not dump core if a phase has no name, use "" name instead 2.2.6 -> 2.2.7 19991215 - added "--time_scale" option to lx; positive values make lx to print relative [to the run start] time for traces, scaled by the specified amount - added "Traces" macro to represent all "traceable" objects in lx - added rptm_mean:hit and rptm_mean:miss to lx's All micro - only redirect standard output when --out option is used (lx); used to redirect error streams as well - try to skip corrupted entries in [otherwise valid] binary logs - implementation of lognormal distribution used incorrect standard deviation leading to occasional huge objects and smaller-than-configured means - changed content sizes for cntOther and cntDownload after fixing the lognormal distribution bug to get the right mean object size - removed percents from Server.direct_access specs; those values are ignored by Polygraph because servers calculate them from scratch to match "global" content distribution - the sum of squared values was not computed correctly leading to incorrect readings for standard and relative deviations for samples with sampled values larger than 46341 2.2.5 -> 2.2.6 19991207 - PolyMix-2 workload now uses 0.4/sec per-robot request rate; use cloning or IP aliases to get more robots (and hence higher request rates) - added "rep_sz_mean" object to lx - added polyfill-2.pg to workloads/ - added pmix2 blob to tools/BB/bb.pl and polished bb.pl's comments - when using --dump, dump transaction meta-info even if there is no message data to dump - recognize KB scale for command line size options - two meant-to-be-independent random number generators used for simulating object sizes got "in-sync", producing wrong size distributions - phases with decreasing request rate were dropping request rate too fast - public_interest for PolyMix-1 and DataComm-1 workloads should be set to 0%, not 50%. - added now-required hit ratio and popularity model specs to datacomm-1.pg - avoid division by zero on corrupted log files 2.2.4 -> 2.2.5 19991129 - added "idle_pconn_tout" field to Agent to specify the maximum time an idle (no pending transactions) persistent connection should remain open (PGL) - support phase-dependent recurrence ratio; useful for populating the cache (fast, with only a few hits) and then doing measurement phase with appropriate hit ratio, all in one run - added "recur_factor" field to Phase (PGL) - added support for priority scheduling: Polygraph runs (N-1) file scans for file descriptors (connections) that require urgent action and then one scan for all file descriptors; N is configurable using the --priority_sched option - added "--priority_sched" option to control priority level for somewhat urgent socket I/O operations; giving higher priority to some operations may improve performance in environments with lots of probably idle connections as Polygraph will spend less time processing those connections; "over-optimized" setting are likely to cause noticeable increase in response time; the default priority is 5; to disable the optimization, set to 1 - added support for assigning the same value to several variables with one assignment operator (PGL): [ a1, a2, a3 ] = 100%; - added support for randomly distributing values of an array among several arrays, each with an optional "portion" quantifier; for example: [ arr1 : 60%, arr2 : 20%, arr3 ] = '10.0.1.1-255'; this feature is very convenient when distributing network addresses among several objects (e.g., servers or pipes) (PGL) - grow storage area for new oids when clients start to run out of them (to make prefetching more aggressive) - optimized initial server scan (when robots are "touching" all servers to make sure robots can later generate hits on every server); the scan is now much faster and has more random access pattern - added reporting of server scan progress - "lock" first phase until all robots are ready to hit all servers - when an advertised oid has not been requested for a while, see if we should grow the bitmap to store more oids (and hence keep an oid for a longer time) instead of reclaiming that stuck oid - changed default for "--new_oids_prefetch" from 256 to 128 because Polygraph is not capable of dynamically adjusting prefetch amount - if load factor reaches zero, stop submitting requests - check for late alarms and other timers and report if we are getting behind the schedule - server now accepts new connections for at most 10msec at a time - do not quit when running out of file descriptors in accept() call - destroy PGL objects after use to reduce memory footprint for large scale configurations - use idle_pconn_tout of 15 seconds for PolyMix-2 servers - changed Robot.open_conn_lmt to 4 in polymix-2.pg - polished phase definitions in polymix-2.pg - check for error status when deleting rules and pipes (piper) - re-ordered screens in polymon - resurrected err_cnt_tbl in lx - the "--phase" option was not supported for non-traceable objects (lx) - fixed introduced in 2.2.4 typo that made servers "blocking" and hence unusable (they would get stuck in the accept(2) call); client-side sockets were also affected but that was not visible to an end-user because blocking robots may still work; the patch was posted to the Polygraph list - portability star, ncurses, #defines timeout causing namespace havoc - bad guys #defined `select' in Solaris system files causing namespace havoc - I/O buffer pool was not reporting memory growth correctly (err.. at all) - prevent coredumps when printing a histogram with max value less than 1 (lx) - True64 requires a special setsysinfo(2) call to raise FD limit beyond the default hard limit of 4K 2.2.3 -> 2.2.4 19991109 - added "piper", a program to configure DummyNet network pipes based on PGL - added 'bwidth' -- a basic PGL type to represent [network] bandwidth small set of operations are supported for now - added NetPipe PGL type to describe network pipes; a pipe specifies maximum network bandwidth, packet delay and loss, and other low -level network parameters/conditions - added pipes.pg with definitions of commonly used network pipes, including modems and various kinds of origin servers - added configure-time check for DummyNet and IPFW interfaces - reduced start-up time in configurations with large number of robots and servers; reduced memory footprint for large configurations (more work may be needed) - print aggregate statistics (min/mean/max/etc) for histograms produced by lx and distr_test - report agent creation progress every 5 seconds rather than every 50 agents - report system resource usage before and after run - adjusted the way stat intervals (i- lines on the console) are printed in a blocking or overloaded environment - increase the maximum number of FD ./configure script can detect to 2^16 (65K) - fixed configuration bug in environments with no rlim_t type - include limits.h file to avoid compilation errors on Solaris - do not print "array probabilities (above) add up to .. less than 100%" if they add up to more than 99% - cntDownload in contents.pg was named cntDowload; some found it funny - lx could not handle logs with combination of duration-based and non-duration based phases - some environments do not know better than #define getc and putc macros causing namespace havoc - rptm_hist was printed twice by lx --objects All - a typo caused size type to do multiplication instead of division (PGL) 2.2.2 -> 2.2.3 19991026 - added "--sample_log" and "--sample_log_size" options to configure sample log; sample log defaults to the general purpose log; if separate log is specified, it is not flushed to disk on periodic basis as general log (to avoid potential run-time overhead) - added "StatsSample" object to PGL to describe the time and size of a statistical sample; Samples must be scheduled(), just like Phases; more work needed to make scheduling more convenient (phase related?) - added "kind" field to label Content object (PGL) - added "cntDowload" content to represent relatively large downloads; use cntDowload in PolyMix-2 - disabled sharing of embedded objects among containers; with no sharing, we have to rely on user specified content configuration with no tools to "tune" the model internally -- resulting (actual) global content distribution may not match the planned one; a likely global distribution is reported so a user can check their specs without running an experiment - disabled client side cache in PolyMix-2 - set embedded recurrence to 100% in PolyMix-2 - added initial support to save experiment "state"; eventually we will use state information to resume experiments from a given point and also for stats analysis; needs more work - added support for collecting low-level statistics in the form of samples; sample collection is controlled using PGL "StatsSample" type and "schedule()" call - for large configurations, inform about the progress in creating agents at startup - prevent int overflows when generating very large files - several bug fixes related to size constants (PGL) - try to handle "unlimited" resources reported by getrlimit(2) correctly 2.2.1 -> 2.2.2 19991020 - added "cache_cap" field to Robot (PGL); the cache is currently used for embedded objects only; it prevents robots from requesting the same embedded objects many times, emulating browser behavior - added "conn_lvl_mean" lx object to report average number of opened concurrent connections; because of Polygraph does not update time on every open() and close(), this statistics may not be accurate - robots now scan all servers (in a semi-random manner) before proceeding with truly random accesses; the initial scan ensures that all connectivity problems are detected early and also helps reach stable HR earlier - inform user when all robots are ready to generate hits on all servers - no "wait_xact_lmt" by default, a robot can queue as many transactions as needed; this is potentially dangerous because a user does not see those queued transactions (yet) and may think that everything is OK; more work is needed - generate new oids only when we know that the xaction will not be queued; this rule prevents robots from thinking an oid has already been seen by a proxy while it was not (because it got queued); the latter used to decreases offered hit ratio - agents should share random number generators to avoid a hard-to-detect artificial condition when all agents are "doing the same" all the time (e.g., all robots send requests to the same subset of servers for a while) - if no new alias is specified, just delete old ones (aka) - make sure that we always #include sys/types.h before inet.h because old BSD systems may not include sys/types.h in inet.h, and they should - removed --out option from aka - fixed a bug that led to an occasional assertion if the FD limit is reached on server side - fixed bugs leading to occasional segv and assertions when poll(2) system call is used - fixed parsing of the --dump option; now all tag combinations are parsed correctly - request rate field of a Robot was misinterpreted unless N/sec format was used (PGL) - due to ioctl(2) interface differences, aka does not work on Linux and probably some other OSes; more work is needed - fixed IP range parsing bug; some valid ranges were declared malformed - logging modifications; binary logs are not compatible with previous versions (again) - updated polymix-2.pg and contents.pg; changed content weights to ensure that an "average" object has 80% chance of being cachable 2.2.0 -> 2.2.1 19991012 - added support for IP ranges using a-b.c-d.e-f.g-h notation, for example '10.0.1-2.50-99' constant specifies an array of 100 addresses; the old range operator (A .. B) is still supported but is less flexible; subnet and/or port specification is also allowed - added "aka", an ingeniously named tool to create [large] number of IP aliases for a given interface; aka accepts IP specs in the PGL format (a-b.c-d.e-f.g-h/subnet) which allows for specifying hundreds of aliases in a compact form; aka will _delete_ all aliases for the interface before setting new ones; aka requires superuser access, of course - added "new_oids_per_msg_max", "new_oids_prefetch", and "new_oids_history" options to control synchronization between robots and servers; may be useful if you get many oid-related errors in large scale experiments and/or when a proxy gets too slow or drops requests - added "open_conn_lmt" and "wait_xact_lmt" fields to Robot (PGL) to limit the number of concurrent connections a single Robot may have at a time, emulating browser behavior (the code was written for version 2.2.0, but lacked PGL hook-ups and was not tested) - removed place() function call and replaced 'host' field in Agent with 'hosts' field to reduce memory footprint when configuring lots of agents on a single machine; place() call used to clone lots of agent specs that would differ only in 'host' field setting; we now specify all the hosts directly (in 'hosts' field) bypassing the cloning step (PGL) - added "conn_close_act_use_hist" and "conn_close_act_ttl_hist" lx objects to filter stats for "actively" closed connections only (i.e., all connections except those that were closed in an "idle" state) - agents where not binding to their addresses when they were supposed to - robots used to maintain one private object space as if there was only one origin server; the bug led to lower than expected hit ratios when robot's public_interest was less than 100% - robots were not setting server world id (wid) for embedded requests - statistics for idle persistent connections closed by Polygraph was not maintained - synchronized workload specs 2.1.0 -> 2.2.0 19990928 - added support for embedded objects: a server populates content with special tags that point to embedded objects (based on PGL Content specs); a robot parses server responses for those tags; a robot requests embedded objects as it detects them; we try to keep average request rate at the specified level (if any) despite these "extra" requests requests for embedded objects do cause bursts in traffic, just like in the real worlds - added may_contain and embedded_obj_cnt fields to Content type (PGL) - added "embed_recur" ratio to Robot; the ratio specifies the portion of embedded objects in a container that are re-requested, on average - added "choice_space" field to Content type (PGL); Choice space specifies the number of oids considered when selecting next embedded oid for a container oid: embedded_oid = uniform(k*container_oid, k*container_oid + choice_space) - added "direct_access" field to Server (PGL); direct_access selector specifies (a) objects that can be accessed not as an embedded part of other object and (b) relative frequency of such an accesses (e.g. 1% of all direct accesses are images and 95% of all direct accesses are HTML pages) - added tools/msl_test tool that attempts to estimate the maximum time a proxy keeps a TCP connection in a TIME_WAIT state. (Glenn Chisholm ) - added rng_test to test the quality of various random number generators that Polygraph is using - added the "--dump" option to control what type of messages and what message parts must be dumped to console. The option accepts a list of flags; the old --prn_* interface was not flexible enough or required too many options; new interface needs more work - renamed "--prn_dump_size" to "--dump_size" - removed "--prn_reqs", "--prn_reps", and "--prn_errs" options - robots must bind(2) to the IP addresses they are running on except when all robots are running on the same address - report approximate time a server waited for an advertised oid to be requested - always randomize IO buffers before use - use agent id along with object id when seeding r.n.g; this prevents skew in pseudo-random numbers when many agents are dealing with the same small subset of oids - use one port manager for all addresses to reduce memory footprint for big scale runs - install SIG_INT (^C) handler _after_ polyclt/srv has configured itself; makes possible to kill a program if configuration takes forever - workload files were missing server side delays (Becky Larsen ) - polymix-2.pg file comments were saying that it is a PolyMix-1 file - fixed typo in [client] "closed conn w/o sending data" error message - polymon.cc::TheSelHostCnt was not defined properly, causing compilation warnings (Ron Jones ) - udp2tcpd.cc was using `int' instead of `socklen_t' causing compilation errors on Linux and others OSes that require socklen_t (Ron Jones ) - a kludge to prevent mysterious (compiler?) bug causing segmentation faults in RecSym::getDistr on some platforms 2.0.0 -> 2.1.0 19990917 - added pop_test program that simulates an LRU-based cache and can be used to estimate [memory] hit ratios depending on popularity model and other related workload parameters - added "pop_model" field to Robot; two popularity models are supported: pmUnif() and pmZipf(skew_factor). - added "public_interest", "recurrence", and "unique_urls" fields to Robot - added working_set_length() procedure call to PGL; use it to limit working set size and hence prevent hit ratio decline during long tests - added a "--prn_dump_size" option to limit the size of HTTP message dumps - added a "--prn_errs" option to print HTTP messages that caused xaction errors - recognize 'All' macro in --phase option (lx) - removed "--unique_urls" option (polyclt) -- it is now Robot's field - removed unused `world' options (polyclt): world_type, world_id, world_urls, unique_urls, order, recur, pop_model, tmp_loc, tmp_loc_delta, tmp_loc_depth, exp_oids, prefilled_cnt; some of these option were already implemented via PGL, and some wait till be implemented/ported/tuned - removed the "dhr" field from Robot specs (the field was ignored anyway) - share URL space among all robots and servers; robots may have a "private" URL subset while "public" subset is shared by all; servers send robots information about the current global state; current limitation is that all polyclt processes should emit request streams with similar rate (per robot differences are OK), otherwise the servers start complaining that some clients are not doing their share in creating the load; solving the latter problem would require more heavy negotiations among clients and servers - make IO buffer content "more random" than it used to be; servers now produce Web objects filled with more random content. Note that servers still may produce different content for the same URL, even if the headers imply that the object has not changed. The latter is probably OK under "normal testing conditions" though. - do NOT make relative URIs artificially long by prepending them with an '/http:/ip-address' prefix; transparent proxies usually receive relative URIs so they do have and advantage of handling shorter request-lines, and we probably should simulate that - HTTP/1.0 connections are non-persistent by default, no need to send *Connection: close header. - write all 'unique ids' (e.g., world id) using hex notation and fixed length fields - write oids in URLs using hex notation and fixed length fields - console now displays minutes since the start of an experiment rather than absolute time; absolute time is logged and is available to lx - when building executables, use CXXFLAGS instead of CFLAGS in Makefiles - "conn_ttl_vs_use" was erroneously named "conn_ttl_vs_sz" - fixed parsing bug that caused assertions on a "''" expression - offered (ideal) hit ratio was not recorded properly 1.3.2 -> 2.0.0 19990907 - new copyright file header and polished licensing terms - added support for PGL configuration language; most experiment parameters are now set via a configuration file - "--pop_model" option defaults to "unif"orm now (used to be "zipf") - added the "--idle_tout" option to the client side; "idle" in "--idle_tout" now means "no network I/O activity" -- Polygraph may still be trying to do something - added the "--stats_cycle" option to specify the duration of a cycle that collects and logs "cheap" stats (defaults to 5 seconds) - added "fill size" experiment goal - removed accept_lmt, rep_sz, obj_bday, obj_life_cycle, obj_life_cycle_var, obj_with_lmt, obj_expire, cool_phase, goal, launch_win, rep_cachable, maximize_req_rate, and perhaps other options; similar functionality is usually provided in PGL - renamed "--dhr" to "--recur" (recurrence ratio or how often a URL is revisited by a robot); in Polygraph 2.0.0 the actual DHR is recurrence/cachability ratio; DHR may be affected by popularity model and other factors; we had to rename "--dhr" because Polygraph can no longer simulate any given DHR independent from other config parameters -- the models are becoming too complex; this is still work in progress and changes are likely - removed `ephemeral' string as a valid port range in --ports - the "--file_scan" option now defaults to poll if poll(2) is supported; Polygraph used to call select(2) by default - added conn_close_frgn_use_hist,conn_close_frgn_ttl_hist, conn_close_lcl_use_hist,conn_close_lcl_ttl_hist, soread_sz_hist, sowrite_sz_hist, and phase_name objects to lx - made 'All' macro a default "object" to be extracted by lx - honor --phase option in trace mode of lx - changed binary log format; new format is not backwards compatible - log errno strings along with errnos; makes error logging portable across OSes - modified logging procedure to flush binary log data periodically - removed "X-ETag" header field - send Proxy-Connection: header when talking to a proxy (used to send Connection: which was a terrible idea and was not honored by some proxies) - added support for Content-Type header field in responses - added support for arbitrary number of user-defined simulation "phases" - cachability is now determined on the server side (again!) - added ./configure script to preconfigure the distribution before `make' - added INSTALL file with generic package installation instructions - added root-level Makefile - added distclean goal to Makefiles to remove most auto-generated files - changed Perl interpreter name in scripts to '/usr/bin/perl'; ideally we need to determine the location at ./configure time - explicitly require Perl version 5.003 or higher in Perl scripts - added DataComm-1 specs to bb.pl - added workloads directory with standard workloads specs written in PGL. - added polymix-1.pg, polymix-2.pg, and datacomm-1.pg workload specs - added simple.pg, a minimalistic workload to use as a "Hello World" example - complain about "uncachable hits" - ignore "false hits" by default - removed hard coded limits on the reply content size; the change may have performance effect for workloads with zipf popularity model - removed 'loop' tool until it gets fixed - applied some HPUX compatibility patches (submitted by James Murphy ) - 'max' values printed by distr_test was greater than real max by one - lots of other additions, modifications, and bug fixes 1.3.1 -> 1.3.2 19990830 - this minor release incorporates all previously released patches for 1.3.1; the pathces fix a few logging bugs and, to the best of our knowledge, do not affect Polygraph run-time performance - replaced the "--prn_trace" lx switch with the "--trace" option; now trace stats can be averaged at a specified time interval; small window sizes emulate old behavior - added "conn_use_mean" object to lx - added datacomm-1 workload specs to bb.pl script - increment error count on each phase/log - fixed generation of class-based object names - fixed lx coredump when extracting class-based stats like class_rptm_mean:hit 1.3.0 -> 1.3.1 19990616 - added the "--notify" option that enables notification messages to be sent to a monitoring program; messages (currently less than 100 bytes) are sent via UDP every 20 seconds. - added "--label" option to specify a [short] run description; the label gets included into notification messages if any. - added udp2tcpd, a daemon that runs on port 18256 (default), listens to notification messages (UDP), and forwards them to polymon(s) via TCP; the intention is to support multiple monitoring programs for the same notification stream without using IP multicast or broadcast - added polymon; a ncurses-based program that monitors run-time status of Polygraph experiments using udp2tcpd daemon - added the err_cnt_tbl option to lx to print error table; note that Unix error codes are not standardized so you may get different descriptions of error messages if you move binary logs around (this is a bug) - fixed "invalid port" assertion in ExpPortMgr - fixed logging of error table (it probably was not logged before) - fixed compile-time warnings reported on Linux - fixed compile-time poll(2) errors on Linux - fixed a few compile-time problems on Solaris - maybe improved timegm() port to True64 and other environments with tm_gmtoff - removed custom polyVersion dependency from Makefile; it caused too many re-compilations and we never used the $CompileTime$ feature anyway - polished comments in Makefiles 1.2.2 -> 1.3.0 19990608 - added poll(2) support; users must add -DLACK_POLL to DEFINES in src/xstd/Makefile in the environments without poll(2); - added the "--file_scan" option to select between poll(2) and select(2); the default is still select, but that is likely to change - added the "--maximize_req_rate" option that enables "maximum request rate" searching mode; must be used together with the "--req_rate" option which specified initial rate; search is done by incrementing or decrementing actual request rate depending on the reply rate; support is rudimental, more work needed - added the "--prn_rep_cnt" option to lx; similar to --prn_time; prepends trace lines with number of replies received since log start - added response time statistics for major request classes: hits, misses, cachable, uncachable, etc. - added several lx objects to extract new stats: conn_use_hist, conn_ttl_vs_use, resp_tm_vs_class, class_rptm_mean - added stats for persistent connections (xactions per connection) - "rep_rate" and "req_rate" objects are now traceable via lx - log errors "histogram" - scan the port range and mark used ports at start-up if port manager is enabled via the "--ports" option; the scan adds about 2.5 seconds to polyclt start but avoids some of the run-time port binding errors - improved random number generation; do not use object ids as r.n.g seeds directly; generate a "good" "uncorrelated" random value and use that instead; prevents "loops" in reply size generation and other oid-dependent models; the loops were visible only on some micro-level workloads though; the fix adds a few seconds to polyclt/srv start time and about 4MB to memory footprint - adjusted histogram dumps so that max column in the output corresponds to max rather than max+1 - avoid floating point exceptions when reading bogus log data - improved handling of incomplete or old binary logs in lx - fixed FP exception error due to wrong double->int cast (a patch was available for 1.2.2) - use "linux" #define to prevent "FD_SETSIZE already defined" warnings on Linux (one can also add -DLINUX in Makefiles) 1.2.1 -> 1.2.2 19990521 - added distr_test tool that can print a histogram for any distribution recognized by Polygraph - added the "--prn_trace" lx option to print trace stats rather than totals; not all object values can be computed in the middle of a log; those that can, will be printed; others will be silently ignored; log files are not joined when trace mode is on - added the "--prn_time" lx option to show log time as a first column of the output (useful together with --prn_trace) - added support for accumulating large sizes (over 2GB); yet untested; old counters used to overflow leading to bogus byte hit ratios; current counters can hold up to about 4 exabytes which should be enough for a while - timegm is missing on Solaris; Solaris users should add -DLACK_TIMEGM to DEFINES in xstd/Makefile to enable a workaround - several bug fixes in Histogram module; trace stats (collected at 90 second intervals) that were based on histograms (e.g., response time percentiles) where not recorded correctly; aggregate stats (e.g., means) and stats averaged over entire phase were not affected - Zipf distribution was not generating the value of '1' when used for persistent connection limits; other zipf fixes - fixed "ambiguous overload for `bool ? Size : int'" compilation errors - zipf distribution now reports "world size" parameter only to be consistent with the way zipf is specified on the command line (used to print alpha=1 as a first parameter) - other minor bug fixes 1.2.0 -> 1.2.1 19990513 - added the "--obj_bday" option to specify object creation times; used to be hard-coded to a randomly (per object) chosen day in the 70s; negative birthdays are relative to program start time; non-negative birthdays are relative to 0, UTC; - accept "zipf" distribution on command line - time parsing routines now accept "year" scale (e.g., const:1.5year) - report Polygraph version in the configuration dump - internal lmt and expires time must be rounded using second resolution for HTTP date comparisons to work correctly - polished last-modified-time calculations to better handle lmts that are close to current time - renamed XSTD_INCLUDE to LIB_INCLUDE in xstd/Makefile 1.1.0 -> 1.2.0 19990501 - added Object Life Cycle model to simulate object modification and expiration; the model is controlled by the following server side options: --obj_life_cycle, --obj_life_cycle_var, --obj_with_lmt, and --obj_expire - support If-Modified-Since requests on server side - Poly-server now adds the Date: header field to all replies - removed "max-age" option from cachable replies because it had priority over the more common Expires: header. As a side effect not all cachable replies have expiration information - Last-Modified header field is now added to replies only if --obj_life_cycle option is enabled; some proxies will not cache replies without a Last-Modified header field -- watch out! - Expires header fields are not added to cachable replies by default; see --obj_expire option to control Expires fields - reordered HTTP reply header fields to follow HTTP recommendations - always use the minimum of FD_SETSIZE and getrlimit(RLIMIT_NOFILE); warn if FD_SETSIZE is smaller; we got tired of coredumps when a tester would forget to increase FD_SETSIZE after increasing kernel limits - date manipulation functions now use "timegm()" call which may not be portable; portability problems will be fixed on-demand - distinguish between "premature end of msg body" and "premature end of msg headers"; useful to diagnose proxy connection resets - detect premature end of headers on client side (we probably would wait forever prior to this fix) - reorganized and optimized HTTP header parsing; parsing speed is increasingly important as we have to recognize more header fields; current improvement (on 2.5 fields) is negligible (about 3%). - fixed a bug with HTTP header parsing (a patch was available for 1.1.0) 1.0p7 -> 1.1p0 19990425 - added a "--version" option to print package version and (c) info - ident-like routines may extract version info from Poly binaries now - added a "--pconn_use_lmt" option to control the number of transactions executed over a single [persistent] connection; Poly generates (and understands) Connection: close/keep-alive headers, and will honor Proxy-Connection: header as well; mutli-token Connection: headers are not supported; the option makes sense on both client and server sides; TCP may need "--nagle off" to utilize the advantage of pconns - added support for user-defined (tabulated) distributions; "tab:filename" or "tab(filename)" can be used wherever a distribution is expected on the command line - changed the default of the "--origin" option to "127.0.0.1:8080" - the "--proxy" option has no default now and implies proxied connection if present; with no "--proxy" option, a direct connection is assumed - removed the "--abs_urls" option; absolute (relative) urls are now generated only for proxied (direct) traffic - added libs target to Makefile to ease re-build of libraries - make will now clean src/xstd directory when making clean in src - the "--rng_seed" option is now honored by all rnd number generators - complain if units (KB, sec, etc.) are specified where none expected - added reporting of source code location in error messages - check for connect errors during first IO on a socket; Poly 1.0 would do an I/O and only then check for an error leading to imprecise error diagnostics - set socket options for accept(2)-ed sockets (server side), do not rely on inheritance; BSD OSes silently suppress inheritance of some flags like TCP_NODELAY (Nagle algorithm) - response time now includes connection setup time - fixed member initialization bug in StatPhase.cc that caused coredumps on Linux 1.0p6 -> 1.0p7 19990405 - added rep_sz_hist, rep_sz_mean, resp_tm_vs_sz, req_bhr, rep_bhr objects to 'lx' (byte HRs are broken for long experiments) - added --track_rports and --rport_wait_min options to track TCP port usage on the proxy side; unfortunately tracking is complicated by known TCP violations of most BSD implementations - fixed 'lx' bug with aggregating reply rate from logs with different phase durations - applied bake-off mods to bb.pl 1.0p5 -> 1.0p6 19990310 - added "lx", a routine to eXtract objects from binary logs; also mergers logs to support multiple client/server experiments - added "exprep.pl" script to generate reports from binary logs of several experiments (needs more work) - replaced all ETag: headers with X-ETag (requests and replies); ETag is not a request header and gets filtered out on some proxies - added "Last-Modified: Wed, 18 Feb 1998 17:53:58 GMT" reply header to cachable replies; some proxies may not cache replies otherwise (a magic constant value should be eventually eliminated) - changed the default for the "--log_size" option to 10KB with no logging and 10MB with logging enabled - various fixes to handle log buffer overflows; we now ignore all new entries if the buffer is full (needs more work) - fixed a lot of bugs in interpreting logged data (recording was OK?) no program was interpreting logs so bugs stayed undetected - mandatory phases should not stop even when too many errors; we used to abort if that happens (needs more work to detect error flood early as we used to) - increased required verbosity level for "got N xactions..." message to 2; a more detailed but less readable progress/goal message is printed with a verbose level of 0. 1.0p3 -> 1.0p5 19990223 - added "--prefilled_cnt" option - added Uniform Popularity Model - minor polish and bug fixes 1.0p3 -> 1.0p4 19990222 - added "--ign_false_hits" option; by default false hits are now treated as xaction errors - added "--prn_false_misses" option; handy to see what hits a proxy has "missed" - added the "--tmp_loc_depth" option that specifies a soft limit on inter-request distances in tmp locality model - added "--tmp_loc_delta" option to specify fuzziness of temporal locality choices - added "--cool_phase" option to specify minimal cool-off phase; handy when you run more than one polyclt and do not want one of them to stop when others are still in their measurement phase - added "--fd_limit" option to lower FD limit supplied by OS (as a safeguard for OSes that crash when we run out of FDs) - added "--phase" option to log reader to show more stats about a particular phase (needs more work) - added support for error limit (given in the --goal option) - added ETag headers to pass object specific info from the client to the server - better facilities for logging and reporting various errors - record ideal (i.e, offered) Hit Ratio - record actual cachability ratio - added lognormal distribution to the list of distributions available on the command line (logn:mean,sdev) - print number of hits, misses, errors, etc. in lr - enforce a safeguard FD limit of (0.97*rcur - 10), where "rcur" is the FD limit enforced by OS according to getrlimit(2) the "--fd_limit" option can be used to lower this value - shutdown nicely on exit or fatal external error - count and report the number of open sockets; the last column of the console output now shows number of open FDs, not just pending xactions - revised traffic model operation and simplified the interface now traffic characteristics are controlled individually rather than via an ugly --world_oids option - constant hit ratio now works correctly regardless of cachability ratio and other traffic parameters - moved --rep_cachable option from the server side to the client side - changed tmp locality model to use future "plan" instead of past "history"; old model was emitting too many too close requests; current model follows the --tmp_loc distribution much better - complain if cachability status of a reply has changed - print various FD limits on start - r.n.g. seed was not properly changed when calculating reply sizes, resulting in same object ids having different content length if requested more than once by a proxy 1.0p2 -> 1.0p3 19990215 - added initial support for Constant Hit Ratio; we have several models (Object Ids distributions) to choose from, see --world_oids - added --world_oids option to specify Object Ids "distribution"; currently supported "distributions" meaningful for --world_oids are: . sequential ("seq"), . MemoryLess Zipf ("MLZipf"), . Hot/Cold ("HoCo"); names will change if we find better ones - changed default behavior when no --world_oids is specified: we used to generate Zipf-like stream of Object Ids by default, now the default is Sequential stream (hence, no hits by default) - removed --world_cap option; we no longer need/use World Capacity - added "Expires: Wed, 17 Feb 2000 03:35:25 GMT" header to cachable responses; this constant should be good enough in most cases and will be changed to a dynamic value in next releases - added "Pragma: no-cache" to uncachable responses - better explanation for errors with distribution specs on the command line - fixed zero Content-Length bug - fixed time accounting bug (visible at least on FreeBSD 3.0) - define _BSD_SIGNALS to make IRIX happy (needs more work) 1.0p1 -> 1.0p2 19990212 - added support for specifying distributions on the command line; currently available distributions: unif, exp, norm, and const - added support for specifying ratios and such on the command line; - "--rep-size" is now of type Size Distribution (default: exp(13KB)) - "--xact_think" is now of type Time Distribution - "--rep_cachable" is now of type Ratio (e.g. 80%) - renamed "--users" to "--robots" - value "0" does not require scale any more ("0" == "0sec" == "0min") - report number of outstanding xactions in i-stats (new last column) - increased verbosity level required for i-stats to 2 - fixed parsing of floating point values on the command line (used to be truncated to int) - #warn, not #err if FD_SETSIZE is #defined suspiciously early 1.0p0 -> 1.0p1 19990207 - added time support in goal option - enabled --launch_win support - enabled --xact_think support (per xaction think times on both sides) - added constant req submission rate mode (--req_rate) - replaced --req_ccl with --users (just a name change) - added support for absolute/relative urls (--abs_urls option) - added Host: header (all requests) - many binary logs fixes and improvements - many stats fixes - disable SO_LINGER if linger_tout == 0 - drastically improved memory management; now Poly runs out of filedescriptors before running out of memory on high loads - account for BSD select(2)'s magic limit on timeout value - internal alarms now use Heap instead of LinkedList