$Id: Release-Notes-1.3.txt,v 1.14 1995/09/05 21:00:00 duane Exp $
TABLE OF CONTENTS
1. Gatherer
IP-based filtering
Username/Passwords
Post-Summarizing
Cache directory cleanup
Limit on retrieval size
Support for HTML-3.0, Netscape, and HotJava DTDs
2. Broker
Brokers.cf
Glimpse 3.0
Verity/Topic
WAIS, Inc.
Displaying SOIF attributes in results
Uniqify duplicate objects
Glimpse inline queries
3. Cache
Persistent disk cache
Common logfile format
Improved Internet protocol support
TTL calculation by regexp
Improved customizability
Security
Performance Enhancements
Portability
Optional Code
4. Miscellaneous
Admin scripts
========================================================================
GATHERER
IP-based filtering
------------------
It is now possible to use an IP network address in a Host
filter file. The IP address is matched using regular
expressions. This means that periods must be escaped. For
example:
Allow 128\.196\..*
Deny .*
Username/Passwords
------------------
It is now possible to gather password-protected documents from
HTTP and FTP servers. In both cases, it is possible to specify
a username and password as a part of the URL. The format is
ftp://user:password@host:port/url-path
http://user:password@host:port/url-path
With this format, the "user:password" part is kept as a part
of the URL string all throughout Harvest. This may enable
anyone who uses your Broker(s) to access password-protected
pages.
It is also possible to have "hidden" username and password
information. These are specified in the gatherer.cf file.
For HTTP, the format is
HTTP-Basic-Auth: realm username password
'realm' is the same as the 'AuthName' parameter given in an
NCSA .htaccess file. In the CERN HTTP configuration, the
realm value is called 'ServerId.'
For FTP, the format in the gatherer.cf file is
FTP-Auth: hostname[:port] username password
Post-Summarizing
----------------
It is now possible to "fine-tune" the summary information
generated by the Essence summarizers. A typical application of
this would be to change the 'Time-to-live' attribute based on
some knowledge about the objects. So an administrator could
use the post-summarizing feature to give quickly-changing
objects a lower TTL, and very stable documents a higher TTL.
Objects are selected for post-processing if they meet a
specified condition. A condition consists of three parts: An
attribute name, an operation, and some string data. For
example:
city == 'New York'
In this case we are checking if the 'city' attribute is equal to
the string 'New York' The for exact string matching, the string
data must be enclosed in single quotes. Regular expressions
are also supported:
city ~ /New York/
Negative operators are also supported:
city != 'New York'
city !~ /New York/
Conditions can be joined with '&&' (logical and) or '||' (logical or)
operators:
city == 'New York' && $state != 'NY';
When all conditions are met for an object, some number of
instructions are executed on it. There are four types of
instructions which can be specified:
1. Set an attribute exactly to some specific string
Example:
time-to-live = "86400"
2. Filter an attribute through some program. The attribute
value is given as input to the filter. The output of the
filter becomes the new attribute value.
Example:
keywords | tr A-Z a-z
3. Filter multiple attributes through some program. In this
case the filter must read and write attributes in the
SOIF format.
Example:
address,city,state,zip ! cleanup-address.pl
4. A special case instruction is to delete an object. To do
this, simply write
delete()
The conditions and instructions are combined together in a
"rules" file. The format of this file is somewhat similar to a
Makefile; conditions begin in the first column and instructions
are indented by a tab-stop. Example:
type == 'HTML'
partial-text | cleanup-html-text.pl
URL ~ /users/
time-to-live = "86400"
partial-text ! extract-owner.sh
type == 'SOIFStream'
delete()
This rules file is specified in the gatherer.cf file with the
Post-Summarizing: tag, e.g.:
Post-Summarizing: lib/myrules
Cache directory cleanup
-----------------------
The gatherer uses a local disk cache of objects it has
retrieved. These objects are stored in the tmp/cache-liburl
subdirectory. Prior to v1.3 this cache directory was left in
place after the gatherer completed. This caused confusion and
problems when users re-ran the gatherer and expected to see new
or changed objects appear.
Now the default behaviour is to remove the cache-liburl
directory after the gatherer completes successfully. Users who
want to leave this directory in place will need to add
Keep-Cache: yes
to their gatherer.cf file.
Limit on retrieval size
-----------------------
The code for retrieving FTP, HTTP, and Gopher objects now stops
transferring after 10M bytes. This is to prevent bogus URL's
from filling up local disk space. This limit can currently
only be changed by modifying the source in src/common/url (look
for "MAX_TRANSFER_SIZE").
Support for HTML-3.0, Netscape, and HotJava DTDs
------------------------------------------------
DTDs for HTML-3.0, Netscape, and HotJava have been added to
the collection in lib/gatherer/sgmls-lib/HTML/. To take advantage
of these DTDs your HTML pages should begin with one of:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 3.0//EN">
<!DOCTYPE HTML PUBLIC "-//Netscape Comm. Corp.//DTD HTML//EN">
<!DOCTYPE HTML PUBLIC "-//Sun Micorsystems Corp.//DTD HotJava HTML//EN">
========================================================================
BROKER
Brokers.cf
----------
Prompted by security concerns, there is a change in the way
that BrokerQuery.pl.cgi connects with a broker. The old method
had the broker hostname and port number passed as CGI
arguments. The new way passes the broker short name instead.
This name is then looked up in the file
$HARVEST_HOME/brokers/Brokers.cf. The CreateBroker program
will add the correct entry to Brokers.cf.
The old method still works for backwards compatibility. With
the new method, the broker name must appear in the Brokers.cf
file. If it does not, the user receives an error message.
The Brokers.cf file may also provide interesting features such as
* quickly relocating brokers to other machines
* using dual brokers for 24hr/day availability
If you change your broker port number (in admin/broker.conf)
then don't forget to change it here as well.
Glimpse 3.0
-----------
Harvest now uses Glimpse 3.0 which includes a number of bugfixes
and performance improvements:
* A new data structure considerably speeds up queries
on large indexes. Typical queries now take less
than one second, even for very large indexes.
* Incremental indexing is now fully supported.
* The on-disk indexing structures have been improved in
several ways. As a result, indexes from previous
versions are incompatible. When upgrading to this
release, you should remove all .glimpse_* files
in the broker directory before restarting the broker.
* Glimpse can now handle more than 64k objects in the
broker.
Verity/Topic
------------
This release includes support for using Verity Inc.'s Topic
indexing engine with the broker. In order to use Topic with
Harvest, a license must be purchased from Verity (see
http://www.verity.com/).
At this point, Harvest does not make use of all features in
the Topic engine. However, does include a number of features
that make it attractive:
* Background indexing: the broker will continue to
service requests as new objects are added to the
database.
* Matched lines (or Highlights): lines containing query
terms are displayed with the result set.
* Result set ranking
* Flexible query operations such as proximity, stemming,
and thesaurus.
WAIS, Inc.
----------
This release includes support for using WAIS Inc.'s commercial
WAIS indexing engine with the broker. To use commercial WAIS
with Harvest, a license must be purchased from WAIS Inc. (see
http://www.wais.com/). The WAIS/Harvest combination offers
the following features:
* Structured queries (not available with Free WAIS).
* Incremental indexing
* Result set ranking
* Use of native WAIS operators, e.g. ADJ to find one
word adjacent to another.
Displaying SOIF attributes in results
-------------------------------------
In v1.2 the Broker allowed specific attributes from matched
objects to be returned in the result set. However, there
was no real support for this in BrokerQuery.pl.cgi.
Now it is possible to request SOIF attributes with the use
of HTML FORM facilities. A simple approach is to include
a select list in the query form. For example:
<SELECT NAME="attribute" MULTIPLE>
<OPTION VALUE="title">Title
<OPTION VALUE="author">Author
<OPTION VALUE="date">Date
<OPTION VALUE="subject">Subject
</SELECT>
In this manner, the user may control which attributes are
displayed. The layout of these attributes in HTML is
controlled by the '<FormatAttribute>' specification in
$HARVEST_HOME/cgi-bin/lib/BrokerQuery.cf.
Uniqify duplicate objects
-------------------------
Occasionally a broker may end up with duplicate entries for
individual URLs. This usually happens when the Gatherer
changes (its description, hostname, or port number). To remedy
this situation, there is a "uniqify" command on the broker
interface. On the admin.html page it is described as "Delete
older objects of duplicate URLs." When two objects with the
same URL are found, the object with the least-recent timestamp
is removed.
Glimpse inline queries
----------------------
In v1.2 using Glimpse with the broker required the broker to
fork a 'glimpse' process for every query. Now the broker can
make the query directly to the 'glimpseserver'. If glimpseserver
is disabled or not running for some reason, the broker will use
the previous approach and spawn a glimpse process to handle the
query.
========================================================================
CACHE
Persistent disk cache
---------------------
Upon startup the cache now "reloads" cached objects from a
previous session. While this adds some delay at startup,
heavily used sites will benefit, especially where filling
the cache with popular objects is expensive or time-consuming.
To disable the persistent disk cache, add the '-z' flag to
cached's command line. This emulates the previous behaviour,
which is to remove all previously cached objects at startup.
Common logfile format
---------------------
The cache now supports the httpd common logfile format which is
used by many HTTP server implementations. This makes the
cache's access logfile compatible with many of the freely
available logfile analyzers. Note that the cache does not
(yet) log the object size for requests which result in a
'TCP_MISS'.
There have been many improvements to the debugging output
as well.
Improved Internet protocol support
----------------------------------
Numerous improvements and bugfixes have been made to HTTP,
FTP, and Gopher protocol implementations. Additionally,
a user-contributed patch for proxying to WAIS servers has
been included.
TTL calculation by regexp
-------------------------
It is now possible to have the cache calculate time-to-live
values based on URL regular expressions. This would allow
an administrator to set large TTL's for images and lower
TTL's for text, for example.
These are specified in the cached.conf file, beginning with
the tag 'ttl_pattern'. For example:
ttl_pattern ^http:// 1440 20% 43200
The second field is a POSIX-style regular expression. Invalid
expressions are ignored.
The third value is an absolute time-to-live, given in minutes.
This value is ignored if negative. A zero value indicates that
an object matching the pattern should not be cached. NOTE: the
absolute TTL is used only if the percent-of-age (described
next) is not used.
The fourth value is a percent-of-age factor. If the object is
sent with valid Last-Modification timestamp information, then
the object's TTL is calculated as
TTL = (current-time - last-modified) * percent-of-age / 100;
If the percent-of-age field is zero, or a last-modification
timestamp is not present, then the algorithm looks at the
absolute TTL value next.
The fifth field is a maximum, upper-bound on the TTL to return
for the percent-of-age method. It is specified in minutes,
with the default being 30 days. This is provided in case a
buggy server implementation returns ridiculous last-modification
data.
Improved customizability
------------------------
More options have been added to the cache configuration file:
* String-based stoplist to deny caching of objects
which contain the stoplist string (e.g.: "cgi-bin").
* Support for "quick aborting." When the client drops
a connection, the cache will abort the data transfer
immediately. Useful for caches behind SLIP/PPP
connections.
* The number of DNS lookup servers is now configurable.
The default is three.
* The trace mail message sent to cs.colorado.edu
(containing only the IP address and port number of
your cache) can now be turned off.
Security
--------
IP-based access controls are now supported. The administrator
may deny access to specific IP networks/hosts, or may only
allow access from specific networks/hosts. Two access control
lists are maintained: one for clients/browsers using the cache
(the "ascii port") and another for the remote instrumentation
interface (cache manager).
Performance Enhancements
------------------------
Several performance enhancements have been made to the cache:
* The LRU replacement algorithm is more efficient and
quicker. In conjunction with the new LRU replacement
policy the default low water mark has been changed
from 80% to 60%.
* The in-memory usage (metadata) of cached objects has
been reduced to 80-100 bytes per object.
* The retrieval of various statistics from the instrumentation
interface is much faster.
* User-configurable garbage collection reduces the number
of times these more expensive operations are performed.
* Cleaned up and reduced overall memory management. Our
checks with Purify report no memory leaks.
Portability
-----------
The TCL libraries are no longer needed to compile the cache.
User-contributed patches have been incorporated for better
support on BSD, Linix, IRIX, and HP-UX systems.
Optional Code
-------------
The following are recent additions to the code. They can be
optionally included by setting '-D' flags in the Makefile.
CHECK_LOCAL_NETS
Define this to optimize retrievals from servers on your
local network. If your cache is configured with a parent,
objects from your local servers may be pulled through the
parent cache. To always retrieve local objects directly
define CHECK_LOCAL_NETS and rebuild the source code. Then
add your local IP network addresses to the cache configuration
file with the 'local_ip' directive. For example:
local_ip 128.138.0.0
local_ip 192.54.50.0
LOG_FQDN
Client IP addresses are logged in the access log file. To
log the fully qualified domain name instead, define LOG_FQDN
and rebuild the code. WARNING: This is not implemented
efficiently and may adversely affect your cache performance.
Before each line is written to the access log file, a call
to gethostbyaddr(3) is made. This library call may block
an arbitrary amount of time while waiting for a reply from
a DNS server. While this function blocks, the cache will
not be able to process any other requests. You have been warned.
APPEND_DOMAIN
Define this and use the 'append_domain' configuration directive
to append a domainname to hostnames without any domain
information.
USE_WAIS_RELAY
Define this and use the `wais_relay' configuration directive
to allow WAIS queries to be cached and proxied.
========================================================================
MISCELLANEOUS
Admin scripts
-------------
A number of sample scripts are provided to aid in administering
your Harvest installation:
RunGatherers.sh: This script can be run from your ``/etc/rc''
scripts to start the Harvest gatherer daemons at boot time.
It must be customized with the directory names of your gatherers.
It is installed in $HARVEST_HOME/lib/gatherer.
RunBrokers.sh: This script can be run from your ``/etc/rc''
scripts to start the Harvest brokers at boot time.
It must be customized with the directory names of your brokers.
It is installed in $HARVEST_HOME/lib/broker.
harvest-check.pl: This Perl script is designed to be run
occasionally as a cron(1) job. It will contact your gatherers
and brokers and report on any which seem to be unreachable.
The list of gatherers and brokers to contact can be specified
at the end of the script.