WAIS & GOPHER
JANUARY 14, 1993
OVERVIEW
Wide Area Information Server (WAIS) is a suite of programs
and protocol for creating and searching full text databases.
The University of Minnesota's Unix Gopher (UMN) server,
gopherd, can easily access WAIS databases thus integrating
these materials into gopher-space.
This document outlines the necessary steps to have your
gopherd server use WAIS databases. It is concerned with only
the most typical of uses.
This is a working document and may contain errors and
omission. If you have any comments, please send them to
Andrew_Gilmartin@Brown.Edu. The latest version of this
document is available via anonymous ftp at ftp.brown.edu in
the /pub/gopher directory.
WAIS AND GOPHER AVAILABILITY
The WAIS software is available via anonymous ftp at
think.com in the /wais directory. The file wais-8-b5.tar.Z
is the Unix source code for the full text indexer, database
server, and a couple of Unix clients. In addition, there are
a number of machine specific clients available there. The
file WAIStation-0-63.sit.hqx is the Macintosh client. To
have gopherd use WAIS databases, however, you only need the
Unix source code.
The Gopher software is available via anonymous ftp at
boombox.micro.umn.edu in the /pub/gopher/Unix directory. The
file gopher gopher1.03.tar.Z contains the latest version of
gopherd. You need at least version 1.0 to use these
instructions.
INSTALLING WAIS AND GOPHERD
Both WAIS and Gopher Unix software are rather complex. It is
best to test and install each separately before attempting
to join them. If you are not familiar with Unix software
installation procedures, I recommend that you work with
someone who is--the process is messy.
Install and test the WAIS software first. While the WAIS
installation is not documented well, the software is very
portable. Follow the wais INSTALLATION instructions very
carefully. You do not need to make the X Windows Systems or
Emacs client; to prevent this remove from the Makefile's
default rule the x and tags dependencies.
I recommend that you install and test gopherd before trying
to link in the WAIS code. Once gopherd is working, you will
have to remake it for WAIS. To incorporate WAIS into
gopherd, carefully follow the "Full Text Indexing Setup for
WAIS" installations in the file gopher/doc/INSTALL.
PROTOCOL LINKS
Gopher can access existing WAIS database servers via the
WAIS protocol. To do this all you need do is to place a WAIS
database source description file (.src) into your Gopher
data directory.
To test your gopherd, copy the following to the file
aesop.src
(:source
:version 3
:ip-address "18.85.0.233"
:ip-name "bloat.media.mit.edu"
:tcp-port 8000
:database-name "Aesop-Fables"
:maintainer "jh@bloat.media.mit.edu"
)
Place this file in the root of the gopher directory (while
testing). If you want to give the file a more descriptive
name create the file aesop.src
Name=Search Aesop's Fables (Testing)
and place this in the root data directory's .cap directory.
Delete the root data directory's .cache file and you are
ready to test.
Using any Gopher client that supports searching, connect to
the root of your Gopher server. You should see a search menu
item named "Search Aesop's Fables (Testing)". If you don't
see this check that you followed each step above
ù Created the WAIS database source description file
aesop.src at the root of the gopher data directory
ù Created the Gopher .cap file aesop.src in the root
gopher directory's .cap directory.
ù Deleted the root gopher directory's .cache file.
To test the search, look for "hare" (a common Aesop
character). After a short while, your Gopher client should
display a list of 10 items, among which is "The-Lion-and-the-
Hare...". Success!
A list of publicly accessible WAIS databases is kept at
quake.think.com. The list, wais-sources.tar.Z, is available
via anonymous ftp in the /wais directory. You can use this
to build your own list of available WAIS databases.
Alternatively, you can use the shell script waisload to do
it for your. Waisload is available via anonymous ftp at
ftp.brown.edu in the /pub/gopher directory.
DISK LINKS
Having tested gopherd with a protocol link you are ready to
test a disk link. A disk link enables you to access WAIS
databases available within gopherd's file system. For
example, you might want to create a full text index of all
the documents within your Gopher data directory.
To test disk links you will construct a simple wais database
of several of your files. Create the following directory
structure at the root of your gopher data directory
./myfiles/
./myfile/.waisindex/
Place several simple text files in ./myfiles. The WAIS
database's index and auxiliary files will be placed in
./myfiles/.waisindex.
To create the database issue the command
waisindex -d ./myfiles/.waisindex/index ./myfiles/*
This will build a WAIS database of all the files in
./myfiles and place the database's index and auxiliary files
in ./myfiles/.waisindex.
To have gopherd recognize the WAIS database files for
searching, create the following .link file in ./myfiles
Name=Search My Files (Testing)
Type=7
Path=7/myfiles/.waisindex/index
Host=+
Port=+
Using any Gopher client that supports searching, connect to
the root of your Gopher server. You should see a folder item
"myfiles". Enter this folder and you should see a menu
listing all your files plus the "Search My Files (Testing)"
search item. If you don't see this make sure that you
carefully followed the steps
ù Created ./myfiles and ./myfiles/.wasindex directories
ù Copied several files to ./myfiles
ù Built the WAIS database
ù Have a .link file in ./myfiles
To test the search, look for a word--other then connectives-
-you know is within one of your files. After a short while,
your Gopher client should display a list of files that
contain the search word. Success!
WAIS INDEX TO WHOLE CWIS
To build an index for the whole content of your CWIS do the
following.
mkdir ./.waisindex
find ./ \
\( \
! -name .\?\* \
! -name core \
! -name adm \
! -name bin \
! -name dev \
! -name etc \
! -name usr \
-print \
\) \
-o -prune \
| waisindex -d ./.waisindex/index -stdin
Note that this uses GNU find, the standard Unix find does
not have the -prune option.