This directory contains a simple remote run command, constructed from 
PEXEC pieces.  This remote run command is simple yet robust, and can be used
to experiment with various enhancements (such as security and heartbeats).

This implemenation makes use of the "forwarding" handlers which accept only as
much data at one time as they can handle.  

The programs are

rrund - The demon; must be running on the "remote" host
rrun  - The remote run command; a rough replacement for rsh/remsh
rrunc - A console that can attach to a rrund

The Programs
------------
The program "rrund" is a demon that runs on the system that you want to be
able to run remote jobs on.  It listens on a known port for connections.
Once a connection is established, it responds to commands including "status"
and "run program".  A simple authentication mechanism is used.

The program "rrun" is a client that attaches to a "rrund" and sends a "run
program" request.

The program "rrunc" is a client that attaches to a "rrund" and sends various
commands; usually "send status", but also "kill process" or "rrund exit".

Running a remote program
------------------------
Once the run request is sent, the client wait to read from stdin (for input to
forward to the program) and on the socket (for messages from the program, via
rrund).  

Data is sent between the rrun client and rrund demon using typed messages.
However, this isn't really a message-passing application, because of the way
data is handled.  This is more of an active-message approach, but with 
important differences.  

First, a header is read.  This header contains a message tag and payload
length.  This is sent at 2 network shorts.  Based on the tag, various routines
are called.  These are (by function, not name)

    STDOUT - Forward to standard output
    STDERR - Forward to standard error
    EXIT_STAT - exit with status.
    QUERY  - Respond to an "are you alive" request (or the response)

On the demon end, the message tags are

    STDIN  - Forward to programs standard input
    SIGNAL - Send signal to program
    QUERY  - Respond to an "are you alive" request (or the response)
    
as well as

    STATUS - Reply with status
    RUN    - Run a program

Attached to the other fd's are "forwarding" handlers that format the data as a
message and send it along the socket.  These use STDOUT/STDERR (at the demon)
and STDIN (at the client).  

Forwarding handlers
-------------------
Rather than read all of an incoming message into a buffer, and then
(attempting) to write it out, we simply indicate that the next "n" bytes
should be sent to another fd.  Both input and output are non-blocking, and
the algorithm is (roughly) this

    nleft = n
    while (nleft > 0) {
        Read upto MIN(MAX_BUF,nleft) bytes.
        Try to write.
	nleft -= n_written
	    If (!wrote all)
		Must wait: Turn off reading on in_fd,
		Turn ON writing out out_fd
		Save information on what's left to write
		return
	}

Note that messages are relatively short (2 byte length) so this can't
monopolize the message traffic.  The "write" handler for this out_fd
essentially continues this process (taking care first of any leftover data)
and unsets writing on out_fd and resets reading on in_fd when the data is
successfully forwarded.

Timeouts
--------
Because remote connections can sometimes fail without the socket giving any
indication (until you try to write to it), an optional bi-directional timeout
is included.  The timeout is reset everytime data arrives, so an active
connection will never generate a timeout message request.  The timeout value
can be set by environment or command-line option.

The "Console"
-------------
One benefit of using PEXEC is that it is easy to create a console application.
This is "rrunc", and can send commands to

      Run 
      Send status of all active jobs
      Stop/signal jobs
      Stop rrund

In many ways, rrun is just a simple version of rrunc that only sends
run commands (along with signal, if it receives a catchable signal from the
user).  Note that signals are sent as "network" signals (so that SIGINT on the
client becomes SIGINT on the destination).

Authentication
--------------
This is a weakness of rrund currently.  The system used here is based on
read-access to common data stored in a file; alternative mechanisms can be
used by changing the "auth.c" module.  
The current system accepts a username and a keyvalue; this pair is checked
against the values that rrund was started with.  This assumes a secure
network (hah!).

We could use ruserok as well, along with the optional "send password" code
like that in serv_p4.c .  Using a ".rrun_rc" file to contain authorized
programs (again like serv_p4) is another possibility.

Termination
-----------
When a program started by rrund terminates, a message containing the exit
status is sent to the rrun client.

Issues
------
Since the socket can become choked when one of the destination fds itself
gets choked, things like the "timeout" queries can get lost.  There are
several responces:
   
   The behavior is OK, since we want to ensure that apps are blocked when
   some part of the chain (of sockets/fds) fills up

   A "backstop" query could try to establish a separate connection if the
   socket is full (since everything is managed by PEXEC).

   Separate sockets could be used for STDIN/OUT/ERR, though I'd like to 
   restrict the number that are used up

