The Weaver

So what is it?

The "Weaver" --- the core of both the MusicWeaver and StreamWeaver suites --- gives you, the user, an easy, graphic, way of connecting small modules together, to work in concert on common data. Any kind of sequential data that needs to be processed in a series of steps can be handled by the scheme --- provided suitable modules for the purpose exist. Because, crucially, data is tagged as to its nature, modules will only conern themselves with data that they know about; different kinds can thus be freely mixed, interacting only when -- and if -- necessary. A notable feature of the Weaver, as compared to other 'flow-diagram' based applications, is that the diagram is always 'live' -- there are no separate 'build' and 'run' modes.

The Weaver program itself is just a manager: it needs some suite of modules to manage to do anything very useful. The MusicWeaver suite is targeted toward the real-time processing of MIDI event streams, and the StreamWeaver provides a visual approach to the piping of data between posix-type commands. Rather than getting into the depths of either package, a simpler starting point for a guided tour is the included pair of 'test' data modules: one simple data generator, and a module to display its output.

Requirements

This version needs the BeOS R4 or later. PPC and Intel versions are available. The main program and the test modules should work with any such system, as will the entire StreamWeaver suite. Most of the MusicWeaver will also, but for MIDI-I/O you will of course need a MIDI interface of some kind.



A Walk Through

[...probably more of a quick trot through, but hopefully it'll give you an idea.]

If you've successfully unzipped the package [and presumably you have, as you're reading this] you should have a directory called "Weaver" containing the Weaver executable itself, some common modules ('JoinPaths', 'Switcher', 'TriggerSwitch', 'BridgeIn', and 'BridgeOut'), some subfolders (including this Documentation) , and a few example 'Diagram Configurations'. Most of the modules that you will be able to drag into a Weaver window to weave diagrams [I am tempted to call them "Tapestries" (:-)] are in their respective sub-folders.

You are not constrained to the exact hierarchy as supplied, but to avoid difficulties -- especially if you wish to exchange diagram files with others, you probably should not stray too far from this arrangement. [See Folder Organization for details.]

First Steps

There are a couple of ways to start things up. To begin with a blank diagram window, just double click on the Weaver icon itself. To start with a previously constructed configuration, double click on that configuration file.

If you're impatient, you can open a configuration now, but for ease of exposition (avoiding details of all the modules involved for the moment) let's begin from the beginning. [Feel free to explore at any time...]

So to start, just double click on the 'W' icon. Two windows should open up: a small one labelled "Weaver Main" at top left, and a larger one marked "Diagram". The former is for overall control (the "Main Window"); it only has a menu ("Global") with four items: "New Diagram", "About...". "Help", and "QUIT". The first opens a new window --- you can have as many separate diagrams as you want ---, the others are obvious. As you notice, the diagram itself has no menu bar. It does however have a popup menu, as we will see shortly.

Diagram Operations

The diagram window is where most of the action takes place. You place elements you want in your configuration there simply by dragging the module's icon from its home drawer (one of the sub-folders here). [I assume all current readers are familiar with basic BeOS operations]. Mouse button one, by the way, is used for all 'arrangement' of the window. For (my) convenience I'll call this the 'left' one. In what follows I'll have to talk also about the 'right' button; if you have one of those one-button thingies, you can understand this to mean 'control' or 'shift' + mouse-button.

Let's look around at the basics first. For simplicity we'll use the 'test' modules for our exploration. Open the folder named 'TEST': inside are the modules 'PacketSource' and 'Counter'. <-- IMG SRC=Images/PacketSource.jpg HSPACE="10" --> <-- IMG SRC=Images/Counter.jpg HSPACE="10" --> Drag the "PacketSource" icon into the diagram window, and drop it somewhere toward the left. You'll notice as you drag it that a little square (in addition to the tracker's ghost-icon) follows your cursor around. The window is "tiled" with a grid, and this square shows you where the centre of your placed element will be.

Having placed the first element, now similarly drag and drop a copy of "Counter" somewhere to the right of this. If you don't like where you actually dropped it, you can simply grab it again and move it somewhere else.

Now we have to connect the two. You'll see a little 'pip' on the right side of the PacketSource element in the diagram. Drag from this point to one of the sides of the Counter (one without the pip); you should see a 'rubber-band' line following you, plus the target square as before. (There were some older graphics cards that had trouble with single-pixel-wide diagonals; this might cause brokem up rubberbanding, but won't affect other operations.) When you release the mouse-button you should see a link between the two elements. (Be sure you release the mouse-button when the 'guide' square is exactly over the middle of the side you wish to connect to.) Data is now flowing, though this is not apparent as yet.

The Counter has a Display Panel that can be brought up to show the count. To do this, move the cursor over the centre of the element and press the right mouse-button. A short menu will pop up -- as well as a (blue) panel of information about the element. Select the "Panel" item from the menu and the Counter's display will open up: it should be clicking away merrily. (All that PacketSource does is to generate a stream of incrementing values -- one every millisecond.) Observe that the element is outlined in red while its panel exists (minimized or not).

You'll have noticed other items on that menu. "Remove" is probably again obvious, but you may not want to do that at this moment... Play around some more first. "Name", when selected, opens up an edit window where you can supply a suitable descriptive label if you wish. Every element must have a unique name (to label its panel if it has one, but more importantly for the sake of saving and loading); it will be given a default one -- like "FLT0' etc -- if you don't wish to. Notice too that the menu for the source element doesn't have a "Panel" item -- simply because there's nothing to see or control for this particular module. In general, menu items are only present when they are applicable to that element at that time.

You can also use the right button on a link. The menu here has items that will be discussed more later. "Cut" should be self-evident, though.

Working with the trivial diagram --- and with data still flowing --- we can explore some of the other basic capabilities. First, drag another copy of the Counter, but this time drop it straight onto the existing link, somewhere between the other two. (Make sure the targetting square is directly over the link line.) The new copy will be inserted into the existing path. If you pop open the panels for each, you'll see them counting along together. (Now you understand why the Counter has an output pip, too! In this case, the incoming data is passed on unchanged; other filters will output a modified data stream.)

Oh yes. pop up the inserted element's menu, and you'll see it has that extra "Elide" item. If you don't know that word (:-))... just try it out... You'll only see that item when the element can be removed from a path, leaving the path itself intact; "Remove" in contrast also deletes any links that may be attached.

Drag yet another counter onto the diagram, but not onto an existing path this time. Then drag from some intermediate point on the path that does exist, and drop on one of the counter's input sides. You have a branch! Data is duplicated on both forks.

Conversely, as each basic "Filter" (like the Counter) has three possible input sides, you can merge up to three paths onto one such element. (This is only true for simple Filters. Multiconnector Modules , for example, only accept one connection for each input they can handle; 'Sources' won't accept any. You can never attach more than one path to the same connection point.) You can try adding another source and connecting it to a free side of one of the existing counters. You'll probably see some flickering of the displayed numbers, as the two data streams are running from different starting points. (Unlike some systems, which allow forests of branches to and from connection points, the Weaver is strictly regulated. This is partly because of the tiled structure, which makes for much more readable diagrams; also because the efficient data transmission mechanism --- designed to minimize copying --- works well with linear chains of filters and identifiable branch points.)

Joining Paths

For merging data streams where there happens to be no convenient filter, the "JoinPaths" module is provided (found in the same drawer as the Weaver itself, as it doesn't really belong in any particular subset); it is simply a null element that passes on all its inputs through its single output, and doesn't care what kind of data is flowing.

Saving the Diagram

Now that you have a simple diagram 'woven', we should look at how to save and reload it. Saving a diagram to a file is managed through the popup menu of the diagram itself. Press the right button on some empty part of the diagram: you will get a menu with "Name Diagram", "Save Diagram", and "Save Diagram As...".

"Name Diagram" lets you enter a specific name for the diagram that will appear in the window tab. This is strictly your choice. (Don't be confused: this label has no connection with the filename under which the diagram might be saved -- unless you coordinate them yourself, of course.)

"Save Diagram As..." opens a File Save Panel where you can specify the folder and the name you wish to save it under. [Note that this Save Panel is somewhat extended from the standard: if you wish to use an existing filename as a base for the name you actually wish to save as, simply click (once) on that file in the panel, and the name will appear in the text pane, ready for editing. (This doesn't quite happen with a normal BeOS panel.)] The default folder is the last one used in any "Save Diagram As..." action; this is persistent (maintained in a attribute of the executable) from session to session, on the assumption that this will be your preferred place for them.

"The "Save Diagram" item uses the current pathname of the diagram (if it has one) so you can quickly re-save the diagram after a change. If the diagram is not already associated with a file (i.e. not already saved or loaded from a previously saved one) "Save Diagram" opens a File Panel in the same way as "Save Diagram As...". When the program thinks that there has been no change since the last save, it puts the item in parentheses; it does not disable it, however, because it does not know about changes you may have made to module settings that you may wish to save.

Reloading a Diagram File

The quickest way to reload a file you have saved is to go to its folder window and double click on its icon. This will open a new diagram window, with a copy of the diagram exactly as you originally created it (including any open panels). If the Weaver is not already running, it will be started. If it is running, you can achieve the same result by dragging the icon into the Weaver Main Window.

You can also drag a configuration into an already open diagram. In this case, however, it does not appear in its original position within the window, but takes the drop point as its new top left-hand corner. (So if you drop it and nothing appears, try scrolling the window around!) This operation won't succeed if there are any overlapping elements. No two elements may have the same name (within a diagram), but if a naming conflict is encountered the Weaver simply supplies a new unique default.

(Although you can open as many diagrams as you like within reason, there are certain fairly obvious limitations on running some combinations of diagrams at the same time. For example, you can load several diagrams all connected to the same MIDI port, but unless they are designed to function together you will probably have total confusion!)

Off You Go...

You've probably got it pretty well figured out by now, so I'll leave you to play with the other facilities and modules yourself. Remember that modules may have differing connection characteristics and so on from those used to this point. We have seen one "Source" (with no inputs allowed) and one "Filter" (that passes input data on to its output, possibly modifying it on the way -- though the Counter does not). There are also some "Multi-connector" modules that can have several input and output connections, each with their own function; see the section below for more specifics.

There are other things you can do with and to diagrams, elements, and links, that were not mentioned in the brief discussion above. You can replace one element with another, route a link path between elements, and so on. Later sections describe all these possibilities.

If you find that you repeatedly use a given configuration of elements as a part -- or repeated parts -- of a diagram, you can create a 'Composite Element' from them, and place instances of this just as you would any other element. Again, see the respective section below.

You will find descriptions of the specific modules of a suite, information on how they are intended to be used, and their differences, in other documentation. Each suite of modules comes with a few examples as well. Refer to the respective README's for pointers to all this.



Multiconnector Modules

Modules that are more complex than simple "Sources" or "Filters" may need more than just a single output or 'through' connection. Such 'Multiconnector' modules are distinguished by having a larger outline, giving room for extra outputs (most have at least one 'source' and one 'through' path). The standard 3x5-tile box provides room for up to three output and five input connections.

Because each connection will have a different function, there are some things you need to know when making a link. As with other types of module, outputs appear at the 'pips', and inputs -- as appropriate for the module -- can be connected at any tile that is not on the output side and is not a corner. However -- unlike simple Filters -- each individual input will only accept one link, because otherwise it would be difficult to specify exactly what you wished each link to be. Simply place a preceding 'JoinPaths' if you want to merge more than one path into a multiconnector element.

You need to be able to control the actual function of each connection, as each will be different. Note that there is no predetermined association between a particular output pip or input location with a function. There are two factors that determine which function is associated with which connector.

In the first place, there is a default ordering (set by the module) . The first output connected will have one function, the second another. Similarly for input connections (independent of the output).

Secondly, after a link is made to another module, you can change which function is associated with that link. If you press the right mouse button when the cursor is over the connector (tile), you will get a popup menu of the possible choices for that link (with the current selection checked). Simply select a different item to change it. (Associations will be swapped if the selected one is already attached to another connector.)

To avoid surprises, it is best to go back and check connections you have made to multiconnector elements, to be sure that they are correct, at least until you're familiar with the element's defaults.



More Diagram Operations

There are some popup menu options and other possible operations that were not covered in the brief review above. These are detailed in the following.

Replacing Elements

If you wish to replace an element that is already in a diagram with one of a different type, you can do so, if the original and replacement have compatible connections. Just drag the replacement as you would normally to place an element, but drop it directly onto the one you wish to replace. An alert will appear to ensure that you really meant to do this, and if you confirm (and it is possible) the new element will take the place of the old.

Having 'compatible' connections means that the new element is able to accept the same number of input and output links as the original currently has. Replacement may also be foiled by other things such as differing sizes. Unfortunately the system can't tell before it tries whether the change is possible, so the confirmation request will always pop up. If it can't be done, though, there is no harm -- the original configuration remains unchanged.

Remember that if you have multiple input or output connections to the element the system cannot tell which should be replaced with which, and assigns them essentially arbitrarily. You always need to check them after replacement and correct them if necessary.

Orienting Elements

When you first place an element, a default orientation is chosen automatically -- outputs to the right if it is a new placement, some appropriate rotation if inserted in a previous link. If you want to orient a newly placed (but not inserted or otherwise linked) element differently, you can select the 'Rotate' item in its popup menu, and choose the rotation you desire (90 degrees in either direction, or 180 degrees reversed). You can do this at any later time too, as long as the element currently has no connections. If you want to rotate an already-linked element, you will have to cut all the connections first. (The menu item only appears of the operation is allowed.)

Routing Link Paths

When you drag a link from one element to another, a default ('shortest') path is chosen. This choice can be aesthetically ugly, or just confusing, so you may want to lay out a more suitable route.

To do this, bring up the popup menu for the path, and select 'Route'. A 'control' window opens at top left, but the diagram itself remains active. Click with the left button, in order, on major points in the diagram that you wish the path to go through (you only usually need one or two points) and then click 'Done' in the control window.

Each point ('tile') is highlighted as it is selected. Unfortunately there is at the moment no way to unselect a tile again -- you just have to Cancel and start over. If you choose an impossible or unavailable tile, though, it is ignored, and the path will still go through the others.

Labelling Link Paths

For convenience only, the links between elements may be labelled in the same way as the elements themselves. The name has no significance to the system, but appears in the info window displayed by the right mouse-button, and is recorded in the configuration file. Unlike elements, there is no prohibition on giving more than one path the same label. All branches of the path will assume the same label -- there is no way to assign different labels to branches.

Select 'Name' from the path's popup menu, and enter the desired label string into the text box.

Accessing Help

A flexible help system is provided for both the Weaver and its modules. (At this writing only some of the modules have help files, but it will extend to them all eventually.) Help is invoked, if available, from an item in the relevant menu.

General help with using the Weaver itself can be found in either the Diagram popup menu, or the menu of the 'Weaver Main' window. (The two are equivalent.)

If a module has help available, a 'Help' item will appear in a corresponding element's popup menu.

As HelpFiles are standard HTML, you can also of course go straight to the 'Help' folder within the Weaver hierarchy, find your topic and select it there.



Multi-Level Diagrams -- Composite Elements

A Weaver diagram, as well as containing the basic elements that correspond directly to dragged-in modules, can have 'Composite Elements' that in fact are sub-diagrams of their own. This lets you package up common groups of elements that you may want to use several times in a configuration. Or you can tuck away parts of the diagram that aren't often accessed.

A sub-diagram is pretty much like any other except that it will usually contain 'Bridge' elements that allow it to connect to its enclosing diagram. At that higher level, the bridges are represented by connector points on the composite element itself. A composite element has a fixed aspect -- six available outputs and six inputs around its edges -- but the number of connections that can actually be made to it corresponds to the current set of bridge elements within.

You can save composite element diagrams, like others, as configuration files. However, when you drag the result into another diagram it is placed like an element, rather than being expanded into its components. In most ways, this composite element behaves the same as any other -- it can be placed, deleted, inserted, moved, and so on. The 'Panel' that is brought up from its popup menu item, though, is its diagram window. One difference from a standard top-level diagram is that if you close this window, the configuration does not also quit -- it continues running uninterrupted until its enclosing top-level diagram is itself exited. Any panels opened for elements within the subdiagram also remain visible.

If you wish to open a Composite diagram file without having already placed it in an existing window, just double-click on it as you would a regular diagram file: it will open 'stand-alone' and can be treated like any other. If you save it again, however, it will retain its composite nature. (It is not possible, by the way, to open a fresh Composite diagram window directly, or to [easily!] convert a regular diagram into one. A blank file -- "comp0" -- is provided in the Weaver drawer as a convenient starting point.)

If you place several instances of one Composite element in a configuration, they are entirely independent of each other. If you edit one of the subdiagrams, or change parameters in its elements, the other instances are entirely unaffected. Beware, though: one thing each of the subdiagrams retains is the filename it was loaded from, so if you save any one of them back to its default, that file will assume the current configuration of that particular instance. [This could be considered a bug. It might change.]

When you save a diagram (at any level), all its sub-diagrams are saved as part of the overall configuration. It is of course possible for a Composite element to contain lower level Composites.

Bridge Elements

To be able to make links to or from other elements in the diagram in which a Composite element has been placed, the Composite must have appropriate 'Bridge' elements placed in its own diagram.
Each 'BridgeIn' element provides one possible input connector on the Composite;
each 'BridgeOut' allows an output.

These elements simply pass the data streams from one level to the other. They do no other processing, and pay no attention to the type of data actually flowing. Within their diagram they can only accept connections of their appropriate direction: a BridgeIn has an output pip from which the data can be taken, but accepts no inputs; a BridgeOut has no output pip at all, but will accept (and merge) inputs on all its four sides if desired.

As with any other element, a Bridge may be given a name, or if not it gets a default one. The corresponding connection made to its Composite gets the same name. You can select which of several Bridges is to represent that link through the popup menu on that connector point (after the link is made). The initial association of the links may be arbitrary, so don't forget to check this if you have more than one BridgeIn or BridgeOut.

You can't make more connections to a Composite than the number of Bridges it contains, and you are limited in any case by geometric constraints to a maximum of six of each kind. You can actually have more Bridges placed than this: you just can't connect them all at one time. If you delete a Bridge element, any corresponding upper level link gets cut too.



Common Modules

There are a few modules, supplied as part of the basic Weaver package, that function with any type of data. JoinPaths and the Bridge modules described above are of course representatives.

Others provide a path-switching facility, so that data can be directed to different regions of a configuration at different times, possibly as the result of specific events in that data.

All these modules are discussed in their own section of the documentation -- Univeral Modules



Folder Organization

The Weaver "suites" are essentially self-contained, and can be placed anywhere you like -- presumably somewhere in the '/boot/home' hierarchy. However, if you have more than one suite -- the MusicWeaver and StreamWeaver for instance -- it is best to keep them in a common top-level folder, preferably named 'Weaver'. In any case, you should make sure that you only have one copy of the Weaver program itself on your system: having more than one may not cause immediate confusion, but may have subtle unexpected effects, especially if you ever want to upgrade.

For Help to function properly, the 'Help' folder must be a direct child of the Weaver folder, and for proper cross-reference the 'Documentation' folder should be also.

You can launch a Diagram file from any folder. Equally, a Diagram file may be saved wherever you want. By default, the File Save Panel originally opens at the Weaver's own folder, but you can navigate anywhere from there before saving it. The Panel remembers where it was left last time when it is next opened (even if the Program has been closed and restarted in the meantime).

Modules similarly can in fact be anywhere, but it will be much the most convenient to keep them in the hierarchy as supplied. When a Diagram file is saved, the full path name of the module is put in the file. However if this was the only way it could locate a module it would restrict Diagram files largely to the machine on which they were created, or to ones with exactly the same file hierarchy; therefore if an original pathname is not found, the system looks for a module of the same name within its own sub-hierarchy. To avoid this search each time, you can simply resave Diagram files, and they will get pathnames appropriate to your system.

For reference, the (current) exact search sequence is as follows: It first looks for a subdirectory of the Weaver app's own directory called 'Modules', and checks to see if it can find the right name there. If not, it then checks all subdirectories of 'Modules'. ('Modules' doesn't need to exist at all, naturally -- it doesn't in the supplied set -- but it may be convenient at some point to have an enclosing directory.) It next checks in its own directory (where 'JoinPaths', 'Switcher' and so on are). Finally it checks all (direct) subdirectories of its own -- which is where the current organization has them. If it can't find the supplied name anywhere, it reports an error.



Cautions

One thing to watch out for when you are constructing diagrams is unintentional loops. Bcause data is carried around the paths in the diagram by 'packets', and each path has (currently) only one packet that circulates, you can jam things up by introducing a loop into a path. This can be troublesome because a packet in one path may have to wait (at a branch for example) for the packet in another to return: if they are the same path, it will never come back! (There is protection against a straightforward endless loop that automatically terminates it eventually, but this cannot avoid a lock up if the loop itself gets jammed.)

Going into much more detail here would be tedious, so I won't. Suffice it to say that if you find that you suddenly aren't getting any output, you may have such a loop lockup. If this happens, you should be able to cut a link or delete the offending element to recover.



What Next?

With this release, the Weaver itself as I envisaged it is essentially complete. The range and capabilities of the module suites (The MusicWeaver and StreamWeaver at the moment, perhaps others to come) will continue expanding of course.

Eventually I'll make the protocol -- both for MIDI and the general module mechanism -- available for others to write their own bits and pieces. I'm leaving commercial niggles aside for now, but I expect to ask some small licence fee --- nothing up front, but some small proportion of sales or shareware fees, I'd guess.

As I stressed at the start, the Weaver should be suitable for many data domains. The data being passed around might be bitmaps, for instance. Depending on the complexity of the operations you wanted to perform, it might not be possible to process bitmaps in real-time, but this probably wouldn't matter for that sort of application, where you would be performing the same sequence of transformations on a large set of images for later presentation.

Another attractive possibility is real-time control. A MIDI track, or some other master sequence could be linked to controller modules that would drive special-purpose hardware, synchronize animation, and so on.

Then there is Scientific Data Acquisition and Analysis. An important application area, if anyone gets around to doing it under BeOS...

So, there are interesting possibilities. Watch this space.



Copyright and Distribution

This documentation, the Weaver program, and all associated modules, are Copyright 1997-1999 by Peter J. Goodeve. All rights are reserved. For further conditions attached to the associated module suites, refer to the relevant READMEs.

You are permitted to distribute the Weaver program, the common modules, and documentation without charge for non-commercial purposes, provided that the whole package is kept intact. For commercial use or distribution (other than charging a reasonable media and copying fee) please contact the author.

Please note that future releases of the MusicWeaver or other module suite may be fully commercial, and distribution will be restricted. However, the firm intention is for the main Weaver program to remain freely distributable.. Only the modules themselves will require a licence fee for use. Developers interested in writing their own modules for MIDI or any other purpose, please contact the author.



Author:

				Pete Goodeve
				Berkeley, California

		e-mail:	pete@jwgibbs.cchem.Berkeley.EDU
				pete.goodeve@computer.org
========================