[programmer/prog-exa-bnet.tex]
\label{prog_exabnet}
\section{Overview}

Complex experiments feature a lot of front-end systems running in parallel.
These take data and mark them with trigger information, or just with time stamps.
To completely analyze such data, all portions belonging to the same event 
(or time stamp), must be combined in one processing unit. 
Such task is usually called "event building".

To support event building functionality in \dabc, a special sub-framework called 
BNET ("Building NETwork") was introduced. 
Its main purpose is to simplify the experiment-specific implementation of  
such event building, distributed over several network nodes.

A typical event building network contains several \strong{readout} nodes, 
each connected to several data sources.
A readout node reads data from its data sources and combines
together data parts which logically belong together; this is called \strong{subevent building}.
In case of a triggered system it combines together data with the same trigger number;
in case of time-stamped data it combines together data which belongs to the same 
time interval. 
Because there are several readout nodes, building a complete event requires to 
bring together all data of the same trigger number (or time interval, resp.) 
into the same \strong{builder} node. 
Typically the system has not only a single builder node, 
but several of them; so full connectivity between all readouts and 
all builder nodes is necessary.
Once all corresponding subevents have been delivered to the same builder node, 
the complete event may be build and eventually stored on disk or tape. 

For such use case,
the BNET framework defines a programming interface 
to implement the functional units (i.~e.~ applications and modules), 
and it already provides several important components.
BNET also defines the topology of these functional units
which can be customized up to a definite level.


\section{Controller application}

The event building task is usually distributed over several nodes 
which must be controlled and synchronized.
Therefore in BNET all nodes are classified by their functionality 
in two kinds: \strong{controller} node and
\strong{worker} nodes. Workers perform all 
data transport, and run the (sub-)event building code. 
The controller configures and steers all workers. 

The controller node is implemented as \class{bnet::ClusterApplication} class.
Via the control system interface the cluster controller distributes commands
from the operator GUI to all workers. 
It observes the state of all workers, 
and may reconfigure them automatically if errors are detected.

The functionality of \class{bnet::ClusterApplication} is based on state-machine logic 
of \dabc~.
All actions are performed during the execution of a state changing command, implemented  
in virtual method \func{DoStateTransition()}. 
A state transition on the cluster controller node means
that the appropriate state transition is performed on all worker nodes.
Technically speaking: a state machine command which is executed on the cluster 
controller is only completed if the state transition commands on all workers are successfully completed. This
is implemented by means of class \class{dabc::CommandsSet}  
(see method \func{StartClusterSMCommand()} for details).  
  
Class \class{bnet::ClusterApplication} has following configuration parameters:

\begin{tabular}{llll}
\hline
Name &  Type &  Dflt & Description  \\
\hline
\param{NetDevice}           & str  & dabc::SocketDevice  &  device class for network connections \\
\param{NumEventsCombine}    & int  & 1      &  number of events (time frames) combined together  \\   
\param{TransportBuffer}     & int & 8192  &  size of buffer used for data transport cluster wide \\
\hline
\end{tabular}

Class \class{bnet::ClusterApplication} is fully functional and can be used as is for 
a real cluster set-up.
 
 
\section{Worker application}
 
The basic functionality of a worker is implemented in \class{bnet::WorkerApplication} class.
Its main purpose is to instantiate, configure, connect, and run all working modules,
triggered by commands from the cluster controller. 


Main functionality of \class{bnet::WorkerApplication} is implemented 
in virtual method \func{CreateAppModules()} which is called during transition 
from \keyw{Halted} to \keyw{Configured} state. In this method all local modules are instantiated 
and configured. Some of these modules depend on the actual experiment, therefore 
class \class{bnet::WorkerApplication} provides a number of virtual methods
to create experiment-specific components:
\bbul
\item \func{CreateCombiner()}  - create a module combining several data sources to produce a subevent 
\item \func{CreateBuilder()}   - create a module which combines $N$ subevents to a complete event 
\item \func{CreateFilter()}    - optional filter module to filter out events 
\item \func{CreateReadout()}   - creates a readout transport connected to a data source
\item \func{CreateStorage()}   - creates a storage transport to store data on disk/tape 
\ebul

The user  must define his/her own application class
which inherits from \class{bnet::WorkerApplication},
implementing these virtual methods.


Class \class{bnet::WorkerApplication} has the following "public" 
configuration parameters:

\begin{tabular}{llll}
\hline
Name &  Type &  Dflt & Description  \\
\hline
\param{IsGenerator}    & bool & false  &  use generators instead of data sources  \\   
\param{IsSender}       & bool & false  &  is sender module is created (readout functionality)  \\   
\param{IsReceiver}     & bool & false  &  is receiver module is created (event builder functionality)  \\   
\param{IsFilter}       & bool & false  &  is filter module is created (event builder should be true)  \\   
\param{NumReadouts}    & int  & 1      &  number of data inputs  \\   
\param{Inpit0Cfg}      & str  &       &  string parameter to configure input 0 - user specific \\   
\param{Inpit1Cfg}      & str  &       &  string parameter to configure input 1 and so on - user specific  \\   
\param{StoragePar}         & str  &       &  string parameter to configure storage - user specific  \\   
\param{ReadoutBuffer}      & int  & 2048  &  buffer size, used for readout  \\   
\param{ReadoutPoolSize}    & int  & 4MB  &  size of memory pool for readout  \\   
\param{TransportPoolSize}  & int  & 16MB  &  size of memory pool for data transport  \\   
\param{EventBuffer}        & int  & 32768  &  buffer size, used for event building  \\   
\param{EventPoolSize}      & int  & 4MB  &  size of memory pool for event building  \\   
\hline
\end{tabular}

There are also number of "private" parameters which
are not seen by control system and cannot be configured via XML file:

\begin{tabular}{llll}
\hline
Name &  Type &  Dflt & Description  \\
\hline
\param{CfgNodeID}    & int &  &  node id (starts from 1 for workers)  \\   
\param{CfgNumNodes}  & int &  &  number of nodes in configuration  \\   
\param{CfgSendMask}  & str & &  string in form of "xxox" defines which nodes are sender "x" or not "o" \\   
\param{CfgRecvMask}  & str & &  string in form of "xxox" defines which nodes are sender "x" or not "o"  \\   
\param{CfgClusterMgr} & str  & & name of cluster controller node \\   
\param{CfgNetDevice}   & str  & & name of configured network device, same as cluster param \param{NetDevice} \\   
\param{CfgEventsCombine}  & int  &  & number of events combined together, same as cluster param \param{NumEventsCombine}  \\   
\param{CfgReadoutPool}    & str &  &  name of memory pool, used for readout ("ReadoutPool" or "TransportPool")  \\   
\param{CfgConnected}      & bool &  &  true when local configuration of application completed  \\   
\hline
\end{tabular}
 
These parameters are set during initialization phase.
Some of them like \param{CfgEventsCombine} should be used by modules for it's configuration.  

If required, the user subclass of \class{bnet::WorkerApplication}
may define additional configuration parameters.


\section{Combiner module}

The combiner module merges together several data sources and produces 
subevent packets.
Here a "subevent" means that data from all sources which belong to the same
event (or time frame, resp.) are put
into the same \class{dabc::Buffer} object.
This buffer object should have a  header with a unique identifier of type
\decl{bnet::EventId}; this is a 64-bit unsigned integer. 

\begin{small}
\begin{verbatim}
  ...
  dabc::Buffer* buf = fPool->TakeBuffer(bufsize);
  buf->SetHeaderSize(sizeof(bnet::EventId));
   *((bnet::EventId*) buf()->GetHeader()) = evid++;
  ...
\end{verbatim}
\end{small}

The subevent identifier number should 
be subsequently increasing without a gap. When
no data for the current identifier is available, an empty buffer with no data and
correct header must be delivered to the output.

Class \class{bnet::CombinerModule} provides the prototype 
of a combiner module. It uses the following single parameter:

\begin{tabular}{llll}
\hline
Name &  Type &  Dflt & Description  \\
\hline
\param{NumReadouts}    & int  & 1   &  number of data inputs  \\   
\hline
\end{tabular}

Actually, parameter \param{NumReadouts} may not be defined in the configuration of the module itself.
Since class \class{bnet::WorkerApplication} already has a parameter of such name,
its value will be directly used for the module configuration.     

When implementing an experiment-specific combiner class, 
one should either derive it from \class{bnet::CombinerModule} class,
or start "from scratch" by
subclassing \class{dabc::ModuleSync} or \class{dabc::ModuleAsync}.
One may add more experiment-specific
parameters to the module. 


\section{Network topology}

The connection topology of the event building network is defined by parameters 
\param{IsSender} and \param{IsReceiver} of \class{bnet::WorkerApplication}. 
These parameters configure the roles of each worker node:
\bbul
\item collector of data from data source(s) and sender to event builder
\item receiever of data from collectors and builder of complete events
\item both functions at the same application  
\ebul
It is required that at least one of both parameters has a \keyw{true} value.
During configuration,
the cluster controller establishes the connections between the workers such,
that each sender module is connected with all receiver modules. This guarantees 
that each receiever node gets data from all sources,
necessary to perform the full event building.

The two classes \class{bnet::SenderModule}, and \class{bnet::ReceiverModule},
implement the functionality of data sender, and data receiver, respectively. 
These classes are instantiated by
\class{bnet::WorkerApplication} and should not be modified by the user.     

The subevents buffers, as produced by the combiner module, are delivered 
to the sender module. Based on the event identifier, the buffer is 
send to that specific receiver where the event with such id will be build. 
For now a simple round-robin schedule is used by BNET, but in next 
\dabc~ versions one or several other data transfer schedules will be implemented.
One idea of the BNET framework is that such improvements are possible without
changing the user application code.  


\section{Event builder module}

The task of the receiver module is to collect all buffers of the 
same event identifier and
deliver them at once to the event builder module.

To define an experiment-specific builder module, one can either derive it from
\class{bnet::BuilderModule} class, or implement it "from scratch"
 by subclassing \class{dabc::ModuleSync} or \class{dabc::ModuleAsync}. 
The event builder module has one input and one output. Over the input port 
it gets
$N$ buffers with subevents for the same event identifier. 
Over the output port it should deliver one
buffer with build events. 
 
When the user inherits his/her builder module from \class{bnet::BuilderModule},
it is enough to implement the virtual \func{DoBuildEvent()} method, 
which gets as argument a list of $N$ buffers with subevents. 
The format of the output buffer is completely user-defined. 
It is allowed to fill several events into the same output buffer if necessary.    


\section{Filter module}

This is an optional component of BNET if build events shall be filtered
before they are stored. To implement such filter, 
one can derive it from \class{bnet::FilterModule} and reimplement virtual method
\func{TestBuffer()}. As an alternative, filtering can be implemented directly in the 
event builder module.  


\section{BNET test application}
\label{prog_exabnet_test}
This application may test different asspects of a BNET
without the necessity to have real data sources. The complete source code 
and configuration
examples can be found in {\tt \$DABCSYS/applications/bnet-test} directory.  

The example contains following classes:
\bbul
\item  \class{bnet::TestGeneratorModule}
\item  \class{bnet::TestCombinerModule}
\item  \class{bnet::TestBuilderModule}
\item  \class{bnet::TestFilterModule}
\item  \class{bnet::TestWorkerApplication}
\item  \class{bnet::TestFactory}
\ebul

There are several examples of configuration files. For instance,
the configuration of 4 worker nodes with sender and receiver functionality each
is shown in {\tt SetupBnet.xml}:

\begin{small}
\begin{verbatim}
<?xml version="1.0"?>
<dabc version="1">
  <Context host="lxi008" name="Controller:41">
    <Run>
      <runfunc value="RunTestBnet"/>
    </Run>
    <Application class="bnet::Cluster">
       <NetDevice value="dabc::SocketDevice"/>
    </Application>
  </Context>
  <Context host="lxi009" name="Worker1:42"/>
  <Context host="lxi010" name="Worker2:42"/>
  <Context host="lxi011" name="Worker3:42"/>
  <Context host="lxi012" name="Worker4:42"/>
  <Defaults>
    <Context name="*">
      <Run>
        <logfile value="test${DABCNODEID}.log"/>
        <loglevel value="1"/>
        <lib value="libDabcBnet.so"/>
      </Run>
    </Context>
    <Context name="Worker*">
       <Run>
          <lib value="${DABCSYS}/applications/bnet-test/libBnetTest.so"/>
       </Run>
       <Application class="bnet::TestWorker">
         <IsGenerator value="true"/>
         <IsSender value="true"/>
         <IsReceiver value="true"/>
         <NumReadouts value="4"/>
      </Application>       
    </Context>
  </Defaults>
</dabc>
\end{verbatim}
\end{small}

Here one can see cluster controller apllication in the beginning, configured to
use \class{dabc::SocketDevice} for workers connections. And there are four workers
with the same configurations parameters which can be found in <Defaults> section.
In section <Context name="Worker*"> one sees, 
that \param{IsGenerator}, \param{IsSender} and \param{IsReceiver} parameters are 
all set to \keyw{true}.
This defines the so-called "all-to-all" topology, i.~e.~ each node communicates with all other nodes including itself.   
Parameter \param{NumReadouts}=4 means that there are 4 inputs on each combiner, 
resulting  in 16 data sources for the complete system.

To run this example, one should specify correct host names for all contexts and 
start it with \verba{run.sh SetupBnet.xml} command.      

This can be used as template for developing a user-specific application.
One can change functionality of combiner and builder modules, and provide a 
real readout instead of the generator module. 


\section{BNET for MBS application}
\label{prog_exabnet_mbs}
This is a ready-ro-use implementation of distributed event building for MBS.
The source code can be found in {\tt \$DABCSYS/plugins/bnet-mbs} directory.
It contains following classes:
\bbul
\item  \class{bnet::MbsCombinerModule}
\item  \class{bnet::MbsBuilderModule}
\item  \class{bnet::MbsWorkerApplication}
\item  \class{bnet::MbsFactory}
\ebul

Class \class{bnet::MbsCombinerModule} combines together  
events with the same event identifier from all inputs. 
In the cluster application parameter \param{NumEventsCombine} defines how many
events should be bundled together in one buffer. It is crucial that transport
buffer size is big enough for such number of subevents. During initialisation,
cluster parameter \param{NumEventsCombine}  is copied to each
worker parameter \param{CfgEventsCombine}, which is finally used in \class{bnet::MbsCombinerModule}: 
   
\begin{small}
\begin{verbatim}
bnet::MbsCombinerModule::MbsCombinerModule(...  
  ...
  fCfgEventsCombine = GetCfgInt(CfgEventsCombine, 1, cmd);
  ...
\end{verbatim}
\end{small}

For the moment \class{bnet::MbsCombinerModule} skips an event, 
if it is not present on all local data inputs. 

Class \class{bnet::MbsBuilderModule} builds MBS events from
the buffers as delivered by the receiver module. It also takes 
application parameter \param{CfgEventsCombine} to tell how many real MBS events
are contained in the input buffers.
  
Application class \class{bnet::MbsWorkerApplication} implements several methods to
correctly instantiate combiner and builder modules.
It also implements virtual method \func{CreateReadout()}, where the input transport 
for the combiner module is created. In case of MBS there are three possibilities:
\bnum
\item when \param{IsGenerator}=true module \class{mbs::GeneratorModule} connected as data input 
\item when the appropriate readout parameter (like \param{Input0Cfg} for the first input) 
is a filename with ".lmd" suffix, the specified file will be used as data input
\item otherwise, the value of readout parameter (like \param{Input0Cfg}) will be used as MBS server name for connecting of \class{mbs::ClientTransport} to the appropriate data input         
\enum
In virtual method \func{CreateStorage()} an output {\tt\*.lmd} file will be created, if 
parameter \param{StoragePar} value is not empty.

Example file {\tt \$DABCSYS/applications/bnet-mbs/SetupBnetMbs.xml}
shows the configuration for an MBS event building with 2 readout nodes, connected with 2 generators each and 2 event builder nodes. 
This configuration file can be customised via {\tt <Variables>} definitions 
in the beginning:
\begin{small}
\begin{verbatim}
<?xml version="1.0"?>
<dabc version="1">
  <Variables>
     <node0 value="lxi008"/>
     <node1 value="lxi009"/>
     <node2 value="lxi010"/>
     <node3 value="lxi011"/>
     <node4 value="lxi012"/>
     <custport value="16015"/>
  </Variables>
  ...
</dabc>
\end{verbatim}
\end{small}

Here \keyw{node0} specifies the node where the cluster controller will run, \keyw{node1} and \keyw{node3} are used as readout nodes,
\keyw{node2} and \keyw{node4} as builder nodes. On all four worker nodes one MBS generator
application will be started. To run the application, just type \verba{run.sh SetupBnetMbs.xml}.  
