[programmer/prog-exa-bnet.tex]
\label{prog_exabnet}
\section{Overview}
Complex experiments feature a lot of front-end systems running in parallel.
These take data and mark them with trigger information, or just with time stamps.
To completely analyze such data, all portions belonging to the same event
(or time stamp), must be combined in one processing unit.
Such task is usually called "event building".
To support event building functionality in \dabc, a special sub-framework called
BNET ("Building NETwork") was introduced.
Its main purpose is to simplify the experiment-specific implementation of
such event building, distributed over several network nodes.
A typical event building network contains several \strong{readout} nodes,
each connected to several data sources.
A readout node reads data from its data sources and combines
together data parts which logically belong together; this is called \strong{subevent building}.
In case of a triggered system it combines together data with the same trigger number;
in case of time-stamped data it combines together data which belongs to the same
time interval.
Because there are several readout nodes, building a complete event requires to
bring together all data of the same trigger number (or time interval, resp.)
into the same \strong{builder} node.
Typically the system has not only a single builder node,
but several of them; so full connectivity between all readouts and
all builder nodes is necessary.
Once all corresponding subevents have been delivered to the same builder node,
the complete event may be build and eventually stored on disk or tape.
For such use case,
the BNET framework defines a programming interface
to implement the functional units (i.~e.~ applications and modules),
and it already provides several important components.
BNET also defines the topology of these functional units
which can be customized up to a definite level.
\section{Controller application}
The event building task is usually distributed over several nodes
which must be controlled and synchronized.
Therefore in BNET all nodes are classified by their functionality
in two kinds: \strong{controller} node and
\strong{worker} nodes. Workers perform all
data transport, and run the (sub-)event building code.
The controller configures and steers all workers.
The controller node is implemented as \class{bnet::ClusterApplication} class.
Via the control system interface the cluster controller distributes commands
from the operator GUI to all workers.
It observes the state of all workers,
and may reconfigure them automatically if errors are detected.
The functionality of \class{bnet::ClusterApplication} is based on state-machine logic
of \dabc~.
All actions are performed during the execution of a state changing command, implemented
in virtual method \func{DoStateTransition()}.
A state transition on the cluster controller node means
that the appropriate state transition is performed on all worker nodes.
Technically speaking: a state machine command which is executed on the cluster
controller is only completed if the state transition commands on all workers are successfully completed. This
is implemented by means of class \class{dabc::CommandsSet}
(see method \func{StartClusterSMCommand()} for details).
Class \class{bnet::ClusterApplication} has following configuration parameters:
\begin{tabular}{llll}
\hline
Name & Type & Dflt & Description \\
\hline
\param{NetDevice} & str & dabc::SocketDevice & device class for network connections \\
\param{NumEventsCombine} & int & 1 & number of events (time frames) combined together \\
\param{TransportBuffer} & int & 8192 & size of buffer used for data transport cluster wide \\
\hline
\end{tabular}
Class \class{bnet::ClusterApplication} is fully functional and can be used as is for
a real cluster set-up.
\section{Worker application}
The basic functionality of a worker is implemented in \class{bnet::WorkerApplication} class.
Its main purpose is to instantiate, configure, connect, and run all working modules,
triggered by commands from the cluster controller.
Main functionality of \class{bnet::WorkerApplication} is implemented
in virtual method \func{CreateAppModules()} which is called during transition
from \keyw{Halted} to \keyw{Configured} state. In this method all local modules are instantiated
and configured. Some of these modules depend on the actual experiment, therefore
class \class{bnet::WorkerApplication} provides a number of virtual methods
to create experiment-specific components:
\bbul
\item \func{CreateCombiner()} - create a module combining several data sources to produce a subevent
\item \func{CreateBuilder()} - create a module which combines $N$ subevents to a complete event
\item \func{CreateFilter()} - optional filter module to filter out events
\item \func{CreateReadout()} - creates a readout transport connected to a data source
\item \func{CreateStorage()} - creates a storage transport to store data on disk/tape
\ebul
The user must define his/her own application class
which inherits from \class{bnet::WorkerApplication},
implementing these virtual methods.
Class \class{bnet::WorkerApplication} has the following "public"
configuration parameters:
\begin{tabular}{llll}
\hline
Name & Type & Dflt & Description \\
\hline
\param{IsGenerator} & bool & false & use generators instead of data sources \\
\param{IsSender} & bool & false & is sender module is created (readout functionality) \\
\param{IsReceiver} & bool & false & is receiver module is created (event builder functionality) \\
\param{IsFilter} & bool & false & is filter module is created (event builder should be true) \\
\param{NumReadouts} & int & 1 & number of data inputs \\
\param{Inpit0Cfg} & str & & string parameter to configure input 0 - user specific \\
\param{Inpit1Cfg} & str & & string parameter to configure input 1 and so on - user specific \\
\param{StoragePar} & str & & string parameter to configure storage - user specific \\
\param{ReadoutBuffer} & int & 2048 & buffer size, used for readout \\
\param{ReadoutPoolSize} & int & 4MB & size of memory pool for readout \\
\param{TransportPoolSize} & int & 16MB & size of memory pool for data transport \\
\param{EventBuffer} & int & 32768 & buffer size, used for event building \\
\param{EventPoolSize} & int & 4MB & size of memory pool for event building \\
\hline
\end{tabular}
There are also number of "private" parameters which
are not seen by control system and cannot be configured via XML file:
\begin{tabular}{llll}
\hline
Name & Type & Dflt & Description \\
\hline
\param{CfgNodeID} & int & & node id (starts from 1 for workers) \\
\param{CfgNumNodes} & int & & number of nodes in configuration \\
\param{CfgSendMask} & str & & string in form of "xxox" defines which nodes are sender "x" or not "o" \\
\param{CfgRecvMask} & str & & string in form of "xxox" defines which nodes are sender "x" or not "o" \\
\param{CfgClusterMgr} & str & & name of cluster controller node \\
\param{CfgNetDevice} & str & & name of configured network device, same as cluster param \param{NetDevice} \\
\param{CfgEventsCombine} & int & & number of events combined together, same as cluster param \param{NumEventsCombine} \\
\param{CfgReadoutPool} & str & & name of memory pool, used for readout ("ReadoutPool" or "TransportPool") \\
\param{CfgConnected} & bool & & true when local configuration of application completed \\
\hline
\end{tabular}
These parameters are set during initialization phase.
Some of them like \param{CfgEventsCombine} should be used by modules for it's configuration.
If required, the user subclass of \class{bnet::WorkerApplication}
may define additional configuration parameters.
\section{Combiner module}
The combiner module merges together several data sources and produces
subevent packets.
Here a "subevent" means that data from all sources which belong to the same
event (or time frame, resp.) are put
into the same \class{dabc::Buffer} object.
This buffer object should have a header with a unique identifier of type
\decl{bnet::EventId}; this is a 64-bit unsigned integer.
\begin{small}
\begin{verbatim}
...
dabc::Buffer* buf = fPool->TakeBuffer(bufsize);
buf->SetHeaderSize(sizeof(bnet::EventId));
*((bnet::EventId*) buf()->GetHeader()) = evid++;
...
\end{verbatim}
\end{small}
The subevent identifier number should
be subsequently increasing without a gap. When
no data for the current identifier is available, an empty buffer with no data and
correct header must be delivered to the output.
Class \class{bnet::CombinerModule} provides the prototype
of a combiner module. It uses the following single parameter:
\begin{tabular}{llll}
\hline
Name & Type & Dflt & Description \\
\hline
\param{NumReadouts} & int & 1 & number of data inputs \\
\hline
\end{tabular}
Actually, parameter \param{NumReadouts} may not be defined in the configuration of the module itself.
Since class \class{bnet::WorkerApplication} already has a parameter of such name,
its value will be directly used for the module configuration.
When implementing an experiment-specific combiner class,
one should either derive it from \class{bnet::CombinerModule} class,
or start "from scratch" by
subclassing \class{dabc::ModuleSync} or \class{dabc::ModuleAsync}.
One may add more experiment-specific
parameters to the module.
\section{Network topology}
The connection topology of the event building network is defined by parameters
\param{IsSender} and \param{IsReceiver} of \class{bnet::WorkerApplication}.
These parameters configure the roles of each worker node:
\bbul
\item collector of data from data source(s) and sender to event builder
\item receiever of data from collectors and builder of complete events
\item both functions at the same application
\ebul
It is required that at least one of both parameters has a \keyw{true} value.
During configuration,
the cluster controller establishes the connections between the workers such,
that each sender module is connected with all receiver modules. This guarantees
that each receiever node gets data from all sources,
necessary to perform the full event building.
The two classes \class{bnet::SenderModule}, and \class{bnet::ReceiverModule},
implement the functionality of data sender, and data receiver, respectively.
These classes are instantiated by
\class{bnet::WorkerApplication} and should not be modified by the user.
The subevents buffers, as produced by the combiner module, are delivered
to the sender module. Based on the event identifier, the buffer is
send to that specific receiver where the event with such id will be build.
For now a simple round-robin schedule is used by BNET, but in next
\dabc~ versions one or several other data transfer schedules will be implemented.
One idea of the BNET framework is that such improvements are possible without
changing the user application code.
\section{Event builder module}
The task of the receiver module is to collect all buffers of the
same event identifier and
deliver them at once to the event builder module.
To define an experiment-specific builder module, one can either derive it from
\class{bnet::BuilderModule} class, or implement it "from scratch"
by subclassing \class{dabc::ModuleSync} or \class{dabc::ModuleAsync}.
The event builder module has one input and one output. Over the input port
it gets
$N$ buffers with subevents for the same event identifier.
Over the output port it should deliver one
buffer with build events.
When the user inherits his/her builder module from \class{bnet::BuilderModule},
it is enough to implement the virtual \func{DoBuildEvent()} method,
which gets as argument a list of $N$ buffers with subevents.
The format of the output buffer is completely user-defined.
It is allowed to fill several events into the same output buffer if necessary.
\section{Filter module}
This is an optional component of BNET if build events shall be filtered
before they are stored. To implement such filter,
one can derive it from \class{bnet::FilterModule} and reimplement virtual method
\func{TestBuffer()}. As an alternative, filtering can be implemented directly in the
event builder module.
\section{BNET test application}
\label{prog_exabnet_test}
This application may test different asspects of a BNET
without the necessity to have real data sources. The complete source code
and configuration
examples can be found in {\tt \$DABCSYS/applications/bnet-test} directory.
The example contains following classes:
\bbul
\item \class{bnet::TestGeneratorModule}
\item \class{bnet::TestCombinerModule}
\item \class{bnet::TestBuilderModule}
\item \class{bnet::TestFilterModule}
\item \class{bnet::TestWorkerApplication}
\item \class{bnet::TestFactory}
\ebul
There are several examples of configuration files. For instance,
the configuration of 4 worker nodes with sender and receiver functionality each
is shown in {\tt SetupBnet.xml}:
\begin{small}
\begin{verbatim}
\end{verbatim}
\end{small}
Here one can see cluster controller apllication in the beginning, configured to
use \class{dabc::SocketDevice} for workers connections. And there are four workers
with the same configurations parameters which can be found in section.
In section one sees,
that \param{IsGenerator}, \param{IsSender} and \param{IsReceiver} parameters are
all set to \keyw{true}.
This defines the so-called "all-to-all" topology, i.~e.~ each node communicates with all other nodes including itself.
Parameter \param{NumReadouts}=4 means that there are 4 inputs on each combiner,
resulting in 16 data sources for the complete system.
To run this example, one should specify correct host names for all contexts and
start it with \verba{run.sh SetupBnet.xml} command.
This can be used as template for developing a user-specific application.
One can change functionality of combiner and builder modules, and provide a
real readout instead of the generator module.
\section{BNET for MBS application}
\label{prog_exabnet_mbs}
This is a ready-ro-use implementation of distributed event building for MBS.
The source code can be found in {\tt \$DABCSYS/plugins/bnet-mbs} directory.
It contains following classes:
\bbul
\item \class{bnet::MbsCombinerModule}
\item \class{bnet::MbsBuilderModule}
\item \class{bnet::MbsWorkerApplication}
\item \class{bnet::MbsFactory}
\ebul
Class \class{bnet::MbsCombinerModule} combines together
events with the same event identifier from all inputs.
In the cluster application parameter \param{NumEventsCombine} defines how many
events should be bundled together in one buffer. It is crucial that transport
buffer size is big enough for such number of subevents. During initialisation,
cluster parameter \param{NumEventsCombine} is copied to each
worker parameter \param{CfgEventsCombine}, which is finally used in \class{bnet::MbsCombinerModule}:
\begin{small}
\begin{verbatim}
bnet::MbsCombinerModule::MbsCombinerModule(...
...
fCfgEventsCombine = GetCfgInt(CfgEventsCombine, 1, cmd);
...
\end{verbatim}
\end{small}
For the moment \class{bnet::MbsCombinerModule} skips an event,
if it is not present on all local data inputs.
Class \class{bnet::MbsBuilderModule} builds MBS events from
the buffers as delivered by the receiver module. It also takes
application parameter \param{CfgEventsCombine} to tell how many real MBS events
are contained in the input buffers.
Application class \class{bnet::MbsWorkerApplication} implements several methods to
correctly instantiate combiner and builder modules.
It also implements virtual method \func{CreateReadout()}, where the input transport
for the combiner module is created. In case of MBS there are three possibilities:
\bnum
\item when \param{IsGenerator}=true module \class{mbs::GeneratorModule} connected as data input
\item when the appropriate readout parameter (like \param{Input0Cfg} for the first input)
is a filename with ".lmd" suffix, the specified file will be used as data input
\item otherwise, the value of readout parameter (like \param{Input0Cfg}) will be used as MBS server name for connecting of \class{mbs::ClientTransport} to the appropriate data input
\enum
In virtual method \func{CreateStorage()} an output {\tt\*.lmd} file will be created, if
parameter \param{StoragePar} value is not empty.
Example file {\tt \$DABCSYS/applications/bnet-mbs/SetupBnetMbs.xml}
shows the configuration for an MBS event building with 2 readout nodes, connected with 2 generators each and 2 event builder nodes.
This configuration file can be customised via {\tt } definitions
in the beginning:
\begin{small}
\begin{verbatim}
...
\end{verbatim}
\end{small}
Here \keyw{node0} specifies the node where the cluster controller will run, \keyw{node1} and \keyw{node3} are used as readout nodes,
\keyw{node2} and \keyw{node4} as builder nodes. On all four worker nodes one MBS generator
application will be started. To run the application, just type \verba{run.sh SetupBnetMbs.xml}.