This is log of changes in the dabc project, which is started in winter 2013.
In some cases it is the best place to find documentation about current functionality
of the framework.
2.07.2013
1. Fix error in command channel - submitted commands not always was send to remote.
2. Tag version number 2.2.0 - this is mark for important milestone achieved.
1.07.2013
1. Move DABC/ROOT interface methods to DabcRoot class (instead of using
dabc_root namespace). It is necessary, while rootmap for libraries works only
when at least one class is specified. All methods in DabcRoot are static.
Actiavte http server in ROOT will look like:
root [0] DabcRoot::StartHttpServer(8095);
Actiavte connection to DABC master node:
root [1] DabcRoot::ConnectMaster("localhost:4444");
2. Adjustment to be able use RootSniffer and browser with go4.
3. Fix problem with thread usage when hierarchy was created. Now method really
called from the proper thread context and one not need to use locking.
28.06.2013
1. Implement automatic reconnect of command channel when server disappears.
2. Objects request via command channel works! It means, one could access data
from running ROOT analysis, which automatically connects to http server and
provides its hierarchy and objects via such connection. One could have many
such connection in the http server. This if first, but most important step to
control many-node application (ROOT, Go4, DABC) via single http server!
3. First real test to run two ROOT analsysis, connected to the same http server.
Organize better folders for browser, suppress main debug output.
4. Prepare DABC code for using within go4analysis. Idea to reuse RootSniffer
and just provide information from go4 object manager.
5. Today a first day when ROOT, Go4 and DABC with arbitrary combination of nodes
can provide data via central DABC web server to any normal browser like Mozilla
or IE or Chrome. This allows to build complex heterogene systems with unified
user monitoring and control interface! It took only 17 days to implement!
A huge amount of work need to be done, but all building blocks are already exist
and proved to be working.
27.06.2013
1. In http::Server add possibility to select part of objects hierarchy, displayed
in the browser. One also can specify how top folder is called. This allows in case
of ROOT browser shows only ROOT-relevant structures with name "ROOT".
2. Fix error in dabc.js, when object version number was kept in string form. In this
case comparasion between versions was not working correctly. One should use
explicit conversion to number like MyCall(Number(ver)).
3. Change functions name which must be called from ROOT script.
Now user should do from ROOT session:
root [0] gSystem->Load("libDabcRoot")
root [1] dabc_root::StartHttpServer(8095);
root [2] .x $ROOTSYS/tutorials/hsimple.C
Reduce debug output.
4. First try to connect ROOT analysis with DABC application, which has web server running.
With commands:
root [0] gSystem->Load("libDabcRoot")
root [1] dabc_root::ConnectMaster("localhost:4444");
root [2] .x $ROOTSYS/tutorials/hsimple.C
One create communication channel, which can be used from server side to browse
objects in the analysis. Theoretically, it should be transporent for the user
if objects exists in the same application as server or objects received from
such special channel from remote.
5. Fix error in version control of hierarchy. Now one really see in browser
all objects from slave process. Last small step remains -
provide getbinary request to remote node.
26.06.2013
1. Now is possible define master for any item in the hierarchy. Idea of master item is
that such master object should be updated together with item itself (when master is
changed). Server-client protocol will provide information about actual master version
as well.
2. Browser request streamer info only when we sure that streamer info is required.
Potentially many streamerinfos can be used - will be required when many ROOT analysis
connected to same http server.
3. First time continiousely running hsimple.C with web server!
Now arbitrary ROOT script can be monitored via web broser!
4. Add support of TProfile and, partially, TCanvas. TCanvas cannot be updated.
5. Provide only browsing for TTree/TBranch/TLeaf classes
6. When ROOT object requested from the browser, it is located again in ROOT structures.
This is needed while object may disapper and full hierarchy scan may take too much time.
Commands from browser executed about every 0.1 s, hierarchy update performed once a 3 s.
25.06.2013
1. Generate dictionary and rootmap for libDabcRoot. This will allow to use
part of DABC functionality directly from ROOT.
24.06.2013
1. Update hierarchy via xml file and build tree structure direct in the browser.
Use dtree.jquery for that. This will allow to fully customize look and fill
of the hierarchy browser. The only communication format between server and browser
will be xml.
2. Made hierarchy.htm as main DABC page, remove no longer used files.
3. Enable display of streamer infos, assign images for ROOT objects.
4. Smooth open/close logic, but it should be rewritten - dabc.js should
fully control itself how items are open or closed.
21.06.2013
1. Create BinDataContainer and BinData classes for manipuldation of
binary data in dabc::Hierarchy classes. Main motivation is to be able
from any thread keep reference on binary data disregard of changes in hierarchy.
BinData also simplifies version control of binary data.
2. In case of RootSniffer special container is created, which interfaces TBuffer
functionality. By that one do not require memcopy and duplication of memory.
3. Implement GetBinary command for RootSniffer. By that step one completely decouples
http::Server from ROOT functionality.
4. Now it is possible to draw root objects and ratemeters from hierarchy browser.
5. Make more generic method to generate StreamerInfos. Test TGraph.
6. First demonsrtation showing that simple redraw of the object is working.
7. Introduce proper version control of the hierarchy components.
For every element hash property could be set, which must be changed if object is changing.
If hash changed, version of that item will be changed. This change will be propogated
over all dependent hierarchy structures. When browser requests object, it provides
version of last instance it has. Server compare browser object version with
local object version and if they match, returns empty reply. If not, binary buffer
will be requested first from producer and with current version number provided to the
browser. Little bit complex, but prevent gui from unnecessary redraw and prevent server
from extra load if nothing is changing. To force object retrieving, one must specify
extremely high version number.
20.06.2013
1. Create DABC.Manager class for JavaScript applications (name is always the same :))
It should manage most (probably all) draw elements, produce requests and update
draw elements. DABC.DrawElement class contains full description of drawn element -
item name, kind, draw object and so on.
2. Create hierarchy of DABC.DrawElement classes to separate Gauge from Root canvas
from anything else. Only 'simple' classes handled in the main request,
rest will be treated by own-managed asynchronous requests.
3. Extract DABC-specific JavaScript code to dabc.js.
4. Implement binary requests, central binary callback and
first primitive version of drawing histogram from DABC browser!
19.06.2013
1. Change logic how hierarchy is managed. There is dabc::Object::BuildHierarchy()
method, which responsible to create hierarchy below. Method executed in thread
context of object.
18.06.2013
1. First example, which demonstrate how JSRootIO could work inside DABC.
rootio.htm page shows how object, generated inside DABC application,
can be streamed, reconstructed and displayed in web browser.
Code is very-very preliminary and done only as demonstrator. Special file
is required to provide streamer infos to the browser. Nevertheless, it is
prove of principal that web-server for arbitrary ROOT analysis can be implemented!
2. Found simple way to generate streamer infos on server side.
Therefore special ff.root file no longer required.
3. Several fixes in JSRootIO to be able work with dummy JSROOTIO.RootFile.
4. Introduce dabc_root::RootSniffer class, which will deliver ROOT objects hieararchy
to DABC framework.
17.06.2013
1. Principal possibility shown to transport ROOT-objects from DABC into browser
without intermediate file. Around such feature one could build application,
which can deliver any object from running ROOT code into web browser.
14.06.2013
1. First working ratemeter in the browser!
Very simple JustGage element was used, can be always replaced by more advanced
widget. Basic logic of widgets organisations and data exchange between
browser and server is exercised. Many other widget types (commands, info fields, ...)
can be done in similar way.
2. Version number switched to 2.1.1.
Odd second number indicates that now many new features can appear -
mostly in http interface.
13.06.2013
1. Put node hierarchy as separate request. In future can update it like gauge
periodically, one could easely insert hierarchy in many different pages.
2. Understand how correctly select items in hierarchy and how react on click.
12.06.2013
1. Implement html hierarchy display with jquery.treeview java script.
2. Make in http::Server internal cash of files like images or scripts.
11.06.2013
1. Mongoose web browser (from https://github.com/valenok/mongoose) integrated into
DABC. New libDabcHttp plugin is introduced, which should provide http protocol
to DABC applications. For a moment it is just helloword application, should
provide functionality similar to command shell, ncurses shell, go4-dabc browser.
2. Show application hierarchy in form of xml file.
10.06.2013
1. Minimalistic structure browser in dabc_shell. Menus from ncurses library are used.
Real functionality will be provided in parallel to other implementations.
Idea to show some rates, infos, submit commands in such shell.
7.06.2013
1. Introduce primitive command shell to remote dabc node. Via command channel
any kind of command can be executed.
2. First working code for transport and update of hierarchy structure over
command channel. By this one should be able to inspect any remote dabc node
in the future, even several of them simultaneous. This command shell will be
very minimalistic, more advanced text-based interface will be tested with ncurses.
Later plugin for go4 will be done!
3. Allow to transport raw data together with xml command description over command
channel. This allows potentially to transport any kind of data via the channel -
not only xml-based representation, which is not veru efficient for big raw objects.
6.06.2013
1. Implement unreliable (datagram) socket communication.
Idea to provide udp and multicast messages, where no any connections are required.
Usage of datagram socket does not require connection manager and no any kind of control.
Include appropriate example in net-test.
5.06.2013
1. Introduce new command channel, based on TCP sockets (old was using UDP). There is
one worker per connection and all commands to remote node are sequentialized via that worker.
At the moment functionality of old and new command channels are similar and they can be
easily replaced. In future command channel can be used to communicate with dabc
nodes from external application (command shell, go4 plugin and so on).
2. Introduce ModuleAsync::ActivateOutput/ActivateInput to generate events for
specified output/input ports. Required when processing of events was stopped
when ProcessRecv/ProcessSend methods return false.
3. ModuleAsync::TakeBuffer() not always requires handle to be created. In simple case
direct call to default memory pool (with name "Pool") will be performed. This should
exclude complex rules when handle is required or not. From this moment meaning of handle
is provide self-filled queue of buffers, which can be used like any other input port.
4.06.2013
1. Fix (implement) missing cleanup in several components - Port, PoolHandle, NetworkTransport.
Cleanup mostly required to correctly release cross-references before objects can be destroyed.
Without correct cleanup manager can not destroy referenced objects - in most cases MemoryPool.
31.05.2013
1. Provide version control of dabc::Hierarchy structure. Each element in hierarchy includes
version number, which indicate when last changes were performed. This version gives
possibility to extract only changed items relative to specified version in the past.
2. Implement conversion of dabc::Hierarchy structure into xml and reconstruct
hierarchy from xml. This will allow to transport and reconstruct hierarchy on
any remote location.
28.05.2013
1. Introduce dabc::Hierarchy class, which should represent in the future
objects hierarchy of remote (or local) DABC process.
27.05.2013
1. Activate shared-memory observer in hadaq plugin only when it really enabled
24.05.2013
1. Make doxygen docu more readable
21.05.2013
1. In mbs::ServerTransport implement correct stop of transport when EOF buffer has been seen.
2. Make transport kind of mbs::ServerTransport blocking by default. In this case every buffer
should be delivered to at least one client. If no one client is connected, transport will block.
One can specify unblocking mode for transport server if configure:
Stream server will be always unblocking.
17.05.2013
1. In input and output ports properties "rate" attribute can be specified.
When specified, data rate parameter with such name will be created and assigned to port.
2. Create dabc::RepeaterModule. It's only functionality - translate data from each input to
correspondent output. Thus number inputs and outputs should be the same.
3. Remove old latex-based documentation, start long transition to doxygen-based one.
At least 1.8.0 version of doxygen is required.
To generate documentation files, call "make doxy".
4. Produce 2.0.2 tag - currently working version of HADAQ readout
16.05.2013
1. Allow to specify auto="false" for every input/output port to avoid automatic port connection.
2. Adjust most of MBS examples to current modules creation logic.
Main advantage now - one do not need special init functions for different use-cases.
3. Remove mbs::GeneratorModule - one can use "lmd://Generator" input instead.
4. Implement of automatic reconnection of any module input/output.
For that "reconnect" attribute should be specified for the port:
Property specifies interval (in seconds) how often reconnect should be performed.
Reconnect will not be done when auto="false" specified.
5. Fix error that MemoryPool was stopped when no any connection was available.
Thus when new connection was coming, pool failed to provide
new buffer to modules and transports.
15.05.2013
1. Make more comfortable way to specify affinity of the threads and complete process.
2. Already before it was possible to specify taskset arguments for complete dabc process in
section in the form like . This is used in dabc_run script.
3. One can specify default affinity for every new created thread, providing "affinity"
parameter in section. Value can be following:
- unsigned value with processors mask
- string like "xxxoooxxxss" with allowed symbols 'x', 'o' and 's'.
'x' - enabled, 'o' - disabled, 's' - special purpose
first element in string corresponds to first processor
- string like "-N" where N is processors number which should be reserved
for special purposes. These processors could be later assigned
with SetAffinity("+M") call (M section of xml file:
Default kind of threads layout is balanced.
2. Allow to configure "timeout", "thread", "device", "useackn" and "timeout" attributes
for port or for connection. These attributes will be used for transport creation for
particular port. For instance, one could specify thread attribute:
to define which thread will be used for the transport.
13.05.2013
1. Add possibility to create devices and threads automatically via xml file.
Xml code should look like:
10.05.2013
1. Fix error in transport start/stop logic. Transport was started before connection to
the module was established, therefore many first buffers were lost. Was happened, while
transport was reacting on connection activating with memory pool.
2. Fix flushing error in hadaq::MbsTransmitterModule. When EOF marker was appearing,
rest of data in output iterator was not finalized and was not delivered to the output.
3. Unified typedefs and structures HADAQ between DABC and Stream analysis.
Idea to use as much as possible common code in both frameworks.
8.05.2013
1. Memory pool, modules and connections between them could be created automatically,
based on information from xml file. For module one should specify class tag.
In addition, thread tag can be provided. For instance:
...
Local connection between modules can be configured via simplified url:
...
...
As alternative, one could specify connection between modules in extra node:
hadaq plugin is first plugin with such examples.
2. Provide new configuration files for hadaq plugin, showing auto-creation functionality
3. Introduce BasicFile class, which only contains file descriptor and IO object.
Used as basic class for dabc::BinaryFile, mbs::LmdFileNew, hadaq::HldFile.
4. Change implementation of hadaq::HldFile file. Main change - use DABC IO class
to be able switch to any other file system. Also when reading events, first read
as much as possible and than rewind if non-complete events are read.
7.05.2013
1. Tag version 2.0.0 in repository.
3.05.2013
1. Introduce mbs::LmdFileNew class, which should replace LmdFile in next future. Idea
is to use same logic as with dabc::BinaryFile, where different low-level file I/O
can be used. Also like dabc::BinaryFile, only appropriate includes will be required
to use mbs::LmdFileNew classes outside DABC. One more argument for mbs::LmdFileNew -
reduce functionality of lmd storage to absolute minimum, remove never-used index table.
2. Provide LMD random generator input, which could be configured in form
It should replace MBS generator module in the near future.
3. Provide temporary "lmd2" file input and output classes, which are using
mbs::LmdFileNew. New lmd files can be used with go4 as well. Test it.
4. Fix error with MemoryPool::TakeEmpty - pool reference was not set correctly
in buffer, which leads to double delete of memory.
11.04.2013
1. Switch main DABC brunch to version 2.
22.03.2013
1. Make dabc::Manager class interface more clear, most functions must be used via
dabc::mgr instance - means via dabc::ManagerRef class
2. Introduce FileInput class, which handles common functionality for file inputs.
Reused in dabc::BinaryFileInput, mbs::LmdInput, hadaq::HldInput, mbs::TextInput
3. Introduce FileOutput class, which handles mots common tasks of file storage. Use
it for dabc::BinaryFileOutput, mbs::LmdOutput, hadaq::HldOutput.
21.03.2013
1. Introduce RFIO plugin in DABC2. Idea to have example how FileInterface can be
used. Not yet really tested.
20.03.2013
1. Reintroduce BinaryFile class to the DABC. It is generic storage for any kind of
dabc buffer. Now segmented buffers are supported. Most important that BinaryFile.h
include can be directly used in any project without need of any addition library.
One can even use ROOT CINT in the way like:
.L $DABCSYS/include/dabc/BinaryFile.h
dabc::BinaryFile f;
f.OpenReading("file.bin");
while (!f.eof()) {
uint64_t size = maxsize;
f.ReadBuffer(buf, size);
}
f.Close();
2. Replace FileIO class by FileInterface. FileInterface is wrapper for posix file
functions with very similar functions. Already FileInterface itself provides
basic IO functions and used in BinaryFile class. Later RFIO or ROOTD can be
implemented.
3. Implement BinaryFileOutput which can be work as generic output for buffers in
DABC. Syntax is "bin://test.bin?maxsize=30"
4. Replace const char* arguments in several Factory methods.
5. Implement BinaryFileInput.
6. Eliminate outdated FileIO classes, now everything goes via FileInterface.
Eliminate extra ListMatchFiles - implemented via FileIO.
19.03.2013
1. Modify dabc::Buffer class. Now it uses reference on MemoryPool to access
its functionality. When release buffer, memory pool mutex will be locked twice -
first time to release memory structures and second time to release reference
on pool itself. Solves very nasty problem during application cleanup,
when memory pool potentially could remains hanging.
2. Replace in many places const char* argument by const std::string&
3. Remove direct pointer on module from module item, module is always must be
direct parent of item.
15.03.2013
1. Make majority methods of Module and Port classes protected, some of them private.
Only reference classes will provide public, thread-safe methods.
2. Like Ports, now Timers and UserItems not accessible via pointer in the module.
From now only module methods should be used for any kind of send/recv/timer or
user event generation.
3. Remove InputPort.h OutputPort.h Timer.h PoolHandle.h files. All these classes are
now "hidden" from normal user and should not be directly appear in the code.
4. Make central dabc::Exception class with kind field to simplify handling of the exceptions.
14.03.2013
1. Change logic how factories are instantiated,
2. Remove libraries handles list from the factory, it was used for symbols finding.
3. Hide all PoolHandle methods from user,
only Module methods can be used for correspondent functionality
4. Remove direct usage of InputPort/OutputPort classes in user code - everything done
via indexes or names. Later classes will be removed completely. Still one need to
hide many module/ports methods
13.03.2013
1. Change syntax of DOUT1(("something")) to DOUT1("something"). As before, command will not be
compiled when make debug=0 command is used.
12.03.2013
1. Modify EZCA plugin - remove static variables, use DataInput interface in proper way,
improve compilation
2. Fix timeout probelms in DataTransports
3. Allow to change EZCA config parameters
4. Use EZCA group operations - should improve performance when using many variables.
5. Use mbs::WriteIterator in EZCA plugin, fix severeal errors in iterator itself.
6. Rebuild EZCA descriptors only when variables list changed - mean once at the beginning
7. Adjust simple go4 analysis in EZCA plugin to be able work with current readout.
11.03.2013
1. Next step to make PoolHandle like input port, which delivers buffers from the memory
pool. No any requests is necessary. Till now possible to request buffers directly from
the pool, but most of transport and standard modules will use new feature.
2. Continue API transition, step by step hide direct port methods from user.
8.03.2013
1. Rename PortBase -> Port class.
2. Modify BuffersQueue class, use it everywhere where it makes sencse.
3. Start transition to module API, where user do not have access to pointers to the
objects like I/O port, pool handle, timer and so on. Either port index or port name
should be used. Also all main methods like Send/Recv should be protected to avoid
usage from outside module code.
7.03.2013
1. First workign setup with MBS combiner module. Main change - do not block event loop by
event building forever. Use ProcessRecv/ProcessSend methods, they can reinject events itself.
2. Strictly check version 2 of xml file.
3. In mbs::ServerTransport correctly treat second request from the client
4. In epics plugin eliminate applications class - everything can be done with simple init
function. Simplify a lot class structure, still to be checked with epics.
5. Adjust mbs plugin, remove old examples and old eventapi
6. Remove old port, pool handle, transport and local transport classes.
6.03.2013
1. Adjust mbs plugin modules to new scheme, need to test them
2. First tests working with hadaq simulators!
3. Add possibility to balance performance between modules by limiting loop max count
per port. In this case ProcessRecv/ProcessSend calling sequence will be broken and
events are rescheduled. Probably, very important for hadaq
4. Debug and improve hadaq::CombinerModule and hadaq::UspTransport classes.
Now with minimal queue length everything works smoothly. Also correct treatment of
events id overflow is implemented.
5. Test hld and lmd output for hadaq event builder module - works!
6. In hadaq transport use simple recv() function - socket is already bind to the port.
5.03.2013
1. Change const char* to std::string in modules constructors and all related methods
2. Adjust hadaq plugin to new organization, need to be tested.
3. By default, module creates input and output ports and pool handle (if conbfigured)
4. Allow to put configuration parameters as node attributes in xml files like
Should significantly simplify xml files. Old syntax is also valid:
5. Start transition of roc::CombinerModule to new scheme, remove some old features.
4.03.2013
1. For time been include roc-udp plugin into dabc2 - just for convenience duting
migration. Convert UdpTransport to SocketAddon.
2. In NewDataInput allow to use queues of the buffers for transport addon.
Use such feature for the roc udp transport
3. First debugging of the roc-udp plugin - seems to be, it is working!
4. Eliminate SocketWorker classes - now all codes based on Addons.
1.03.2013
1. Refactor hadaq::UdpDataTransport - use InputTransport and SocketAddon instead.
Need to be verified with real setup.
28.02.2013
1. Debug network transport for socket, including acknowledge feature. Is working now.
2. Introduce verbs::WorkerAddon class, which replaces specialized verbs::Worker
Redo all relevant verbs classes. Adjust NetworkTransport for verbs.
27.02.2013
1. Refactor command channel, socket device, network transport and socket transport.
In all cases SocketAddon class used insted of SocketWorker.
Still a lot of debugging to be done.
26.02.2013
1. Significant redesign of mbs::ServerTransport. Now it consists from ServerAddon, which
handles connection and OutputAddon which handles data transfer to client.
Many clients organized as many slave connection to server transport. Number of connections
can be increased or descreased dynamically. Tested with go4 and with dabc client.
22.02.2013
1. Deliver disconnect events to module also in non-running state.
2. First working version of mbs::ClientTransport.
3. Split DataTransport on two independend InputTransport and OutputTransport.
Both classes should be used with DataInput/DataOutput classes.
4. Adjust DataInput interface to work not only with blocking interfaces but also
with call-back functionality - like DataInput already.
21.02.2013
1. Implementing full pallette of WorkerAddon for SocketThread. All new classes should
replace SocketWorker classes in dabc.
2. Rewrite mbs::ClientTransport class with SocketIOAddon. Try to use DataInputTransport.
Only implementation, not yet tested.
20.02.2013
1. First working implementation of NewDataTransport, based on NewTransport.
Here normal ModuleAsync is used to provide event loop, queues and so on.
Dramatically simplifies way how port and transport connected and work together.
2. First adoption of NewDataTransport to lmd file input/output.
First running example with such new transports.
3. Introduce di_DfltBufSize return value for DataInput object. Allows to use default
buffers, provided by the memory pool. Thus configuration can be shifted to memory pool.
4. Allow to specify transport kind directly in the xml file - via url tag.
By one string one can specify type of transport and basic parameters. Like
output lmd file is set with
19.02.2013
1. New event-generation model. Before each send/recv operation produces event,
which should be handled. This can leads to unlimited grow of thread events queue, if
event processing done not proper. To cover most typical tasks, new models for i/o ports
introduced. There are four models now:
enum EventsProducing {
SignalNone = 0, // port will not produce any events
SignalConfirm, // next event can be produced when previous event is received, confirmed and
// event consumer performed next operation
SignalOperation, // next event can be produced when event consumer performs next operation
SignalEvery // every operation will produce event
}
2. By default use SignalConfirm kind for i/o ports. This kind is useful when in
each event processsing function many buffers can be handled - is a situation in most cases.
3. Implement burst mode in memory pool. When module permanently requests buffers from the pool,
maximum number of buffers will be send.
4. Support of new pool handle in ModuleSync
15.02.2013
1. Introduce WorkerAddon class, which should contain all device-specific
code for sockets and IB.
2. MemoryPool class now derived from ModuleAsync - means just another source of
buffers. All requests now will be processed via ports, keep for the moment
direct TakeBuffer methods.
14.02.2013
1. Introduce InputPort and OutputPort
2. Introduce BufferQueue for connection between ports
3. Add support of new i/o ports into both modules kinds
4. Generate i/o events only when consumer react on previous event
5. Improve by factor 3 speed of buffers propogation over modules
6. Exclude mutex from queue when ports comunicating in the same thread
7. Use thread for ModuleItems only when required. For instance i/o ports not
necessary need to be assigned to the thread. Also in case of Timer,
one can use Module thread. This should simplify start/stop procedure.