This is log of changes in the dabc project, which is started in winter 2013. In some cases it is the best place to find documentation about current functionality of the framework. 10.10.2013 1. Small changes in readme. 2. Create 203-05 tag. 9.10.2013 1. Implement Start/Stop HldFile commands in hadaq::CombinerModule. This gives possibility to start HLD files from browser 2. In CommandDef use RecordField class to specify dflt, min, max values for arguments. 3. Add in dabc.js, CommandDrawElement support for min,max, dflt values. Support simple string input for command argument. 4. Provide possibility to configure info parameter for module, port, transport. This parameter can be used for debug or error output. Aim to replase DOUT/EOUT by IOUT macros, where all advantages for conditional compilation can be used. 5. Remove arguments from DataOutput::Write_Init() method. Now all transport parameters should be extracted from url string. 6. Provide info parameter in the hadaq transmitter module. 8.10.2013 1. Small adjustment in JSRootIO - block context menu when touch selection is active. Also adjust css file. 2. Remove most debug ouput in DabcRoot plugin. Debug ouput can be enabled via DabcRoot::EnableDebug() method. 7.10.2013 1. Create hsimple.htm for fast JSRootIO tests - it reads hsimple.root file and shows automatically hpx histogram. 2. Introduce new handling of touches events - one could react on double-tap and selection with two fingers. Not clear why coordinates in d3.touches are wrong, need to verify on iOS. 4.10.2013 1. Allow to publish normal parameters in hierarchy. Parameters remain as they are and just all changes are recorded in hierarchy, which can be published under any path (not necessary under /DABC folder). Thus all low-level parameter functionality remains as it was before. 2. Optiomize parameter events generation. Now each events from parameter to worker should be confirmed. Also if parameter changed in worker thread, it will be immediately processed - no any event submission/processing. Same should be done in the future for global parameter events. 2.10.2013 1. Small update of doxygen documentaion, add ROOT plugin info. 2. Create intermediate 2.3.5 version tag. 30.09.2013 1. Implement primitive way to set arguments for the command. At the moment only int argument are supported. 2. Also result of command execution is returned. At the moment full command is stored in form of xml. 3. Full functionality (with typing of arguments, ranges, info, return values) will be implemented after clear formatting rule will be defined. 27.09.2013 1. Introduce CommandDef class. It will be used to publish command with different arguments, which than can be executed by module where command was published 2. Provide primary implementation of execute query like: http://dabc_server:port/FESA/Test/CmdReset/execute At the first stage command created, based on its definition. Than it is submitted to the producer. Theoretically, any number of arguemnts and returns value can be implemented in the future. 3. Provide preliminary draw element for command execution in dabc.js 4. Delete CommandsSet class, any complex commands should be implemented individually. 26.09.2013 1. Generate 'hash' field for the binary data only when binary data is requested. If next request comes, new hash is calculated and compared with previous one. Only if hash changed, new binary data is generated and returned. 'hash' attributes will be no longer propogated via names lists. 2. In JavaScript code make axis time format more like to original ROOT graphic. 3. After that when trying to open correspondent page in browser, one will be requested to type user and password for "your@domain". Was tested with Firefox. 2. Small fix in mongoose.c to be able open file when creating context. 3. Switch to mongoose 4.2 version, one problem is resolved there, another need to be fixed 4. Fix few issues with mongoose 4.2 version, hope to get them fixed soon. 5. Try to use https protocol. For that two parameters should be provided in config file: After that one could use address like: https://hostname:8091 Creation of valid certificate is still open question - fake certificate from mongoose web site IE does not like, but works 6. Link libDabcFesa with libDabcRoot directly when ROOT is used for compilation 23.09.2013 1. First try to implement display of single element without hierarchy on the left side. Works with several workarounds, which should be cleaned up later. Now one can specify as url name of element like: http://dabc_server:port/FESA/Test/BeamRate/ With additional switches one can enable/disable history view for the elements. 2. In different tooltips use full path to the element - this is more clear than relative path to current page. 3. Fix small problem with log history display. 19.09.2013 1. Change the way how hierarchy items addressed via http interface. Now each subdirectory in hierarchy can be addressed and disaplyed just like top level directory. For instance, if there is 'FESA' subdirectory, its content can be displayed like: http://dabc_server:port/FESA/ It is important to specify slash at the end. 2. To get hierarchy below specified selected item, just do: wget http://dabc_server:port/FESA/h.xml 3. To get value of selected item, one should do: wget http://dabc_server:port/FESA/Test/BeamRate/get.xml One could specify version number and history length 4. There is also "get.bin" and "image.png" - of course, it only will work for items which really provide binary data. 5. Small changes in mongoose server to avoid that it reads files - only http::Server class will do files reading from preselected directories. 6. In dTree add possibility to directly open subfolders (by clicking item) 18.09.2013 1. When reading hierarchy data from binary buffer, one can avoid most part of decoding. Feature is used, when data series only for concrete element is necessary. This is possible while every object (like fields map, history, hierarchy element) is streamed in standard way - size of object is always written in the front. Therefore to skip data for such object it is enough to read size and jump over data. 2. For series extraction from files no-delete flag implemented for hierarchy streamer. In such case none of the existing elements in hierarchy will be deleted. 3. Implementation of HierarchyReading is in very experimental state and requires good user interface for testing. But major functions are there. Open question is how to map all this story to SQL databases? 17.09.2013 1. Implement FileInterface::mkdir method, which able to create new directory, including all missing parents directories. Normally one should not use ".." symbol inside full path. 2. Very first implementation of hieararchy storage in binary files. At the moment storage is triggered by the Publisher, but performed in module where hierarchy is created. One can specify interval how often storage file should be recreated and full hierarchy must be dump again. 3. Start implementation of HierarchyReading class to read data, stored in by HierarchyStorage 16.09.2013 1. Optimize LogDrawElement - it is now derived from HistoryDrawElement 2. Introduce GenericDrawElement - text output of all availible fields in element. Used when kind of the element is not specified. 3. In fesa::Monitor set dabc:error field when any error is detected. 4. Use JavaSscript-like date format in record to store date and time. It is number of milliseconds since 1.1.1970 in GMT. When converted into string (xml), human-readable format (ISO 8601) is used. It is like '2013-09-16T12:42:30.884Z' and directly can be interpreted in the JavaScript, using 'Date' class. 5. Introduce dabc::HierarchyStore class which will be used to write hierarchy log into the binary files. 11.09.2013 1. Major functionality of managing of published hierarchies implemented in dabc::Publisher class. It creates and updates central hierarchy (only nameslist), which is later used to discover which entries are availible and to which modules/workers they are belong. 2. Every slave connection provide dabc::Publisher with remote hierarchy, which mare anaged separately. As result, dabc::Publisher has N separate hierarchies, where top branches like /MBS or /FESA or /Go4 can be the same. Here main task of publisher correctly merge them in one big structure. 3. dabc::PublisherRef::Get method implemented, which runs synchron (blocking) and return all fields of requested entry. Behind the scene publisher first identify node and module where entry is published, than redirect command to the module and finally in respective module thread command is executed. 4. dabc::PublisherRef::Get also can be used to request recorded history for the entry. The result returned in binary form and can be used for analysis. 5. dabc::PublisherRef::GetBinary implemented to provide special data which is not published as field. 6. As result, hierarchy management moved from http::Server and can be used in different parts of application code. In the future subscription (asynchron) modus will be implemented. 7. Remove old code with hierarchy, created in command channel and with "GetBinary" request. Now everything handled via publisher with new and much more powerfull interface. 5.09.2013 1. Unify address convention of dabc application. Now any module in the system can be identify with address like: "dabc://nodename:port/module". Such address can be supplied to any command as receiver. If command channel do not have connection to remote dabc node, it will try to establish it. 2. In case of master/client connection client not necessarly have socket port open for connection. In such case address formed from nodename_pidNNN, where NNN is process id (unique on the same host). Such address can only be used from the master like: "dabc://nodename_pid127/module". Command channel will never try automatically create connection if port number in address is not specified. 3. In hierarchy add additional version, which is changed only when names list is changed (so-called DNS version). It will be used to request only names list (and its changes) by publisher. 4. First step in introducing publisher/subscriber mechanism for hierarchy structure. Up to know only http::Server was managing global hierarchies, in future any module will be able to subscribe for any item in hierarchy. Also direct get methods will be implemented. 5. Hierarchy in subscriber will be organized in other way. Each module will publish its hierarchy, using some global 'mount' point. Publisher will be able to combine all independent hierarchies in global structure, merging such hierarchies together. For instance, two mbs::StatusReadout modules could publish there hierarchies like: Publish(fHierarchy1,"/MBS/server1"); Publish(fHierarchy2,"/MBS/server2"); Accordingly, subscriber will use global names disregard of actual server which provides structures. This will be the of publisher to collect necessary description (names lists). 6. Old scheme is still woring (for demonstarion purposes). I expect smooth transition in next weeks from old to new scheme, adding more new features. 7. Main motivation for new functionality - also be able to provide clear storage facility for all published hierarchies. 3.09.2013 1. Use new dabc::Record class for hierarchy. Main change - now user explicitely should mark his/her changes in hierarchy. There is no autofield, which was used before to trigger version tagging and increment. 2. History now preserved and recorded in binary form. Any kind of fields can be recorded in history. 3. Automatic creation of time field is now possible without history object. 30.08.2013 1. Introduce new RecordField, which can preserve native data types like int or doubles. In addition, 1-dim arrays are supported. 2. RecordFieldsMap organizes map between field name (std::string) and field value, stored in RecordField class. 3. Introduce dabc::iostream class, which will be used to stream RecordField and RecordFieldsMap to/from binary form. Main motivation - in addition to xml representation one should be able to store Record in compact binary form. 4. Use new record class for Command, Cfg and Parameter cases. 27.08.2013 1. Introduce arrays in the record fields. Array implemented in JSON-like style. Integer array will look like [1,5,7,10], string array like ["abc", "cdf"]. Special symbols like '"' or '\' or '[' or ']' should be treated correctly. 2. In xml syntax there are two variants to set field as array. First variant is as before - via attribute: or via list of items like: Internally and on any transport cases array will be presented in JSON-like form. 3. Introduce fesa::Monitor module, which can monitor any number of FESA services in asynchron mode. List of services should be specified in form of array in Services field of module. Example is: As one could see, not necessaraly quotes should be used - only if special symbols are necessary. 4. In fesa::Monitor module all availible fields in each service entry will be published. Maybe it is overkill and in the future only small fields (up to certain size) will be exported. Need to be tested. 5. In fesa::Player remove asynchronous mode of parameter request. 6. For all dabc-relevant hierarchy attributes add 'dabc:' prefix. It allow to distinguish them from all other non-dabc attributes. 23.08.2013 1. First working code for synchronous readout of FESA variable from test server. 2. Also asynchronous readout of FESA double variable is implemented. Introduce mutex to allow work from several threads. 3. Show correct time format in tooltip (when time axis is specified). Fix problem for axis value near 0 - now it is shown as 0 in tooltip. 22.08.2013 1. Fix small problem with gauge display. 2. Introduce UTC time for any kind of time recording. 3. Fix several problems with time display on JavaScript side. 31.07.2013 1. In RootSniffer try to use asynchronous timer - it is working! Means one can get access to all ROOT objects without frequent calling of gSystem->ProcessEvents(). Significant drawback - one never can be sure that object (like histogram or canvas) is consistent during such interrupted call, but there is no other chance to get in between. If user wants, he could always introduce gSystem->ProcessEvents call and be on the safe side. 2. In Hist1DPainter, when do optimization, draw maximum value from interval. It is little bit longer, but shows real histogram content. Anyhow, 200000 bins histogram drawn in 7 sec (old code was never finished) 30.07.2013 1. Primitive, non-flexible but working way to generate png image from ROOT canvas. One could think to implement images generation with other kind of libraries in the future, but for a momement just reuse existing ROOT ASImage functionality. 29.07.2013 1. In xml file in parameters list add parameter. This activates connection to master DABC process. It works similar to ROOT or Go4 connections. 2. Fix memory leak in dabc_root::BinaryProducer and improve command channel processing (avoid several intermediate buffers). 3. Fix small error in XmlEngine. 26.07.2013 1. Introduce mbs::Player class to display MBS status record. 2. Copy code from MBS m_daq_rate to show different ratemeters, based on the status record. Provide four different logs for showing rate output. 3. Found proper way to display log messages in browser, using
tags 4. Provide checkbox to enable/disable history drawing. Works for ratemeter and log messages. 5. Read MBS status via SocketAddon. This prevents from thread blocking due to any communication problem. 6. Add possibility to force log recording even when value is not changed by next SetField call. Works only for so-called autofields. 7. Extract all history-relevant variables into special class - it instantiated only when history is configured for the hierarchy item. 8. Improve ratemeter drawing. 25.07.2013 1. Introduce hidden flag for the object. It hides object from hierarchy scan. Use that flag for some special parameters like command definitions. 2. Fix very old memory leak in XmlOutputStream. Need to be fixed in ROOT. 3. Load ROOT scripts ourselfs, excluding duplication. Provide spinner to be able change number of items in grid. 4. Introduce permanent flag for dabc::Hierarchy, which protect element from be deleted when synchronization with other hierarchy is done. 5. In dabc.js provide generic HistoryDrawElement which communicates with server to get history records. Specialized implementations like RateHistoryDrawElement and LogHistoryDrawElement provides specific output. 6. Primitive but working implementation for LogHistoryDrawElement - show arbitrary log text in scrolled space. 7. Even more primitive LogDrawElement - just text output. 24.07.2013 1. In dabc::Url class provide method to extract options part like name?part1&var=3 2. Implement history generation for simple hierarchy item with any attributes. Provide simple time stamp for each element in history 3. Implement gethistory request for items, which are providing history. One can limit number of items in request. One can detect if there is gap in history data relative to previous dataset. Works only for local hierarchy objects. Request look like: http://server:8090/gethistory?/Server/App/Rate&ver=100&limit=100 4. In dabc.js implement drawing of history in form of TGraph object. Fix several issues with graph update in painter. 5. Implement automatic recording of history when selected attribute is changed. Normally "value" recorded, but can be any. By such feature not only changes during hierarchy update can be detected (update done about every second), but also by arbitrary number of direct write operations. One can also record time. 6. Implement duplication of history on server side - if gethistory request goes to remote node, local history cache will be produced first. By such feature any browser requests between two hierarchy updates can be processed locally. 7. As result, flexible history book-keeping is possible now. Depending from application needs, one could record and display history over arbitrary variable in the system with arbitrary (~ms) time precision. 8. Cleanup code how http::Server produces binary and history requests. Now complex logic is moved to the place where it should be - to dabc::Hierarchy class. By this change one could easily reuse code in other than http implementations. 23.07.2013 1. Remove transient flag in reference class. By this C++-conform code can be used. Main problem was assignment operator - it requires increment of reference counter. But in most cases original object will be cleared and reference counter will be decreased. To optimize this, transient flag in reference class was introduced. But one cannot really control code where and when reference from source object can be MOVED and when COPIED. Therefore only left-shift operator allows to move reference, all other operations will create copy of reference class. In most importnat case of dabc::Buffer class this does not lead to performance penalty, while dabc::Buffer is created without mutex - instance can be used only inside single thread. 2. Remove owner flag from reference - ownership is only can be set for child objects. User can always call dabc::Reference::Destroy to initiate object destroy, but real destructor will be called only when all reference are disapper. 3. Fix memory leak with Hierarchy class - in present state Reference could not automatically cleanup object, which has children. Will be implemented soon. 4. Try to provide more logical methods to operate with childs in object. Now all kind of remove methods call RemoveChild or RemoveChilds. If cleanup flag specified (default), object will be destroyed when parent is owner of the object. 5. Add DebugObject method, which could help to understand which objects are not cleaned correctly. Activated only with make extrachecks=true compile flags. Actually, only this helped to find memory leak. 6. Provide automatic cleanup of objects with auto-destroy flag even if such objects has childs. Before any child was blocking its parent while parent refcounter was always bigger than 1. Now cleanup can be started if refcounter equal to number of childs. 7. Use auto-destroy with hierarchy classes - now any release will cause deletion of not-used hierarchy structures - elegant solution for potential memory-leaks problem. 8. Reduce number of Reference constructor signatures - there were too many. Special case of reference creation without mutex hidden in protected methods. No need of Reference::Ref() method - now assignment operator works as expected (no any more transient objects) 9. Go back to non-recursive mutexes - using them, one has better chance to detect deadlocks. 10.First step in implementing history recording for dabc::Hierarchy. If attribute of hierarchy item changes, one can produce diff string with only changed attributes and adding time stamp when change was record. Just in debug output. 22.07.2013 1. Replace Hierarchy::BinData and Command::RawData by the dabc::Buffer class 2. Provide special header for BinData, where version, master_version and other binary-data relevant fields are collected. 3. In dabc.js script automatically unzip data, which are compressed on server side. Means, not only ROOT objects but any future components in DABC can be zipped before transported to the browser. 19.07.2013 1. Create tag with version 2.3.0. Start redesign of buffer classes. 2. Buffer is now normal reference on BufferContainer class. One can use same buffer only in the same thread - buffer fields are not protected with the mutex. When buffer sends to other thread, it's disappers from sending thread. One can do buf.Duplicate() - in this case new BufferContainer will be created and one can keep original buffer in sending thread. One should not forget - if buffers reference same memory, one not allowed to change memory from both threads. 3. Buffer::CanSafelyChange() returns true when user can modify buffer content without any doubts. It happens when refcnt==1 for each memory segments. 4. Main motivation for Buffer changes - allow to use Buffer class in other places like RawData in command or BinData of hierarchy. 18.07.2013 1. Introduce FESA plugin in DABC. At the moment it is just dummy code, which creates simple 2D beam profile array. Plugin demonstrates how arbitrary binary data (not necessarly ROOT) can be created in DABC and exported to the browser. Version control is provided. 2. Provide in dabc.js code to display FESA.2D objects. ROOT painter is used to draw profile in form of 2D histopgram in col mode 3. In RootSniffer extract code which responds for master production and placed it in separate class BinaryProducer. Idea to reuse code at other places, where ROOT classes need to be involved 4. Add in FESA plugin demonstrator showing how arbitrary ROOT objects can be add to hierarchy and provided to browser. This solution could simplify GUI development - one could use ROOT graphics directly. 17.07.2013 1. Use proper method to create patterns in canvas. 2. Fix several small problems with TH2 drawing. 16.07.2013 1. Improve log-scales handling, add appropriate context-menu commands. 2. Implement draw optimization for TH1 histograms - at maximum draw array with number of points on window will be created. 3. Improve algorithm of loading necessary scripts. 4. Go away from ROOT style of drawing - one could not see more than one histogram. First try to use HTML tabele to display more than one object at once. 15.07.2013 1. Fix problem with stat-box drawing, fix cleanup problem by stat box redraw 2. Implement toggling of col, colz, stat-box properly 3. Improve zooming selection - now one could select only X or only Y axis. 4. Implement autozoom for 1-d and 2-d histograms 5. Try to optimize 2-D histo draw - create bins array with necessary info. 12.07.2013 1. Provide optionally normal style for tooltips. In case of TH2 objects improve drawing performace - avoid creation of double number of elements. 2. Add into context menu switch for "col" options. Fix problem with TH2 draw. 3. Fix drawing errors and problems with TH2 and TAxis classes. 4. Bring together vis (svg:g) frame(svg:frame) and pad - all are collected in top svg:g entity. Therefore only single arguments need to be deliver in all draw methods. This makes much more easier to understand logic behind. 5. Provide drawing (not yet update) of colz axis. 6. Define central drawObjectInFrame function, where all different classes must be handled only once. Therefore from now it should be easy to add any new classes. 7. Better drawing of stat box (alignment, 2d statistic) 8. Normal filling of stat box for TH1/TH2 hsitograms 11.07.2013 1. Implement drawing of THStack. 2. Try to avoid bins array usage and generate draw elements directly and only required amount. Done as test for TH1 with line drawing 3. Add support for TProfile - via normal hist painter. 4. Fix several problems with TGraph draw, files/rf107.root, files/ct.root are now working. 5. Add support for TF1 - create Func1DPainter class. 6. Put protection when TF1.evalPar() returns NaN value. At least, fillrandom.root file with such funstion now working. 7. Move functionality to draw TH2 histograms in 3D to Hist2DPainter. One can reuse range, selected in 2D view. 10.07.2013 1. Introduce GraphPainter for TGraph objects. Implement graph and multigraph drawing. Do not use tooltip (svg:title) for graphs. One may need to improve speed of tooltip generation, and also graphic. 2. Split HistPainter on generic part and Hist1DPainter and Hist2DPainter. Many functions remain in basic class. 3. Implement Hist2DPainter. Linear code (no any long loops) for producing tooltip information. 9.07.2013 1. First approximation for HistPainter and PavePainter. One could select ranges, create and toggle stat box. Information in stat box can be updated. One could extend menu items. 2. Remove tooltip from Bertrand - it eat full time and cannot be used as is. 8.07.2013 1. Experimental code in JSRootD3Painter.js. Try to zoom in without remake of drawn objects, but via d3.transition approach. One should get better control how objects are modified. 2. Try to introduce gStyle object in JSRootD3Painter.js - use to create stat box for histogram. 3. Provide possibility to select between experimental and normal painter. 4. Very-very first HistPainter class in JSRootD3ExpPainter.js - try to move all graphical handling out of histogram itself. Painter will keep all options, which we could change later interactive. 5.07.2013 1. Implement context menu for drawn object. This allows to implement different functionality without complex gui with collapsibles. 2. Change hist paint mathod - keep hist.redraw() as it was, introduce hist.rebuild() method which should recreate histogram. 3. With few modifications allow to work with local files. Web server, which does not support ranges, should also work now. 4. First try to generate stats box on browser side. Question, how far one should go with duplication of ROOT functionality? 4.07.2013 1. Way to update histogram content was too simple and do not scale in the future. Therefore code was ralled back to previous state 2. Clone of Bertrand repository was created in https://github.com/linev/JSRootIO.git master branch will contain actual code, used in dabc. 3. Allow in http::Server specify JSROOTIOSYS directory - location of JSRootIO files. If not specified, local copy in plugins/http/JSRootIO will be used 4. For TH1 classes found correct way to redraw histogram with selected range. 3.07.2013 1. In full browser do not automatically show hierarchy of remote clients. Only when clients="true" is specified, server will provide view of remote structures. 2. Set default DABC command channel port to 1237. 3. In JSRootIO found simple way to update histogram content on drawn object without redraw of complete canvas - it is really simple. Tested with TH1 objects, but should work with all kinds of objects beside TCanvas. 2.07.2013 1. Fix error in command channel - submitted commands not always was send to remote. 2. Tag version number 2.2.0 - this is mark for important milestone achieved. 3. Implement compression/uncompression of ROOT data. Compression level can be configured (now only via xml file). 4. Fix CPU consumption problem with MBS server transport. 5. Add tmout argument to mbs generator input - reduce CPU load of examples. 1.07.2013 1. Move DABC/ROOT interface methods to DabcRoot class (instead of using dabc_root namespace). It is necessary, while rootmap for libraries works only when at least one class is specified. All methods in DabcRoot are static. Actiavte http server in ROOT will look like: root [0] DabcRoot::StartHttpServer(8095); Actiavte connection to DABC master node: root [1] DabcRoot::ConnectMaster("localhost:4444"); 2. Adjustment to be able use RootSniffer and browser with go4. 3. Fix problem with thread usage when hierarchy was created. Now method really called from the proper thread context and one not need to use locking. 28.06.2013 1. Implement automatic reconnect of command channel when server disappears. 2. Objects request via command channel works! It means, one could access data from running ROOT analysis, which automatically connects to http server and provides its hierarchy and objects via such connection. One could have many such connection in the http server. This if first, but most important step to control many-node application (ROOT, Go4, DABC) via single http server! 3. First real test to run two ROOT analsysis, connected to the same http server. Organize better folders for browser, suppress main debug output. 4. Prepare DABC code for using within go4analysis. Idea to reuse RootSniffer and just provide information from go4 object manager. 5. Today a first day when ROOT, Go4 and DABC with arbitrary combination of nodes can provide data via central DABC web server to any normal browser like Mozilla or IE or Chrome. This allows to build complex heterogene systems with unified user monitoring and control interface! It took only 17 days to implement! A huge amount of work need to be done, but all building blocks are already exist and proved to be working. 27.06.2013 1. In http::Server add possibility to select part of objects hierarchy, displayed in the browser. One also can specify how top folder is called. This allows in case of ROOT browser shows only ROOT-relevant structures with name "ROOT". 2. Fix error in dabc.js, when object version number was kept in string form. In this case comparasion between versions was not working correctly. One should use explicit conversion to number like MyCall(Number(ver)). 3. Change functions name which must be called from ROOT script. Now user should do from ROOT session: root [0] gSystem->Load("libDabcRoot") root [1] dabc_root::StartHttpServer(8095); root [2] .x $ROOTSYS/tutorials/hsimple.C Reduce debug output. 4. First try to connect ROOT analysis with DABC application, which has web server running. With commands: root [0] gSystem->Load("libDabcRoot") root [1] dabc_root::ConnectMaster("localhost:4444"); root [2] .x $ROOTSYS/tutorials/hsimple.C One create communication channel, which can be used from server side to browse objects in the analysis. Theoretically, it should be transporent for the user if objects exists in the same application as server or objects received from such special channel from remote. 5. Fix error in version control of hierarchy. Now one really see in browser all objects from slave process. Last small step remains - provide getbinary request to remote node. 26.06.2013 1. Now is possible define master for any item in the hierarchy. Idea of master item is that such master object should be updated together with item itself (when master is changed). Server-client protocol will provide information about actual master version as well. 2. Browser request streamer info only when we sure that streamer info is required. Potentially many streamerinfos can be used - will be required when many ROOT analysis connected to same http server. 3. First time continiousely running hsimple.C with web server! Now arbitrary ROOT script can be monitored via web broser! 4. Add support of TProfile and, partially, TCanvas. TCanvas cannot be updated. 5. Provide only browsing for TTree/TBranch/TLeaf classes 6. When ROOT object requested from the browser, it is located again in ROOT structures. This is needed while object may disapper and full hierarchy scan may take too much time. Commands from browser executed about every 0.1 s, hierarchy update performed once a 3 s. 25.06.2013 1. Generate dictionary and rootmap for libDabcRoot. This will allow to use part of DABC functionality directly from ROOT. 24.06.2013 1. Update hierarchy via xml file and build tree structure direct in the browser. Use dtree.jquery for that. This will allow to fully customize look and fill of the hierarchy browser. The only communication format between server and browser will be xml. 2. Made hierarchy.htm as main DABC page, remove no longer used files. 3. Enable display of streamer infos, assign images for ROOT objects. 4. Smooth open/close logic, but it should be rewritten - dabc.js should fully control itself how items are open or closed. 21.06.2013 1. Create BinDataContainer and BinData classes for manipuldation of binary data in dabc::Hierarchy classes. Main motivation is to be able from any thread keep reference on binary data disregard of changes in hierarchy. BinData also simplifies version control of binary data. 2. In case of RootSniffer special container is created, which interfaces TBuffer functionality. By that one do not require memcopy and duplication of memory. 3. Implement GetBinary command for RootSniffer. By that step one completely decouples http::Server from ROOT functionality. 4. Now it is possible to draw root objects and ratemeters from hierarchy browser. 5. Make more generic method to generate StreamerInfos. Test TGraph. 6. First demonsrtation showing that simple redraw of the object is working. 7. Introduce proper version control of the hierarchy components. For every element hash property could be set, which must be changed if object is changing. If hash changed, version of that item will be changed. This change will be propogated over all dependent hierarchy structures. When browser requests object, it provides version of last instance it has. Server compare browser object version with local object version and if they match, returns empty reply. If not, binary buffer will be requested first from producer and with current version number provided to the browser. Little bit complex, but prevent gui from unnecessary redraw and prevent server from extra load if nothing is changing. To force object retrieving, one must specify extremely high version number. 20.06.2013 1. Create DABC.Manager class for JavaScript applications (name is always the same :)) It should manage most (probably all) draw elements, produce requests and update draw elements. DABC.DrawElement class contains full description of drawn element - item name, kind, draw object and so on. 2. Create hierarchy of DABC.DrawElement classes to separate Gauge from Root canvas from anything else. Only 'simple' classes handled in the main request, rest will be treated by own-managed asynchronous requests. 3. Extract DABC-specific JavaScript code to dabc.js. 4. Implement binary requests, central binary callback and first primitive version of drawing histogram from DABC browser! 19.06.2013 1. Change logic how hierarchy is managed. There is dabc::Object::BuildHierarchy() method, which responsible to create hierarchy below. Method executed in thread context of object. 18.06.2013 1. First example, which demonstrate how JSRootIO could work inside DABC. rootio.htm page shows how object, generated inside DABC application, can be streamed, reconstructed and displayed in web browser. Code is very-very preliminary and done only as demonstrator. Special file is required to provide streamer infos to the browser. Nevertheless, it is prove of principal that web-server for arbitrary ROOT analysis can be implemented! 2. Found simple way to generate streamer infos on server side. Therefore special ff.root file no longer required. 3. Several fixes in JSRootIO to be able work with dummy JSROOTIO.RootFile. 4. Introduce dabc_root::RootSniffer class, which will deliver ROOT objects hieararchy to DABC framework. 17.06.2013 1. Principal possibility shown to transport ROOT-objects from DABC into browser without intermediate file. Around such feature one could build application, which can deliver any object from running ROOT code into web browser. 14.06.2013 1. First working ratemeter in the browser! Very simple JustGage element was used, can be always replaced by more advanced widget. Basic logic of widgets organisations and data exchange between browser and server is exercised. Many other widget types (commands, info fields, ...) can be done in similar way. 2. Version number switched to 2.1.1. Odd second number indicates that now many new features can appear - mostly in http interface. 13.06.2013 1. Put node hierarchy as separate request. In future can update it like gauge periodically, one could easely insert hierarchy in many different pages. 2. Understand how correctly select items in hierarchy and how react on click. 12.06.2013 1. Implement html hierarchy display with jquery.treeview java script. 2. Make in http::Server internal cash of files like images or scripts. 11.06.2013 1. Mongoose web browser (from https://github.com/valenok/mongoose) integrated into DABC. New libDabcHttp plugin is introduced, which should provide http protocol to DABC applications. For a moment it is just helloword application, should provide functionality similar to command shell, ncurses shell, go4-dabc browser. 2. Show application hierarchy in form of xml file. 10.06.2013 1. Minimalistic structure browser in dabc_shell. Menus from ncurses library are used. Real functionality will be provided in parallel to other implementations. Idea to show some rates, infos, submit commands in such shell. 7.06.2013 1. Introduce primitive command shell to remote dabc node. Via command channel any kind of command can be executed. 2. First working code for transport and update of hierarchy structure over command channel. By this one should be able to inspect any remote dabc node in the future, even several of them simultaneous. This command shell will be very minimalistic, more advanced text-based interface will be tested with ncurses. Later plugin for go4 will be done! 3. Allow to transport raw data together with xml command description over command channel. This allows potentially to transport any kind of data via the channel - not only xml-based representation, which is not veru efficient for big raw objects. 6.06.2013 1. Implement unreliable (datagram) socket communication. Idea to provide udp and multicast messages, where no any connections are required. Usage of datagram socket does not require connection manager and no any kind of control. Include appropriate example in net-test. 5.06.2013 1. Introduce new command channel, based on TCP sockets (old was using UDP). There is one worker per connection and all commands to remote node are sequentialized via that worker. At the moment functionality of old and new command channels are similar and they can be easily replaced. In future command channel can be used to communicate with dabc nodes from external application (command shell, go4 plugin and so on). 2. Introduce ModuleAsync::ActivateOutput/ActivateInput to generate events for specified output/input ports. Required when processing of events was stopped when ProcessRecv/ProcessSend methods return false. 3. ModuleAsync::TakeBuffer() not always requires handle to be created. In simple case direct call to default memory pool (with name "Pool") will be performed. This should exclude complex rules when handle is required or not. From this moment meaning of handle is provide self-filled queue of buffers, which can be used like any other input port. 4.06.2013 1. Fix (implement) missing cleanup in several components - Port, PoolHandle, NetworkTransport. Cleanup mostly required to correctly release cross-references before objects can be destroyed. Without correct cleanup manager can not destroy referenced objects - in most cases MemoryPool. 31.05.2013 1. Provide version control of dabc::Hierarchy structure. Each element in hierarchy includes version number, which indicate when last changes were performed. This version gives possibility to extract only changed items relative to specified version in the past. 2. Implement conversion of dabc::Hierarchy structure into xml and reconstruct hierarchy from xml. This will allow to transport and reconstruct hierarchy on any remote location. 28.05.2013 1. Introduce dabc::Hierarchy class, which should represent in the future objects hierarchy of remote (or local) DABC process. 27.05.2013 1. Activate shared-memory observer in hadaq plugin only when it really enabled 24.05.2013 1. Make doxygen docu more readable 21.05.2013 1. In mbs::ServerTransport implement correct stop of transport when EOF buffer has been seen. 2. Make transport kind of mbs::ServerTransport blocking by default. In this case every buffer should be delivered to at least one client. If no one client is connected, transport will block. One can specify unblocking mode for transport server if configure: Stream server will be always unblocking. 17.05.2013 1. In input and output ports properties "rate" attribute can be specified. When specified, data rate parameter with such name will be created and assigned to port. 2. Create dabc::RepeaterModule. It's only functionality - translate data from each input to correspondent output. Thus number inputs and outputs should be the same. 3. Remove old latex-based documentation, start long transition to doxygen-based one. At least 1.8.0 version of doxygen is required. To generate documentation files, call "make doxy". 4. Produce 2.0.2 tag - currently working version of HADAQ readout 16.05.2013 1. Allow to specify auto="false" for every input/output port to avoid automatic port connection. 2. Adjust most of MBS examples to current modules creation logic. Main advantage now - one do not need special init functions for different use-cases. 3. Remove mbs::GeneratorModule - one can use "lmd://Generator" input instead. 4. Implement of automatic reconnection of any module input/output. For that "reconnect" attribute should be specified for the port: Property specifies interval (in seconds) how often reconnect should be performed. Reconnect will not be done when auto="false" specified. 5. Fix error that MemoryPool was stopped when no any connection was available. Thus when new connection was coming, pool failed to provide new buffer to modules and transports. 15.05.2013 1. Make more comfortable way to specify affinity of the threads and complete process. 2. Already before it was possible to specify taskset arguments for complete dabc process in section in the form like . This is used in dabc_run script. 3. One can specify default affinity for every new created thread, providing "affinity" parameter in section. Value can be following: - unsigned value with processors mask - string like "xxxoooxxxss" with allowed symbols 'x', 'o' and 's'. 'x' - enabled, 'o' - disabled, 's' - special purpose first element in string corresponds to first processor - string like "-N" where N is processors number which should be reserved for special purposes. These processors could be later assigned with SetAffinity("+M") call (M section of xml file: Default kind of threads layout is balanced. 2. Allow to configure "timeout", "thread", "device", "useackn" and "timeout" attributes for port or for connection. These attributes will be used for transport creation for particular port. For instance, one could specify thread attribute: to define which thread will be used for the transport. 13.05.2013 1. Add possibility to create devices and threads automatically via xml file. Xml code should look like: 10.05.2013 1. Fix error in transport start/stop logic. Transport was started before connection to the module was established, therefore many first buffers were lost. Was happened, while transport was reacting on connection activating with memory pool. 2. Fix flushing error in hadaq::MbsTransmitterModule. When EOF marker was appearing, rest of data in output iterator was not finalized and was not delivered to the output. 3. Unified typedefs and structures HADAQ between DABC and Stream analysis. Idea to use as much as possible common code in both frameworks. 8.05.2013 1. Memory pool, modules and connections between them could be created automatically, based on information from xml file. For module one should specify class tag. In addition, thread tag can be provided. For instance: ... Local connection between modules can be configured via simplified url: ... ... As alternative, one could specify connection between modules in extra node: hadaq plugin is first plugin with such examples. 2. Provide new configuration files for hadaq plugin, showing auto-creation functionality 3. Introduce BasicFile class, which only contains file descriptor and IO object. Used as basic class for dabc::BinaryFile, mbs::LmdFileNew, hadaq::HldFile. 4. Change implementation of hadaq::HldFile file. Main change - use DABC IO class to be able switch to any other file system. Also when reading events, first read as much as possible and than rewind if non-complete events are read. 7.05.2013 1. Tag version 2.0.0 in repository. 3.05.2013 1. Introduce mbs::LmdFileNew class, which should replace LmdFile in next future. Idea is to use same logic as with dabc::BinaryFile, where different low-level file I/O can be used. Also like dabc::BinaryFile, only appropriate includes will be required to use mbs::LmdFileNew classes outside DABC. One more argument for mbs::LmdFileNew - reduce functionality of lmd storage to absolute minimum, remove never-used index table. 2. Provide LMD random generator input, which could be configured in form It should replace MBS generator module in the near future. 3. Provide temporary "lmd2" file input and output classes, which are using mbs::LmdFileNew. New lmd files can be used with go4 as well. Test it. 4. Fix error with MemoryPool::TakeEmpty - pool reference was not set correctly in buffer, which leads to double delete of memory. 11.04.2013 1. Switch main DABC brunch to version 2. 22.03.2013 1. Make dabc::Manager class interface more clear, most functions must be used via dabc::mgr instance - means via dabc::ManagerRef class 2. Introduce FileInput class, which handles common functionality for file inputs. Reused in dabc::BinaryFileInput, mbs::LmdInput, hadaq::HldInput, mbs::TextInput 3. Introduce FileOutput class, which handles mots common tasks of file storage. Use it for dabc::BinaryFileOutput, mbs::LmdOutput, hadaq::HldOutput. 21.03.2013 1. Introduce RFIO plugin in DABC2. Idea to have example how FileInterface can be used. Not yet really tested. 20.03.2013 1. Reintroduce BinaryFile class to the DABC. It is generic storage for any kind of dabc buffer. Now segmented buffers are supported. Most important that BinaryFile.h include can be directly used in any project without need of any addition library. One can even use ROOT CINT in the way like: .L $DABCSYS/include/dabc/BinaryFile.h dabc::BinaryFile f; f.OpenReading("file.bin"); while (!f.eof()) { uint64_t size = maxsize; f.ReadBuffer(buf, size); } f.Close(); 2. Replace FileIO class by FileInterface. FileInterface is wrapper for posix file functions with very similar functions. Already FileInterface itself provides basic IO functions and used in BinaryFile class. Later RFIO or ROOTD can be implemented. 3. Implement BinaryFileOutput which can be work as generic output for buffers in DABC. Syntax is "bin://test.bin?maxsize=30" 4. Replace const char* arguments in several Factory methods. 5. Implement BinaryFileInput. 6. Eliminate outdated FileIO classes, now everything goes via FileInterface. Eliminate extra ListMatchFiles - implemented via FileIO. 19.03.2013 1. Modify dabc::Buffer class. Now it uses reference on MemoryPool to access its functionality. When release buffer, memory pool mutex will be locked twice - first time to release memory structures and second time to release reference on pool itself. Solves very nasty problem during application cleanup, when memory pool potentially could remains hanging. 2. Replace in many places const char* argument by const std::string& 3. Remove direct pointer on module from module item, module is always must be direct parent of item. 15.03.2013 1. Make majority methods of Module and Port classes protected, some of them private. Only reference classes will provide public, thread-safe methods. 2. Like Ports, now Timers and UserItems not accessible via pointer in the module. From now only module methods should be used for any kind of send/recv/timer or user event generation. 3. Remove InputPort.h OutputPort.h Timer.h PoolHandle.h files. All these classes are now "hidden" from normal user and should not be directly appear in the code. 4. Make central dabc::Exception class with kind field to simplify handling of the exceptions. 14.03.2013 1. Change logic how factories are instantiated, 2. Remove libraries handles list from the factory, it was used for symbols finding. 3. Hide all PoolHandle methods from user, only Module methods can be used for correspondent functionality 4. Remove direct usage of InputPort/OutputPort classes in user code - everything done via indexes or names. Later classes will be removed completely. Still one need to hide many module/ports methods 13.03.2013 1. Change syntax of DOUT1(("something")) to DOUT1("something"). As before, command will not be compiled when make debug=0 command is used. 12.03.2013 1. Modify EZCA plugin - remove static variables, use DataInput interface in proper way, improve compilation 2. Fix timeout probelms in DataTransports 3. Allow to change EZCA config parameters 4. Use EZCA group operations - should improve performance when using many variables. 5. Use mbs::WriteIterator in EZCA plugin, fix severeal errors in iterator itself. 6. Rebuild EZCA descriptors only when variables list changed - mean once at the beginning 7. Adjust simple go4 analysis in EZCA plugin to be able work with current readout. 11.03.2013 1. Next step to make PoolHandle like input port, which delivers buffers from the memory pool. No any requests is necessary. Till now possible to request buffers directly from the pool, but most of transport and standard modules will use new feature. 2. Continue API transition, step by step hide direct port methods from user. 8.03.2013 1. Rename PortBase -> Port class. 2. Modify BuffersQueue class, use it everywhere where it makes sencse. 3. Start transition to module API, where user do not have access to pointers to the objects like I/O port, pool handle, timer and so on. Either port index or port name should be used. Also all main methods like Send/Recv should be protected to avoid usage from outside module code. 7.03.2013 1. First workign setup with MBS combiner module. Main change - do not block event loop by event building forever. Use ProcessRecv/ProcessSend methods, they can reinject events itself. 2. Strictly check version 2 of xml file. 3. In mbs::ServerTransport correctly treat second request from the client 4. In epics plugin eliminate applications class - everything can be done with simple init function. Simplify a lot class structure, still to be checked with epics. 5. Adjust mbs plugin, remove old examples and old eventapi 6. Remove old port, pool handle, transport and local transport classes. 6.03.2013 1. Adjust mbs plugin modules to new scheme, need to test them 2. First tests working with hadaq simulators! 3. Add possibility to balance performance between modules by limiting loop max count per port. In this case ProcessRecv/ProcessSend calling sequence will be broken and events are rescheduled. Probably, very important for hadaq 4. Debug and improve hadaq::CombinerModule and hadaq::UspTransport classes. Now with minimal queue length everything works smoothly. Also correct treatment of events id overflow is implemented. 5. Test hld and lmd output for hadaq event builder module - works! 6. In hadaq transport use simple recv() function - socket is already bind to the port. 5.03.2013 1. Change const char* to std::string in modules constructors and all related methods 2. Adjust hadaq plugin to new organization, need to be tested. 3. By default, module creates input and output ports and pool handle (if conbfigured) 4. Allow to put configuration parameters as node attributes in xml files like Should significantly simplify xml files. Old syntax is also valid: 5. Start transition of roc::CombinerModule to new scheme, remove some old features. 4.03.2013 1. For time been include roc-udp plugin into dabc2 - just for convenience duting migration. Convert UdpTransport to SocketAddon. 2. In NewDataInput allow to use queues of the buffers for transport addon. Use such feature for the roc udp transport 3. First debugging of the roc-udp plugin - seems to be, it is working! 4. Eliminate SocketWorker classes - now all codes based on Addons. 1.03.2013 1. Refactor hadaq::UdpDataTransport - use InputTransport and SocketAddon instead. Need to be verified with real setup. 28.02.2013 1. Debug network transport for socket, including acknowledge feature. Is working now. 2. Introduce verbs::WorkerAddon class, which replaces specialized verbs::Worker Redo all relevant verbs classes. Adjust NetworkTransport for verbs. 27.02.2013 1. Refactor command channel, socket device, network transport and socket transport. In all cases SocketAddon class used insted of SocketWorker. Still a lot of debugging to be done. 26.02.2013 1. Significant redesign of mbs::ServerTransport. Now it consists from ServerAddon, which handles connection and OutputAddon which handles data transfer to client. Many clients organized as many slave connection to server transport. Number of connections can be increased or descreased dynamically. Tested with go4 and with dabc client. 22.02.2013 1. Deliver disconnect events to module also in non-running state. 2. First working version of mbs::ClientTransport. 3. Split DataTransport on two independend InputTransport and OutputTransport. Both classes should be used with DataInput/DataOutput classes. 4. Adjust DataInput interface to work not only with blocking interfaces but also with call-back functionality - like DataInput already. 21.02.2013 1. Implementing full pallette of WorkerAddon for SocketThread. All new classes should replace SocketWorker classes in dabc. 2. Rewrite mbs::ClientTransport class with SocketIOAddon. Try to use DataInputTransport. Only implementation, not yet tested. 20.02.2013 1. First working implementation of NewDataTransport, based on NewTransport. Here normal ModuleAsync is used to provide event loop, queues and so on. Dramatically simplifies way how port and transport connected and work together. 2. First adoption of NewDataTransport to lmd file input/output. First running example with such new transports. 3. Introduce di_DfltBufSize return value for DataInput object. Allows to use default buffers, provided by the memory pool. Thus configuration can be shifted to memory pool. 4. Allow to specify transport kind directly in the xml file - via url tag. By one string one can specify type of transport and basic parameters. Like output lmd file is set with 19.02.2013 1. New event-generation model. Before each send/recv operation produces event, which should be handled. This can leads to unlimited grow of thread events queue, if event processing done not proper. To cover most typical tasks, new models for i/o ports introduced. There are four models now: enum EventsProducing { SignalNone = 0, // port will not produce any events SignalConfirm, // next event can be produced when previous event is received, confirmed and // event consumer performed next operation SignalOperation, // next event can be produced when event consumer performs next operation SignalEvery // every operation will produce event } 2. By default use SignalConfirm kind for i/o ports. This kind is useful when in each event processsing function many buffers can be handled - is a situation in most cases. 3. Implement burst mode in memory pool. When module permanently requests buffers from the pool, maximum number of buffers will be send. 4. Support of new pool handle in ModuleSync 15.02.2013 1. Introduce WorkerAddon class, which should contain all device-specific code for sockets and IB. 2. MemoryPool class now derived from ModuleAsync - means just another source of buffers. All requests now will be processed via ports, keep for the moment direct TakeBuffer methods. 14.02.2013 1. Introduce InputPort and OutputPort 2. Introduce BufferQueue for connection between ports 3. Add support of new i/o ports into both modules kinds 4. Generate i/o events only when consumer react on previous event 5. Improve by factor 3 speed of buffers propogation over modules 6. Exclude mutex from queue when ports comunicating in the same thread 7. Use thread for ModuleItems only when required. For instance i/o ports not necessary need to be assigned to the thread. Also in case of Timer, one can use Module thread. This should simplify start/stop procedure.