-----------------------
 README for macro/prod
-----------------------

A set of macros to run various kinds of simulation either locally or submit them for larger amounts of events to Kronos cluster at GSI.

See also 

 https://wiki.gsi.de/foswiki/bin/view/Linux/SlurmUsage
 http://wiki.gsi.de/cgi-bin/view/Linux/GridEngine (outdated)

for general description of usage of GSI Gridengine and Slurm and

 https://panda-wiki.gsi.de/foswiki/bin/view/Computing/PandaRootSimulationKronos
 http://panda-wiki.gsi.de/cgi-bin/viewauth/Computing/PandaRootSimulationPrometheus (outdated)

for a nicer look-and-feel of the information given below.


############
  CONTENTS
############

- Files in directory

- Generator setup

- Run locally in shell
  - Full Simulation
  - Fast Simulation
  - Examples

- Submit to cluster
  - Prerequisits
  - Most important slurm commands
  - Full Simulation
  - Fast Simulation
  - Examples
  - Resubmission of jobs
  - Examples

- Analysing data
  - Run locally
  - Examples
  - Submission of analysis jobs
  - Examples



######################
  FILES IN DIRECTORY
######################

Simulation
- prod_sim.C
- prod_aod.C
- prod_fsim.C

Analysis macros 
- prod_ana.C
- quickana.C

Analysis example task 
(-> https://panda-wiki.gsi.de/foswiki/bin/view/Computing/PandaRootRhoTutorial#A_3._Analysis_in_a_Task)
- prod_myana.C
- PndProdAnaTask.cxx
- PndProdAnaTask.h
- macroprodLinkDef.h
- CMakeLists.txt

Example decay file for EvtGen
- D0toKpi.dec

Scripts to run all automatically
- runall_prod.sh

Scripts to (re-)submit jobs to Kronos
- jobsim_kronos.sh
- jobfsim_kronos.sh
- resub.pl

Script to submit analysis job for many input files
- jobana_kronos.sh 
- anasub.pl

Helper macro to merge output
- merge.C



###################
  GENERATOR SETUP
###################

When running any kind of simulation, there always is a certain event generator used. The info below explains
how the configuration of parameter <gen> looks like for all of them.

1. EvtGen: Either  <gen> = "yourdecayfile.dec" or "yourdecayfile.dec:initialResonance"
   Example:  <gen> = "D0toKpi.dec:pbarpSystem"

2. DPM  : Keyword DPMx; "DPM" : Inelastic only, "DPM1" : Elastic + inelastic only, "DPM2" : Elastic only

3. FTF  : Keyword FTFx; "FTF" : Elastic + inelastic, "FTF1" : Inelastic only  (*** The other way around than DPM!! ***)

4. BOX  : Configuration string of form "BOX:type[pdg,mult]:p[min,max]:tht[min,max]:phi[min,max]"



########################
  RUN LOCALLY IN SHELL
########################

-----------------
 Full Simulation
-----------------

The full simulation runs in two steps utilizing the macros prod_sim.C and prod_aod.C. The shell script runall_prod.sh
runs both.

> ./runall_prod.sh <prefix> <nevts> <gen>

with parameters

 <prefix>    - Prefix for output files
 <nevts>     - Number of events to simulate
 <gen>       - Generator setting (see above)


-> Produces output files <prefix>_sim.root, <prefix>_par.root, <prefix>_pid.root in current directory.


-----------------
 Fast Simulation
-----------------

For performing fast simulation, only the macro prod_fsim.C needs to be run.

> root -l -b -q prod_fsim.C("<prefix>", <nevts>, "<gen>", <pbeam>)

with parameters

 <prefix>    - Prefix for output files
 <nevts>     - Number of events to simulate
 <gen>       - Generator setting (see above)
 <pbeam>     - Beam momentum in GeV/c


-> Produces output file <prefix>_fsim.root current directory.

----------
 Examples
----------

Full Sim:

> ./runall_prod.sh PiPlus  1000 BOX:type[211,1]:p[0.05,8]
> ./runall_prod.sh D0Sim   1000 D0toKpi.dec:pbarpSystem   12.
> ./runall_prod.sh DpmBkg  1000 DPM                       12.
> ./runall_prod.sh FtfInel 1000 FTF1                      12.

Fast Sim:

> root -l -b -q prod_fsim.C("PiPlus",  1000, "BOX:type[211,1]:p[0.05,8]", 15.)
> root -l -b -q prod_fsim.C("D0Sim",   1000, "D0toKpi.dec:pbarpSystem",   12.)
> root -l -b -q prod_fsim.C("DpmBkg",  1000, "DPM",                       12.)
> root -l -b -q prod_fsim.C("FtfInel", 1000, "FTF1",                      12.)



#####################
  SUBMIT TO CLUSTER
#####################

General information about submitting jobs on the Prometheus/ cluster can be found here:
https://panda-wiki.gsi.de/foswiki/bin/view/Computing/PandaRootSimulationPrometheus
http://wiki.gsi.de/cgi-bin/view/Linux/GridEngine

** For Kronos cluster take a look here: **
https://panda-wiki.gsi.de/foswiki/bin/view/Computing/PandaRootSimulationKronos
https://wiki.gsi.de/foswiki/bin/view/Linux/KronosCluster
https://wiki.gsi.de/foswiki/bin/view/Linux/SlurmUsage


**************************************************************************************************************************
*** In the following the actions to be taken for KRONOS are given only, since Prometheus will be decommissioned soon.  ***
*** Refer to an older revision of this README file for information about Prometheus usage.                             *** 
**************************************************************************************************************************

--------------
 Prerequisits
--------------


BEFORE being able to submit jobs to Kronos, you have to 

1. Login to an lxbk machine by ssh yourname@kronos.hpc.gsi.de locally from a lxpool.gsi.de machine.  

2. Create a work directory on the data storage element NYX, e.g. /lustre/nyx/panda/yourname/pandaroot

3. Install PandaRoot in that place, since your GSI linux home directory is not mounted on Kronos nodes.

   There are several versions of the external packages and FairRoot pre-installed under /cvmfs. Take a look with
   
   > ls /cvmfs/fairroot.gsi.de/fairsoft/
   > ls /cvmfs/fairroot.gsi.de/fairroot/

   and set the environment variables SIMPATH to an appropriate fairsoft and FAIRROOTPATH to a corresponding fairroot
   version before compiling PandaRoot.


-------------------------------
 Most important slurm commands
-------------------------------

Some nice condensed information about slurm can be found under
https://rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/


- Job array submission with time setting, partition (sth like queue) setting 

  > sbatch --time=01:00:00 --partition=debug --array=1-10 my_slurm_job_script.sh <parameters> <for> <job> <script>

- Check job status

  > squeue -u <your_user_name>

- Cancel (kill) jobs

  > scancel -u <username>             # kills all your jobs
  > scancel <jobid>                   # kills a single job with id <jobid>
  > scancel -t PENDING -u <username>  # kills all your pending jobs
   

-----------------
 Full Simulation
-----------------

This command submit jobs with number <min> ... <max>:

-> sbatch -a<min>-<max> jobsim_kronos.sh <prefix> <nevts> <gen> <pbeam> [opt] [mode]

with parameters

 <min>       - Minimum job number
 <max>       - Maximum job number
 <prefix>    - Prefix for output files
 <nevts>     - Number of events to simulate
 <gen>       - Generator setting (see above)
 <pbeam>     - Beam momentum in GeV/c
 [opt]       - Optional options: if contains 'savesim', sim output is copied as well, 'ana' runs prod_ana.C in addition.
 [mode]      - Optional mode number for analysis. Stored in TTree produced by PndSimpleCombinerTask when using ana_prod.C.


Produces output files data/<prefix>_<run>_pid.root (and data/<prefix>_<run>_sim.root, if save=="saveall") and log files,
with <run> running from <min> to <max>. 


-----------------
 Fast Simulation
-----------------

Submission of fast simulation jobs works in exactly the same way with script jobfsim_kronos.sh.

-> sbatch -a<min>-<max> jobfsim_kronos.sh <prefix> <nevts> <gen> <pbeam> [opt] [mode]

with same parameters

 <min>       - Minimum job number
 <max>       - Maximum job number
 <prefix>    - Prefix for output files
 <nevts>     - Number of events to simulate
 <gen>       - Generator setting (see above)
 <pbeam>     - Beam momentum in GeV/c
 [opt]       - Optional options: if contains 'ana', runs prod_ana.C in addition.
 [mode]      - Optional mode number for analysis. Stored in TTree produced by PndSimpleCombinerTask when using ana_prod.C.


Produces output files data/<prefix>_<run>_fsim.root and log files, with <run> running from <min> to <max>. 


----------
 Examples
----------

Full Sim:

> sbatch -a1-20 jobsim_kronos.sh PiPlus  1000 BOX:type[211,1]:p[0.05,8]
> sbatch -a1-20 jobsim_kronos.sh D0Sim   1000 D0toKpi.dec:pbarpSystem   12. savesim:ana 10
> sbatch -a1-20 jobsim_kronos.sh DpmBkg  1000 DPM                       12. ana 20
> sbatch -a1-20 jobsim_kronos.sh FtfInel 1000 FTF1                      12. 

Fast Sim: (just replace jobsim... with jobfsim...)

> sbatch -a1-20 jobfsim_kronos.sh PiPlus  1000 BOX:type[211,1]:p[0.05,8]
> sbatch -a1-20 jobfsim_kronos.sh D0Sim   1000 D0toKpi.dec:pbarpSystem   12. savesim:ana 10
> sbatch -a1-20 jobfsim_kronos.sh DpmBkg  1000 DPM                       12. ana 20
> sbatch -a1-20 jobfsim_kronos.sh FtfInel 1000 FTF1                      12. 


----------------------
 Resubmission of jobs
----------------------

When submitting many jobs/job arrays, it sometimes happens that some of the jobs randomly fail. Usually resubmitting the same jobs
again leads to success. In order to automatically resubmit only the failed jobs the perl script 'resub.pl' can be used.

It looks for missing output files and output files being too small (size < 10kB) to contain reasonable output and resubmits exactly 
the according jobs. The input parameter either is the complete sbatch command in quotos "sbatch -a...", or a ascii file containing
a bunch of sbatch commands. The file has to have the ending '.jobs'

NOTE: In case some jobs got stuck in the queue, delete them with 'scancel -u <username>' before using resub.pl.

-> ./resub.pl "<cmd>" [check]

Parameters:

  "<cmd>"  : full sbatch command line in quotes. The first parameters must be '-a', the job script has to have a name matching 'job...sh'
             it's also possible to give the name of a file 'somename.jobs' containing sbatch commands, then resub is applied to all of them (except commented by #) 
  [check]  : optional parameter; if set to 'check', only prints the resubmission commands for checking.
  

----------
 Examples
----------

Full Sim

> ./resub.pl "sbatch -a1-20 jobsim_kronos.sh  DPM10GeV 1000 DPM 10.0 saveall" check
> ./resub.pl "sbatch -a1-20 jobsim_kronos.sh  FftInel  1000 FTF1 10.0 saveall" 

Fast Sim

> ./resub.pl "sbatch -a1-20 jobfsim_kronos.sh SingleK  1000 box:type[321,1] 10.0 saveall" 
> ./resub.pl my_fsim_study.jobs



##################
  ANALYSING DATA
##################

-------------
 Run Locally
-------------

In order to run over a number of your resulting output (_pid.root) files, the macro prod_ana.C can be used as follows:

-> root -l -b -q 'prod_ana.C("data/<prefix>", <min>, <max>, <mode>, <nevts>)'

Adds several input files <prefix>_<min>_pid.root, ... , <prefix>_<max>_pid.root and <prefix>_<min>_par.root as par file to FairRunAna to run analysis.
Broken or missing files are skipped automatically. The macro is suitable for both full and fast sim.

***                                                                                                 ***
*** The macro has to be configured for reconstruction, and the fastsim flag has to be set properly. ***
***                                                                                                 ***

Parameters:

 <prefix>    - Prefix for output files, or ROOT file name
 <min>       - Mininum run number
 <max>       - Maximum run number
 <mode>      - Arbitrary mode number; default=0. Stored in TTree produced by PndSimpleCombinerTask.
 <nevts>     - Number of events to analyze, 0 = default -> all events

The format of the output file name depends on the input <prefix>

- "myinput", min=1, max=10               -> myinput_ana_1_10.root 
- "myinput_2_pid.root", min/max ignored  -> myinput_2_pid_ana.root
- "myinput", min = max = 3               -> myinput_3_ana.root                       


----------
 Examples
----------

> root -l -b -q  'prod_ana.C("DPM10GeV", 1, 10)' 
> root -l -b -q  'prod_ana.C("D0Sim",    1, 20, 1)' 


--------------------------
 Submitting analysis jobs
--------------------------

Submitting the analysis macro from above as job to Kronos (e.g. to split to several jobs) can be done from macro/prod by

-> sbatch jobana_kronos.sh <prefix> <min> <max> <mode>

***                                                                                                  ***
*** prod_ana.C has to be configured for reconstruction, and the fastsim flag has to be set properly. ***
***                                                                                                  ***

Parameters:

 <prefix>    - Prefix for output files
 <min>       - run from this job number (default: 1)
 <max>       - .. to this (default: 20)
 <mode>      - arbitrary mode number; default=0. Stored in TTree produced by PndSimpleCombinerTask.


----------
 Examples
----------

> sbatch jobana_kronos.sh DPM10GeV 1 10 
> sbatch jobana_kronos.sh D0Sim    1 20 1 



---------------------------------
 Split to multiple analysis jobs
---------------------------------

In case a large number of output files have to be analysed, so that it is reasonable to split into several jobs,
the perl script 'anasub.pl' is useful. It works for both full and fast simulation input.


-> ./anasub.pl <pref> [mode] [num] [min] [max]

Parameters:
  <pref>  : prefix of output filenames of pattern 'data/<pref>_<run>_pid.root; if 'check_' is prepended only prints commands
            if the name of a file 'some_name.jobs' containing many sbatch commands is given, analysis submission is performed for all of them.
  [mode]  : mode number to be written in TTree; default: 0. mode = -1 counts up the mode for each line if filename is given as input
  [num]   : number of runs per job to be analysed; default: 50
  [min]   : first run number; if omitted all found files are considered
  [max]   : last run number; if omitted, all runs starting from [min] are considered


When parameters [min] and [max] are not given, the scripts searches for all files with the according pattern and finds limiting numbers automatically.


----------
 Examples
----------

> ./anasub.pl check_DPM10GeV             // prints all submit commands for available files 
> ./anasub.pl DPM10GeV 0 20              // submits analysis jobs on bunches of 20 inputs each
> ./anasub.pl DPM10GeV 0 25 201 400      // submits jobs in bunches of 25 for runs 201 to 400
> ./anasub.pl mystudy.jobs -1            // submits jobs according to all sbatch commands in file mystudy.jobs; run mode increased automatically