macro/prodgsi
----------------------

A set of macros to run various kinds of simulation either locally or submit them for larger amounts of events to Prometheus-Cluster@GSI.

See also 

  http://wiki.gsi.de/cgi-bin/view/Linux/GridEngine 

for general description of usage of GSI GridEngine and

  http://panda-wiki.gsi.de/cgi-bin/viewauth/Computing/PandaRootSimulationPrometheus

for a nicer look-and-feel of the information given below.


######
FILES:
######

Simulation
- prod_sim.C
- prod_sim_pgun.C
- prod_dig.C
- prod_rec.C
- prod_pid.C
- prod_pid_mva.C

Analysis example for multiple files using PndSimpleCombiner(Task)
- prod_ana.C

Example decay file for EvtGen
- D0toKpi.dec

Scripts to run all at automatically
- runall_prod_pgun.sh
- runall_prod.sh

Scripts to (re-)submit jobs to Prometheus
- job_prod_pgun.sge
- job_prod.sge
- resub.pl

Script to submit analysis job for many input files
- job_ana.sge 
- anasub.pl


####################
Run LOCALLY IN SHELL
####################
-------------
Box Generator
-------------

-> ./runall_prod_pgun.sh <prefix> <N_ev> <PDG code> <max mom> <mult>

with parameters
 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <PDG code>  - PDG code of particle(s) to be created
 <max mom>   - Generate particles with p = 0.01 ... <max mom>, isotropic in cos(theta) and phi
 <mult>      - Multiplicity in single events

Produces output files <prefix>_sim.root, <prefix>_par.root, <prefix>_dig.root, <prefix>_rec.root, <prefix>_pid.root in current directory.

Example:
./runall_prod_pgun.sh PiPlus 1000 211 1

Result: 
PiPlus_sim.root, ... , PiPlus_pid.root


--------
EvtGen 
--------

-> ./runall_prod.sh <prefix> <N_ev> <decay> <pbar mom> <res>

with parameters
 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <decay>     - EvtGen decay file
 <pbar mom>  - Momentum of antiproton beam
 <res>       - Initial resonance used in decay file 

Produces output files <prefix>_sim.root, <prefix>_par.root, <prefix>_dig.root, <prefix>_rec.root, <prefix>_pid.root in current directory.

Example:
./runall_prod.sh D0Kpi 1000 D0toKpi.dec 10.0 pbarpSystem 

Result: 
D0Kpi_sim.root, ... , D0Kpi_pid.root


--------
DPM
--------

-> ./runall_prod.sh <prefix> <N_ev> DPM <pbar mom>

Runs DPM generator in _inelastic_ mode. Parameters as above, just replace name of decay file with key word 'DPM'.

 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <pbar mom>  - Momentum of antiproton beam

Produces output files <prefix>_sim.root, <prefix>_par.root, <prefix>_dig.root, <prefix>_rec.root, <prefix>_pid.root in current directory.

Example:
./runall_prod.sh DPM10GeV 1000 DPM 10.0  

Result: 
DPM10GeV_sim.root, ... , DPM10GeV_pid.root


####################
SUBMIT TO CLUSTER
####################

General information about submitting jobs on cluster can be found here:
http://wiki.gsi.de/cgi-bin/view/Linux/GridEngine

BEFORE being able to submit jobs to Prometheus, you have to 

1. login to a lxsub machine by ssh yourname@pro.hpc.gsi.de  

2. create a work directory on HERA, e.g. /hera/panda/yourname/pandaroot

3. install PandaROOT in that place, since your GSI linux home directory is not mounted on Prometheus nodes.
   (SIMPATH is /cvmfs/fairroot.gsi.de/fairsoft/apr13/ for apr13 external packages)


-------------
Box Generator
-------------

-> qsub -t <min>-<max> job_prod_pgun.sge <prefix> <N_ev> <PDG code> <max mom> <mult> <saveall>

Submits an array of jobs with array numbers <min> to <max>

with parameters
 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <PDG code>  - PDG code of particle(s) to be created
 <max mom>   - Generate particles with p = 0.01 ... <max mom>, isotropic in cos(theta) and phi
 <mult>      - Multiplicity in single events
 <saveall>   - "saveall" copies all output stages (par, sim, dig, rec, pid) to data dir; w/o parameter only pid+par is copied

Produces output files M..._sim.root, M..._par.root, M..._dig.root, M..._rec.root, M..._pid.root and log files in data directory, 
with '...' running from <prefix>_<min> to <prefix>_<max>. 

Example 1:
qsub -t 1-20 job_prod_pgun.sge PiPlus 1000 211 1 saveall

Result 1: 
data/MPiPlus_1_par.root, data/MPiPlus_1_sim.root, data/MPiPlus_1_dig.root, data/MPiPlus_1_rec.root, ... , data/MPiPlus_20_pid.root 

Example 2:
qsub -t 1-20 job_prod_pgun.sge PiPlus 1000 211 1

Result 2: 
data/MPiPlus_1_par.root, data/MPiPlus_1_pid.root, ... , data/MPiPlus_20_par.root, data/MPiPlus_20_pid.root 

--------
EvtGen 
--------

-> qsub -t <min>-<max> job_prod.sge <prefix> <N_ev> <decay> <pbar mom> <res>

Submits an array of jobs with array numbers <min> to <max>

with parameters
 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <decay>     - EvtGen decay file
 <pbar mom>  - Momentum of antiproton beam
 <res>       - Initial resonance used in decay file 
 <saveall>   - "saveall" copies all output stages (sim, dig, rec, pid) to data dir; w/o parameter only pid is copied

Produces output files M..._sim.root, M..._par.root, M..._dig.root, M..._rec.root, M..._pid.root and log files in data directory, 
with '...' running from <prefix>_<min> to <prefix>_<max>. 

Example 1:
qsub -t 1-20 job_prod.sge D0Kpi 1000 D0toKpi.dec 10.0 pbarpSystem saveall

Result 1: 
data/MD0Kpi_1_par.root, data/MD0Kpi_1_sim.root, data/MD0Kpi_1_dig.root, ... , data/MD0Kpi_20_pid.root 

Example 2:
qsub -t 1-20 job_prod.sge D0Kpi 1000 D0toKpi.dec 10.0 pbarpSystem

Result 2: 
data/MD0Kpi_1_par.root, data/MD0Kpi_1_pid.root, ... , data/MD0Kpi_20_pid.root 


--------
DPM
--------

-> qsub -t <min>-<max> job_prod.sge <prefix> <N_ev> DPM <pbar mom> <dummy> <saveall>

Submits an array of jobs with array numbers <min> to <max>

Runs DPM generator in _inelastic_ mode. Parameters as above, just replace name of decay file with key word 'DPM'.
 <prefix>    - Prefix for output files
 <N_ev>      - Number of events to simulate
 <pbar mom>  - Momentum of antiproton beam
 <dummy>     - if running DPM and using switch 'saveall', some dummy string must be used
 <saveall>   - "saveall" copies all output stages (sim, dig, rec, pid) to data dir; w/o parameter only pid is copied

Produces output files M..._sim.root, M..._par.root, M..._dig.root, M..._rec.root, M..._pid.root and log files in data directory, 
with '...' running from <prefix>_<min> to <prefix>_<max>. 

Example 1:
qsub -t 1-20 job_prod.sge DPM10GeV 1000 DPM 10.0 summy saveall 

Result 1: 
data/MDPM10GeV_1_par.root, data/MDPM10GeV_1_sim.root, data/MDPM10GeV_1_dig.root, ... , data/MDPM10GeV_20_pid.root 

Example 2:
qsub -t 1-20 job_prod.sge DPM10GeV 1000 DPM 10.0

Result 2: 
data/MDPM10GeV_1_par.root, data/MDPM10GeV_1_pid.root, ... , data/MDPM10GeV_20_pid.root 



---------------------
---------------------
RESUBMISSION OF JOBS
---------------------
---------------------

When submitting many jobs/job arrays, it happens quite often that some of the jobs randomly fail. Usually resubmitting the same jobs
again leads to success. In order to automatically resubmit only the failed jobs the perl script 'resub.pl' can be used.

It looks for missing output files and output files being too small (size < 10kB) to contain reasonable output and resubmits exactly 
the according jobs.

NOTE: In case some jobs got stuck in the queue, delete them with 'qdel' before using resub.pl.

-> ./resub.pl "<cmd>" [check]

Parameters:

  "<cmd>"  : full qsub command line in quotes. 
  [check]  : optional parameter for only printing the resubmitted jobs for checking.
  

Example 1:
./resub.pl "qsub -t 1-20 job_prod.sge DPM10GeV 1000 DPM 10.0 summy saveall" check

Result 1: prints out all run numbers found to be failed and prints submit commands.

Example 2:
./resub.pl "qsub -t 1-20 job_prod.sge DPM10GeV 1000 DPM 10.0"

Result 2: prints out all run numbers found to be failed and re-submits the according jobs. 



####################
ANALYSING RESULTS
####################

RUN LOCALLY:

In order to run over a number of your resulting output (_pid.root) files, the macro prod_ana.C can be used as follows, e.g. from the data dir:

-> root -l -b -q '../prod_ana.C("<prefix>", <min>, <max>, <N_ev>)'

Adds several input files <prefix>_<min>_pid.root, ... , <prefix>_<max>_pid.root and <prefix>_<min>_par.root as par file to FairRunAna to run analysis.
Broken or missing files are skipped automatically.

Parameters:
 <prefix>    - Prefix for output files
 <min>       - run from this job number
 <max>       - .. to this
 <N_ev>      - Number of events to analyze, 0 = default -> all events

Example:
root -l -b -q  '../prod_ana.C("MDPM10GeV", 1, 10)' 

Result: 
data/MDPM10GeV_ana_1_10.root


SUBMITTING ANALYSIS JOBS:

Submitting the analysis macro from above as job to Prometheus (e.g. to split to several jobs) can be done from macro/prod by

-> qsub job_ana.sge <prefix> <min> <max> 

Parameters:
 <prefix>    - Prefix for output files
 <min>       - run from this job number
 <max>       - .. to this

Example:
qsub job_ana.sge MDPM10GeV 1 10 

Result: 
data/MDPM10GeV_ana_1_10.root


-------------------------------
SPLIT TO MULTIPLE ANALYSIS JOBS
-------------------------------

In case a large number of output files have to be analysed, so that it is reasonable to split that into several jobs,
the perl script 'anasub.pl' is useful.


-> ./anasub.pl <pref> [num] [min] [max]

Parameters:
  <pref>  : prefix of output filenames of pattern 'data/<pref>_<run>_pid.root; if 'check_' is prepended only prints commands
  [num]   : number of runs per job to be analysed; default: 50
  [min]   : first run number; if omitted all found files are considered
  [max]   : last run number; if omitted, all runs starting from [min] are considered
  
 
Example 1:
./anasub.pl check_MDPM10GeV 

Result 1: Searches for all output files data/MDPM10GeV_<run>_pid.root and only prints submission commands for analysis jobs in packages of 50 runs.

Example 2:
./anasub.pl MDPM10GeV 20

Result 2: Searches for all output files and submits analysis jobs in packages of 20 runs, i.e. 1..20, 21..40, 41..60, ...

Example 3:
./anasub.pl MDPM10GeV 25 201 400 

Result 3: Submits analysis jobs for runs 200 to 400 in packages of 20 runs.