Return to README CVS log

Up to [Local Repository] / badi / public_scripts / parallelstarter

Annotation of badi/public_scripts/parallelstarter/README, revision 1.1.1.1

1.1       adi         1: parallelstarter quick how-to
                      2: ----------------------------
                      3: 
                      4: parallelstarter is a script to start a process multiple times in parallel. It controls
                      5: the number of processes that run simultaniousely and it can call custom scripts to set
                      6: up an environment for each process. There are three examples included, one for the matsim
                      7: benchmark test, one for biogeme and one for a database backup script called db_backup.
                      8: The processes you start get their command line arguments from a file. Each line in this
                      9: file contains the arguments for one process to start; thus the number of lines give the 
                     10: number of processes to start in total.
                     11: 
                     12: System requirements: GNU/Linux
                     13: 
                     14: 
                     15: Installation/ Setup:
                     16: - check out the latest parallelstarter from IVT CVS
                     17: - cd into the checked out directory
                     18: - run ./install_parallelstarter
                     19: - cd to your working direcotory
                     20: - run setup_parallelstarter in the working directory you want to collect the output of
                     21:   your called processes. The script creates some softlinks to the predefined processes
                     22:   matsim, biogme or db_backup, depending the argument you pass to it.
                     23: - edit parallelstarter.setup to fit your needs:
                     24:   In this file you control the number of simultaniousely running processes,
                     25:   the command to start one and the command line arguments that are identical for all processes.
                     26: - you are now ready to call parallelstarter in this directory
                     27: 
                     28: For your own process setup, do the following:
                     29: - edit parallelstarter.prepare_process_start:
                     30:   This file is a script to copy or link the files or directories needed to run one process. Predefined matsim 
                     31:   for example expects a template directory, where the matsim Benchmark.jar lies (get ist from 
                     32:   http://matsim.org/benchmark). This is the working directory you would use normally to start the process 
                     33:   without parallelstarter. 
                     34:   This script should create a working directory in the directory you call parallelstarter for each process, change into
                     35:   it if the process writes to the current dir, it should set up links to libraries, copy or link config files etc. See
                     36:   the example for the matsim benchmark test by calling "setup_parallelstarter matsim".
                     37: - edit parallelstarter.cleanup:
                     38:   This file is a script for postprocessing, after all runs have finished.
                     39: - The example of biogeme generates its argument list file automatically, see the code for more details.
                     40:   
                     41: Using parallelstarter:
                     42: - Create a file with the differing command line arguments for your runs. biogeme and db_backup does this on its own, they 
                     43:   need a different config file from which they create the argument list file automatically. For matsim benchmark, you can
                     44:   test different JVM parameters for example by listing them in this file, each combination of switches on one line.
                     45:   It's name is configured in parallelstarter.setup.
                     46:   Each line in this file will trigger one run to start. The full command line for a process is put together as follows:
                     47: 
                     48:        $RUN_BIN $COMMON_LEADING_ARGS <one line from $ARGLIST_FILE> $COMMON_POST_ARGS
                     49:        ^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^                ^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^
                     50:        (the underlined options are set up in parallelstarter.setup)
                     51: 
                     52: - start your runs: parallelstarter  [--quiet --quiet-run --rundir-prefix --runid-offset]
                     53:   where the options have the following effect:
                     54:        --quiet                 suppress all output but errors
                     55:        --quiet-run             suppress output from the started process
                     56:        --rundir-prefix         prefix each run's working directory with this string
                     57:        --runid-offset          start numbering each run's working directory with this number
                     58:        --help                  all switches shortly explained