Utilities for submitting jobs to cluster

Introduction

This package provides utilities for using/running the qsub for job submission to a cluster of machine. Documentation on qsub can be found here and was initially developped to be use on Sun Grid Engine aka SGE. The system of commands – qsub, qstat, … – are also used in the in2p3 Computing Center. For more details on how to run batch jobs, refer to the following howto.

Repository content

For the time being, there are mainly two python3 scripts :

qsubmit.py

the script send job submission given a configuration file where several options, interactive commands can be set. It can be run on a local machine and will use a remote connection via paramiko python package to properly send jobs to CC.

qsurvey.py

the script runs as a daemon and looks for the job submission status by sending a summary of number of jobs, job loads… via a notification popup.

Installation

First, you need to clone this repository by doing

git clone https://github.com/xgarrido/snemo_qsubmit

To use the scripts, you need a recent version of python i.e. at least, version 3.X. You also need to have =paramiko= package in order to remotely access servers. The easiest way to get it is to use =pip= tool and to do pip install paramiko. Except from these external dependencies, you just have to run ./qsubmit.py within the directory where you have cloned snemo_qsubmit repository. See below for the different options.

Script usage

Both scripts come with different running options. You may run ./qsubmit.py --help to get an overview of the different options.

qsubmit.py

• --log option provides logging capabilities. The option must be followed by the logging priority that can be critical,error,warning,info,debug,
• --config followed by the configuration file will load the job setup such as CPU/memory resources, job duration, script content… For a complete example of configuration, see =snemo_sensitivity_study.conf= file.
• --hostname, --username, --password allow you to define your login/password to get connected to the hostname machine.
• --test this mode only runs the full process without sending job to the server. This has to be used for debugging purposes.