jobber - A tool for managing task-parallel computations

Introduction

The main idea of jobber is to enable filling compute nodes with single-core tasks in an easy-to-use way. For reasons of simplicity jobber knows only one method for specifying tasks to be processed which is a task list.

Parallel execution of tasks from a given task list is possible at two levels: one node can be filled by one instance of jobber, several nodes can be filled by running independent instances of jobber simultaneously. Technically this is possible by employing synchronization mechanisms of shared file-systems. A practical consequence is that all jobber instances processing the same task list must have access to the file system on with the task list is stored.

In addition jobber provides an environment for working with task queues. In particular, jobber can report on task failures and repeat tasks that failed. See section Examples for more information on task queue management actions.

This page describes how to use jobber interactively on the command line. When used in conjunction with a batch system most commands will still be executed on the command line (cf. our batch job examples page).

Download

jobber is written in perl. It is installed on RRZ HPC systems. For use outside RRZ the current standalone version of jobber can be downloaded from the author's web page.

Two parallel computing basics

Task parallelism

In parallel computing work needs to be decomposed such that calculations can be carried out simultaneously. Parallelizing a sequential program is demanding. Task parallel problems are also called trivially parallel or embarrassingly parallel problems, because a given program can remain unchanged and parallelism can be exploited by running that program simultaneously on separate data sets. Classic examples of this type of parallel computations are parameter studies and the generation of individual pictures for an animation.

Generally, independent tasks of any kind can be executed simultaneously. Tasks are independent if there is no communication or coordination needed between them.

Task granularity

Typically some overhead is introduced by parallelization. While overhead is expected to be minimal for task parallel computations, it should be mentioned that the file-system based synchronization used by jobber does introduce overhead. As a consequence jobber is not suited for managing parallel execution of very short running tasks. In practice, execution time per tasks should be at least a few minutes.

jobber overview

jobber can:

process a list of tasks in chunks
execute tasks in parallel by
- employing its --parallel option
- starting several jobber instances
be used
- standalone (e.g. on a personnel computer, explained on this page)
- in conjunction with a batch system (see our batch job examples page)

Internally jobber:

works with a task queue
parallel processing of the task queue is based on synchronization mechanisms of a shared file system

jobber command line syntax:

prerequisite
```
   module load jobber
```

execute number_of_tasks or all tasks from task_list

   jobber [options] task_list all
   jobber [options] task_list number_of_tasks

task queue management

   jobber [options] task_list action [more actions]

built-in help

   jobber --help
   jobber --help=-100 | pod2text

Task list and task queue

Task list

The command line parameter task_list is the name of a file in which the tasks to be processed are specified. A task is specified on a single line in task_list. The line will be executed by the shell and may contain arbitrary commands. Every line, including the last one, must end with a line break. Continuation lines are not supported. Lines commented via leading # or containing only white space are ignored.

A task_list must be prepared before jobber can be used. The list can, for example, be created with a text editor or be generated with a script.

Task queue

Internally jobber maps the task list to a task queue. The queue needs to be initialized with the init action:

jobber task_list init

The minimal sequence of commands for using jobber looks like this:

jobber example.list init  # create/generate example.list
jobber example.list 42    # run 42 tasks from example.list

The command for running 4 tasks simultaneously would be:

jobber --parallel=4 example.list 42

For maintaining the queue jobber creates two files in the directory where task_list is located. The files are called task_list.cont and task_list.queue, for the example above:

example.list.cont
example.list.queue

Examples

Examples for almost all options and actions of jobber are given on these pages: