Topic: "META: running a large number of jobs conveniently"

Speaker: Sergey Mashchenko, SHARCNET

Zoom link

Recording

If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools for speeding up the workflow (the built-in Job Array feature of our scheduler, SLURM, GNU Parallel, or GLOST) leave a lot to be desired. Here we introduce our own home-grown package METAdesigned to overcome the shortcomings of the existing tools. META can significantly accelerate your workflow involving running lots of jobs (short or long, serial or parallel, CPU or GPU based). Features like the ability to resubmit all the jobs which either failed or never ran, using a single command, are likely unique to this package. In this webinar we will cover the main features of the package, and will also discuss the typical scenarios when META can become very useful or even essential.

Zoom account is required to attend the webinar. Need help attending a webinar? See the SHARCNET Help Wiki.