Topic: "Parallel Job Orchestration with GNU Parallel"

Speaker: Ramses van Zon, SciNet

Video link

--- 

Many research computing projects involve workloads of large amounts of relatively short, independent computations.  Such workloads are a common reason for researchers to move to shared resources. Despite being called an "embarrasingly parallel" problem, orchestrating large amounts of small computational jobs is surprisingly subtle, particularly when dealing with shared resources with their own usage policies and implementations.  To deal with this, many tools have been created, used for a while, and then abandoned. Here, we will focus on a tool for job orchestration that is very versatile and actively maintained, GNU Parallel, and show how it helps for a number of use cases.

The Compute Ontario Colloquia are weekly Zoom presentations on Advanced Research Computing, High Performance Computing, Research Data Management, and Research Software topics, delivered by staff from three Compute Ontario consortia (CAC, SciNet, SHARCNET) and guest speakers. The series began January 2023 and superseded similar series previously delivered by individual consortia (e.g. General Interest Seminars by SHARCNET or User Group Meeting TechTalks by SciNet). The colloquia are one hour long and include time for questions. No registration is required. Presentations are usually recorded and uploaded to the hosting consortium video channel.