Topic: "Serial Farms: Package options and when to switch to farming"

Speaker: Tyler Collins, SHARCNET

Registration link

Recording

--- 

Small jobs are convenient to submit individually, but at scale they can overload a scheduler, inflate queue times, and ultimately reduce throughput. This webinar examines practical strategies for consolidating large numbers of short tasks, including job arrays, task bundling techniques, and wrapper-based aggregation. We will discuss how these approaches differ in overhead, portability, scheduler behavior, and job failure handling. The session will also provide guidance on recognizing when packaging options no longer yield sufficient throughput and when transitioning to a serial farming model becomes advantageous. Serial farms can mitigate scheduler pressure, improve wait times on busy clusters, and offer more predictable performance. Examples will be provided throughout the Webinar and shared on GitHub for future reference.

Zoom account is required. Webinar registration is required. VERY IMPORTANT: During registration, you have to provide the email address which is associated with your Zoom account, otherwise you won't be able to register and/or attend the webinar!  Need help attending a webinar? See the SHARCNET Help Wiki.