Topic: "Squeeze more juice out of a single GPU in deep learning"

Speaker: Weiguang Guan, SHARCNET

Video link

Recording

 

--- 

It’s well known that GPUs can significantly accelerate neural network training. However, not everyone knows that a single GPU is sufficient to train most neural networks except for a few large ones (like LLM). In fact, a GPU is under-utilized in most cases. In this talk, we are addressing the under-utilization issue and proposing a way to make full use of the GPU capacity. The goal is to increase the throughput with a single GPU. We will use a small NN training as an example to illustrate how to achieve the goal by splitting a physical GPU into multiple logical GPUs and then running a particular training process per logical GPU.

The Compute Ontario Colloquia are weekly Zoom presentations on Advanced Research Computing, High Performance Computing, Research Data Management, and Research Software topics, delivered by staff from three Compute Ontario consortia (CAC, SciNet, SHARCNET) and guest speakers. The series began January 2023 and superseded similar series previously delivered by individual consortia (e.g. General Interest Seminars by SHARCNET or User Group Meeting TechTalks by SciNet). The colloquia are one hour long and include time for questions. No registration is required. Presentations are usually recorded and uploaded to the hosting consortium video channel (colloquia hosted by SHARCNET go to our youtube channel).