Topic: "File Management - Packing Small Files"

Speaker: Ching-Hsing Yu, SciNet

Video link

--- 

Managing large amounts of data can be a challenging task. Processing large numbers of files incurs heavy overhead of I/O communications. In this talk, we will explore several options such as using Apptainer Overlay and SQLite to pack and reduce a large number of files to few files, and hence, improving IO performance. Python scripts will be used throughout the presentation.

The Compute Ontario Colloquia are weekly Zoom presentations on Advanced Research Computing, High Performance Computing, Research Data Management, and Research Software topics, delivered by staff from three Compute Ontario consortia (CAC, SciNet, SHARCNET) and guest speakers. The series began January 2023 and superseded similar series previously delivered by individual consortia (e.g. General Interest Seminars by SHARCNET or User Group Meeting TechTalks by SciNet). The colloquia are one hour long and include time for questions. No registration is required. Presentations are usually recorded and uploaded to the hosting consortium video channel.