Purpose

We aim to study methods to reduce Memory bandwidth cost and overhead by optimizing AI models written in Python through:

• Kernel level optimization such as Triton, CUDA kernel, CUDA Graph
• Computational graph optimization via IR and graph compilers like TorchInductor, TVM
• Model design considering hardware, such as FlashAttention, Ring Attention

Many companies, including OpenAI, are enhancing performance through numerous low-level optimizations that are often omitted in papers, making these technologies seem rarer and less mentioned.

We seek to gather individuals interested in learning and sharing knowledge about these valuable technologies.

Plan

We plan to communicate through Slack on a regular basis and have a weekly online meeting. The regular meetings will involve presentations by at least two members, followed by sharing what everyone has studied over the week.

1. Basics (2024.04 ~ 2024.06)

https://horace.io/brrr_intro.html

Individual Practice

• Watch lectures at https://github.com/cuda-mode and practice individually
• Practice profiling and optimizing other functions and models not mentioned in the presentation

Weekly Meeting