https://docs.google.com/presentation/d/15jNNCisvF09H6NB-Wc_JYFeZihCYxeZtvoHrZSGD_5E/edit?usp=sharing

https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

How to time CUDA GEMM accurately

• https://github.com/ai-compiler-study/test_attn/blob/69d660ff0d2d15f9a7dd0855b59890cd1769f04f/benchmark_attention.py#L53
• https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964/7
• torch.cuda.synchronize
• https://blog.speechmatics.com/cuda-timings