This course is part of the "LRZ AI Training Series", a series of courses aiming at the needs and expectations of data analytics, big data & AI users at LRZ. While focusing on these particular users and their use cases, this session as well as all other courses offered as part of the AI Training Series are, of course, open to all interested parties.

This course for academic participants from Germany will be organised as a hybrid event with the possibility to attend at LRZ in Garching near Munich or online. 

Contents

The aim of this course is to give an overview of the LRZ AI Systems, and provide participants with the knowledge and skills necessary to efficiently utilise them. The course consists of mini lectures, demos and hands on sessions (breaks included) covering the following topics:

• Resources overview of the LRZ AI Systems

• Fundamentals of Deep Learning

• Distributed Training of Neural Networks

Three blocks of content, devoting roughly an hour each to the first two, and two and a half hours to the third one (B=Beginner's, I=Intermediate, A=Advanced content):

• Overview of LRZ AI Systems (1h)
  • Hardware overview (B)
  • Access mode for the different resources (B)
  • Execution Mode (software stack) (B) + (I)

15min Break

• Fundamentals of Deep Learning (1h)

  • Introduction to Neural Networks (B)

  • Training Neural Networks (B)

  • Introduction to Convolutional Neural Networks (B)

  • Introduction to Transformers (B)
  • Exercises: Training Convolutional Neural Networks and Transformers on GPUs (I)

1h Break

• Distributed Deep Learning Training Part I (1h)
  • Motivation for Distributed Deep Learning Training (B)
  • Overview of Techniques for Distributed Deep Learning Training (B)

 15min Break

• Distributed Deep Learning Training Part II (1.5h)
  • Data Parallelism (I)
  • Exercise: Data Parallelism (I)
  • Model Parallelism - Pipeline Parallelism and Tensor Parallelism (A)
  • Demo: Pipeline Parallelism (A)