Third HSE-Yandex autumn school on generative models

23 Nov 2021, 09:45 → 26 Nov 2021, 22:00 Europe/Moscow

Princeton (YSDA Moscow)

Alexey Naumov, Dmitriy Vetrov (HSE), Andrey Ustyuzhanin (Yandex School of Data Analysis (RU)), Natalia Talaikova

HSE University and Yandex invite you to the joint third autumn school on generative models aimed at young undergraduate/graduate students planning to develop a research career. This intense four-day workshop will consist of 3 interdisciplinary mini-courses, a list of invited talks, and a lively discussions. Key topics of this school are:

• Generative models basics
• Advanced generative models 
• Efficient sampling for generative models

Due to the pandemic, the school will run in the mixed format: in addition to a zoom meeting room, a physical room is booked to allow participants to follow the event from the YSDA campus.

The school is organised by three HSE laboratories:

• LAMBDA (Laboratory of Methods for Big Data Analysis)
• DeepBayes (Centre of Deep Learning and Bayesian Methods)
• HDI Lab (International laboratory of stochastic algorithms and high-dimensional inference)

Yandex is an industry partner of the school.

Pre-requisites:

• Statistics 101
• machine learning 101
• neural networks basic level
• PyTorch

Mini-courses:

• Generative models by Denis Derkach (HSE University)
• Diffusion Generative Models by Dmitry Vetrov (HSE University) 
• Efficient Sampling Techniques by Alexey Naumov (HSE University)

Invited talks (TBC):

• Dmitry Vetrov (HSE)
• Alexey Naumov (HSE)
• Denis Derkach (HSE)
• Sergey Shirobokov (ICL, Twitter)
• Maxim Panov (Skoltech)
• Mikhail Burtsev (MIPT)
• Mikhail Lazarev (HSE)
• Nikita Kazeev (HSE)

Registration:

Please register for the school using this form.

School materials:

Please note that most of the materials (slides) will be in English, although the primary language of the school will be Russian. All materials of the school are available via this page timetable. It contains slides of the lectures and talks. All seminar materials are available via the github repository.

This school is supported by the RSF grant N19-71-30020, "Applications of probabilistic artificial neural generative models to the development of digital twin technology for Non-linear stochastic systems."

Tuesday 23 November

1 Welcome

Speaker: Andrey Ustyuzhanin (Yandex School of Data Analysis (RU))

1-opening.pdf

1-opening.pptx

2 Generative Models Intro

Speaker: Denis Derkach (National Research University Higher School of Economics (RU))

1-Intro.pdf

1-Intro.pptx

11:20 Coffee Break

LAMBDA lab: Introduction to Generative Models. Practice

Convener: Artem Ryzhikov

13:00 Lunch

3 GANs Introduction

Speaker: Denis Derkach (National Research University Higher School of Economics (RU))

2-GANs.pdf

2-GANs.pptx

15:20 Tea Break

LAMBDA lab: Generative Adversarial Networks. Practice

Convener: Artem Ryzhikov

Wednesday 24 November

Deep Bayes: Diffusion Generative Models-1

Convener: Dmitriy Vetrov (HSE)

11:20 Coffee Break

Deep Bayes: Diffusion Generative Models-1. Practice

Convener: Mr Victor Oganesyan (HSE University)

13:00 Lunch

Deep Bayes: Diffusion Generative Models-2

Convener: Dmitriy Vetrov (HSE)

4 Generative adversarial networks

Speaker: Mr Artem Maevskiy (HSE University)

15:20 Tea Break

Deep Bayes: Diffusion Generative Models-2. Practice

Convener: Victor Oganesyan (HSE University)

Thursday 25 November

HDI lab: Efficient sampling methods-1

Convener: Alexey Naumov

11:20 Coffee Break

HDI lab: Efficient sampling methods-2

Convener: Alexey Naumov

13:00 Lunch

HDI lab: Efficient sampling methods-3

Convener: Mr Sergey Samsonov (HSE)

15:20 Tea Break

HDI lab: Efficient sampling methods-4

Convener: Mr Sergey Samsonov (HSE)

Friday 26 November

5 Monte Carlo Variational Auto-Encoders

Speaker: Panov Maxim

6 Survey of methods of k-means clustering with optimal transport

TBA

Speaker: Suvorikova Alexandra

7 RICH GAN

Speaker: Mokhnenko Sergey

8 The Robustness of Deep Networks: A Geometrical Perspective

Speaker: Evgenii Burnaev

12:10 Coffee Break

9 Resolution-robust Large Mask Inpainting with Fourier Convolutions

Speaker: Roman Suvorov, Aleksei Silvestrov

10 Problems with Deep Learning: new AI winter or new synthesis?

Speaker: Mikhail Burtsev

13:30 Lunch

11 Material generation with AI

Speaker: Lazarev Mikhail

12 Uncertainty of Generative models

Speaker: Agatha Shishigina

15:30 Tea Break

13 Black-Box Optimization with Local Generative Surrogates

Speaker: Sergey Shirobokov

14 Weather tuning via surrogates

Speaker: Sergey Popov

15 Tackling the Challenge of Uncertainty Estimation and Robustness to Distributional Shift in Real-World applications

While much research has been done on developing methods for improving robustness to distributional shift and uncertainty estimation, most of these methods were developed only for small-scale regression or image classification tasks. Limited work has examined developing standard datasets and benchmarks for assessing these approaches. Furthermore, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction. In this work, we propose the Shifts Dataset for evaluation of uncertainty estimates and robustness to distributional shift. The dataset, which has been collected from industrial sources and services, is composed of three tasks, with each corresponding to a particular data modality: tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction. All of these data modalities and tasks are affected by real, `in-the-wild' distributional shifts and pose interesting challenges with respect to uncertainty estimation. We hope that this dataset will enable researchers to meaningfully evaluate the plethora of recently developed uncertainty quantification methods, assessment criteria and baselines, and accelerate the development of safe and reliable machine learning in real-world risk-critical applications.

An additional challenge to uncertainty estimation in real world tasks is that standard approaches, such as model ensembles, are computationally expensive. Ensemble Distribution Distillation (EnDD) is an approach that allows a single model to efficiently capture both the predictive performance and uncertainty estimates of an ensemble. Although theoretically principled, this work shows that the original Dirichlet log-likelihood criterion for EnDD exhibits poor convergence when applied to large-scale tasks where the number of classes is very high. Specifically, we show that in such conditions the original criterion focuses on the distribution of the ensemble tail-class probabilities rather than the probability of the correct and closely related classes. We propose a new training objective which resolves the gradient issues of EnDD and enables its application to tasks with many classes, as we demonstrate on the ImageNet, LibriSpeech, and WMT17 En-De datasets containing 1000, 5000, and 40,000 classes, respectively.

Speaker: Andrey Malinin

Organization: The end

Evening session: School dinner