Topic

Study Links

Deliverable

Estimated Time

Transfer Learning

Yandex Course; NLP ImageNet Moment; Illustrated BERT; fastai NLP;

Optional, but Recommended: Domain Adaptation Sections 3.4 and 4 from Ruder’s Thesis; Section 7 on fine-tuning pretrained models

Notebook: Try out BERT on an email classification problem. Write a 200-500 word blog on Transfer Learning

4-6 hours

Sub-word Models

cs224n slides; BERT Pytorch implementation - check the tokenizer; Mikolov paper; Words vs Morpheme: Paper

Notebook: Sub-word sentence classification model on a standard dataset; comparing to full word for code-mixed data

10 hours

Multi-Task Learning (MTL); NER

Socher Guest Lecture on MTL; decaNLP; Flair; Haptik Chatbot NER; HMTL by HuggingFace; 

Cyclic Learning Rate Scheduler

Notebook: Yandex Homework;
Optional: Tutorial implementing NER for a chat corpus with a list of custom entities

5-7 hours

~No New Topics~

Yes, you should write a blog - Rachel Thomas;
Generating text with GPT-2

Blog or new experiment notebook: Based on any 2 or more themes

3-7 hours

Model Interpretations and Visualisation

Kaggle Learn module on the topic; example: BERT Multi-lingual Vocabulary; GPT-2 demo

Notebook or blog: Explain BERT and USE predictions with one or more methods

6-10 hours