Course Information

• Course: LIN 393, Computational Linguistics with Minimal Supervision, 39435
• Semester: Fall 2019
• Webpage: http://jessyli.com/courses/lin393
• Canvas: https://utexas.instructure.com/courses/1254549
• Meeting: Tuesday/Thursday 2:00PM-3:30PM, Compling Lab, RLP 4.422
• Instructor: Jessy Li; email: jessy@austin.utexas.edu
  • Office hours: Thu 3:30pm-4:30pm, Fri 10:00am-11:00am, 3:00pm-4:00pm, RLP 4.728

Course Description

Recent progress in machine learning, especially in deep learning, has led to encouraging advances in Natural Language Processing: state-of-the-art results on benchmark datasets are getting renewed at a rapid pace. However, most of these methods rely on a large amount of labeled training data. Yet, high quality labeled textual data is expensive and difficult to build; the amount of labeled data is destined to be small compared to the vastness and dynamics of language (i.e., the number of different languages, possible genres/domains, and language evolution). In this seminar, we discuss several ways to address this bottleneck, including semi-supervised learning, noisy supervision, transfer learning, active learning, and domain adaptation. We focus on the unique challenges of natural language understanding and generation tasks.

Prerequisites

Graduate standing. Prior coursework in Computational Linguistics/Natural Language Processing/Machine Learning/a related field in AI, or instructor consent.

Texts

There will be no textbook for this seminar; reading material will consists of technical papers discussed in each meeting.

Organization and Content

• Research seminar: we will discuss a mix of theoretical and computational papers in discourse and pragmatics. Topics and readings will be given on the schedule page. For each paper, we will:
  • Give an initial summary of the paper;
  • Discuss comments regarding the paper, which may include strengths, weaknesses, and technical questions.
• Course project: each student will be involved in one course project throughout the semester (individually or in teams of 2). For the first half of the semester, explore potential topics and/or conduct preliminary experiments, and settle on a project. For the second half of the semester, work on the project. Specific deliverables include:
  • Midterm presentation (10 mins) and progress report (2-4 pages).
  • Final presentation (15 mins) and technical report (8 pages).

Grading Policy

• Class participation (20%)
  • Each class member will be leading some papers. A sign-up sheet will be distributed at the beginning of the semester.
• Class project (80% total):
  • Midterm report 25%
  • Midterm presentation 5%
  • Final report 40%
  • Final presentation 10%

Extension Policy

If you turn in your assignment late, expect points to be deducted. Extensions will be considered on a case-by-case basis. If you anticipate that you will need an extension for some assignment, let me know in advance.

By default, 5 points (out of 100) will be deducted for lateness, plus an additional 1 point for every 24-hour period beyond 2 that the assignment is late. The maximum extension penalty is 40 points if handed in before the last day of class. Resubmissions of assignments are allowed; extension penalty applies for post-deadline resubmissions. Note that there are always some points to be had, even if you turn in your assignment late. So if you would like to know if you should still turn in the assignment even though it is late, the answer is always yes.

Academic Dishonesty Policy

You are encouraged to discuss your project with classmates. All written work must be your own. Students caught cheating will automatically fail the course. If in doubt, ask the instructor.

Schedule

Note that this is a preliminary schedule that is subject to change.

• Week 1
  • 8/29: Course introduction; plan for the semester.
• Week 2: Active Learning
  • 9/3 We will go through Burr Settles’ survey on Active Learning
  • 9/5 Fang, Li and Cohn, Learning how to Active Learn; Michael will lead the discussion.
• Week 3: Active Learning / Semi-supervised Learning
  • 9/10 Lowell, Lipton and Wallace, Practical Obstacles to Deploying Active Learning, Jiyang will lead the discussion.
  • 9/12 We will read Xiaojun Zhu’s survey; let’s start with sections 1-6.1. Alex will lead the discussion.
    • Zhu and Ghahramani’s label propagation paper
    • Zhu’s encylopedia article on semi-supervised learning, pages 1142-1147.
• Week 4: Semi-supervised Learning
  • 9/17 We will read Section 13 of Xiaojun Zhu’s survey, and Kipf & Welling Semi-Supervised Classification with Graph Convolutional Networks. Shivangi will lead the discussion.
    • Blog posts on GCN: part 1 and part 2, and Thomas Kipf’s blog post
    • Working with directed graphs: Marcheggiani and Titov, Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
  • 9/19 We will read Miyato, Dai and Goodfellow, Adversarial Training Methods for Semi-Supervised Text Classification. Lixiang will lead the discussion.
• Week 5: Semi-supervised Learning
  • 9/24 We will read Athiwaratkun et al., There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average. Michael will lead the discussion.
  • 9/26 We will read Xia et al., Generalized Data Augmentation for Low-Resource Translation. Quang will lead the discussion.
• Week 6: Semi-supervised Learning / Domain Adaptation
  • 10/1 We will read Oliver et al., Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. Yiran will lead the discussion.
  • 10/3 We will read Ruder and Plank, Strong Baselines for Neural Semi-Supervised Learning under Domain Shift. Ryan will lead the discussion.
• Week 7: Domain Adaptation
  • 10/8 We will read Han and Eisenstein, Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling. Shailee will lead the discussion.
    • If you’re curious: Kim et al., Frustratingly Easy Neural Domain Adaptation
  • 10/10 Shrey Desai will present an overview and his recent work on domain adaptation:
    • Adaptive Ensembling: Unsupervised Domain Adaptation for Political Document Analysis at EMNLP’19
    • Evaluating Lottery Tickets Under Distributional Shifts at DeepLo’19
• Week 8: Project presentations
  • 10/15
  • 10/17
  • 10/18 Midterm report due at 11:59pm
• Week 9: Domain Adaptation/Transfer Learning
  • 10/22 We will read Zhang et al., Curriculum Learning for Domain Adaptation in Neural Machine Translation. Debadatta will lead the discussion.
  • 10/24 We will go through the NAACL tutorial on transfer learning (excluding the hands-on component). Elisa will lead the discussion.
  • 10/25 Info: Nazneen Rajani FAI talk
  • Office hour changes: Wed 11:30-1:00, Thu 3:30-5:00
• Week 10: Transfer Learning
  • 10/29 We will read Peters, Ruder and Smith, To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. Jiyang will lead the discussion.
  • 10/31 We will read Hsieh et al., On the Robustness of Self-Attentive Models. Quang will lead the discussion.
    • A related paper: Jin et al.,Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
• Week 11: Transfer Learning/Weak Supervision
  • 11/5 We will read Correia, Niculae and Martins, Adaptively Sparse Transformers. Yiran will lead the discussion.
  • 11/7 We will read Prabhumoye et al., Style Transfer Through Back-Translation. Alex will lead the discussion.
    • A blog post on VAEs
• Week 12: Weak Supervision
  • 11/11 Info: Diyi Yang talk 1:30pm GDC 3.516
  • 11/12 We will read Khetan, Lipton and Anandkumar, Learning From Noisy Singly-labeled Data. Shivangi will lead the discussion.
  • 11/14 We will read Ruber and Carenini, Predicting Discourse Structure using Distant Supervision from Sentiment. Ryan will lead the discussion.
  • 11/15 Info: Yoav Artzi FAI talk
• Week 13: Out-of-distribution Detection
  • 11/18 Info: Jason Baldridge Linguistics Colloquium talk
  • 11/19 We will read Hendrycks and Gimpel, A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Shailee will lead the discussion.
  • 11/21 We will read Nguyen and O’Connor, Posterior calibration and exploratory analysis for natural language processing models. Lixiang will lead the discussion.
  • 11/22 Info: Jacob Andreas FAI talk
• Week 14: Applications
  • 11/26 We will read Schuster et al., Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing. Debadatta will lead the discussion.
  • 11/28 Thanksgiving break
• Week 15: Project presentations
  • 12/3
  • 12/5
  • 12/6 Info: Dan Roth FAI talk
• Final report due 12/12 at 11:59pm