Syllabus

Course Information

• Course: LIN 393, Social Applications and Impact of NLP
• Semester: Spring 2023
• Webpage: http://jessyli.com/courses/lin393
• Canvas: https://utexas.instructure.com/courses/1359609
• Meeting: Tuesday/Thursday 2:00 p.m.-3:30 p.m., RLP 4.422
• Instructor: Jessy Li; email: jessy@austin.utexas.edu
  • Office hours: Tuesday 9-10, Wednesday 12-1, RLP 4.728 or Zoom (link on Canvas)

Course Description

Natural language processing (NLP) techniques have advanced our ability to process text at a massive scale, and shed light into our society. In this seminar, we discuss (1) models and techniques that are designed to handle temporal & geological variations of language, and the noisy nature of social media text; (2) the application of NLP to understand emotions and social discourse; (3) the ethical implications of large-scale models, such as misinformation and model bias.

Prerequisites

Graduate standing. Prior coursework in Computational Linguistics/Natural Language Processing/Machine Learning/a related field in AI, or instructor consent.

Texts

There will be no textbook for this seminar; reading material will consists of technical papers discussed in each meeting.

Organization and Content

• Research seminar: we will discuss a mix of theoretical and computational papers in discourse and pragmatics. Topics and readings will be given on the schedule page. For each paper, all students are required to prepare:
  • 3 bullets about what you like or don’t like about the paper;
  • Any technical questions.
• Course project: each student will be involved in one course project throughout the semester (individually or in teams of 2). For the first half of the semester, explore potential topics and/or conduct preliminary experiments, and settle on a project. For the second half of the semester, work on the project. Specific deliverables include:
  • Midterm presentation (7 mins) and report (2-4 pages).
  • Final presentation (12 mins) and report (6-8 pages).

Grading Policy

• Class participation (5%)
• Discussion points (35%)
  • Please submit your discussion points before class; each submission accounts for 1.5%
• Class project (60% total):
  • Midterm report & presentation 25%
  • Final report & presentation 35%

Extension Policy

All extensions will be negotiated on a case-by-case basis. You must talk to the instructor before the deadline in question. If you did not obtain a permission on extension, then by default, 10 points (out of 100) will be deducted for lateness, plus an additional 5 point for every 24-hour period beyond 2 that the assignment is late.

Notice about students with disabilities

The University of Texas at Austin provides upon request appropriate academic accommodations for qualified students with disabilities. Please contact the Division of Diversity and Community Engagement, Services for Students with Disabilities.

Notice about missed work due to religious holy days

A student who misses an examination, work assignment, or other project due to the observance of a religious holy day will be given an opportunity to complete the work missed within a reasonable time after the absence, provided that he or she has properly notified the instructor. It is the policy of the University of Texas at Austin that the student must notify the instructor at least fourteen days prior to the classes scheduled on dates he or she will be absent to observe a religious holy day. For religious holy days that fall within the first two weeks of the semester, the notice should be given on the first day of the semester. The student will not be penalized for these excused absences, but the instructor may appropriately respond if the student fails to complete satisfactorily the missed assignment or examination within a reasonable time after the excused absence.

Schedule

• Week 1
  • 1/10 During the first class, we will work together to decide the exact topics of the semester that are of interest.
  • 1/12 Webson and Pavlick, Do Prompt-Based Models Really Understand the Meaning of their Prompts?
• Week 2
  • 1/17 Wang et al., Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
    • Boshi Wang and Sewon Min will join our discussion!
    • Background reading: Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • 1/19 Shaikh et al., On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
• Week 3
  • 1/24 Pyatkin et al., Reinforced Clarification Question Generation with Defeasibility Rewards for Disambiguating Social and Moral Situations
    • Valentina Pyatkin will present and join our discussion!
    • Background reading: Jiang et al., Can Machines Learn Morality? The Delphi Experiment
  • 1/26 Jin et al., When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment
• Week 4
  • 1/31 Classes canceled. Stay safe and warm!
  • 2/2 Classes canceled. Stay safe and warm!
• Week 5
  • 2/7 Gonen and Goldberg, Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them
    • Background reading: Blodgett et al., Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  • 2/9 Bommasani and Liang, Trustworthy Social Bias Measurement
• Week 6
  • 2/14 Ravfogel et al., Linear Adversarial Concept Erasure
    • Background reading: Ravfogel et al., Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
  • 2/16 Perez et al., Red Teaming Language Models with Language Models
• Week 7
  • 2/21 Cheng et al., Social norm bias: residual harms of fairness-aware algorithms
    • Myra Cheng will join our discussion!
  • 2/23 Kozyreva et al., Resolving content moderation dilemmas between free speech and harmful misinformation
• Week 8
  • 2/28 Project proposal presentations
  • 3/2 Project proposal presentation
  • 3/5 Project proposal due
• Week 9
  • 3/7 Das et al., The state of human-centered NLP technology for fact-checking
    • Anubrata Das will join our discussion!
  • 3/9 Gupta et al., Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection
    • Soumyajit Gupta will join our discussion!
• Week 10
  • Spring break
• Week 11
  • 3/21 Glockner et al., Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation
  • 3/23 Gabriel et al., Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines
• Week 12
  • 3/28 Miller, Explanation in artificial intelligence: Insights from the social sciences
  • 3/30 Zhou et al., Navigating the Grey Area: Expressions of Overconfidence and Uncertainty in Language Models
• Week 13
  • 4/4 Yang et al., How Many and Which Training Points Would Need to be Removed to Flip this Prediction?
    • Necessary background: Koh and Liang, Understanding Black-box Predictions via Influence Functions
  • 4/6 Guu et al., Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs
• Week 14
  • 4/11 Santurkar et al., Whose Opinions Do Language Models Reflect?
  • 4/13 Pan et al., Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark
• Week 15
  • 4/18 Project presentations
  • 4/20 Project presentations
• Final project report due: 4/30 at 11:59pm, hard deadline