LSLT: Chenglei Si (CS/LING)

Hot Takes on Modern Language Models

Abstract: Large language models have taken the field of natural language processing by storm. In fact, they exhibit so strong performance on a variety of tasks to the extent that there starts to be hypes calling these models conscious, sentient, or general intelligence. At the same time, criticisms arise, arguing these language models are just stochastic parrots and are not trustworthy. In this talk, I will present views from both sides and give my own takes on this debate.
In particular, I will first give a brief introduction on state-of-the-art language models, their emergent capabilities and several cool applications powered by them. Next, I will introduce my new paper that systematically examines the reliability of the popular GPT-3 model, where I will present ample empirical evidence to show that GPT-3 is actually more reliable than many people think, such as being able to generalize out-of-domain, minimizing discrimination and social biases, providing calibrated uncertainty measures, and can be easily updated to reflect new knowledge conflicting with their memorization. Lastly, I conclude by telling a story of how I spent two years on a project trying to inject more linguistic inductive biases into the design of language models. Retrospectively, although this attempt leads to empirical gains in the short term, I will share my thinking of why I think linguistic knowledge has a diminishing role in the advancement of modern language models in the long term. If you want to get up-to-date about the latest (and most exciting) trends in NLP, this is the talk that you should not miss.