Coffee Chat with Chris Potts (Linguistics, CS)
/
All smiles when computational linguistics is the topic on the table!
Thoughts from Regina T.H. Ta ‘23
Professor Chris Potts joined SymSys Society for a virtual coffee chat! We discussed the current limits of computational linguistics, how we encode emotional expression into language, and what’s missing from deep learning data sets. We also debated whether machines could become “superstitious”: observing correlations and applying unknown causes.
For me, a highlight of the chat was our discussion on online translation--and how we can make it more robust! Fun fact from Prof. Potts: if you put “soy milk” into Google Translate, the Spanish translation ends up being: “I am milk!” The challenge is building translations that tune into the pragmatics, as well--not just the semantics.
We also gathered course recommendations for those interested in the intersection between linguistics and CS: take Linguist 130A for a theoretical foundation, then try CS 224U for a computational implementation!
Big thanks to all who joined us!
If you’re curious for more, here are Prof. Potts’ responses to questions that students submitted:
------------------------------
Since our brains are models of computation and even babies unconsciously use Bayesian probabilities to process language, what stops humans from being computers themselves?
I suppose my view on this is that the human brain just simulates or approximates the computational operations that we can define crisply in abstract terms. This is the best that can be done with a bunch of nerve fibers and other gunk! It's mysterious that we can have even partial knowledge of these computational operations.
As someone who studies AI and language, do you have special insights about the phenomenon wherein AI speakers like Alexa can’t understand people with thick accents in English, and how CS might be able to solve that problem?
It seems like data is the primary problem here. These systems just aren't trained on enough data of the right type. It's not that some accents are intrinsically easier or harder -- even if there are differences of that type, they aren't large enough to account for the observed differences in performance. That said, as we figure out how to get more data of the right kind, we might discover that there are intrinsic biases in the systems themselves. This was more obviously a factor in the days of lots of hand-built features than it is now in the era of deep learning, but it would be too hasty to assume that such biases aren't latent in today's models as well.
What are some findings of your research so far?
That's a big one! In recent work:
* I am proud that we settled (I hope!) a long-standing question about whether neural networks can truly learn to do reasoning about equality: https://arxiv.org/abs/2006.07968
* I'm also really excited by the mix of formal results and practical insights emerging from work in this mode: https://www.aclweb.org/anthology/2020.blackboxnlp-1.16/
Atticus Geiger is one of the authors with me on both these papers. He was a SymSys undergrad, and I feel like you can trace some of the above all the way back to his (award-winning) SymSys honors thesis.
Can your research be applicable globally even though every culture expresses emotion differently?
I hope so! On the linguistic side, I want to be offering insights about language in a very general sense. On the NLP side, I am striving for modeling approaches that are robust to variation as well!
How would you describe computational research methods, specifically in relation to your work and SymSys? Would you say it is especially applicable in what you are studying?
I would say that, at this point, knowing about computational methods is part of scientific literary. Not everyone needs to be a programmer or anything, but awareness of what is involved in programming and writing programs is important. Stanford is great for this because there are many, many contexts in which one can learn these skills.
For me, computational methods are vital because I can process lots and lots of data in interesting ways to learn things about how people use language.
What's your thoughts on the current big-Transformer stack approach to language?
I think it's great for lots of reasons. It has lifted all boats. It has led to sharing of model parameters that is very productive. It means you can do meaningful things with small datasets, via fine-tuning of large models. And contextual word representations are much closer to how linguists think about words and phrases that static vectors are.
Do you think that models in the future should incorporate more linguistic priors? If so, which approaches do you think hold the most promise?
Tough to say! The Transformer is a triumph of low-bias model structure with hardly any priors.
Does language cause intelligence or vice versa?
I am sure it's a back and forth. Language is clearly a cognitive tool. Maybe a more controversial question would be whether *communication* with language is a necessary piece.
What's a current limit of computational linguistics that you're most excited about pushing past?
I think we need models that are more grounded in aspects of the human experience that go beyond mere streams of text. For more nuance on this, perhaps check out his blog post I did:
“Is it possible for language models to achieve language understanding?” https://chrisgpotts.medium.com/is-it-possible-for-language-models-to-achieve-language-understanding-81df45082ee2
When you first started working computational methods into linguistics, what was your long-term project or goal, if you had one?
I started using computational methods because I wanted to learn more about what people mean when they swear! To do this, I needed to see a lot of instances of swears in the real world, and computational linguistics is the tool-kit for that. I later realized that I was doing what had recently (at the time) been established as "sentiment analysis" within NLP.
Are you studying and finding emotions based on the connotations of words, or also the tone of voice? Is it possible to use computational methods/CS to analyze tone of voice yet?
I think there is progress on this in speech research. Some if it is covered in CS224s, I believe: http://web.stanford.edu/class/cs224s/syllabus/
How supportive would you say is the Stanford Symsys community to your research, and are there others studying similar things?
I feel like SymSys is my most natural intellectual home. It's the only place in the world that brings together a real mix of people doing work that's computational, linguistic, social, psychological, philosophical, and on and on, and I love how many creative ways people in the SymSys community are finding to connect this work to things with real social significance and impact.
Do you think traditional linguistic theory on semantics/pragmatics has/will have a role to play in the development or analysis of natural language understanding technologies, and if so, what does/will it look like?
I think so. NLPers tend to overlook, or forget about, all the things they do that are shaped by linguistic theory. I expect that to continue, with pragmatics being the latest place where we see these rich influences.
I should add that I don't think this is a requirement, for linguistics or any science. We shouldn't measure the worth of a science based on how much it contributes to engineering efforts. I myself am always seeking such connections, but I am glad that there are many linguists who are not doing that, but rather trying to understand and document languages for their own sake.