Learning Language through Interaction

Machine learning-based natural language processing systems are amazingly effective, when plentiful labeled training data exists for the task/domain of interest. Unfortunately, for broad coverage (both in task and domain) language understanding, we're unlikely to ever have sufficient labeled data, and systems must find some other way to learn. I'll describe a novel algorithm for learning from interactions, and several problems of interest, most notably machine simultaneous interpretation (translation while someone is still speaking).

This is all joint work with some amazing (former) students He He, Alvin Grissom II, John Morgan, Mohit Iyyer, Sudha Rao and Leonardo Claudino, as well as colleagues Jordan Boyd-Graber, Kai-Wei Chang, John Langford, Akshay Krishnamurthy, Alekh Agarwal, Stéphane Ross, Alina Beygelzimer and Paul Mineiro.

On the Synergy of Linguistics and Machine Learning in Natural Language Processing

One way to provide deeper insight into data and to build more powerful, robust models is bridging between linguistic knowledge and statistical learning. I’ll present model-based approaches that incorporate linguistic knowledge in novel ways.

First, I’ll show how linguistic knowledge comes to the rescue in processing languages which lack large data resources. I’ll describe a new approach to cross-lingual knowledge transfer that models the historical process of lexical borrowing between languages, and I will show how its predictions can be used to improve statistical machine translation systems.

In the second part of my talk, I’ll argue that linguistic insight helps improve learning also in resource-rich conditions. I’ll present three methods to integrate linguistic knowledge in training data, neural network architectures, and into evaluation of word representations. The first method uses features quantifying linguistic coherence, prototypicality, simplicity, and diversity to find a better curriculum for learning distributed representations of words. Distributed representations of words capture which words have similar meanings and occur in similar contexts. With improved word representations, we improve part-of-speech tagging, parsing, named entity recognition, and sentiment analysis. The second describes polyglot language models, neural network architectures trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. Finally, the third is an intrinsic evaluation measure of the quality of distributed representations of words. It is based on correlations of learned vectors with features extracted from manually crafted lexical resources. This computationally inexpensive method obtains strong correlation with performance of the vectors in a battery of downstream semantic and syntactic evaluation tasks. I’ll conclude with future research questions.

Toward Natural Language Inference Across Languages

Natural Language processing tasks as diverse as automatically extracting information from text, answering questions, translating or summarizing documents, all require the ability to compare and contrast the meaning of words and sentences. State-of-the-art techniques rely on dense vector representations which capture the distributional properties of words in large amounts of text in a single language. We seek to improve these representations to capture not only similarity in meaning between words or sentences, but also inference relations such as entailment and contradiction, and enable comparisons not only within, but also across languages.

In this talk, we will present novel approaches to inducing word representations from multilingual text corpora. First, we will show that translations in e.g. Chinese can be used as distant supervision to induce English word representations that can be composed into better representations of English sentences (Elgohary and Carpuat, ACL 2016). Then we will show how sparsity constraints can further improve word representations, and enable the detection not only semantic similarity (do "cure" and "remedy" have the same meaning?), but also entailment (does "antidote" entail "cure"?) between words in different languages (Vyas and Carpuat, NAACL 2016).

Text Analysis to Support the Privacy of Internet Users

Although research shows that Internet users care about their privacy, they do not have the time to read and understand the privacy policies of all the websites they visit or all the apps that they use. Fixing this fracture in online notice and choice is the goal of the Usable Privacy Policy Project, an NSF-funded project to extract salient details from privacy policies and present them to Internet users in ways that are responsive to their needs. I will present my ongoing work as the lead for the project's crowdsourcing and natural language processing efforts. Our results have shown that crowdworkers can answer questions about privacy policies with high accuracy, and automated methods can be used to identify legally-relevant topics and user choices in policy text. I will then present some vignettes from my research on online social network privacy and entity linking, along with a long term goal of "user-oriented natural language processing", to break down the complex texts that computer users are obligated to read and automatically find the details that affect them the most.

Topic Models for Identifying Public Health Trends

Twitter and other social media sites contain a wealth of information about populations and has been used to track sentiment towards products, measure political attitudes, and study social linguistics. In this talk, we investigate the potential for Twitter and social media to impact public health research. Broadly, we explore a range of applications for which social media may hold relevant data. To uncover these trends, we develop new topic models that can reveal trends and patterns of interest to public health from vast quantities of data.

Processing Arabic Social Media: Challenges and Opportunities

We recently witnessed an exponential growth in Arabic social media usage. Processing such media is of great utility for all kinds of applications ranging from information extraction to social media analytics for political and commercial purposes to building decision support systems. Compared to other languages, Arabic, especially the informal variety, poses a significant challenge to natural language processing algorithms since it comprises multiple dialects, linguistic code switching, and a lack of standardized orthographies, to top its relatively complex morphology. Inherently, the problem of processing Arabic in the context of social media is the problem of how to handle resource poor languages. In this talk I will go over some of our insights to some of these problems and show how there is a silver lining where we can generalize some of our solutions to other low resource language contexts.

Analyzing Formality in Online Communication

Full natural language understanding requires comprehending not only the content or meaning of a piece of text or speech, but also the stylistic way in which it is conveyed. To enable real advancements in dialog systems, information extraction, and human-computer interaction, computers need to understand the entirety of what humans say, both the literal and the non-literal. This talk presents an in-depth investigation of one particular stylistic aspect, formality. First, we provide an analysis of humans' subjective perceptions of formality in four different genres of online communication. We highlight areas of high and low agreement and extract patterns that consistently differentiate formal from informal text. Next, we develop a statistical model for predicting formality at the sentence level, using rich NLP and deep learning features, and then evaluate the model's performance against human judgments across genres. Finally, we apply our model to analyze language use in online debate forums. Our results provide new evidence in support of theories of linguistic coordination, underlining the importance of formality for language generation systems.

This work was done with Ellie Pavlick (UPenn) during her summer internship at Yahoo Labs.

Machine Translation is Too Slow

We're trying to make machine translation output less terrible, but we're impatient. A neural translation system took two weeks to train in 1996 and two weeks to train in 2016 because the field used twenty years of computing advances to build bigger and better models subject to the same patience limit. I'll talk about multiple efforts to make things faster: coarse-to-fine search algorithms and sparse gradient updates to reduce network communication.

Algorithmic Bias in Artificial Intelligence: The Seen and Unseen Factors Influencing Machine Perception of Images and Language

The success of machine learning has recently surged, with similar algorithmic approaches effectively solving a variety of human-defined tasks. Tasks testing how well machines can perceive images and communicate about them have exposed strong effects of different types of bias, such as selection bias and dataset bias. In this talk, I will unpack some of these biases, and how they affect machine perception today. I will introduce and detail the first computational model to leverage human Reporting Bias—what people mention—in order to learn ground-truth facts about the visual world.

Quantifying the White Space

Behavioral assessment and measurement today are typically invasive and human intensive (for both patient and clinician). Moreover, by their nature, they focus on retrospective analysis by the patient (or the patient’s loved ones) about emotionally charged situations—a process rife with biases, not repeatable, and expensive. We examine all the data in the “white space” between interactions with the healthcare system (social media data, wearables, activities, nutrition, mood, etc.), and have shown quantified signals relevant to mental health that can be extracted from them. These methods to gather and analyze disparate data unobtrusively and in real time enable a range of new scientific questions, diagnostic capabilities, assessment of novel treatments, and quantified key performance measures for behavioral health. These techniques hold special promise for suicide risk, given the dearth of unbiased accounts of a person’s behavior leading up to a suicide attempt. We are beginning to see the promise of using these disparate data for revolution in mental health.

Large Scale Learning for Temporal Expressions

Temporal expressions are words or phrases that refer to dates, times or durations. Social media especially contains time-sensitive information about various events and requires accurate temporal analysis. In this talk, I will present our work on TweeTIME, a minimally supervised time resolver that learns from large quantities of unlabeled data and does not require any hand-engineered rules or hand-annotated training corpora. This is the first successful application of distant supervision for end-to-end temporal recognition and normalization. Our proposed system outperforms all previous supervised and rule-based systems in the social media domain. I will also present ongoing work applying deep learning methods for resolving time expressions and discuss opportunities and challenges that a deep learning system faces when extracting time sensitive information from text.

Social Networks, Social Meaning

Language is socially situated: both what we say and what we mean depend on our identities, our interlocutors, and the communicative setting. The first generation of research in computational sociolinguistics focused on large-scale social categories, such as gender. However, many of the most socially salient distinctions are locally defined. Rather than attempt to annotate these social properties or extract them from metadata, we turn to social network analysis, which has been only lightly explored in traditional sociolinguistics. I will describe three projects at the intersection of language and social networks. First, I will show how unsupervised learning over social network labelings and text enables the induction of social meanings for address terms, such as “Ms” and “dude”. Next, I will describe recent research that uses social network embeddings to induce personalized natural language processing systems for individual authors, improving performance on sentiment analysis and entity linking even for authors for whom no labeled data is available. Finally, I will describe how the spread of linguistic innovations can serve as evidence for sociocultural influence, using a parametric Hawkes process to model the features that make dyads especially likely or unlikely to be conduits for language change.

Rich Morphological Modeling for Multi-lingual HLT Applications

In this talk, I will discuss a number of projects aimed at improving HLT applications across a broad range of typologically diverse languages by modeling morphological structure. These include the creation of a very large, normalized morphological paradigm database derived from Wiktionary, consensus-based morphology transfer via cross-lingual projection, and approaches to lemmatization and morphological analysis and generation based on recurrent neural network architectures. Much of this work falls under the umbrella of the UniMorph project at CLSP, led by David Yarowsky and supported by DARPA LORELEI, and was developed in close collaboration with John Sylak-Glassman.

Linguistic Typology Meets Universal Dependencies

Current work on universal dependency schemes in NLP does not make reference to the extensive typological research on language universals, but could benefit since many principles are shared between the two enterprises. We propose a revision of the syntactic dependencies in the Universal Dependencies scheme (Nivre et al. 2015, 2016) based on four principles derived from contemporary typological theory: dependencies should be based primarily on universal construction types over language-specific strategies; syntactic dependency labels should match lexical feature names for the same function; dependencies should be based on the information packaging function of constructions, not lexical semantic types; and dependencies should keep distinct the “ranks” of the functional dependency tree.

Multimedia Integration: Event Extraction and Beyond

Multimedia research is becoming increasingly important, as we are immersed in an ever-growing ocean of noisy, unstructured data of various modalities, such as text and images. A major thrust of multimedia research is to leverage multimodal data to better extract information, including the use of visual information to post-process or re-rank natural language processing results, or vice versa. In our work, we seek to tightly integrate multimodal information into a flexible, unified approach that jointly utilizes text and images. Here we focus on one application: improving event extraction by incorporating visual knowledge with words and phrases from text documents. Such visual knowledge provides a means to overcome the challenges that the ambiguities of language introduce. We first discover named visual patterns in a weakly-supervised manner in order to avoid the requirement of parallel/well-aligned annotations. Then, we propose a multimodal event extraction algorithm where the event extractor is jointly trained with textual features and visual patterns. We find improvements of 7.1% and 8.5% absolute F-score gain on event trigger and argument labeling, respectively. Moving forward, we intend to extend the idea of tight integration of multimodal information to other tasks, namely image and video captioning.

Conversational markers of social dynamics

Can conversational dynamics—the nature of the back and forth between people—predict the outcomes of social interactions? In this talk I will introduce a computational framework for modeling conversational dynamics and for extracting the social signals they encode, and apply it in a variety of different settings. First, I will show how these signals can be predictive of the future evolution of a dyadic relationship. In particular, I will characterize friendships that are unlikely to last and examine temporal patterns that foretell betrayal in the context of the Diplomacy strategy game. Second, I will discuss conversational patterns that emerge in problem-solving group discussions, and show how these patterns can be indicative of how (in)effective the collaboration is. I will conclude by focusing on the effects of under and over-confidence on the dynamics and outcomes of decision-making discussions.

This talk includes joint work with Jordan Boyd-Graber, Liye Fu, Dan Jurafsky, Srijan Kumar, Lillian Lee, Jure Leskovec, Vlad Niculae, Chris Potts and Justine Zhang.

Speech translation for documentation of endangered languages

Most of the world's languages do not have a writing system, so recent documentation efforts for endangered languages have switched focus to annotating corpora with translations. This talk will present work on modelling parallel speech without access to transcriptions, both using a neural attentional model (Long et al, NAACL 2016) and an unsupervised probability model (Anastasopoulos et al, EMNLP 2016), as well as some recent work on using translations for term discovery (Anastasopoulos et al, SCNLP 2017).

NLP Applications to a Corpus of Corporate Bankruptcy Documents

Data extraction from legal text presents a number of challenges that can be addressed using Natural Language Processing (NLP) methods. I discuss several applications that arise from a corpus of approximately 50 million pages of bankruptcy documents. These constitute substantially all documents from the universe of Chapter 11 cases filed between 2004 and 2014 that involved firms with over $10 million in assets. Examples of NLP applications include various classification issues (nested-phrase docket entries, financial reports, and legal writing), Part-of-Speech tagging, Optical Character Recognition, and quasi-tabular text.

From Strings to Things: Populating Knowledge Graphs from Text

The Web is the greatest source of general knowledge available today but its current form suffers from two limitations. The first is that text and multimedia objects on the Web are easy for people to understand but difficult for machines to interpret and use. The second is the Web's access paradigm, which remains dominated by information retrieval, where keyword queries produce a ranked list of documents that must be read to find the desired information. I'll discuss research in natural language understanding and semantic web technologies that addresses both problems by extracting information from text to produce and populate Web-compatible knowledge graphs. The resulting knowledge bases have multiple uses, including (1) moving the Web's access paradigm from retrieving documents to answering questions, (2) embedding semi-structured knowledge in Web pages in formats designed for computer to understand, (3) providing intelligent computer systems with information they need to perform their tasks, (4) allowing the extracted data and knowledge to be more easily integrated, enabling inference and advanced analytics and (5) serving as background knowledge to improve text and speech understanding systems. I will also cover current work on applying the techniques to extract and use cybersecurity-related information from documents, the Web and social media.

Towards Natural Dialogue with Robots

Robots can be more effective teammates with people if they can engage in natural language dialogue. In this talk, I will address one fundamental research problem to achieving this goal: understanding how people will talk to robots in collaborative tasks, and how robots could respond in natural language to maintain an effective dialogue that stays on track. The unique contribution of this research is the adoption of a multi-phased approach to building spoken dialogue systems that starts with exploratory data collection of human-robot dialogue with a human “wizard” standing in for the robot’s language processing behind the scenes, and ends with training a dialogue system that automates away the wizard.

With the ultimate goal of an autonomous conversational robot in mind, I will focus on the initial experiments that aim to collect computationally tractable human-robot dialogue without sacrificing naturalness. I will show how this approach can efficiently collect dialogue in the navigation domain, and in a form suitable for training a conversational robot. I will also present a novel annotation scheme for dialogue semantics and structure that captures the types of instructions that people gave to the robot, showing that over time these can change as people better assess the robot’s capabilities. Finally, I’ll place this research effort in the broader context of enabling better teaming between people and robots.

This is joint work with colleagues at ARL and at the USC Institute for Creative Technologies.

Offline Evaluation of Search Systems Using Online Data

Evaluation of search effectiveness is very important for being able to iteratively develop improved algorithms, but it is not always easy to do. Batch experimentation using test collections—the traditional approach dating back to the 1950s—is fast but has high start-up costs and requires strong assumptions about users and their information needs. User studies are slow and have high variance, making them difficult to generalize and certainly not possible to apply during iterative development. Online experimentation using A/B tests, pioneered and refined by companies such as Google and Microsoft, can be fast but is limited in other ways.

In this talk I present work we have done and work in progress on using logged online user data to do evaluation offline. I will discuss some of the user simulation work I have done with my students in the context of evaluating system effectiveness over user search sessions (in the context of the TREC Session track), based on training models on logged data for use offline. I will also discuss work on using historical logged data to re-weight search outputs for evaluation, focusing on how to collect that data to arrive at unbiased conclusions. The latter is work I am doing while on sabbatical at Spotify, which provides many motivating examples.

Retrieving Complex Answers through Knowledge Graph and Text

We all turn towards Wikipedia with questions we want to know more about, but eventually find ourselves on the limit of its coverage. Instead of providing "ten blue links" as common in Web search, why not answer any web query with something that looks and feels like Wikipedia? This talk is about algorithms that automatically retrieve and identify relevant entities and relevant relations and can identify text to explain this relevance to the user. The trick is to model the duality between structured knowledge and unstructured text. This leads to supervised retrieval models can jointly identify relevant Web documents, Wikipedia entities, and extract support passages to populate knowledge articles.

Universal Decompositional Semantics

The dominant strategy for capturing a symbolic representation of natural language has focussed on categorical annotations that lend themselves to structured multi-class classification. For example, predicting whether a given syntactic subject satisfies the definition of the AGENT thematic role. These annotations typically result from professionals coming to mutual agreement on semantic ontologies. The JHU Decompositional Semantics Initiative (decomp.net) is exploring a framework for semantic representation utilizing simple statements confirmed by everyday people, e.g., "The [highlighted syntactic subject] was aware of the [eventuality characterized by the salient verb]". This is conducive to a piece-wise, incremental, exploratory approach to developing a meaning representation. The resulting data relates to recent work in natural language inference, and common sense, two topics of increasingly larger interest within computational linguistics.

Cooperative and Competitive Machine Learning through Question Answering

My research goal is to create machine learning algorithms that are interpretable to humans, that can understand human strengths and weaknesses, and can help humans improve themselves. In this talk, I'll discuss how we accomplish this through a trivia game called quiz bowl. These questions are written so that they can be interrupted by someone who knows more about the answer; that is, harder clues are at the start of the question and easier clues are at the end of the question: a player must decide when it has enough information to "buzz in". Our system to answer quiz bowl questions depends on two parts: a system to identify the answer to questions and to determine when to buzz. We discuss how deep averaging networks—fast neural bag of words models—can help us answer questions quickly using diverse training data (previous questions, raw text of novels, Wikipedia pages) to determine the right answer and how deep reinforcement learning can help us determine when to buzz.

More importantly, however, this setting also helps us build systems to adapt in cooperation and competition with humans. In competition, we are also able to understand the skill sets of our competitors to adjust our strategy to optimize our performance against players using a deep mixture of experts opponent model. The game of quiz bowl also allows opportunities to better understand interpretability in deep learning models to help human players perform better with machine cooperation. This cooperation helps us with a related task, simultaneous machine translation.

Finally, I'll discuss opportunities for broader participation through open human-computer competitions: http://hcqa.boydgraber.org/

Compositional Lexical Entailment for Natural Language Inference

In this talk, I will discuss my thesis work on training computers to make inferences about what is true or false based on information expressed in natural language. My approach combines machine learning with insights from formal linguistics in order to build data-driven models of semantics which are more precise and interpretable than would be possible using linguistically naive approaches. I will begin with my work on automatically adding semantic annotations to the 100 million phrase pairs in the Paraphrase Database (PPDB). These annotations provide the type of information necessary for carrying out precise inferences in natural language, transforming the database into a largest available lexical semantics resource for natural language processing. I will then turn to the problem of compositional entailment, and present an algorithm for performing inferences about long phrases which are unlikely to have been observed in data. Finally, I will discuss my current work on pragmatic reasoning: when and how humans derive meaning from a sentence beyond what is literally contained in the words. I will describe the difficulties that such "common-sense" inference poses for automatic language understanding, and present my on-going work on models for overcoming these challenges.

Duolingo: Improving Language Learning and Assessment with Data

Student learning data can and should be analyzed to develop new instructional technologies, such as personalized practice schedules and data-driven proficiency assessments. I will describe several projects at Duolingo—the world's most popular language education platform with more than 200 million students worldwide—where we combine vast amounts of learner data with machine learning, computational linguistics, and psychometrics to improve learning, testing, and engagement.

Data Science @WaPo: How data science can help publishers succeed

The data science team at the Post has built a Big Data platform that helps to develop applications for personalization, newsroom productivity improvement and targeted advertisement. They ingest data from the digital side (washingtonpost.com and apps), paper, and external sources into the platform. They build various applications using the data stored in the platform. These applications are built to enhance user experience, perform targeted advertisement and improve newsroom work productivity. They also build tools to help newsroom adapt to the demands of digital journalism.

Sam will cover some of these applications in detail and share challenges and insights learned from the projects:

• Clavis is an audience targeting platform and is at the root of personalization efforts. Clavis analyzes everything they publish, builds user profiles, and provides personalized content and brand messaging.
• Virality is a machine learning based system that predicts the popularity of articles based on content, meta data, site traffic and social chatters.
• ModBot is an automatic comment moderation system that helps us to maintain a high quality comment section.
• Heliograf is an automated storytelling agent that automates the writing of data-driven articles and frees up reporters to focus on the high-quality stories.
• Bandito allows newsroom to test different headlines and other UX elements, determines the best performing experience and serves it to as much traffic as fast as possible.
• Headliner is an automated system that proposes several headlines for an article.

Computational Approaches to the Study of Translation (and other Crosslingual Language Varieties)

Translated texts, in any language, have unique characteristics that set them apart from texts originally written in the same language; to some extent, they form a sub-language, called "translationese". Some of the properties of translationese stem from interference from the source language (the so-called "fingerprints" of the source language on the translation product); others are source-language-independent, and are presumably universal. These include phenomena resulting from three main processes: simplification, standardization and explicitation.

I will describe research that uses standard (supervised and unsupervised) text classification techniques to distinguish between translations and originals. I will focus on the features that best separate between the two classes, and how these features corroborate some (but not all) of the hypotheses set forth by Translation Studies scholars. More generally, I will show how computational methodologies shed light on pertinent Translation Studies questions.

Translation is only one instance of language that is affected by the interaction of more than one linguistic system. Another instance is the language of advanced, highly-fluent non-native speakers. Are translations and non-native language similar? In what respects? And are such similarities the result of interference or of more "universal" properties? I will discuss recent work that uses text classification to address these questions. In particular, I will describe work that addresses the identification of the source language of translations, and relate it to the task of Native Language Identification.

Separating the Sheep from the Goats: On recognizing the Literal and Figurative Usages of Idioms

Typically, we think of idioms as colorful expressions whose literal interpretations don’t match their underlying meaning. However, many idiomatic expressions can be used either figuratively or literally, depending on their contexts. In this talk, we survey both supervised and unsupervised methods for training a classifier to automatically distinguish usages of idiomatic expressions. We will conclude with a discussion about some potential applications.

Bootstrapping Language Acquisition

Recent work with Abend, Kwiatkowski, Smith, and Goldwater (2016) has shown that a general-purpose program for inducing parsers incrementally from sequences of paired strings (in any language) and meanings (in any convenient language of logical form) can be applied to real English child-directed utterance from the CHILDES corpus to successfully learn the child's ("Eve's") grammar, combining lexical and syntactic learning in a single pass through the data.

While the earliest stages of learning necessarily proceed by pure "semantic bootstrapping", building a probabilistic model of all possible pairings of all possible words and derivations with all possible decompositions of logical form, the later stages of learning show emergent effects of "syntactic bootstrapping" (Gleitman 1990), where the program's increasing knowledge of the grammar of the language allows it to identify the syntactic type and meaning of unseen words in one trial, as has been shown to be characteristic of real children in experiments with nonce-word learning. The concluding section of the talk considers the extension of the learner to a more realistic semantics including information structure and conversational dynamics.

Event Semantics in Text Constructions, Vision, and Human-Robot Dialogue

“Ok, robot, make a right and take a picture” – a simple instruction like this exemplifies some of the obstacles in our research on human-robot dialogue: how are make and take to be interpreted? What precise actions should be executed? In this presentation, I explore three challenges: 1) interpreting the semantics of constructions in which verb meanings are extended in novel usages, 2) recognizing activities and events in images/video by employing information about the objects and participants typically involved, and 3) mapping natural language instructions to the physically situated actions executed by a robot. Throughout these distinct research areas, I leverage both Neo-Davidsonian styles of event representation and the principles of Construction Grammar in addressing these challenges for interpretation and execution.

Text summarization and categorization for scientific and health-related data

The increasing amount of unstructured health-related data has created a need for intelligent processing to extract meaningful knowledge. This knowledge can be utilized to promote healthcare and wellbeing of individuals. My research goal in this dissertation is to develop Natural Language Processing (NLP) and Information Retrieval (IR) methods for better understanding, summarizing and categorizing scientific literature and other health-related information.

First, I focus on scientific literature as the main source of knowledge distribution in scientific fields. It has become a challenge for researchers to keep up with the increasing rate at which scientific findings are published. As an attempt to address this problem, I propose summarization methods using citation texts and discourse structure of the papers to provide a concise representation of important contributions of the papers. I also investigate methods to address the problem of citation inaccuracy by linking the citations to their related parts in the target paper, capturing their relevant context. In addition, I raise the problem of the inadequacy of current summarization evaluation metrics for summarization in the scientific domain and present a method based on semantic relevance for evaluating the summaries.

In the second part, I focus on other significant sources of health-related information including clinical narratives and social media. I investigate categorization methods to address the critical problem of medical errors which is among leading causes of death worldwide. I demonstrate how we can effectively identify significant reporting errors and harmful cases through medical narratives which could help prevent similar future problems. These approaches include both the carefully designed feature-rich methods and more generalizable neural networks. Mental health is another significant dimension of health and wellbeing. Suicide, the most serious challenge in mental health, accounts for approximately 1.4% of all deaths worldwide and approximately one person dies by suicide every 40 seconds. I present both feature-rich and data-driven methods, to capture mental health conditions, such as depression, self-harm, and suicide, based on the general language expressed on social media. These methods have clear clinical and scientific applications and can help individuals with mental health conditions.

The Knowledge and Language Gap in Medical Information Seeking

Interest in medical information retrieval has raised significantly in the last few years. The Internet has become a primary source for consumers looking for health information and advice; however, their lack of expertise causes a language and knowledge gap that affects their ability to properly formulate their information needs. Health experts also struggle to efficiently search the large amount of medical literature available to them, which impacts their ability of integrating the latest research findings in clinical practice. In this dissertation, I propose several methods to overcame these challenges, thus improving search outcomes. For queries issued by lay users, I introduce query clarification, a technique to identify the most appropriate expert expression that describes their information need; such expression is then used to expand the query. I experiment with three existing synonym mappings, and show that the best one leads to a 7.3% improvement over non-clarified queries. When a classifier that predicts the most appropriate mapping for each query is used, an additional 5.2% improvement over non-clarified queries is achieved. Furthermore, I introduce a set of features to capture semantic similarity between consumer queries and retrieved documents, which are then exploited by a learning to rank framework. This approach yields a 26.6% improvement over the best known results on a dataset designed to evaluate medical information retrieval for lay users. To improve literature search for medical professionals, I propose and evaluate two query reformulation techniques that expand complex medical queries with relevant latent and explicit medical concepts. The first is an unsupervised system that combines a statistical query expansion with a medical terms filter, while the second is a supervised neural convolutional model that predicts which terms to add to medical queries. Both approaches are competitive with the state of the art, achieving up to 8% improvement in inferred nDCG. Finally, I conclude my dissertation by showing how the convolutional model can be adapted to reduce clinical notes that contain significant noise, such as medical abbreviations, incomplete sentences, and redundant information. This approach outperforms the best query reformulation system for this task by 27% in inferred nDCG.

Gradient Symbolic Representations in Grammar: The case of French Liaison (a co-authored study by Paul Smolensky, Matthew Goldrick & Eric Rosen)

In Gradient Symbolic Computation, representations are structures in which a given position hosts a blend of symbols each with its own numerical activity level. A goal of Gradient Symbolic Computation for linguistic theory is to enable resolution of a ubiquitous type of impasse: two conflicting structures for analyzing the same linguistic phenomenon persist indefinitely because each can handle data that the other cannot. The hypothesis is that a single theory deploying a numerically weighted blend of the two proposed structures can simultaneously account for both bodies of data. I will describe an attempt at testing this hypothesis: blending two theories of French liaison, a phenomenon that has been argued to be regulated by prosodic boundaries, and one in which child data play a provocative role.

Multilingual Summarization and Evaluation Using Wikipedia Featured Articles

Multilingual text summarization is a challenging task and an active area of research within the natural language processing community. In this talk I will give an overview of the use of Wikipedia featured articles are used to create datasets comprising about 40 languages for the training and testing of automatic single document summarization methods. These datasets were used in 2015 and 2017 MultiLing Workshop's single document summarization task. I will give an overview of the methods used to both generate and to evaluate the summaries submitted for the tasks. Systems overall performance are measured using automatic and human evaluations and these data are analyzed to evaluate the effectiveness of the automatic methods for multi-lingual summarization evaluation. Thus, the results not only suggest which approaches to automatic text summarization generalize across a wide range of languages but also which evaluation metrics are best at predicting human judgments in the multilingual summarization task. This talk is based on a soon to appear book chapter to be published by World Scientific Press. The chapter is jointly written with Jeff Kubina, Peter Rankel, and Julia Yang.

Dynamic Search Models and Applications

Dynamic search is an information retrieval task that involves a sequence of queries for a complex information need (e.g. searching for one-week tour plans in Italy). It is characterized by rich user-system interactions and temporal dependency between queries and between consecutive user behaviors. Dynamic search is a trial-and-error process, which matches well with the family of Reinforcement Learning (RL) algorithms: the RL algorithms learn from repeated, varied attempts which are continued until success. The learner/agent learns from its dynamic interactions with the world. These similarities inspire me to model dynamic search using RL frameworks.

In particular, I model dynamic search as a dual-agent stochastic game, one of the standard variants of Partially Observable Markov Decision Process (POMDP), where the user agent and the search engine agent work together to jointly maximize their long-term rewards. In the framework, users’ search statuses, such as exploitation, exploration, struggle etc. are modeled as hidden states, which can only be estimated through user interaction data. In each search iteration, one search algorithm is picked from a set of candidates to maximize a reward function. My work provides a general framework to model dynamic search. It enables the use of Reinforcement Learning algorithms for optimizing retrieval results. The experiments on the Text REtrieval Conference (TREC) 2011–2014 Session Track datasets show a statistically significant improvement over the state-of-the-art dynamic search algorithms.

The design of states, actions, and rewards is quite flexible when using POMDPs in dynamic search. In the thesis, I also examine all available design choices from related work and compare their retrieval accuracy and efficiency. My analysis reveals the effects of these choices, which enables me to recommend practical design choices. The finding again proves that modeling dynamic search using POMDPs is promising, however, it also shows that this approach is computational demanding.

To improve the efficiency of above dynamic search models, I propose another RL framework, direct policy learning, which finds optimal policies for the best search engine actions directly from what is observed in the user and search engine interactions via gradient descent. The proposed framework greatly reduces the model complexity than the POMDP framework. It is also a flexible design, which includes a wide range of features describing the rich interactions in dynamic search. The framework is shown to be highly effective and efficient on the TREC Session Tracks.

In addition to the dynamic search frameworks, I propose predictive models to detect user struggling states in search. Most prior work uses effort-based features to detect user struggle. However, recent studies suggest that there might be a gap between user effort and user struggle. In this work, I take a psychological perspective to bridge this gap. I demonstrate that after removing the biases introduced by different searcher motivations, user struggle can be tightly connected to user effort.

At last, I implement a dynamic search tool to support the annotation task for the TREC Dynamic Domain Tracks. This serves as my first trial of implementing dynamic search algorithm and struggling detection in practice. The results show that my algorithm is also effective in real life settings.

The research in this thesis is among the first in this field and serves as one step towards solving dynamic search and developing real-world applications.

Fantastic online comments and how to find them

I provide an overview of my current research on discourse and computational methods to analyze social media language. The first part of the talk will be devoted to outlining the two frameworks for this research: rhetorical relations and sentiment analysis. Rhetorical relations are the fundamental building blocks of discourse, connecting propositions to make coherent text. I will describe existing research on rhetorical relations, and present a study on how relations are signalled by discourse markers and other linguistic devices. Then I introduce my work on sentiment analysis, and on the role that rhetorical relations and other contextual factors play in the interpretation of sentiment and opinion.

The second part of the talk is devoted to describing a current project, analyzing online news comments in terms of constructiveness and toxicity. Using a large corpus of comments, I describe how we have modelled constructiveness in terms of rhetorical and argumentation relations, and toxicity as a type of extreme negative sentiment.

Challenges in End-to-End Generation

Progress in NMT has led to optimism for text generation tasks such as summarization and dialogue, but it has been more difficult to quantify the successes and challenges in this space. In this talk, I will survey some of the recent advances in neural NLG, and present a successful implementation of these techniques for the 2017 E2E NLG challenge (Gehrmann et al, 2018). Despite success on these small scale examples, though, we see that similar models fail to scale to a more realistic data-to-document corpus. Analysis shows systems will need further improvements in discourse modeling, reference, and referring expression generation (Wiseman et al, 2017). Finally, I will end by presenting recent work in unsupervised NLG that shows promising results in neural style transfer using a continuous GAN-based text autoencoder (Zhao et al 2017).

Towards Easier Machine Translation

Much has been made about the accuracy improvements from neural systems, but the transition to NMT will potentially make the technology of translation more accessible in transformative ways. I will present three projects from Harvard with this goal, including: 1) OpenNMT; a collaborative open-source project to provide benchmark NMT components, 2) Sequence distillation; a research project to build smaller, faster, on-device NMT systems, and 3) Sequence Vis; a visualization framework designed to support inspection and debugging of translation output. These projects aim to make translation systems easier-to-extend, easier-to-run, and easier-to-understand.

A Mental Space Analysis of Tense and Modality

This talk presents a progress report on developing a mental space analysis of tense and modality, based on prior cognitive semantic and typological research in those domains. Mental spaces are a model to represent "alternative realities", including past and future times and unrealized events, the focus of our interest. Mental spaces are evoked by specific grammatical constructions including tense, mood and modality, complement-taking predicates and conditional constructions. Following Cutrer (1994), we analyze these constructions as providing access paths for the hearer from one space (viewpoint) to another space (focus), which may be a new space or one already established in the discourse. We propose a revised (and simplified) version of Cutrer's analysis of tense, based on Comrie's (1981) analyses. We also adopt Fillmore's (1990) analysis of conditionals, treating his concept of epistemic stance as a relation between the hypothetical space and the "reality" (speaker's belief) space. We extend epistemic stance to model epistemic modality, following Boye (2012), and use a mental space analysis of Clark's (1996) theory of common ground (shared knowledge) extended to individual knowledge to model Boye's theory of the relationship between epistemic modality and evidentiality. Finally, we argue that hypothetical and "reality" (speaker belief) spaces are of the same kind, differing only in epistemic stance, and that they should ultimately be embedded in an interactional model of the negotiation of shared knowledge, as indicated by the grouping of epistemic, evidential and (knowledge) interactional categories in a single grammatical category (see Palmer 2001).

What do neural networks learn about language?

Neural network models have redefined the state-of-the-art in many areas of natural language processing. Much of this success is attributed to their ability to learn representations of their input, and this has invited bold claims that these representations encode important semantic, syntactic, and morphological properties of language. For example, when one research group recently suggested that "prior information regarding morphology ... among others, should be incorporated" into neural models, a prominent deep learning group retorted that it is "unnecessary to consider these prior information" when using neural networks. In this talk I’ll try to tease apart the hype from the reality, focusing on two questions: what do character-level neural models really learn about morphology? And what do LSTMs learn about negation?

This is work with Clara Vania, Federico Fancellu, Yova Kementchedjhieva, Andreas Grivas, and Bonnie Webber.

Demographic bias in social media language analysis: a case study of African-American English

We conduct a case study of dialectal language in online conversational text by investigating African-American English (AAE) on Twitter, through a demographically supervised model to identify AAE-like language associated with geo-located messages. We verify that this language follows well-known AAE linguistic phenomena -- and furthermore, existing tools like language identification, part-of-speech tagging, and dependency parsing fail on this AAE-like language more often than text associated with white speakers. We leverage our model to fix racial bias in some of these tools, and discuss future implications for fairness and artificial intelligence.

Coptic Scriptorium: A Linked Digital Environment for Coptic Studies

The Coptic language represents the last phase of the Ancient Egyptian phylum of the Afro-Asiatic language family, forming part of the longest continuously documented human language on Earth. Despite its high value for historical, comparative and typological linguistics, as well as its cultural importance as the heritage language of Copts in Egypt and in the diaspora, digital resources for the study of Coptic have only recently become available.

Coptic Scriptorium is an interdisciplinary Digital Humanities project dedicated to providing language documentation resources for Coptic, including linguistically and philologically annotated corpora, online lexical resources and automatic tools for processing the language. In this talk, we will present some of our work in building and using Coptic corpora to explore the world of first millennium Egypt and its language. We will feature the technologies and infrastructures that can enhance research in under-resourced languages as well as the benefits of collaboration across institutions and humanities disciplines, including some examples of our research using the digital research environment. The talk will discuss some of the challenges and solutions we have found and are working on for creating tools such as an online dictionary for Coptic, formulating annotation guidelines, building the first syntactically annotated datasets for the language and linking information about entities such as people and places to work in other projects using Linked Open Data standards.

UCCA: A computational approach to cross-linguistic semantic representation

Analyzing large volumes of text in various languages and mapping them into a representation that reflects content, rather than form, is one of the great challenges NLP must face. The talk will present UCCA (Universal Conceptual Cognitive Annotation), an approach to structural semantic representation that emphasizes cross-linguistic applicability and accessibility to non-expert annotators, which represents a step towards meeting this challenge. The first part of the talk will introduce the UCCA scheme and show how it provides a typologically motivated characterization of abstract semantic structure. The second part will discuss a transition-based approach to UCCA parsing, and experiments on leveraging annotated data in other formalisms (such as AMR and SDP) to improve UCCA parsing through multi-task learning. The talk will conclude with an overview of how UCCA is being applied to text-to-text generation tasks, such as machine translation and text simplification, and their evaluation.

All of UCCA's resources are freely available at http://www.cs.huji.ac.il/~oabend/ucca.html.

Joint work with Leshem Choshen, Daniel Hershcovich, Ari Rappoport and Elior Sulem.

Reasoning-Driven Question Answering

Most current methods for Question Answering (QA) treat it as yet another machine learning problem. I argue that this leads to only shallow understanding of text, which is reflected in poor generalization. In this talk, I will show how techniques involving Knowledge Representation, Reasoning, and Logic can be combined with machine learning to improve QA systems. First, I will introduce a formulation for abductive reasoning in natural language and show its effectiveness in two different domains. Next, I will show a few recent theoretical results on limitations of multi-step reasoning. Finally, I will introduce MultiRC, a reading comprehension playground where questions can only be answered based on information from multiple sentences.

De-identification of Clinical Narratives

De-identification is the task of finding and removing patient identifying private health information from electronic health records. This is a sensitive and complicated task: failure to remove all private health information from records results in violations of privacy, whereas overzealous de-identification can remove medically-salient text from the records and render the data unusable for future research. Ambiguities between private health information and medical concepts can exacerbate this problem. Misspellings and foreign names render dictionaries inadequate. Short, telegraphic style of clinical narratives and the density of jargon therein limits transfer of methods from open domain.

In this talk, we will present two natural language processing approaches to de-identification. First, we will define and use "local context". We will show that local context goes a long way towards effective de-identification, even when the concepts are presented in ungrammatical and fragmented narrative text. Next, we will take a deep learning approach to learning long-distance contextual features for de-identification. We will show that featureless recurrent neural networks (RNNs) can adequately capture long-distance contextual information, giving comparable de-identification performance to methods with manually designed features. Addition of manually-designed features to featureless RNNs gives them the boost they need in order to outperform the state of the art.

On the Syntactic Abilities of Recurrent Neural Networks

Recent technological advances have made it possible to train recurrent neural networks (RNNs) on a much larger scale than before. These networks have proved effective in applications such as machine translation and speech recognition. These engineering advances are surprising from a cognitive point of view: RNNs do not have the kind of explicit structural representations that are typically thought to be necessarily for syntactic processing. In this talk, I will discuss studies that go beyond standard engineering benchmarks and examine the syntactic capabilities of contemporary RNNs using established cognitive and linguistic diagnostics. These studies show that RNNs are able to compute agreement relations with considerable success across languages, although their error rate increases in complex sentences. A comparison of the detailed pattern of agreement errors made by RNNs to those made by humans in a behavioral experiment reveals some similarities (attraction errors, number asymmetry) but also some differences (relative clause modifiers increase the probability of attraction errors in RNNs but decrease it in humans). Overall, RNNs can learn to exhibit sophisticated syntactic behavior despite the lack of an explicit hierarchical bias, but their behavior differs from humans in important ways.

Learning to Navigate in Real Urban Environments Using Natural Language Directions

According to the Universal Postal Union (UPU), the majority of people in many developing countries do not have a set address. With very few alternatives, they often rely on natural language (NL) description of the path to their house. E.g., "Turn right after bar and it will be the first house after the school". What if we could automate the interpretation of such directions, allowing robots or autonomous vehicles to automatically navigate based on free NL descriptions of such routes?

The task of following NL navigation instructions requires composition of language and domain knowledge, and it raises several challenges including: grounding language to physical objects; resolving ambiguity; avoiding cascading errors; and many more. The main datasets collected for the NL navigation task so far (SAIL and HCRC) present a very simplistic, unrealistic depiction of the world, with a small fixed set of entities that are known to the navigator in advance. Such representations bypass the great complexity of navigation based on real urban maps, where an abundance of previously ungrounded and unseen entities are observed at test time.

In this work, we redefine the task of NL navigation by endorsing the complexity and challenge presented by real urban environments and we present OpenStreetSet - a novel data-set with large real urban maps and richer information layers than ever before. We present an effective baseline architecture for the NL navigation task, which augments a standard encoder-decoder model with an entity abstraction layer, attention over words and worlds, and a constantly updating world-state. Our experiments show that this architecture is indeed better-equipped to treat the grounding challenge in realistic urban settings than standard sequence-to-sequence architectures.

Resource Building for Parsing Amharic Text: Morphological-rich language with less resource

In recent years, different language processing applications are demanding the state-of-the-art parsers. Specifically, high-quality parsers are required for applications like question answering, machine translation and information summarization. In order to train or develop efficient parser then it has become a trained to create high-quality treebank, a linguistically annotated corpus which includes morphological annotations and syntactic annotations. Treebanks play a role in promoting research in parsing natural languages but also contributes for linguistics theory and corpus-based language analysis. Furthermore, treebanks are important resources for the respective language to building and testing data-driven tools serving as gold standard. Such a resource has been developed for highly resourced language. However, building a resource for a morphologically-rich and less resourced language is difficult. In this presentation I will talk about the general tasks involved in the development of a treebank for Amharic parser. I will explain the problems at each stages and the solution we proposed.

Effective deep near-take detection

Deep learning models produce image representations that excel both at semantic and perceptual similarity. Networks trained on the classic ImageNet benchmark are commonly fine-tuned over task-specific data and tend to produce highly accurate solutions for a variety of problems, ranging from image classification to content based search. It has been observed, however, that the deepest architectures such as ResNet, while superior at semantic classification are not as as effective at providing perceptually sensitive representations when compared to older architectures, such as AlexNet or VGG. This has implications for complex solutions for which both of these modalities are important. In this work we investigate the problem of improving the perceptual sensitivity of deeper architectures with application to near-take detection, where the objective is to identify clusters of images with strong visual similarity. We show that with an appropriate modification of the learning objective, deeper architectures such as ResNet substantially improve in their perceptual representations, which maintaining good semantic representations.

Commonsense Reasoning without Commonsense Knowledge

Reading comprehension tasks measure the ability to answer questions that require inference from a free text story. This talk explores the two machine learning approaches included in MITRE’s submission to a shared task, SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge. The first system is a logistic regression over simple features like question length and lexicalized patterns. The second is a recurrent neural network with attention, which uses a pretrained semantic space and learns to align words of the story, passage, and answers. The resulting ensemble system answers reading comprehension questions with 82.27% accuracy and ranked second in the task. This strong performance, despite limited use of external knowledge sources or explicit reasoning, raises questions about “commonsense knowledge” in this task.

Decompositional Event Semantics

In natural language understanding, traditional approaches to mapping observed linguistic forms (e.g., a sentence) to symbolic representations of meaning typically require the construction of an underlying ontology of abstract semantic categories. Such ontologies can be challenging to establish, expensive to annotate training data for, and may still suffer from issues of label ambiguity, incomplete coverage, or sparsity. The Decompositional Semantics Initiative is a collection of efforts to provide light-weight alternatives to ontology-backed semantic representations of text; these decompositional representations instead consist of layers of independent, rapidly-annotated properties (entailments) based on common-sense questions that can be posed to crowdsource workers. In this talk, I will present my work on two particular efforts under this initiative: Semantic Proto-Role Labeling and Event Factuality Prediction. This work involves each stage of the decompositional semantics life-cycle: from the formulation of semantic targets, to large-scale data collection, and finally to the development of state-of-the-art, linguistically-informed predictive models. I will also discuss recent follow-up work on analytically probing the capabilities of these models, as well as a newly-developed cross-lingual application.

Screening for cancer using a learning internet advertising system

Studies have shown that the traces people leave when browsing the internet may indicate the onset of diseases such as cancer. Specifically, queries to search engines were found to be indicative of future cancer diagnosis for several types of cancer.

In my talk I will discuss two studies showing that the adaptive engines of advertising systems working in conjunction with clinically verified questionnaires can identify people who are suspected of having one of three types of solid tumor cancers. First, a classifier trained to predict suspected cancer inferred from questionnaire response using past queries reached an Area Under the Curve of 0.64. Second, using a conversion optimization mechanism, both Bing and Google advertisement systems learned to identify people who were likely to have symptoms consistent with suspected cancer, such that after a training period of approximately 10 days, 11% of people it selected for showing of targeted campaign ads were found to have suspected cancer. People who received information that their symptoms were consistent with suspected cancer increased their searches for healthcare utilization and maintained it for longer than people whose symptoms were not associated with suspected cancer, indicating that the questionnaires provided useful information to people who completed them.

These results demonstrate the utility of using search engine queries to screen for possible cancer and the application of modern advertising systems to identify people who are likely suffering from serious medical conditions.

Modeling early phonetic learning from spontaneous speech

Most models of language acquisition have used idealized data. Phonetic category learning models, in particular, have been trained on inputs whose "acoustics" are artificially constructed to follow Gaussian distributions and abstract away from the time-varying nature of the speech signal. Furthermore, the acoustic dimensions used in phonetic learning models are typically hand-selected to be linguistically relevant.

This talk describes research that builds toward a theory of phonetic learning from naturalistic speech. The first half of the talk considers ways in which context might constrain phonetic category learning from speech that is highly variable, and finds that context can still be very helpful for learning from spontaneous speech -- but only when it is used in very specific ways. The second half of the talk raises the question of whether we should be thinking in terms of early phonetic category learning at all.

This is joint work with Kasia Hitczenko, Thomas Schatz, Stephanie Antetomaso, Emmanuel Dupoux, Micha Elsner, Sharon Goldwater, Reiko Mazuka, and Kouki Miyazawa.

Beyond the average case: machine learning for atypical examples

Although machine learning systems have improved dramatically over the last decade, it has been widely observed that even the best systems fail on atypical examples. For example, prediction models such as image classifiers have low accuracy on images from minority cultures, and generative models such as dialogue systems are often incapable of generating diverse, atypical responses. In this talk, I will discuss two domains where high performance on typical examples is insufficient.

The first is learning prediction models that perform well on minority groups, such as non-native English speakers using a speech recognition system. We demonstrate that models with low average loss can still assign high losses to minority groups, and this gap can amplify over time as minority users that suffer high losses stop using the model. We develop an approach using distributionally robust optimization that learns models that perform well over all groups and mitigate the feedback loop.

The second domain is learning natural language generation (NLG) systems, such as a dialogue system. It has been frequently observed that existing NLG systems which produce high-quality samples rely heavily on typical responses such as "I don't know" and fail to generate the full diversity of atypical but valid human responses. We carefully quantify this problem through a new evaluation metric based on the optimal classification error between human- and model-generated text and propose a new, edit-based generative model of text whose outputs are both diverse and high-quality.

Machine Learning in Medicine: Disease Prediction and Biomarker Discovery

Machine learning has seen great progress in the past decade. In parallel, Electronic Health Records (EHR) systems are accumulating clinical and medical data at unprecedented scales. The intersection of the two phenomena has enabled multitude of machine-learning-assisted medical care, with potential to impact and improve healthcare research, and delivery for millions of individuals. In this talk, we first briefly review the data landscape of healthcare, including modalities and quantities of data available to various machine learning tasks, and discuss the implications of this data on different research areas. We will then focus on a number of recent work at my research lab at NYU Langone Medical Center on the topics of biomarker discovery and disease classification. Our discussion includes classification of lung cancer genomic mutation and subtype using histopathology images; deep learning on clinical notes for disease prediction; and biomarker discovery using EHR time series.

Recovering Syntactic Structure from Surface Features

We show how to predict the basic word-order facts of a novel language, and even obtain approximate syntactic parses, given only a corpus of part-of-speech (POS) sequences. We are motivated by the longstanding challenge of determining the structure of a language from its superficial features. While this is usually regarded as an unsupervised learning problem, there are good reasons that generic unsupervised learners are not up to the challenge. We do much better with a supervised approach where we train a system – a kind of language acquisition device – to predict how linguists will annotate a language. Our system uses a neural network to extract predictions from a large collection of numerical measurements. We train it on a mixture of real treebanks and synthetic treebanks obtained by systematically permuting the real trees, which we can motivate as sampling from an approximate prior over possible human languages.

Universal Semantic Parsing with Neural Networks

Natural language understanding requires the ability to comprehend text, reason about it, and act upon it intelligently. While simplistic frameworks such as end-to-end sequence-to-sequence architectures or even bag-of-words models can go a long way, symbolic meaning representation is inevitably needed for some applications, and may provide an invaluable inductive bias for others. We construct such graphical meaning representations from text, with a focus on a particular semantic representation scheme called Universal Conceptual Cognitive Annotation (UCCA), whose main design principles are support for all linguistic semantic phenomena, ease of annotation, cross-linguistic applicability and stability, and a modular architecture of different layers of meaning. We develop a general directed acyclic parser supporting the graph structures UCCA exhibits. Subsequently, we apply the parser to three other representation schemes in three languages, demonstrating its flexibility and universality. We show that training the parser in a multitask setting on all of these schemes improves its UCCA parsing accuracy, by effectively learning generalizations across the different representations. inally, in an empirical comparison of the content of semantic and syntactic representations, we discover several aspects of divergence. These have profound impact on the potential contribution of syntax to semantic parsing, and on the usefulness of each of the approaches for semantic tasks in natural language processing.

A Multifactorial, Multitask Approach to Automated Speaker Profiling

Automated Speaker Profiling (ASP) refers broadly to the computational prediction of speaker traits based on cues mined from the speech signal. Accurate prediction of such traits can have a wide variety of applications such as automating the collection of customer metadata, improving smart-speaker/voice-assistant interactions, narrowing down suspect pools in forensic situations, etc.

Approaches to ASP to date have primarily focused on single-task computational models—i.e. models which each predict one speaker trait in isolation. Recent work however has suggested that using a multi-task learning framework, in which a system learns to predict multiple related traits simultaneously, each trait-prediction task having access to the training signals of all other trait-prediction tasks, can increase classification accuracy along all trait axes considered.

Likewise, most work on ASP to date has focused primarily on acoustic cues as predictive features for speaker profiling. However, there is a wide range of evidence from the sociolinguistic literature that lexical and phonological cues may also be of use in predicting social characteristics of a given speaker. Recent work in the field of author profiling has also demonstrated the utility of lexical features in predicting social information about authors of textual data, though few studies have investigated whether this carries over to spoken data.

In this dissertation I focus on prediction of five different social traits: sex, ethnicity, age, region, and education. Linguistic features from the acoustic, phonetic, and lexical realms are extracted from 60 second chunks of speech taken from the 2008 NIST SRE corpus and used to train several types of predictive models. Naive (majority class prediction) and informed (single-task neural network) models are trained to provide baseline predictions against which multi-task neural network models are evaluated. Feature importance experiments are performed in order to investigate which features and feature types are most useful for predicting which social traits.

Results presented in chapters 5-7 of this dissertation demonstrate that multi-task models consistently outperform single-task models, that models are most accurate when provided information from all three linguistic levels considered, and that lexical features as a group contribute substantially more predictive power than either phonetic or acoustic features.

Natural Language Processing in Therapy: What Is and What Can Be

Natural language processing (NLP) can potentially transform a broad array of applications where human interactions are largely mediated by conversation. For example, automatically analyzing dialogue in counseling sessions can not only help understand and guide behavior, but also improve patient outcomes.

In this talk, I will first present how language technology can guide the course of therapy sessions. I will describe recent work on automated helpers that can provide realtime feedback and guidance to therapists, and also alert them to potentially important forthcoming cues from a patient.

Following this introduction, I will focus on the some of the technical challenges that need to be addressed before the potential of NLP can be realized. Specifically, I will focus on the question of how we can mitigate the need for massive annotated datasets using knowledge stated in the form of logic. To this end, I will present a new learning framework that can exploit domain knowledge in logic to improve the accuracy and self-consistency of neural models and demonstrate promising results on several tasks that call for reasoning about language.

The Humble Preposition and the Sins of Traditional Grammar

The general public has an insatiable appetite for books about English grammar and usage. But few of the people who buy such books appreciate how much they are being cheated. The available books, plagiarizing each other shamelessly, repeat inadequate analyses dating from the 18th century. Few topics illustrate this better than the treatment of prepositions. Traditional grammars undercount prepositions massively, distinguish prepositions from other categories irrationally, ignore cogent refutations spread across two centuries, and overlook most of the really interesting aspects of prepositions and the phrases they head. This talk presents an uncompromising critique of the state of the art, and an informal reanalysis distinct from both traditional and generative grammar.

Multi-objective Hyperparameter Optimization of Deep Neural Networks

Deep neural network models are full of hyperparameters. To obtain a good model, one must carefully experiment with hyperparameters such as the number of layers, the number of hidden nodes, the type of non-linearity, the learning rate, and the drop-out parameter, just to name a few. I will discuss general hyperparameter optimization algorithms, based on evolutionary strategies or Bayesian techniques, that can automate this laborious process. Further, I will argue for the necessity of a multi-objective approach: we desire models that are not only optimized for accuracy, but also respect practical computational constraints such as model size and run-time. We will present results for machine translation and language modeling.

This is joint work with Takahiro Shinozaki, Tomohiro Tanaka, and Takafumi Moriya at the Tokyo Institute of Technology, Shinji Watanabe and Xuan Zhang at JHU, and Michael Denkowski at Amazon.

Building Compositional, Interpretable and Robust Visual Recognition through Language

The principle of compositionality of language states that complex meaning can be derived from simpler constituents, from a simple vocabulary complex ideas can emerge. Models for visual recognition have seen remarkable success at finding isolated concepts but still struggle to deal with evolving vocabularies and compositionally derived concepts. In this talk I will argue that imbuing visual recognition models with compositionality borrowed from the language domain has the potential to address several shortcomings in current visual recognition models, including issues with fairness, robustness, interpretability, and generalization. I will present several recent works from the vision and language research group at the University of Virginia addressing some of these challenges.

Enhancing Human-AI Collaboration in Language Technology

The goal of the human-AI collaboration is to take advantage of the complementary nature of humans and AI systems, such that they can work together to achieve better application goals. In language technology, the questions of enhancing human-machine collaboration include (1) how can we build AI systems that will help humans acquire language skills effectively? Moreover, (2) how can we build explainable AI systems that can be trusted by human users in real-world NLP applications? In this talk, I present some recent work as examples to answer these two questions. The first part of this talk will demonstrate that recent progress on text generation can help enhance human writing skills via a collaborative writing system. Then, in the second part, I will show our recent work on interpreting predictions from neural text classifiers with human-understandable explanations.

The Data-Driven Analysis of Literature

Literary novels push the limits of natural language processing. While much work in NLP has been heavily optimized toward the narrow domains of news and Wikipedia, literary novels are an entirely different animal—the long, complex sentences in novels strain the limits of syntactic parsers with super-linear computational complexity, their use of figurative language challenges representations of meaning based on neo-Davidsonian semantics, and their long length (ca. 100,000 words on average) rules out existing solutions for problems like coreference resolution that expect a small set of candidate antecedents.

At the same time, fiction drives computational research questions that are uniquely interesting to that domain. In this talk, I'll outline some of the opportunities that NLP presents for research in the quantitative analysis of culture—including measuring the disparity in attention given to characters as a function of their gender over two hundred years of literary history (Underwood et al. 2018)—and describe our progress to date on two problems essential to a more complex representation of plot: recognizing the entities in literary texts, such as the characters, locations, and spaces of interest (Bamman et al. 2019) and identifying the events that are depicted as having transpired (Sims et al. 2019). Both efforts involve the creation of a new dataset of 200,000 words evenly drawn from 100 different English-language literary texts and building computational models to automatically identify each phenomenon.

This is joint work with Matt Sims, Ted Underwood, Sabrina Lee, Jerry Park, Sejal Popat and Sheng Shen

Sentence-level Semantic Inference: From Diverse Phenomena to Applications

Many NLP tasks involve understanding meaning at the sentence level. In order to analyze models developed for these tasks, we should decompose sentence-level semantic understanding into a diverse array of smaller, more-focused, fine-grained types of reasoning. This will help improve our understanding of the sentence-level reasoning capabilities of our NLP systems. In this talk, we will focus on Natural Language Inference (NLI), the task of determining if one sentence (hypothesis) can likely be inferred from another (context/premise). NLI has traditionally be used to evaluate how well different models understand language and the relationship between texts. We investigate whether 10 recent NLI datasets require models to reason about both texts, or if the datasets contain biases or statistical irregularities that allow a model to correctly label a context-hypothesis pair by only looking at a hypothesis. In the most popular dataset that we consider, a hypothesis-only model outperforms the majority baseline by over 2x. We will also discuss our recently released dataset, the Diverse NLI Collection (DNC), that can be used to shed light on a model’s ability to capture or understand a diverse array of semantic phenomena that are important to Natural Language Understanding. We will demonstrate how a variant of the DNC has been used to evaluate whether a Neural Machine Translation encoder captures semantic phenomena related to translation. With the remaining time, we will discuss how lessons from these studies can be applied to real-world uses cases of sentence-level semantic inference. This talk is based on work that has appeared at NAACL, ACL, StarSem, and EMNLP.

The Promise of Big Data: Blending Noisy Organic Signals with Traditional Movement Variables to Predict Forced Migration in Iraq

Worldwide displacement due to war and conflict is at all-time high. Unfortunately, determining if, when, and where people will move is a complex problem. This talk will describe a multi-university project that develops methods for blending variables constructed from publicly available organic data (social media and newspapers) with more traditional indicators of forced migration to better understand when and where people will move. I will demonstrate our approach using a case study involving displacement in Iraq, and shows that incorporating open-source generated conversation and event variables maintains or improves predictive accuracy over traditional variables alone. I will conclude with a discussion on strengths and limitations of leveraging organic big data for societal—scale problems.

Algorithmic measures of language mirror human biases

Following progress in computing and machine learning and the emergence of big data, artificial intelligence has become a reality. Reliance on machine learning can be seen in diverse areas ranging from cancer detection and job candidate selection to recidivism prediction. Machine learning can detect patterns in large swaths of data and suggest solutions to hard problems in ways that would elude the best human experts. However, machine learning is not the panacea it was expected to be. For example, Google Translate converts the gender-neutral Turkish sentences “O bir doktor. O bir hemşire” to the English sentences “He is a doctor. She is a nurse.” What is the reasoning behind making these biased decisions? What types of biased outcomes should we expect to observe in language-based AI technologies? This talk introduces a method, the Word Embedding Association Test (WEAT), an adaptation of the Implicit Association Test (IAT) to machine learning. WEAT operates by analyzing language models that are trained on billions of sentences collected from the internet to probe whether measures of individuals’ implicit attitudes and beliefs can be corroborated by measures of association in language as it resides in large datasets.

Language and Context

I'll start this talk by summarizing the latest breakthrough in natural language processing: contextual word representations, giving a historical perspective and some intuitive explanation for their impact on NLP. I'll then turn to a discussion of some key challenges that remain—despite great excitement and the emergence of a new paradigm for tackling NLP problems, our systems are still missing some fundamental aspects of linguistic intelligence, resulting in inconsistent performance. I'll argue that the notion of "context" must be radically expanded if NLP is to become broadly useful and reliable. I'll present two recent projects: a method that adapts NLP models to the specific communicative context of their inputs (often called "domain"; examples include biomedical articles and web reviews),* and a new conceptual formalism and dataset to make NLP systems more "aware" of social context.**

* Collaborators: Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, and Doug Downey (AI2)

** Collaborators: Maarten Sap, Saadia Gabriel, Lianhui Qin (UW); Dan Jurafsky (Stanford); and Yejin Choi (UW/AI2)

NLP Evaluations That We Believe In

With all of the modeling advancements in recent years, NLP benchmarks have been falling over left and right: "human performance" has been reached on SQuAD 1 and 2, GLUE and SuperGLUE, and many commonsense datasets. Yet no serious researcher actually believes that these systems understand language, or even really solve the underlying tasks behind these datasets. To get benchmarks that we actually believe in, we need to both think more deeply about the language phenomena that our benchmarks are targeting, and make our evaluation sets more rigorous. I will first present ORB, an Open Reading Benchmark that collects many reading comprehension datasets that we (and others) have recently built, targeting various aspects of what it means to read. I will then present contrast sets, a way of creating non-iid test sets that more thoroughly evaluate a model's abilities on some task, decoupling training data artifacts from test labels.

Using an Inverted Index Synopsis for Query Latency and Performance Prediction

Predicting the query latency by a search engine has important benefits, for instance, by allowing the search engine to adjust its configuration to address long-running queries without unnecessarily sacrificing its effectiveness. However, for the dynamic pruning techniques that underlie many search engines, achieving accurate predictions of query latencies is difficult. In this talk I will discuss how index synopses – which are stochastic samples of the full index – can be used for attaining accurate timings. Experiments using the TREC ClueWeb09 collection, and a large set of user queries, show that using a small random sample it is possible to very accurately estimate properties of the larger index, including sizes of posting list unions and intersections. I will also show that index synopses facilitate two use cases: (i) predicting the query latencies on the full index and classifying long-running queries can be accurately achieved using index synopses; (ii) the effectiveness of queries can be estimated more accurately using a synopsis index post-retrieval predictor than a pre-retrieval predictor. This work is partially supported by the Italian Ministry of Education and Research (MIUR) in the framework of the CrossLab project (Departments of Excellence).

Neural Mechanisms of Multiword Expressions

Recently there has been an increase in psycholinguistics and neurolinguistics research using naturalistic stimulus following Willems’ (2015) encouragement to probe the neural bases of language comprehension with greater ecological validity. Along with naturalistic stimulus, applying tools from computational linguistics to neuroimaging data can help us gain further insight into real time language processing. In this talk, I will discuss Multiword Expressions (MWEs). From a processing perspective, human sentence comprehension must involve both memory retrieval and structural composition and MWE processing can be an instance of memory retrieval. MWEs have also posed a challenge in the past for computational linguistics since they comprise a heterogeneous family of word clusters and are difficult to detect in natural language data. Using Association Measures like Pointwise Mutual Information and Dice’s Coefficient, we can capture these finer-grained distinctions within MWEs. I present a neuroimaging study where fMRI recordings of naturalistic narrative comprehension is used to investigate to what extent these computational measures and their underlying cognitive processes are observable during sentence processing.

Aspects of Advanced Proficiency in Second Language Acquisition

In this talk I will present a study on two topics related to acquisition of English as a second language in advanced non-native speakers: semantic infelicities in second language stemming from the atypical nature of English indefinite pronouns, and native language cognate effects on second language lexical choice. First, drawing on studies in semantic typology, I will lay out empirical evidence for theoretical hypotheses on the nature of challenges posed by indefinite pronouns to English learners with varying degree of proficiency. I will then suggest and evaluate an automatic approach for detection of infelicitous usage patterns, demonstrating encouraging initial results obtained with deep learning architectures on this task involving nuanced semantic anomalies. Second, I will show that utilizing a large corpus of highly competent non-native speakers and a set of carefully selected lexical items, it is possible to reveal that the lexical choices of non-natives are affected by cognates in their native language. This effect is sufficiently powerful to reconstruct the phylogenetic language tree of the Indo-European language family solely from the frequencies of specific lexical items in the English of authors with various native languages.

Toward Robust Abstractive Multi-Document Summarization and Information Consolidation

Humans can consolidate textual information from multiple sources and organize the content into a coherent summary. Can machines be taught to do the same? The most important obstacles facing multi-document summarization include excessive redundancy in source content, less understood sentence fusion, and the looming shortage of training data. In this talk, I will present our recent work tackling these issues through decoupling of content selection and surface realization. I will describe a lightly-supervised optimization framework using determinantal point processes (DPP) for content selection. I will further present a new method leveraging DPP to select self-contained, summary-worthy text segments to be highlighted on the source documents to make it easier for users to browse through a large amount of text. Finally, I will discuss challenges and opportunities for pushing forward research on abstractive multi-document summarization.

Adapting Transformer models for Document-level Natural Language Tasks

Representation learning forms the foundation of today’s natural language processing systems; Transformer models have been extremely effective at producing word- and sentence-level contextualized representations, achieving state-of-the-art results in many NLP tasks. However, applying these models to produce contextualized representations of the entire documents faces challenges. These challenges include lack of inter-document relatedness information, decreased performance in low-resource settings, and computational inefficiency when scaling to long documents.

In this talk, I will describe 3 recent works on developing Transformer-based models that target document-level natural language tasks. I will first introduce SPECTER, a method for producing scientific document representations using a Transformer model that incorporates document-level relatedness signals and achieves state-of-the-art results in multiple document-level tasks. Second, I will describe TLDR, the task of extreme summarization for scientific papers. I will introduce SciTLDR, our new multi-target summarization dataset of scientific papers as well as a simple yet effective training strategy for generating summaries in low-resource settings. Finally, I will discuss the practical challenges of scaling existing Transformer models to long documents, and our proposed solution, Longformer. Longformer introduces a new sparse self-attention pattern that scales linearly with the input length while capturing both local and global context in the document, and achieves state-of-the-art results in both character-level language modeling and document NLP tasks.

Four awesome GUCL alumni will be joining us to tell us about their journey from grad school to where they are today, and to answer all of your questions about either side of a computer science or linguistics degree.

Knowledge-Rich Neural Text Comprehension and Reasoning

Enormous amounts of ever-changing knowledge are available online in diverse textual styles (e.g., news vs. science text) and diverse formats (knowledge bases vs. web pages vs. textual documents). This talk presents the question of textual comprehension and reasoning given this diversity: how can AI help applications comprehend and combine evidence from variable, evolving sources of textual knowledge to make complex inferences and draw logical conclusions? I present question answering and fact checking algorithms that offer rich natural language comprehension using multi-hop and interpretable reasoning. Recent advances in deep learning algorithms, large-scale datasets, and industry-scale computational resources are spurring progress in many Natural Language Processing (NLP) tasks, including question answering. Nevertheless, current models lack the ability to answer complex questions that require them to reason intelligently across diverse sources and explain their decisions. Further, these models cannot scale up when task-annotated training data are scarce and computational resources are limited. With a focus on textual comprehension and reasoning, this talk will present some of the most recent efforts in my lab to integrate capabilities of symbolic AI approaches into current deep learning algorithms. I will present interpretable algorithms that understand and reason about textual knowledge across varied formats and styles, generalize to emerging domains with scarce training data (are robust), and operate efficiently under resource limitations (are scalable).

Operationalizing legal and societal constraints in information systems

Data protection laws define requirements for respectful processing of user data. However, the lack of technical interpretations of many of those requirements has inhibited their adoption. In this talk, I will discuss our recent efforts to operationalize several data protection principles from GDPR—including the data minimization principle and the right to be forgotten—in the context of automated profiling and decision-making systems. We propose new models that enable the adoption of those principles and evaluate the outcomes in the domain of recommendation and search systems. Our results demonstrate remaining practical challenges and some of the impacts the implementation of the legal principles might have on the digital ecosystem.

Syntactic Nuclei in Dependency Parsing – A Multilingual Exploration

Standard models for syntactic dependency parsing take words to be the elementary units that enter into dependency relations. In this talk, I will discuss whether there are any benefits from enriching these models with the more abstract notion of nucleus proposed by Tesnière. More precisely, I will show how the concept of nucleus can be defined in the framework of Universal Dependencies and how we can use composition functions to make a transition-based dependency parser aware of this concept. Experiments on 12 languages show that nucleus composition gives small but significant improvements in parsing accuracy. Further analysis reveals that the improvement mainly concerns a small number of dependency relations, including nominal modifiers, relations of coordination, main predicates, and direct objects.

This is joint work with Ali Basirat to be presented at EACL 2021.

Operationalizing legal and societal constraints in information systems

In this presentation I will address the explainability of web search engines. I will present recent work in which we added explainable elements to the deep relevance matching model (DRMM). In a user study we showed that a visualization of these explainable elements on the search engine result page improved the assessability of the search results: users judge our proposed interface significantly more explainable and easier to assess than a regular search engine result page. This indicates that the explainability of the search engine result page leads to a better user experience. Our current research follows up on these results in the direction of professional search contexts. Explainable search is particularly relevant for professional search contexts, where users are critical towards search results and have the need to be in control. In these contexts, trust is even more important than in generic web search.

On the semantics and pragmatics of diachronic pathways in modality

One of the most intensively studied domains of diachronic (historical) change is the domain of modality. To a large extent, changes in the form and meaning of modal expressions (words like 'can', 'ought', and 'going to') have shaped our conception of how languages more generally change over time. Change is thought to create meanings that are abstract rather than concrete, and to be propelled by enrichment of meanings with originally defeasible pragmatic inferences. The centrality of pragmatic inferencing, in particular, has been challenged in recent years by works demonstrating that it is neither sufficient nor necessary as an explanation of language change. My goal in this talk is to provide additional support for this position by examining well-known as well as novel cases of change in the domain of modality.

I will focus on two pathways in particular: the pathway from motion to desire (where 'come' and 'go' become bouletic) and the pathway from temporal to modal meaning (joint work with Elena Herburger). The empirical focus will be on the evolution of Modern Hebrew over the past 130 years, from the time it was revived as a language with native speakers until the present day.

A "comparative concepts" approach to American Sign Language verbs

The cognitive linguistics literature on American Sign Language divides verbs into categories such as "indicating verbs" and "depicting constructions". One downside of this terminology is that a term like "indicating verb" is a "typologically idiosyncratic" notion (Franz Dotter, p.c.) which is a barrier to cross-linguistic comparison. This is unfortunate, as sign languages are infrequently included in cross-linguistic typological surveys. In this talk, I assess ASL verb categories from a functional-typological perspective informed by "comparative concepts", which are based in semantics and designed for cross-linguistic research.

Socio-pragmatic analyses of functional morphemes

All phenomena in linguistics pragmatics have important (but often neglected) sociolinguistic dimensions relating to style, agency, and variation. This creates opportunities to achieve deeper results by bringing pragmatic and sociolinguistic theories together. In this talk, I'll highlight a range of recent socio-pragmatic analyses of functional elements of language, from Eric Acton, Lelia Glass, Sunwoo Jeong, and others, and I will offer an extended case study involving English singular 'some', drawing on ongoing work with Lewis Esposito. I'll use these examples as a chance to conjecture about why functional elements are rich in social meaning. In addition, all my examples follow a general pattern of using semantic meaning and broadly Gricean ideas as foundational ingredients, and so I am hoping we can reflect on alternative perspectives and modes of analysis in this area as well.

When do languages use the same word for different meanings? The Goldilocks Principle in colexification

It is common for languages to express multiple meanings with the same word, a phenomenon known as colexification. For instance, the meanings FINGER and TOE colexify in the word 'dedo' in Spanish, while they do not colexify in English. Colexification has been suggested to follow universal constraints. In particular, previous work has shown that related meanings are more prone to colexify. This tendency has been explained in terms of the cognitive pressure for simplicity, since expressing related meanings with the same word makes lexicons easier to learn and use. The present study examines the interplay between this pressure and a competing universal constraint, the functional pressure for languages to maximize informativeness. We hypothesize that colexification follows a Goldilocks principle: meanings are more likely to colexify if they are related (fostering simplicity), but not so related as to become confusable and cause misunderstandings (fostering informativeness). We find support for this principle in data from over 1200 languages and 1400 meanings. Our results thus suggest that universal principles shape the lexicons of natural languages. More broadly, they contribute to the growing body of evidence suggesting that languages evolve to strike a balance between competing functional and cognitive pressures.

Computational frameworks for garden-variety polysemy: Word vectors are word story vectors, and that is okay.

Many words are polysemous, and most of them in a manner that is entirely idiosyncratic, not regular. That makes them no less interesting: We can ask, for example, what influences the meaning of a word in a particular context, or what makes a listener perceive a group of usages as similar, or what information should be considered part of a stored word meaning in the mind. Word embeddings give us a window into garden-variety polysemy: They are computed as an aggregate of many observed word uses, from different speakers, and they have been shown, time and again, to be able to mimic human behavior on word meaning-related tasks. Interestingly, they also pick up what could be called traces of stories: text topics, judgments and sentiment, and cultural trends. This holds for all types of word embeddings, from count-based vectors to word type embeddings and word token embeddings. In this talk, I trace the many ways in which stories show up in word vectors, and I argue that this is actually an interesting signal and not a bug. Contextualized word embeddings are particularly interesting for lexical semantics because they give us embeddings of individual word uses, and hence possibly a much more detailed window into polysemy. Because they are relatively recent, we don't have a standard way yet to use them for lexical semantics. I report on a recent study that we did on how to make contextualized embeddings usable for lexical semantics, and show examples of contextualized embeddings also exhibiting "story traces".

Verbing, Linguistic Innovation and Constructions

I was in law school when they Clarence Thomas-ed Thurgood Marshall’s seat. (@mayaharris 10/12/20)

In this talk, I present findings of a project on English denominal verb (DV) meaning conducted at CU Boulder with PhD student Allen Hsiao (Michaelis & Hsiao, 2021). DVs are produced by an implicit syntactic category shift, conversion, in which a noun acquires the inflectional and combinatory properties of a verb (Valera, 2015: 322). How? DVs present a continuum of conventionality ranging from conventional verb-phrase replacements like paint, trash, pocket, mother to evanescent innovations like adulting, criming and Clarence-Thomas-ing. For Clark and Clark (1979), interpretations of innovative DVs either (a) directly reflect criterial features of their source nouns or (b) are ad hoc, derived from “moment-to-moment cooperation,” including gestures, allusions, and “other momentarily relevant facts about the conversation” (1979: 783). We note that certain DVs, which I will call hybrid DVs, have properties of both conventional and novel DVs, and so challenge a dual-route model. Hybrid DVs include: those with idiosyncratic meanings and correspondingly weird direct objects (like key and door), those that are ambiguous or uninterpretable outside of a particular argument-structure configuration (like clock, railroad, and geek), and those whose source nouns are proper names (like Houdini and Bork). To cover the DV spectrum, including the hybrid cases, we propose a unified approach that leaves fewer interpretive factors to vagaries of context. In our framework, denominal meanings, both conventional and novel, come from reconciling the results of four interpretive strategies: nominal frame computation (Fillmore, 2006; Andor, 2010), verb-construction integration (Michaelis, 2004; Goldberg, 2006; Michaelis, 2011), co-composition (Pustejovsky, 1998; Pustejovsky, 2012) and, finally, conceptual blending (Fauconnier and Turner, 2004). This framework allows for considerable interpretive flexibility, while placing greater reliance on structured aspects of interpretation, like metonymy, metaphor and constructional meaning.

Detecting the “Fake News” Before It Was Even Written, Media Literacy, and Flattening the Curve of the COVID-19 Infodemic

Given the recent proliferation of disinformation online, there has been growing research interest in automatically debunking rumors, false claims, and "fake news". A number of fact-checking initiatives have been launched so far, both manual and automatic, but the whole enterprise remains in a state of crisis: by the time a claim is finally fact-checked, it could have reached millions of users, and the harm caused could hardly be undone.

An arguably more promising direction is to focus on analyzing entire news outlets, which can be done in advance; then, we could fact-check the news before it was even written: by checking how trustworthy the outlet that has published it is (which is what journalists actually do). We will show how we do this in the Tanbih news aggregator (http://www.tanbih.org/), which aims to limit the impact of "fake news", propaganda and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which are arguably the best way to address disinformation in the long run. In particular, we develop media profiles that show the general factuality of reporting, the degree of propagandistic content, hyper-partisanship, leading political ideology, general frame of reporting, stance with respect to various claims and topics, as well as audience reach and audience bias in social media.

Another important observation is that the term "fake news" misleads people to focus exclusively on factuality, and to ignore the other half of the problem: the potential malicious intent. Thus, we detect the use of specific propaganda techniques in text, e.g., appeal to emotions, fear, prejudices, logical fallacies, etc. We will show how we do this in the Prta system (https://www.tanbih.org/prta), another media literacy tool, which got the Best Demo Award (Honorable Mention) at ACL-2020; an associated shared task got the Best task award (Honorable Mention) at SemEval-2020.

Finally, at the time of COVID-19, the problem of disinformation online got elevated to a whole new level as the first global infodemic. While fighting this infodemic is typically thought of in terms of factuality, the problem is much broader as malicious content includes not only "fake news", rumors, and conspiracy theories, but also promotion of fake cures, panic, racism, xenophobia, and mistrust in the authorities, among others. Thus, we argue for the need of a holistic approach combining the perspectives of journalists, fact-checkers, policymakers, social media platforms, and society as a whole, and we present our recent research in that direction (https://covid19.tanbih.org/).

Combating Disinformation on Social Media and Its Challenges

In recent years, disinformation and fake news have become a global phenomenon, particularly so during the pandemic, COVID-19. The wide dissemination of fake news can have detrimental societal effects on individuals and the society. Despite the recent progress in detecting fake news, it remains a challenging task due to its complexity, diversity, multimodality, speed, and costs of fact-checking or annotation. In this talk, we look at some challenges and lessons learned when exploring effective and explainable detecting strategies of fake news, and discuss challenges in fake news research and the need for interdisciplinary research.

Effectiveness and Efficiency Advancements in Conversational Search

In a conversational context, a user converses with a system through a sequence of natural-language questions, i.e., utterances. Starting from a given subject, the conversation evolves through sequences of user utterances and system replies. We aim at improving both the quality (effectiveness) of the replies and the processing time (efficiency) required to search those replies. We address the quality aspect by proposing an adaptive utterance rewriting strategy based on the current utterance and the dialogue evolution of the user with the system. Retrieving relevant documents for a question is difficult due to the informal use of natural language in speech and the complexity of understanding the semantic context coming from previous questions/utterances. In our system, a classifier identifies utterances lacking context information as well as the dependencies on the previous utterances. Our modular architecture performs: (i) automatic utterance understanding and rewriting, (ii) first-stage retrieval of candidate passages for the rewritten utterances, and (iii) neural re-ranking of candidate passages to retrieve the most relevant documents as replies. Rapid response is fundamental in search applications; it is particularly so in interactive search sessions, such as those encountered in conversational settings. To address the efficiency aspect, we exploit the temporal locality of conversational queries and propose and evaluate a client-side document embedding cache to improve responsiveness. By leveraging state-of-the-art dense retrieval models to abstract document and query semantics, we cache the embeddings of documents retrieved for a topic introduced in the conversation, as they are likely relevant to successive queries. Our document embedding cache implements an efficient metric index answering nearest-neighbor similarity queries by estimating the approximate result sets returned. It significantly improves the responsiveness of conversational systems while likewise reducing the number of queries managed on the search back-end.

AI meets human rights meets diplomacy: Lessons from 9 months at the State Department

A common refrain in the technology world is that policymakers need more technological expertise. Those of us from technical backgrounds may want to help, but what “policy” work consists of and how people come to contribute to it can seem opaque. This non-technical talk will attempt to demystify tech policy work and shed light on where technical expertise does and does not matter. I will discuss my experiences as a AAAS Science and Technology Policy Fellow in the Department of State working at the intersection of computer science and human rights. Anecdotes about working as a computer scientist in the federal government will abound, but I will also draw out more general lessons from those experiences.

Neural Information Retrieval: the Good, the Bad, and the Ugly

Over the past few years, neural ranking methods powered by transformer models have substantially improved search across a variety of settings. These neural IR methods can be divided into three families: reranking approaches powered by cross-encoders, dense retrieval approaches powered by bi-encoders that leverage an approximate nearest neighbor index, and learned sparse retrieval approaches that combine a traditional inverted index with learned term weights. While all of these approaches work well in different settings, none are both fast and robust. In this talk, I will draw on my past work and experience writing “Pretrained Transformers for Text Ranking” to characterize these approaches and their trade-offs. In the first part of the talk, I will describe some of my work on neural ranking methods and views on why they work well. The remaining parts of the talk will investigate where these methods fail and why: are we a few tweaks away from solving their shortcomings or are there more fundamental issues that should be addressed?

Social Data Analytics with Citizen-AI Collaboration Networks to Rapidly Augment Public Services

Digital technologies such as social media and the web have revolutionized information generation and consumption by the public, generating massive unstructured big data. This unstructured data presents an opportunity to understand the risks and gather real-time public behavior to help improve the quality of service provided by public services in our communities. This talk will present some illustrative use cases of emergency management and methods that demonstrate both the challenges and solutions to processing big social data for understanding relevant human behaviors by intent mining and machine learning techniques. In addition, this talk will argue for the need to tap into Citizen-AI collaboration networks to effectively and reliably scale the social data analytics capabilities for public service organizations.

Towards Inclusive and Equitable Human Language Technologies

The societal needs of language technology users are rapidly changing alongside the technological landscape, calling for a deeper understanding of the impact of natural language processing models on user behaviors. As these technologies are increasingly deployed to users, it is ever more important to design culturally responsible and inclusive language technologies that can benefit a diverse population. Towards this end, I present three of our recent works: 1) Discourse-aware generation models for automatic social media moderation and mediation, 2) Equitable and human-like dialogue generation models based on learning theory, and 3) Sign language processing. Finally, I describe my research vision: Building inclusive and collaborative communicative systems and grounded artificial intelligence models by leveraging the cognitive science of language use alongside formal methods of machine learning.

Building a Broad Infrastructure for Uniform Meaning Representations

Impressive progress has been made in many aspects of natural language processing (NLP) in recent years because of a powerful NLP research paradigm in which machine learning algorithms are trained to “replicate” what humans do by analyzing large amounts of human generated language data. Many of the recent breakthroughs have been achieved through training language models on massive amounts of raw text that can be “fine-tuned” or “prompted” to perform a wide range of NLP tasks. For many other NLP applications, however, a more expressive semantic representation is needed to support NLP systems that require sophisticated reasoning and higher levels of interpretability. In this talk I will discuss efforts by an international team of researchers in building a broad infrastructure for Uniform Meaning Representations (UMRs), a cross-lingual document-level representation that we believe can be used to power the development of next-generation NLP systems. I will present aspects of the UMR representation, report current progress, and end with an open invitation to join this collaborative effort.

Bringing Document Collections to Life With the Help of Large Language Models

When discussing Large Language Models (LLMs), corpora are often only mentioned for their potential to serve as training data. However, LLMs can play a crucial role in enriching collections to make them more suitable to their users. In this talk, I will examine two scenarios in which LLMs have been employed to augmenting corpora. First, I will discuss how generative models can be used to improve answers to general domain user questions. Then, I will give an overview of recent initiatives at Semantic Scholar that leverage LLMs to enrich scholarly documents. Overall, these two research directions show that LLMs can significantly help exploring, consuming, or augmenting existing corpora.

Exploring the Interplay between Explicit and Implicit Knowledge in Hybrid NLP

With increasingly vast storehouses of textual data readily available, the field of Natural Language Processing offers the potential to extract, organize, and repackage knowledge revealed either directly or indirectly. Though for decades one of the holy grails of the field has been the vision of accomplishing these tasks with minimal human knowledge engineering through machine learning, with each new wave of machine learning research, the same tensions are experienced between investment in knowledge engineering and integration know-how on the one hand and production of knowledge/insight on the other hand. Recent work in neural-symbolic approaches to NLP have again explored this tension, in some cases reporting advances from incorporation of formal representations of language and knowledge and in other cases revealing challenges in identifying high utility abstractions and strategic exceptions that frequently require exogenous data sources and the interplay between these formal representations and bottom-up generalities that are apparent from endogenous sources. Couched within these tensions, this talk reports on recent work towards increased availability of formal representations of language and knowledge as well as explorations within the space of tensions to use this knowledge in effective ways.

Take a Fresh Look at Recommender Systems from an Evaluation Standpoint

Recommendation has become a prominent area of research in the field of Information Retrieval (IR). Motivated by a few counter-intuitive observations reported in recent studies, in this talk, we take a fresh look at recommender systems from an evaluation standpoint. Rather than examining metrics like recall, hit rate, or NDCG, or perspectives like novelty and diversity, the key focus here is on how these metrics are calculated when evaluating a recommender algorithm. Specifically, the commonly used train/test data splits and their consequences are re-examined. We begin by examining common data splitting methods, such as random split or leave-one-out, and discuss why the popularity baseline is poorly defined under such splits. We then move on to explore the two implications of neglecting a global timeline during evaluation: data leakage and oversimplification of user preference modeling. Afterward, we present a new perspective on recommender systems, including techniques for evaluating algorithm performance that more accurately reflect real-world scenarios; this new approach to evaluation could also affect the way in which user preferences are learned.

Democratizing Language: Towards Language Models for Everyone

Language models have largely been a "success story" in AI. They have enabled progress in numerous language processing applications, and opened the door to the integration of AI in other domains such as social sciences and humanities. Yet, these models mainly follow the strategy of one-size-fits-all, which does not account for individual or group differences. In this talk, I will explore the interplay between language and people, highlighting the need for cross-cultural language models that can capture the diverse ways in which different groups use language. I will show how we can develop models to identify and represent words that are used in significantly different ways by speakers from different groups, and also talk about downstream applications that can effectively leverage these people-centric language representations.

Codex to corpus: Processing and annotation for an open and extensible edition of the Florentine Codex

The Florentine Codex is a bilingual text produced in Mexico in the 16th century. It describes the lives and beliefs of the Indigenous people who were living in the Valley of Mexico before the arrival of the Spanish. The text is in Nahuatl, the lingua franca of the area, and is accompanied by a translation or summary in Spanish. In this talk we describe the processing of the Nahuatl text and some linguistic issues that will need to be addressed in the annotation, including relational nouns, functional incorporation and subordination and clause structure. We also describe efforts to translate from the Nahuatl of the era to modern varieties of Nahuatl spoken in the Sierra of Puebla.

Offensive Language Identification in Social Media

Offensive language is pervasive in social media. Individuals frequently take advantage of the perceived anonymity of computer-mediated communication, using this to engage in behavior that many of them would not consider in real life. Online communities, social media platforms, and technology companies are well aware of the problem and they have been investing heavily in ways to cope with abusive behavior in social media. One of the most widely-used strategies for tackling this problem is to use computational methods to identify such content in user-generated content (e.g. posts, comments, microblogs, etc.). In this talk, I discuss some of the challenges and opportunities of using NLP to recognize offensive content online. I present the OLID taxonomy created to annotate offensive language datasets. I discuss some of the lessons learned from the two editions of OffensEval organized at SemEval. Finally, I also present TBO, a novel target-based taxonomy annotation taxonomy presented at ACL 2023 and new experiments on vicarious offense to appear at EMNLP 2023.

Re-Thinking Re-Ranking

Re-ranking systems take a "cascading" approach, wherein an initial candidate pool of documents are ranked and filtered to produce a final result list. This approach exhibits a fundamental relevance misalignment problem: the most relevant documents may be filtered out by a prior stage as insufficiently relevant, ultimately reducing recall and limiting the potential effectiveness. In this talk, I challenge the cascading paradigm by proposing methods that efficiently pull in additional potentially-relevant documents during the re-ranking process, using the long-standing Cluster Hypothesis. I demonstrate that these methods can improve the efficiency and effectiveness of both bi-encoder and cross-encoder retrieval models at various operational points. Cascading is dead, long live re-ranking!

Collective Emotions and Polarization on Social Media

Social media has linked people on a global scale, transforming how we communicate and interact by sharing not only ideas, but also emotions and feelings. The massive interconnectedness created new vulnerabilities in the form of societal conflict, mistrust and deteriorating mental health. I describe the tools my group developed to recognize emotions and moral language in online discussions at scale and show how they help elucidate phenomena like affective polarization. Affective polarization is a growing problem in a democracy, and it means that liberals and conservatives not only disagree on policy issues but also dislike and distrust each other. I show that affective polarization exists in online interactions, with same-ideology users, e.g., liberals or conservatives, expressing warmer feelings toward each other than to opposite-ideology users. I also show that emotions structure social networks: interactions between users who are close to each other within the network elicit positive emotions, while more distant interactions have more anger and disgust. These findings are consistent across diverse datasets and languages, spanning discussions on topics such as the Covid-19 pandemic, abortion, and the 2017 French Election. Our research provides new insights into the complex social dynamics of collective emotions with implications for political discourse.

An encoder-centric view of information retrieval

Modern large language models like GPT-4 and Bard are currently in the spotlight due to their strong performance on many information seeking tasks. These impressive models rely on retrieval to generate up-to-date responses across topics, which is often powered by neural methods.

In this talk, I will describe my encoder-centric view of neural methods for information retrieval with a focus on how such approaches relate to large language models (LLMs). LLMs are effective reranking methods when provided with candidate documents, but when and how should they serve as encoders to power efficient first-stage retrieval? I will address this question while describing my recent work on encoders for learned sparse retrieval and dense retrieval, which provide paths to leveraging a LLM's strong language processing for search.

Making LLMs Right

Large language models (LLMs) like ChatGPT have been criticized for their propensity to "hallucinate" and state incorrect information. However, the errors these systems make are not random: there are certain capabilities, like summarization, they can do quite reliably, and others, like arithmetic, that are fundamentally unreliable. In this talk, I argue that paying attention to this divide in capabilities can allow us to make LLMs right. First, I will discuss how we use LLMs as building blocks in systems that can do sound reasoning over natural language. For instance, we can use them to translate a natural language problem definition into a formal specification, or break a reasoning problem down into steps that are easily checkable. LLMs can do these steps reliably even if they struggle with the full end-to-end task. I will present a new dataset we have constructed to benchmark LLMs' capabilities on challenging reasoning tasks presented in narratives, like solving murder mysteries. Second, I will discuss how we can figure out post-hoc whether LLMs' generations are right. Our approach is inspired by human fact-checking: first, dissect an LLM's "claim" into pieces, then explain whether those pieces are right or wrong. Finally, I will discuss ongoing work on how to integrate this error detection capability into LLMs to improve the state of the art.

Grammar as information

From a general cognitive perspective, which has gained progressively wider acceptance over the past 20 years, language has been approached as a coded communication system that must be learned from often sparse and biased input and processed efficiently by speakers under ‘noisy’ conditions that may be non-optimal in other respects. This talk focuses on some intricate organizational properties of these coding systems that contribute to the remarkable uniformity in learning outcomes and success in achieving communicative goals. It outlines novel reanalyses of a range of longstanding issues in the language sciences. These include interactions between frequency and function in gender and classifier systems, the role played by probabilistic factors in governing the occurrence of elements traditionally described as ‘optional’, and the distribution and composition of inflection classes. Each of these studies exhibit a common methodology, which performs computational analyses of large corpora to produce human-interpretable distributional measures. A salient feature of this methodology is the way that it highlights the role that grammatical structure plays as a conduit of information in a host of ‘gradient’ phenomena, most prominently in modeling the network of interrelationships between word-form distribution, word ‘meaning’, and form frequency.

Bigger is not always better: The importance of human-scale language modeling for psycholinguistics

Scaling is the trend toward producing ever-larger models, both in terms of parameter counts and in terms of the amount of data they are trained on. While largely beneficial to performance on downstream benchmarking tasks, scaling has several downsides for both computational psycholinguistics and natural language processing research. In the first half of this talk, I will discuss the scientific and practical challenges presented by scaling for connectionist modeling, as well as the benefits that would result from human-scale language modeling research. In the second half, I will report on takeaways from one effort to bring about human-scale language modeling: The BabyLM Challenge, a shared task that challenged members of the community to train a language model on 100 million words or less.

A Sanity Check on Emergent Properties

One of the frequent points in the mainstream narrative about large language models is that they have "emergent properties", but there is a lot of disagreement about what that even means. If they are understood as a kind of generalization beyond training data - as something that a model does without being explicitly trained for it - I argue that we have not in fact established the existence of any such properties, and at the moment we do not even have the methodology for doing so.