Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality

In the NLP literature, the thematic fit estimation task is defined as the task in which a system has to predict how likely a candidate argument (e.g. cop) is to fit a given a verb-specific role (e.g. the agent of to arrest) (Santus et al., 2017). Because of the scarcity of benchmark datasets, thematic fit models are currently evaluated by measuring the correlation between their output and human ratings for isolated verb-filler pairs (Sayeed et al., 2016). However, such evaluation does not account for the dynamic nature of argument expectations: there is robust psycholinguistic evidence that human update their predictions on upcoming arguments during sentence processing, depending on the way other verb arguments are filled (Bicknell et al., 2010; Matsuki et al., 2011). Consider, for example, how the expectation for the patient of to check would change if we use journalist or mechanic as agents. In this paper we introduce DTFit (Dynamic Thematic Fit), a dataset of human ratings for verb-role fillers in a given event context, with the aim of providing a rigorous benchmark for context-sensitive argument typicality modeling. The dataset accounts for the plausibility of patient, instrument and location roles, given the agent and the predicate.