US Patent 11989525 Techniques for generating multi-modal discourse trees

Methods for generating and utilizing a multi-modal discourse tree (MMDT) are provided herein. An extended discourse tree (EDT) may be generated (e.g., from a discourse tree (DT) or a communicative DT (CDT)) from a corpus of text. Data records (e.g., records contained numerical data) may be linked to the extended discourse tree to generate a multi-modal discourse tree. The multi-modal discourse tree may link any suitable text/records from disparate sources. For example, entities identified from elementary discourse units of the EDT may be matched to an entity of a data record. Causal links may be identified between EDTs and/or data records. Rhetorical relationships can be identified for each entity/causal link match to incorporate the data records with the EDT to generate a MMDT. The MMDT may be used to classify subsequent input, to generate answers to subsequent questions, to navigate the corpus of text and/or data records, or the like.