Validity Checking and Repairing of Machine Generated Transfer-Type Word Problems

Abstract

Machine generation of arithmetic word problems (AWPs) is challenging as these problems require the correct use of quantities and mathematical relationships among them. While state-of-the-art deep-learning (DL) models excel at generating text with language variations, the mathematical validity of generated problems often remains unchecked. Metrics such as BLEU-4, METEOR, and ROUGE-L exist to assess the language quality of generated problems, but checking the end-to-end mathematical validity of AWPs is less explored. This work focuses on transfer-case (TC)-AWPs (problems involving object transfer among agents). Though we train them with a dataset of valid problems, DL systems generate valid, near-valid, and invalid problems. Near-valid cases are invalid problems that are grammatically correct but mathematically incorrect. The proposed work focuses on validity-checking of TC-AWPs and repairing the near-valid cases. Detecting valid/near-valid problems requires manual effort and is error-prone. Encoding the relevant domain knowledge as an ontology is very helpful in these tasks. We propose leveraging an extended TC-ontology, previously developed to solve TC-AWPs, for automated validity-checking and repairing near-valid problems. We construct a problem-specific representation (ontology Assertional-Box) of an auto-generated problem by leveraging a sentence-classifier and BERT language models (LMs). The training set for these LMs is problem-texts where sentence-parts are annotated with ontology class-names. The proposed approach ensures that TC-AWPs produced in the output are always valid. We also briefly discuss how our ontology-based approach can be adapted to generate TC-AWPs that contain multiple object-transfers and are guaranteed to be valid. Adopting this approach to generate other types of AWPs is interesting future work.