Nishant Balepur

2024

1. EMNLP 2024

   A SMART Mnemonic Sounds like “Glue Tonic”: Mixing LLMs with Student Feedback to Make Mnemonic Learning Stick

   Nishant Balepur, Matthew Shu, Alexander Hoyle, and 4 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

   Abs TL;DR Bib PDF

   Keyword mnemonics are memorable explanations that link new terms to simpler keywords.Prior work generates mnemonics for students, but they do not train models using mnemonics students prefer and aid learning.We build SMART, a mnemonic generator trained on feedback from real students learning new terms.To train SMART, we first fine-tune LLaMA-2 on a curated set of user-written mnemonics.We then use LLM alignment to enhance SMART: we deploy mnemonics generated by SMART in a flashcard app to find preferences on mnemonics students favor.We gather 2684 preferences from 45 students across two types: **expressed** (inferred from ratings) and **observed** (inferred from student learning), yielding three key findings.First, expressed and observed preferences disagree; what students *think* is helpful does not always capture what is *truly* helpful.Second, Bayesian models can synthesize complementary data from multiple preference types into a single effectiveness signal.SMART is tuned via Direct Preference Optimization on this signal, which resolves ties and missing labels in the typical method of pairwise comparisons, augmenting data for LLM output quality gains. Third, mnemonic experts assess SMART as matching GPT-4 at much lower deployment costs, showing the utility of capturing diverse student feedback to align LLMs in education.

   We generate mnemonics by aligning an LLM using preferences from 47 GRE test-takers on the mnemonics they like and which mnemonics aid learning

   @inproceedings{balepur-etal-2024-smart,
     title = {A {SMART} Mnemonic Sounds like {``}Glue Tonic{''}: Mixing {LLM}s with Student Feedback to Make Mnemonic Learning Stick},
     author = {Balepur, Nishant and Shu, Matthew and Hoyle, Alexander and Robey, Alison and Feng, Shi and Goldfarb-Tarrant, Seraphina and Boyd-Graber, Jordan Lee},
     editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung},
     booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
     month = nov,
     year = {2024},
     address = {Miami, Florida, USA},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.emnlp-main.786},
     tldr = {We generate mnemonics by aligning an LLM using preferences from 47 GRE test-takers on the mnemonics they like and which mnemonics aid learning},
     pages = {14202--14225},
   }

2. ACL 2024

   Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?

   Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger

   In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

   Best Paper Award (4%) and Oral (7%) at MASC-SSL 2024

   Abs TL;DR Bib PDF

   Multiple-choice question answering (MCQA) is often used to evaluate large language models (LLMs). To see if MCQA assesses LLMs as intended, we probe if LLMs can perform MCQA with choices-only prompts, where models must select the correct answer only from the choices. In three MCQA datasets and four LLMs, this prompt bests a majority baseline in 11/12 cases, with up to 0.33 accuracy gain. To help explain this behavior, we conduct an in-depth, black-box analysis on memorization, choice dynamics, and question inference. Our key findings are threefold. First, we find no evidence that the choices-only accuracy stems from memorization alone. Second, priors over individual choices do not fully explain choices-only accuracy, hinting that LLMs use the group dynamics of choices. Third, LLMs have some ability to infer a relevant question from choices, and surprisingly can sometimes even match the original question. We hope to motivate the use of stronger baselines in MCQA benchmarks, the design of robust MCQA datasets, and further efforts to explain LLM decision-making.

   We find that LLMs don’t need the question in multiple-choice question answering to do better than random chance, and explore how

   @inproceedings{balepur-etal-2024-artifacts,
     title = {Artifacts or Abduction: How Do {LLM}s Answer Multiple-Choice Questions Without the Question?},
     author = {Balepur, Nishant and Ravichander, Abhilasha and Rudinger, Rachel},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.acl-long.555},
     pages = {10308--10330},
     tldr = {We find that LLMs don't need the question in multiple-choice question answering to do better than random chance, and explore how}
   }

2023

1. EMNLP 2023

   Expository Text Generation: Imitate, Retrieve, Paraphrase

   Nishant Balepur, Jie Huang, and Kevin Chen-Chuan Chang

   In The 2023 Conference on Empirical Methods in Natural Language Processing: EMNLP 2023, Aug 2023

   Abs TL;DR Bib PDF Video

   Expository documents are vital resources for conveying complex information to readers. Despite their usefulness, writing expository documents by hand is a time-consuming and labor-intensive process that requires knowledge of the domain of interest, careful content planning, and the ability to synthesize information from multiple sources. To ease these burdens, we introduce the task of expository text generation, which seeks to automatically generate an accurate and informative expository document from a knowledge source. We solve our task by developing IRP, an iterative framework that overcomes the limitations of language models and separately tackles the steps of content planning, fact selection, and rephrasing. Through experiments on three diverse datasets, we demonstrate that IRP produces high-quality expository documents that accurately inform readers.

   We design an iterative planning, retrieval, and generation system to produce factual expository texts

   @inproceedings{balepur2023expository,
     title = {Expository Text Generation: Imitate, Retrieve, Paraphrase},
     author = {Balepur, Nishant and Huang, Jie and Chang, Kevin Chen-Chuan},
     booktitle = {The 2023 Conference on Empirical Methods in Natural Language Processing: EMNLP 2023},
     year = {2023},
     video = {https://youtu.be/6elNaka-JKM},
     tldr = {We design an iterative planning, retrieval, and generation system to produce factual expository texts}
   }