From human-in-the-loop to LLM-in-the-loop for high quality legal dataset

Annotating legal documents with rhetorical structures is difficult and time-consuming, especially if done com-pletely manually. This paper explores two methodologies for optimal results: first, a human-in-the-loop ap-proach based on a multi-step annotation process with domain experts reviewing and revising datasets itera-tively. To enhance interpretability, eXplainable Artificial Intelligence (XAI) models are incorporated, aidingin understanding decision-making processes. Second, an LLM-in-the-loop method has humans leveraginggenerative large language models (LLMs) to assist experts by automating repetitive annotation tasks undersupervision. Further research is proposed to develop interaction models that effectively balance automationwith human guidance and accountability.