Vickey Ghimire

Iterative prompting is a technique commonly used to progressively refine the output produced by Large Language Models (LLMs) to obtain more comprehensive content. This method is particularly useful in the context of education for drafting and refining lists of topics to be included in course outlines, syllabi, and lecture plans. However, one of the main challenges in iterative prompting is determining whether, at each iteration, the model outputs truly new material or merely rephrases existing content. While iterative refinement is a common practice, it often results in repetition, which requires educators to conduct time-consuming manual checks, thereby limiting the benefits provided by LLM-assisted course design. To mitigate this, we introduce a framework that applies a novelty metric, integrating both lexical and semantic similarity measures, to automatically categorize generated outputs as novel, redundant, or requiring human judgment. We tested this framework across seven widely used LLMs (ChatGPT o3, Claude 4 Sonnet, Gemini 2.5 Pro, DeepSeek R1, Grok 3, Qwen 2.5, and Llama 3.3) to create a Java programming course outline over multiple iterations. Results from conversations containing over 23,751 individual topics showed that DeepSeek R1 produced the highest percentage of novel content. Moreover, our method successfully automated classification for 96.7% of the outputs, substantially reducing the proportion of items that require manual review.

Large Language Models (LLMs) have been explored as a valuable tool for creating course outlines. To this end, iterative prompting can be utilized to correct and expand the output. However, iterative prompting often results in LLMs producing redundant content that requires extensive manual review. In this context, distinguishing between genuinely novel content and rephrased existing information creates a trade-off between LLM capabilities and validation efforts. To address this, this paper proposes a novelty metric that combines lexical similarity mea- sures with semantic analysis to automatically classify generated content as novel or repeated, thus simplifying human review. We evaluated this approach using five widely used LLMs (ChatGPT 4o, Claude 3.7 Sonnet, Gemini 2.5 Flash, DeepSeek v3, and DeepSeek r1) to create a Java programming course outline over multiple iterations. Our methodology automated the classification of 90.52% of the generated content, significantly reducing the number of items requiring manual review. Moreover, our findings show that the proposed metric can also be utilized for ranking the performance of models in the context of iterative prompting.

OpenAI’s new “memory" feature enables ChatGPT to provide more personalized and relevant interactions by storing user information from the prompts and using it across conversations. While offering improved responses, the memory feature poses privacy and security challenges. This paper reports a three-fold study investigating ChatGPT’s memory feature in more detail. First, we utilized the Knowledge-Attitude-Behavior model and distributed a survey to over 135 users to assess their awareness of ChatGPT’s memory functionality, attitudes toward privacy implications, and the behavioral changes prompted by perceived risks. Secondly, memory content from over 55 user accounts was analyzed to evaluate the accuracy, relevance, and privacy of the stored data. Finally, we studied the distribution of the stored data across key categories to obtain insights into what kind of information ChatGPT considers relevant and stores. The findings reveal gaps in user understanding of the memory feature, the need for greater transparency, and the challenges of personalizing LLM agents while safeguarding privacy