On 30th October 2026, MSc candidate Sara Đurović defended her thesis entitled “Evaluation of Synthetic Mental Health Datasets Generated by Large Language Models” within the Information Technologies Master’s programme (Software Engineering module) at the Faculty for Information Systems and Technologies, University of Donja Gorica, under the mentorship of Prof. Dr. Tomo Popović and co-mentorship of Dr. Stevan Čakić.
The research and experimental work were partially supported by NCC Montenegro through HPC infrastructure and expertise within the EuroCC3 project. Through its support for academic research and the development of AI and HPC competencies, NCC Montenegro contributes to advancing the application of advanced computing technologies in natural language processing and responsible AI development.
The thesis investigates how the quality and research usability of synthetic text generated by large language models can be assessed, using mental health diary entries as a sensitive domain where access to real-world data is limited, and focusing on a multidimensional evaluation framework rather than a single quality metric.
ABSTRACT – This thesis presents a methodological evaluation of the quality and research usability of three synthetic mental health diary datasets produced through workflows based on GPT-5.5, Claude Sonnet 4.6 and Gemini 3.1 Pro. Each dataset contains 1,000 entries across 100 journals, sharing a common data schema and co-generated labels, with the journal treated as the unit of statistical independence. The evaluation combined cluster-aware permutation inference with Benjamini-Hochberg correction, a classifier two-sample test against a human reference corpus (SemEval-2026 Task 2), cross-dataset transfer experiments, a TF-IDF versus RoBERTa comparison, and zero-shot mmBERT transfer between paired English and Montenegrin text. The datasets differed significantly in seven of nine labels, and synthetic text remained clearly separable from human-written text even after controlling for text length (balanced accuracy 0.9855–0.9942). All 30 cross-dataset comparisons showed performance drops (26.04%–84.99%), while RoBERTa outperformed the TF-IDF baseline (macro-F1 0.7872 vs. 0.7118). The findings show that the datasets support controlled experiments but are not distributionally interchangeable, and that synthetic data quality requires multidimensional evaluation rather than a single metric. The results do not establish clinical validity of the labels or rank the generators by quality.
