How to assess the reliability and research usability of synthetic text beyond a single performance metric
Large language models can generate realistic-looking text at scale, creating new opportunities for natural language processing (NLP) research in sensitive domains. As part of her MSc thesis at the University of Donja Gorica, Sara Đurović investigated whether synthetic mental-health diary datasets are suitable for controlled research experiments. The work was defended on 30 October 2026 under the supervision of Prof. Dr. Tomo Popović and co-supervision of Dr. Stevan Čakić.
CHALLENGE
Research involving mental-health-related text faces important data availability, privacy and ethical constraints. Although large language models can generate plausible diary entries, linguistic fluency does not guarantee that synthetic data reflect human-written text, support reliable model evaluation, or generalize beyond the dataset used for training.
A central challenge was therefore to distinguish technical consistency from genuine research usability, particularly when labels are generated together with the text rather than independently validated.
SOLUTION
The study assessed three synthetic diary datasets generated through workflows associated with GPT-5.5, Claude Sonnet 4.6 and Gemini 3.1 Pro. Each dataset contained 1,000 entries organized into 100 journals, using a shared data schema. Evaluation accounted for the journal-level grouping of entries to avoid treating related observations as statistically independent.
The evaluation combined structural validation, cluster-aware statistical tests, comparison with human-written reference text from SemEval-2026 Task 2, cross-dataset transfer experiments, a TF-IDF versus RoBERTa comparison, and an exploratory English–Montenegrin zero-shot transfer experiment with mmBERT.
The more computationally demanding RoBERTa and mmBERT experiments were conducted on HPC infrastructure, with logs, manifests and result archives retained to document the analyses.
RESULTS
All three datasets passed structural validation, but differed significantly in seven of nine synthetic label distributions. A classifier distinguishing synthetic from human-written reference text reached balanced accuracy of 0.9855–0.9942 even after controlling for text length, indicating substantial differences between the corpora.
Across 30 cross-dataset transfer comparisons, macro-F1 performance decreased by 26.04%–84.99% relative to corresponding within-dataset evaluations. RoBERTa achieved a higher macro-F1 than the TF-IDF baseline (0.7872 versus 0.7118) on the evaluated task. Exploratory English–Montenegrin transfer also showed limitations in absolute multi-label performance.
These findings demonstrate that the datasets can support controlled NLP experiments, but are not distributionally interchangeable. The work does not establish clinical validity of the synthetic labels or rank the language models by intrinsic quality.
BENEFITS
- More robust dataset evaluation: A multidimensional framework combining structural, statistical, distributional and model-based checks.
- Evidence of limited transferability: Quantified performance losses when models trained on one synthetic dataset are evaluated on another.
- HPC-supported experimentation: Computing resources supported transformer-based analyses and the preservation of documented experimental outputs.
- Responsible AI research: A practical example of how to identify limitations of synthetic data before considering broader uses in sensitive domains.
The research and experimental work were partially supported by the National Competence Centre for High-Performance Computing of Montenegro (NCC HPC Montenegro) within the EuroCC3 project.