Development and validation of a multimodal data collection system for adolescent mental health management

Abstract

Objective

Adolescence is a critical developmental stage during which mental health vulnerabilities often emerge. Traditional self-report methods are insufficient to capture the complexity of emotional and physiological responses, underscoring the need for data-driven, personalized mental health strategies. This study aimed to develop and validate a structured multimodal data collection system for adolescents to support the future advancement of precision mental health care.

Methods

This study was conducted as the baseline phase of a longitudinal panel study designed to construct and validate a structured multimodal dataset for adolescent mental health research. A total of 74 adolescents aged 11–15 years from schools and community facilities in Korea was selected through convenience sampling. Multimodal data were collected by integrating six data types: self-reported surveys, electroencephalography (EEG), heart rate variability (HRV), genotyping, microbiome data, and video-based psychological counseling. Data collection was standardized through a three-phase protocol (pre-, on-site, and post-assessment), and participant privacy was protected via pseudonymization based on international standards. Variables were systematically labeled and structured to enable cross-modality analysis. Statistical analyses, including correlation and descriptive statistics, were performed to examine preliminary relationships across modalities.

Results

The study successfully constructed a comprehensive dataset encompassing biological and psychosocial indicators from 74 adolescents. Preliminary analysis revealed statistically significant associations between survey-based BMI and both genomic data (ρ = 0.30, p < 0.01) and microbiome-based obesity indicators (ρ = 0.27, p < 0.05), whereas other psychological constructs (e.g., stress, resilience) showed non-significant cross-modal correlations.

Conclusions

This study presents a replicable framework for collecting rich, multimodal data from adolescents in real-world settings. By enabling integrative analysis of biological and psychosocial variables, the dataset lays the groundwork for personalized mental health prediction and intervention strategies. Future research should expand longitudinally and optimize context alignment to improve predictive precision and clinical utility.

Introduction

Adolescence is characterized by emotional instability, heightened social sensitivity, and pronounced emotional vulnerability. ¹ Many psychiatric disorders and behavioral problems diagnosed in adulthood are closely associated with psychological changes that occur during adolescence, emphasizing the need for early intervention.^2,3 Recent approaches to mental health management have increasingly highlighted the need for personalized strategies that consider individual biological, psychological, and environmental characteristics. A one-size-fits-all model is inappropriate.^4,5 This perspective aligns well with the emerging paradigm of precision medicine that is gaining traction across various domains of healthcare.

Precision medicine integrates heterogeneous data including genomic information, biosignals, and behavioral patterns when defining disease subtypes and designing personalized prediction and intervention strategies. This paradigm is increasingly accepted when treating chronic diseases such as cancer, cardiovascular conditions, and diabetes.^6,7 However, although mental health conditions may exhibit chronic progression, associated with significant public health implications, mental health has seldom been discussed by those who promote precision medicine, largely because the supposed biological markers thereof are not reproducible and the diagnostic criteria therefore subjective. ⁸ Mental health issues in adolescents are triggered by complex interactions between biological vulnerabilities and psychosocial factors. A precision-based approach is essential. Feasible, data-driven integrative strategies are lacking. ⁹

Most mental health services for adolescents focus on groups at high risk of suicide or addiction. The services do not consider all developmental changes at this stage of life, or the effects of digital environments.^10,11 Existing assessment methods rely on self-reported answers to questionnaires or interviews, which are associated with subjective bias and one-time evaluation. Neither dynamic emotional states that require attention, nor emerging risk signals, are captured in a timely manner. ¹² The validity and reliability of the data modalities used for mental health assessment have not been adequately evaluated. ¹³ Systematic comparisons are lacking. It remains unclear how heterogeneous variables that are supposed to reflect the same contextual phenomena in fact capture the relevant psychological constructs.

In response, multimodal data-driven approaches that integrate the collection and analysis of various data types (text, speech, images, and biosignals) have been developed. ¹⁴ Multimodal data fusion aids an understanding of adolescent emotional states, psychological characteristics, and behavioral patterns. In particular, studies combining unstructured data such as smartphone sensor inputs, voice and text information, and biosignals from wearable devices with self-reported survey data have demonstrated practical potential in predicting adolescents’ depression and anxiety, tracking symptoms, and recognizing emotions, showing notable technological progress in the field.^15,16 Accordingly, multimodal data have been recognized as a promising methodological framework that captures emotional and behavior changes often overlooked by single measures, thereby enhancing the early detection and predictive precision of mental health problems. ¹⁷

Nevertheless, research on multimodal data construction for adolescents remains in its early stages, and there is a notable lack of systematically developed and practically applicable data collection processes. Existing studies have primarily employed two or three modalities that were combined in a parallel manner, with limited temporal alignment and contextual integration across data sources. These approaches have been criticized for their insufficient capacity to capture the developmental characteristics and environmental interactions unique to adolescence.^18,19 Studies utilizing modalities such as speech, video, text, and physiological signals for depression risk detection have consistently reported that unimodal features often lack robustness and that cross-modal interactions are insufficiently captured. ²⁰ Furthermore, the absence of standardized protocols for data labeling, quality control, and the definition of modalities and tasks limits the reproducibility and reliability of multimodal datasets. ²¹ These methodological and ethical challenges are further complicated in adolescence research, given their developmental characteristics and the sensitivity of personal data. Addressing these complexities requires the establishment of a systematic and collaborative data collection process that can be feasibly implemented in real-world research setting, with active engagement from schools and local communities to ensure both technical rigor and ethical integrity. ²²

This study designs and implements an adolescent data collection process applicable to real-world settings. This approach may serve as a foundation for future, multimodal data utilization, and aid the development of personalized mental health care for adolescents in need.

Methods

The preparatory phase

Prior to construction of a multimodal dataset, a systematic preparatory phase explored the research design, the development of data collection protocols, the need for appropriate Institutional Review Board (IRB) approval, the recruitment of participants, and the required equipment. The key variables in terms of physiological, psychological, and genetic factors were established. To ensure data reliability and standardization, all measurements were defined in detail, as were the instruments required by each data item.

This study was designed as the baseline phase of a longitudinal panel study, aiming to construct and validate a structured multimodal dataset for adolescent mental health research. As the same cohort will be used for subsequent follow-up and effectiveness studies, the sample size was determined by referring to previous studies and by considering an expected dropout rate of approximately 30% to maintain statistical power and ensure sample normality.^23,24 Therefore, efforts were made to recruit at least 50 adolescents to secure a sufficient number of valid cases for analysis. The Kongju National University IRB approved the study (approval no. KNU_IRB_2024_071).

We targeted adolescents aged 11–15 years either enrolled in elementary and middle schools of the Chungcheongnam-do region or affiliated with welfare centers/facilities in the Bucheon area. Prior to participation, informational sessions were offered to both the guardians and adolescents. Detailed explanations of the study purpose, procedures, the types of data to be collected, the intended uses thereof, and the measures employed to protect personal information were visually presented. Written informed consent was obtained from both the adolescents and their parents or legal guardians in compliance with ethical requirements. Eligible participants were those who understood the purpose of the study and voluntarily agreed to participate, providing informed consent for the use of personal information together with their parents or legal guardians. Individuals who did not agree to provide personal information during the preliminary screening, who withdrew participation due to personal reasons during the study period, or who had difficulty completing the electroencephalography (EEG) or heart rate variability (HRV) measurements were excluded.

For data collection, bio-signal devices were prepared to measure EEG and HRV, along with testing kits for genotype and microbiome analyses. Additionally, an online survey tool based on Google Forms was developed to assess psychological and emotional characteristics, thus establishing all necessary data collection tools in advance.

Data collection

In coordination with associated institutions, data were collected on single visits between October 2024 and January 2025. Data collection was rigorously standardized. The protocol was continuously refined to address issues encountered during data collection. A multimodal approach was employed, as detailed below.

Here, a “modality” was operationally defined based on the nature of the data and the method of measurement. The six types of data collected were divided into three modalities by the sources of measurement and the biological characteristics, as follows:

- Psychosocial modality: Self-reported and expert-evaluated data, including survey responses and counseling records,

- Bio-signal modality: Signal data (physiological responses); EEG and HRV measurements,

- Biological modality: Genotypic and microbiome data.

The psychosocial modality: Survey data. Each participant used a Google form to self-report on 83 items that explored general characteristics, mental health indicators (stress, anxiety, and depression), resilience, and health-related behaviors (Supplementary 1). Each response was scored using standardized and validated scales that have been widely applied to adolescent populations, including the Patient Health Questionnaire-9 (PHQ-9) for depression, the Generalized Anxiety Disorder-7 (GAD-7) for anxiety, and the Perceived Stress Scale-10 (PSS-10) for stress. Resilience was measured using the Korean Connor-Davidson Resilience Scale-10 (K-CD-RISC-10), which is a licensed instrument and was purchased and used under the appropriate license. ²⁵ In addition, several items related to attention, subjective mental health, and physical activity were adopted from nationally approved public health surveys, such as the Korea Community Health Survey, the Korean Children and Youth Panel Survey, and the National Mental Health Survey of Korea. This ensured both content validity and developmental relevance.^26–34 We required participants to complete all survey fields. A trained researcher was present throughout to support each participant and provide real-time assistance if confusion or hesitation arose. To ensure psychological stability, each survey was administered in a quiet room.

The psychosocial modality: Counseling data. Based on the survey results, one-on-one remote (ZOOM) counseling sessions were scheduled between each participant and a professional counselor. After a participant consented, all sessions were video- and audio-recorded by cameras at defined positions. Participants joined using smartphones, tablets, or computers. The videos recorded only the participants; the audio recordings captured both sides of the conversation. Each session included both a set of common questions and personalized questions suggested by the prior survey responses. During each session, the counselor completed a structured report that documented participant attitudes and behaviors, the key discussion points, observed issues, and suggested interventions. The reports were in a format that permitted qualitative analysis.

The bio-signal modality: EEG data. Each EEG was used to measure cognitive and emotional states. The data were collected using the MINDD SCAN (YEP-119B; Ybrain, Republic of Korea), a 19-channel non-invasive device compliant with the international 10–20 electrode placement standard. To ensure accurate data acquisition in a relaxed setting, all measurements were obtained by a trained researcher, and recordings lasted for at 5 min. The data were processed to remove 60-Hz power line noise generated by the standard 220-V power supply, and additional artifacts (such as eye movements) were then eliminated using EEGLAB software. After preprocessing, segments exhibiting excessive noise were excluded. The final dataset included the frequency-domain features of the Delta, Theta, Alpha, Beta, High Beta, and Gamma bands.

The bio-signal modality: HRV data HRV measurements reveal the emotional state. The data were collected using the HRV sensor integrated into the same MINDD SCAN (YEP-119B; Ybrain, Republic of Korea) used for EEG acquisition. All data were acquired by a trained researcher in a relaxed environment. HRV data were collected over a 3-min period using a clip-type electrode, and included time- and frequency-domain, and nonlinear, indices. The time-domain features, based on the R-R intervals, included the SDNN, RMSSD, and pNN50 parameters that correlate with the levels of stress and other emotional factors. Frequency-domain features were derived by determining the power distributions of heart rate intervals across specific frequency bands: VLF (0.0033–0.04 Hz), LF (0.04–0.15 Hz), and HF (0.15–0.4 Hz). Nonlinear indices, including SD1 and SD2, were extracted using Poincaré plots.

The biological modality: Genotyping data. Genotyping data was collected to explore genetic factors associated with mental health. Saliva samples were collected from participants, from which deoxyribonucleic acid (DNA) was extracted. Genotyping was performed through chip hybridization and chip scanning. Genotyping quality control procedures were applied to remove low-quality single-nucleotide polymorphisms (SNPs), and only high-quality genotyping data were retained for analysis. As a result, 121 genotyping features across six categories were obtained, including nutrient metabolism, dietary habits, and exercise responsiveness, which are relevant to gene-based health management (Figure 1).

Figure 1.

Genotypic data collection and processing.

The biological modality: Microbiome data. The microbiomes of stool samples self-collected by participants were determined. DNA was extracted, and the 16-S ribosomal ribonucleic acid gene region amplified and sequenced. Adapters and primers were trimmed, followed by quality filtration, denoising, chimera removal, and identification of amplicon sequence variants. The alpha and beta diversities were used to evaluate the nature and functional characteristics of the gut microbiota (Figure 2).

Figure 2.

Microbiome data collection and analysis.

Data protection, structuring, and context-level fusion validation

We used a consistent pseudonymization/de-identification procedure to protect personal information, which, nonetheless enabled the analysis and integration of multimodal data. The procedure was developed in accordance with the 2024 Guidelines on Pseudonymized Information Processing issued by the Personal Information Protection Commission and the ISO/IEC 27559:2022 standard (Information security, cybersecurity and privacy protection—Privacy enhancing data de-identification framework).

As data were collected, each participant was assigned a pseudo-identifier (PID) mapped to his/her personal information in a table stored in a physically isolated/encrypted location that only the designated data protection officer could access. The PID served as the file name and internal identifier across all modalities, and was used to link/integrate data from the same individual. The following pseudonymization methods were applied. These depended on the data type.

Names, dates of birth, and phone numbers were removed from all datasets. Quasi-identifiers such as addresses, affiliated institutions, and age were either categorized or generalized. For example, dates of birth were in years only, and regional information was reduced to the city/province level.

For EEG and HRV data, metadata in file headers (names and collection timestamps) were deleted. Only the PID served as the filename and the internal identifier.

Genotypic and microbiome data were received from external agencies in a preprocessed/quality-controlled format; no raw data were delivered. Identifiable elements within these datasets (sample IDs and collection dates) were replaced by the PIDs.

When viewing video data during counseling, participant faces were blurred, and any personal data (such as real names) either muted or edited out. All voice data were voice-morphed, and personal information in the text and counseling records (names, family information, school descriptors) either removed or replaced with symbols such as “[Name].” To allow future analysis of facial expressions, the original (unblurred) videos were stored in encrypted form.

In addition, to ensure efficient utilization and reliability of the collected multimodal data, a consistent naming convention was established. Each variable was assigned a standardized prefix and format, and a variable guide developed. This detailed the definition, measurement unit, and data format for each variable.

We performed an exploratory, context-level correlation analysis to examine whether variables representing the same context exhibited similar patterns across different types of data. The analysis was limited to cases where two or more such variables were present across distinct modalities. Spearman rank correlation analysis was employed because certain variables were ordinal in nature, or grouped, and the distribution was therefore not normal. All analyses employed Python 3.12.0 of the PyCharm environment. The target valuables included attention, resilience, stress, and the body mass index (BMI). The corresponding indicators were:

- Attention: The survey attention score and the concentration index derived from EEG data,

- Resilience: The survey resilience score, the fatigue recovery index of the HRV data, and the fatigue-related index obtained from the microbiome data,

- BMI: The self-reported weight and height, the obesity risk score derived from the genotyping data, and the obesity index obtained from the microbiome data,

- Stress: The survey PSS score, the EEG stress index, the HRV stress resistance index, and the frequency of the Korean word “스트레스” (stress) from the counseling reports.

Results

The multimodal data collection protocol

A standardized data collection protocol was employed during construction of multimodal data. The protocol featured three phases: pre-assessment, on-site assessment, and post-assessment (Figure 3).

Figure 3.

Multimodal data collection protocol.

Pre-assessment. To allow participants to collect stool samples in a comfortable environment, a pre-assessment phase was conducted prior to their visit to the institution. Microbiome testing kits were delivered by mail, and participants were instructed to bring their samples with them at the time of their visits.

On-site assessment. On the day of the institutional visit, each participant completed the survey, and EEG, HRV, and genotypic testing, in environments optimized for each procedure. As ambient noise can compromise data quality, EEG and HRV measurements were conducted in the quietest room available. Some participants found it difficult to provide saliva samples for genotypic testing. We considered that the associated stress might affect EEG results. Therefore, such testing was the final stage of on-site assessment.

Post-assessment. Based on the results collected during on-site assessment, one-on-one ZOOM sessions were scheduled with professional counselors. All sessions were recorded. The counselors documented counseling logs that included the participant's reported mental health concerns, emotional responses, and an overall summary of the session.

The results of multimodal data collection

Ninety-three adolescents aged 11–15 years from Chungcheongnam-do and Bucheon were recruited. Nineteen later withdrew; data were finally collected for 74 of mean age 14.45 years. Of all participants, 63.5% were male, and 10.8% were out-of-school (Table 1).

Table 1.

Sociodemographic characteristics of the study participants.

Variables	n (%)
Age (years)
Mean (SD)	14.45 (1.24)
Range	11–15
Sex
Male	47 (63.5)
Female	27 (36.5)
School attendance status
In school	66 (89.2)
Out of school	8 (10.8)
Household type
Single-parent	7 (9.5)
Two parents	53 (71.5)
Multicultural	8 (10.8)
Other	6 (8.2)

SD: standard deviation.

Seventy-four multimodal datasets were constructed. As certain individuals did not respond in terms of particular modalities, the final numbers of microbiome and psychological counseling datasets were 73 and 70, respectively. The multimodal data established in this study featured six modalities: survey, EEG, HRV, genotyping, microbiome, and psychological counseling data. The key data types and their characteristics in terms of each modality are summarized below.

The psychosocial modality: Survey data. To establish a multidimensional understanding of adolescent mental health, the survey data captured psychological and emotional characteristics, health-related factors, and sociodemographic attributes. Data from 74 participants were collected; high-quality responses were obtained from all individuals. The survey data were used to derive scores on standardized scales or measurements of categorical variables, based on the guidelines of each measurement tool and the objectives of the study. The dataset was systematically curated for future statistical analyses. The key variables of the survey data are summarized in Table 2.

Table 2.

Key variables of the survey data.

Variable	Measurement tool	Description
Sociodemographic characteristics
Sex	Researcher-developed item	Male/Female
Age	Researcher-developed item	Age in years (self-reported)
Household type	Researcher-developed item	Respondent-selected single-parent, two-parent, or multicultural household
School attendance	Researcher-developed item	Respondent-indicated in- or out-of-school status.
Psychological and emotional characteristics
Depression	PHQ-9	4-point Likert scale (score range: 0–27)
Anxiety	GAD-7	4-point Likert scale (score range: 0–21)
Stress	PSS-10	5-point Likert scale (score range: 0–40)
Attention	Seven items from the Korean Children and Youth Panel Survey	4-point Likert scale (score range: 7–28)
Resilience	K-CD-RISC-10	5-point Likert scale (score range: 0–40)
Metacognition	Metacognition scale for adolescents 33	5-point Likert scale (score range: 0–70)
Subjective Mental Health	Four items from the National Mental Health Survey of Korea 2021	5-point Likert scale(1 = very good, 5 = very poor)
Health behaviors
Physical activity	Three items from the Korea Community Health Survey	Number of days with physical activity in the past 7 days
Sleep duration	Researcher-developed item	Average daily sleep duration on weekdays and weekends

The psychosocial modality: Counseling data. To understand comprehensively the psychological and emotional characteristics of the adolescents, and mental health issues, psychological counseling data were collected from 70 participants (Table 3). The counseling sessions were conducted remotely (via ZOOM) and recorded in both video and audio formats. Counselors completed structured reports that were used to construct text data. These reports captured a wide range of psychological and emotional features that are difficult to quantify using standardized measures, including not only mental health conditions (e.g., depression, anxiety, and stress) but also peer and family relationships, academic and career concerns, self-awareness, and patterns of emotional expression. The counseling data are an important resource that reduces the lack of qualitative contextual information in many multimodal analyses.

Table 3.

Key variables of the counseling data.

Variable	Description
Video and audio recordings
Original video file	ZOOM-based counseling session recordings (.mp4)
Extracted audio file	Audio files extracted from counseling videos (.wav)
Video sample unit	Five-second video segments (.mp4) that served as the base units for multimodal analysis
Frame image	Images extracted frame-by-frame from video samples
Counseling content	Transcriptions of speaker utterances in the audio files
Counseling report
General appearance	Outward appearance of a participant
Attitude and behavioral traits	Behavioral characteristics observed during counseling (uncooperativeness, formality, defensiveness, resistance, and variability)
Counseling summary	Summary of the key topics discussed, including mental health, peer and family relationships, academic and career issues, self-awareness, and emotional expression
Follow-up recommendations	Any need for additional psychological counseling or mental health support

The videos were segmented into 5-s units to generate individual samples for subsequent analysis. Each analytical batch featured 16 samples. Each sample contained multimodal data (images, audio, and text). Images were extracted as individual frames and resized and filtered before storage. Audio data were extracted in .wav format. Preprocessing featured format unification and noise reduction. This improved the audio quality and will aid future analysis. Text data were generated by distinguishing the speakers on the audio followed by alignment of segments with the audio timeline, and transcription (Figure 4).

Figure 4.

Multimodal data collection using the counseling videos.

The bio-signal modality: EEG and HRV data. To assess adolescent cognitive and emotional states from a multidimensional perspective, EEG and HRV data were collected. Data acquisition was conducted in a private setting following a standardized measurement protocol. Complete EEG and HRV datasets were obtained from all 74 participants. EEG data reveal stress levels and emotional reactivity. HRV data are affected by stress, the autonomic nervous system balance, and psychological stability. The key variables and characteristics of the bio-signal data are presented in Table 4.

Table 4.

Key variables of the EEG and HRV data.^35–39

Variable	Interpretation
EEG data
Frontal beta power	Increases with elevated stress
High beta power	Increases with elevated stress
Delta power and theta power	Tends to increase as cognitive ability decreases
Theta/beta ratio (TBR)	Commonly elevated in children with Attention Deficit/Hyperactivity Disorder (ADHD)
HRV data
Time-domain indicators
Mean RR	Lower values indicate higher stress
SDNN	Lower values indicate higher stress
RMSSD	Lower values indicate higher stress
pNN50	Higher values indicate higher stress
Frequency-domain indicators
HF	Decreases as stress increases
Nonlinear indicators
SD1SS	Decreases as stress increases

The biological modality: Genotyping data. To assess adolescent health characteristics and physiological traits, genotypic data were obtained using the saliva samples with a focus on genetic variants that are known to negatively affect specific traits. A total of 121 genetic markers of 74 participants was surveyed, and the results divided into six categories: health management (22 items), nutrients (35 items), eating habits (20 items), physical activity (10 items), skin/hair (18 items), and personal characteristics (16 items) (Table 5). Each genetic score was interpreted in reference to the Korean average of 61. Based on the extent of genetic influence, traits were classified into levels described as “safe,” “average,” or “at-risk”, and sometimes—in terms of walking speed, grip strength, hair thickness, sleep type, and heart rate—participants were categorized into specific phenotype-based groups (e.g., slow/normal/fast walkers, short/long sleepers).

Table 5.

Key variables of the genotyping data.

Variable	Description	Related genes		Genetic impact (%)
Health management
Obesity	Caused by a complex interplay among various environmental factors: poor dietary habits, lack of physical activity, excessive energy intake, medication use, and inherent genetic predispositions	· FTO (associated with serious obesity) · BDNF (involved in leptin regulation) · MC4R (a regulator of satiety) · GIPR (a promoter of insulin secretion) · GNPDA2 (a lipogenesis-associated agent)	· HNF4G (a regulator of lipid metabolism) · NEGR1 (a regulator of the body mass index) · NRXN3 (involved in neural signal transmission) · RPTOR (involved in nutrient catabolism) · LOC144233 (associated with obesity)	45
BMI	A representative indicator of obesity	· FTO · BDNF · MC4R	· GIPR · CDKAL1 (involved in insulin secretion)	14.7–32
Abdominal obesity	Individuals are classified as having abdominal obesity if their waist-to-hip ratio is ≥0.9 for men and ≥0.85 for women	· ADAMTSL3 (activator of protein degradation) · CPEB4 (promoter of adipocyte formation) ·FGFR4 (promoter of adipocyte proliferation) · RSPO3 (stimulates adipocyte activity)	· BMP2 (promotes adipocyte formation) · CCDC92 (involved in adipocyte differentiation) · PBRM1 (promotes adipocyte breakdown) · TBX15 (involved in skeletal development)	8.2
Weight regains	Tendency to regain weight after weight loss	· BDNF · NEGR1 · LEP (regulator of leptin production)	· LAMB1 (involved in cell adhesion) · ADRB2 (controller of energy expenditure) · POSTN (involved in wound healing)	50
Exercise-related weight loss effect	Impaired genetic switches regulating energy expenditure may reduce fat-burning efficiency, thereby diminishing the effect of weight loss through exercise	· FTO · BDNF · MC4R	· SEC16B (involved in organelle formation) · TMEM18 (involved in cell migration)	16.1
Nutrients
Calcium levels	Calcium is a major mineral of bones and teeth; it is involved in neural signal transmission, and modulates the contraction and relaxation of muscles and blood vessels	· CASR (detects calcium levels) · CYP24A1 (regulatory genes of active vitamin D metabolism)	· DGKD (regulator of bone mineral density) · GCKR (regulator of glucose metabolism)	39∼45
Vitamin D levels	A fat-soluble vitamin that promotes calcium absorption, strengthens bones, and supports immune function. A deficiency thereof is common among individuals who are not often outdoors	· GC (transports of vitamin D) · CYP2R1 (involved in vitamin D synthesis)	· NADSYN1 (involved in vitamin D metabolism)	70∼77
Magnesium levels	An essential mineral that is a major component of bones and teeth and plays a critical role in regulating muscle and nerve function	· DCDC5 (contributes to bone mineral density) · MECOM (influences magnesium balance)	· MUC1 (involved in mucin production) · SHROOM3 (regulator of kidney function)	29∼35
Vitamin B6 levels	A nutrient involved in the synthesis of neurotransmitters such as serotonin and dopamine that in turn affect hippocampal activation, and therefore memory	· ALPL (involved in vitamin B6 metabolism)	· ADCYAP1R1 (regulator of hormone secretion)	17∼27
Omega-3 fatty acid levels	An essential fatty acid that must be obtained from food. Required for brain and visual development, cardiovascular health, and mitigation of inflammation	· FADS2 (desaturates fatty acids) ·ELOVL2 (involved in fatty acid synthesis) · HERPUD1 (maintains endoplasmic reticulum homeostasis) · GCKR (regulator of glucose metabolism) · LOC157273 (influences DHA levels)	· TM6SF2 (regulator of hepatic lipid metabolism) · LIPC (involved in lipid metabolism) · C11orf10 (involved in lipid transport) · FADS1 (synthesizes unsaturated fatty acids)	75
Omega-6 fatty acid levels	An essential fatty acid in terms of brain development; vascular health; and high-quality skin, hair, and nails	· FADS2 · LIPC	· APOC1P1 (apolipoprotein-related pseudogene) · ASAP3 (promotes fatty acid transport)	52
Iron storage levels	Iron is a mineral, about 70% of which is found in hemoglobin in the blood; it transports oxygen and helps prevent anemia. This refers to the concentration of stored iron in the body	· TF (transports iron) · TFR2 (transports iron)	· TMPRSS6 (regulator of iron absorption)	30∼44
Eating habits
Appetite	Unlike hunger, which is a general state of need during fasting, appetite refers to the desire to consume specific foods even in the absence of physical hunger	· MC4R · GAD2 (produces GABA)	· PRKCA (associated with food addiction)	28
Satiety	A sensation of fullness that promotes satisfaction after eating and helps to regulate food intake	· LEP · FTO · GCKR	· CCNL1 (regulator of leptin) · LEPR (binds leptin)	63
Sweet taste sensitivity	Influenced by environmental factors such as diet and age, and the genetics of taste receptors. A lower sensitivity may increase the wish for sweet foods, potentially associated with weight gain	· FTO · FGF21 (promoter of sugar absorption)	· TAS1R2 (detector of sweet taste)	16
Salty taste sensitivity	Influenced by environmental factors such as diet and age, and the genetics of taste receptors. Sensitivity may increase the secretion of appetite-related hormones that contribute to weight gain	· SCNN1B (involved in sodium transport)		22
Bitter taste sensitivity	Among the five basic tastes, bitterness exhibits the most complex genetic patterns; strong bitterness may suppress gastric secretion, irritate the stomach, and reduce appetite	· TAS2R38 (detects bitter tastes) · PRH1 (detects bitter tastes)	· DIRC3 (detects bitter tastes)	15
Physical activity
Endurance exercise suitability	The ability to sustain physical activity for an extended period by minimizing fatigue	· HIF1A (regulator of muscle fiber composition) · PPARD (involved in lipid metabolism)	· VEGFA (promoter of vascular growth)	28∼47
Aerobic exercise suitability	An indicator that evaluates the capacity essential for performing aerobic exercise	· ACSL1 (regulator of insulin) · SCN10A (encodes the sodium ion channel)	· PRARGC1A (supports energy metabolism)	24∼48
Muscle development capacity	An indicator reflecting the bodily ability to circulate blood and support immune function via muscle-related mechanisms	· SLC30A8 (zinc transporter) · TRHR (involved in muscle metabolism)	· VCAN (promoter of muscle cell growth)	78
Muscular strength exercise suitability	The maximum force a muscle or muscle group exerts in a single contraction. This affects posture maintenance, organ movement, and exercise efficiency	· ACTN3 (generates fast-twitch muscle fibers) · MTHFR (involved in vitamin B metabolism)	· SOD2 (removes reactive oxygen species)	22∼45
Sprinting ability (short-distance)	The fast-twitch muscle contraction power required for short-distance sprints from 50 to 400 m	· ACTN3 · SOD2 · AGT (regulator of blood pressure)	· MCT1 (lactate transporter) · AMPD1 (regulator of muscle energy)	73
Grip strength	The force exerted when gripping with the hand; an indicator of overall muscular function	· MC4R · ABHD17C (involved in energy production) · ADCY9 (involved in energy metabolism) · CRTAC1 (related to cartilage tissue)	· GBF1 (activator of signal transduction) · GNPDA2 (involved in lipid metabolism) · MGMT (involved in DNA repair) · TRIM27 (repressor of transcription)	65
Skin and hair
Skin inflammation	Any inflammatory skin condition caused by a complex interaction between genetic, environmental, and immunological factors	· IL18R1 (regulates immune response) · PBX2 (associated with skin inflammation) · RTEL1 (associated with skin inflammation) ·TMEM232 (associated with skin inflammation)	· ZNF365 (involved in DNA damage repair) · TSBP1 (involved in the development of skin inflammation) · GLB1 (contributes to elastic fiber formation)	75
Personal characteristics
Insomnia	A sleep disorder caused by stress, poor sleep habits, and/or genetic factors, characterized by difficulty falling asleep or frequent awakenings during sleep	· CYCL1 (regulator of sleep) · DAB1 (involved in neural development) · KLHDC8B (involved in cell division)	· WDR27 (influences sleep regulation) · TGFBI (involved in sleep regulation)	59
Sleep habits and duration	Sleep patterns vary individually. Some function well with less than 6 h of sleep, others require more than 10 h to feel rested and function normally	· FOXP2 (associated with language function) · LINCOO243 (influences sleep) · PAX8 (involved in sleep regulation) · LOC101927400 (involved in sleep regulation)	· LOC105377632 (affects sleep duration) MAD1L1 (controls the cell cycle) · TCF4 (regulator of protein activity) · VRK2 (modulator of signal transduction)	9
Pain sensitivity	A protective sensory mechanism; sensitivity varies among individuals	· FAM173B (involved in pain regulation)	· SCN10A (encodes the sodium ion channel)	29

The biological modality: Microbiome data. To assess gut health and mental wellness, microbiome data were derived from stool samples self-collected by participants. A total of 73 datasets were obtained. The focus was on microbial diversity and the distributions of functionally significant gut bacteria. Sixteen indicators were derived, of three categories: the overall results, gut health, and wellness markers (Table 6). All data were scored on a scale from 1 to 100, and participants were categorized into three groups—“safe,” “average,” and “at-risk” by reference to the Korean average.

Table 6.

Key variables of the microbiome data.

Variable	Description	Average score of Koreans
Summary results
Gut healthsummary type	Vulnerability factors among the gut health indicators
Wellnesssummary type	Vulnerability factors among the wellness-related indicators
Gut health score	A score representing the diversity and abundance of gut microbiota, where higher values indicate healthier gut environments	50
Beneficialbacteria score	A score reflecting the distribution of beneficial bacteria that contribute to vitamin synthesis, immune enhancement, and metabolic improvement	65
Harmfulbacteria score	A score indicating the distribution of harmful bacteria associated with diseases such as obesity and diabetes	61
Gut health
Constipation	A predicted constipation risk score based on the distribution of gut microbiota that modulate colonic motility	54
Flatulence	A predicted score for gas production based on the presence of gas-producing gut bacteria	59
Abdominal bloating	A bloating score reflecting gut microbiotic imbalance and abnormal gastrointestinal motility	53
Functional abdominal discomfort	A discomfort score associated with psychological factors and gut microbial imbalance	57
Diarrhea	A diarrhea risk score based on microbial imbalance and infection, diet, and stress	45
Wellness
Happiness index	An indicator of psychological stability and well-being based on the levels of neurotransmitters (such as serotonin) produced by gut microbiota	44
Fatigue index	A measure of fatigue that is influenced by the immune responses and neurotransmitter changes that reflect the gut microbial balance	50
Immunity index	An indicator of immune function based on the activities of microbes that contribute to mucosal immunity and suppress pathogens	37
Obesity index	An obesity risk index based on the relationship between gut metabolites and weight regulation	54
Sleep index	An indicator of sleep quality related to the diversity of gut microbiota that produce substances such as GABA and serotonin	57
Aging index	An indicator of biological aging reflecting the relationship between increased levels of putrefactive or harmful gut bacteria and aging per se	46

Data integration and context-level correlations

The derived multimodal data were integrated in a format that established links across the modalities—survey, EEG, HRV, genotyping, and microbiome data; and psychological counseling—based on the participant-specific PIDs. This enabled seamless integration of diverse data types, both structured and unstructured. Structured data were tabled and unstructured data managed using PID-based metadata and linkage keys. The architecture supports cross-modal connectivity and enables integrated analysis at the participant level. The data can be input to multimodal machine learning models.

Spearman rank correlation analysis was conducted to explore whether indicators that measured the same context exhibited similar trends across different datasets (Figure 5). Relatively consistent correlations among the BMI-related indicators were observed. Significant positive correlations were apparent between the survey-based BMI and genomic data (ρ = 0.30, p < 0.01) and between the survey-based BMI and microbiome-based obesity indicators (ρ = 0.27, p < 0.05). In contrast, indicators of stress, attention, and resilience were generally minimally correlated, with no statistical significance.

Figure 5.

Multimodal data collection and the context-level correlations. Note. The color of each circle corresponds to the specific data source from which the associated variable was extracted. The data sources used for each context-level fusion were as follows: ·Attention: The survey-based attention score and the EEG-based concentration index.·Resilience: The survey-based resilience score, the HRV-based fatigue recovery index, and the microbiome-derived fatigue index.·BMI: The survey-calculated BMI, the genotypic obesity risk score, and the microbiome obesity index.·Stress: The survey PSS score, the EEG-based stress index, the HRV-based stress resistance, and the frequency of the Korean word “스트레스” (stress) from the counseling reports.

To enable systematic data analysis and management, variable names with modality-specific prefixes were assigned to all data types except survey data, and a corresponding variable guide developed. Variables were labeled by their data source as follows: E_ (EEG), H_ (HRV), G_ (genotype), M_ (microbiome), and C_ (counseling). The variable guide included the variable names, the data types, and the value ranges. It was sought to ensure consistency and reproducibility throughout the entire analysis (Supplementary 2).

Discussion

This study constructed a multimodal dataset containing six heterogeneous types of data from individual adolescents: survey responses, EEG, HRV, genotype, microbiome, and psychological counseling records. Mental health issues tend to emerge in various complex ways during adolescence; early intervention is critical. The collection and linking of diverse physiological and psychological data are essential when it is sought to understand individual characteristics and risk factors in a multi-layered manner. ⁴⁰ Most previous multimodal studies evaluated the modalities either in parallel or independently, rendering it difficult to interpret biological factors within their psychosocial contexts. In contrast, this study used structured multimodal data for integrated analysis of adolescent mental health.

Adolescents often do not well-articulate their emotional states. Their statements are easily influenced (and indeed altered) by the social context. Standardized questionnaires do not fully capture internal experiences. ⁴¹ To address this limitation, the present study constructed a dataset that included not only self-reported survey information but also text, audio, and image data collected during psychological counseling. It was thus possible to analyze verbal and nonverbal characteristics in an integrated manner. This approach enabled identification of discrepancies between self-reported and externally observable responses, aiding in the detection of high-risk emotional suppression, avoidance, and defensive reporting. ⁴² The higher-order psychological indicators employed, including metacognition and resilience measured by the survey instruments, can be used to assess the capacity for self-regulation and psychological flexibility.^43,44 When such structured psychological data are integrated with biosignals and qualitative data derived during counseling sessions, a psychosocial context emerges in which the connections between subjective experiences and physiological responses can be interpreted. ⁴⁵ This aids the quantification and interpretation of individual differences in terms of emotional expression strategies.

EEG and HRV biosignals reflect the reactivities of the central and autonomic nervous systems, respectively. Both neurophysiological tools have been widely used to define and detect adolescent mental health conditions early. EEG data reflect brain activities associated with attention, cognitive loads, and emotional responses, and changes in specific frequency bands have been reported to be related to emotional states such as depression, anxiety, happiness, and anger.^46–49 HRV indices including the RMSSD, SDNN, and LF/HF ratio vary with stress reactivity. The levels are significantly correlated with various emotional states including anxiety, depression, and post-traumatic stress disorder.^17,50 However, in previous studies, EEG and HRV data were often collected in different environments or at various times, rendering it difficult to interpret the relationship between the two physiological signal groups in any contextualized manner. ⁵¹ Such disjointed research yields only a fragmented understanding of emotional responses. The signal data are interpreted in isolation, rendering it difficult to capture interplays among physiological responses across the various modalities. ⁵² In this study, EEG and HRV were measured sequentially in the identical setting, yielding the empirical foundation required when interpreting the interactions between the two signal types under relatively consistent emotional conditions. This rendered it possible to analyze precisely any discrepancies or individual variations in how physiological signals responded to emotional states.^53,54 This enhances the utility of biosignal-based, nonverbal indicators in terms of adolescent mental health research.

Genotypic and microbiome data afford essential insights when seeking to understand the underlying multi-layered mechanisms of adolescent mental health. Genotyping yields information on emotional reactivity and temperamental vulnerability by identifying genetic variations associated with depression, anxiety, and impulsivity.^55,56 Genotyping data obtained during adolescence thus serve as important biological indicators that enable early prediction and personalized intervention. ⁵ The microbiome, through the gut–brain axis, reflects environmental factors associated with emotional regulation, and certain gut microbial communities have been reported to show significant associations with conditions such as ADHD and depression.^57,58 It reflects lifestyle factors such as diet, sleep, and stress and can be modified through behavioral regulation, making it a biological indicator with high potential for intervention. ⁵⁹ In this study, both biological modalities were examined in the same individuals at the same times. This enabled integrative analysis of the interactions between fixed genetic traits and modifiable environmental factors. The empirical evidence showed that the biological pathways of mental health are neither linear nor unique being, rather, components of an interactive system. When both biological and psychosocial variables are subjected to multimodal integrative analysis, the interpretability and applicability of genotypic data improve, in turn advancing precision psychiatry. ⁶⁰

The correlations among variables reflecting the same context across the different data modalities revealed that most associations were not significant. This does not necessarily indicate inconsistencies among the variables but, rather, that each modality uniquely captures the same context at a different level of observation. ⁶¹ For example, stress may be a subjective perception in self-reported questionnaires, a physiological response in EEG data, and an expert interpretation in counseling records.^62,63 Such results suggest that theoretical and methodological considerations of how best to interpret and operationalize specific data are more important than the extent of agreement among the indicators per se. ⁶⁴ Furthermore, the low correlations should not be attributed to a single cause but, rather, to multiple factors, including differences in sensitivity and specificity across the tools, the multidimensional nature of psychological constructs, and the mismatch between physiological responses and self-awareness. ⁶⁵ Accordingly, the growing need for integrative interpretation of heterogeneous data has brought increased attention to digital phenotyping, with empirical evidence demonstrating its effectiveness in enhancing predictive accuracy and early detection of mental health conditions.^66,67 Digital phenotyping features continuous, ecologically valid assessments of psychological states using everyday data sources such as smartphones, biosignals, and language. Such an approach aligns closely with the goals of precision medicine, which emphasizes data-driven prediction and personalized intervention. ⁶⁸ In this context, the structured collection of diverse data from the same individual and the empirical analysis of interrelationships among such data constitute a foundational step toward future, precise mental health care. ⁶⁹ Future research should employ longitudinal designs that align multimodal data both temporally and contextually and also consider developmental changes and environmental influences. Also, it is essential to quantify the contribution of each modality and to develop systematic interpretive frameworks. This would improve clinical applicability and predictive precision.

This study empirically demonstrated that precise mental health assessment was possible using a structured multimodal design that integrated biological with unstructured data. However, several practical challenges emerged. First, some participants found saliva and stool collections psychologically uncomfortable. Second, the quality of an EEG or HRV measurement may be compromised by noise, unfamiliar equipment, or participant tension. Multimodal data collection requires not only technically accurate devices but also participant-sensitive environments and carefully designed protocols. ⁷⁰ Third, as counseling was online, all of platform resolution, camera positioning, and network stability may have limited the clarity of speech and compromised the capture of subtle facial expressions. Some participants, especially those unfamiliar with virtual communication or constrained in terms of physical movement, seemed to be “stiff”; expression was inhibited. All of pre-session adaptation, standardization of the measurement environment, and improvements in equipment are required to enhance the reliability and sensitivity of multimodal data collected via online counseling. In addition, a limitation of this study is that participants were recruited from specific regional populations, which may limit the generalizability of the findings to broader adolescent groups. Nevertheless, this study provides an important empirical foundation for developing standardized and ethically grounded multimodal data collection processes in real-world adolescent populations. Future research should build upon these findings by broadening participant diversity, enhancing environmental control, and refining data collection procedures to ensure the reproducibility and scalability of multimodal research frameworks.

Conclusions

This study developed and implemented a standardized multimodal data collection protocol integrating surveys, EEG, HRV, genotypic, microbiome, and counseling records to enhance comprehensive assessment of adolescent mental health. The protocol addresses the limitations of traditional self-report-based methods and provides a structured, data-driven framework incorporating biological, psychological, and social indicators. Technical and ethical considerations identified during implementation, including standardized environmental conditions and participant-oriented procedures, offer practical guidance for the reliable acquisition of high-quality multimodal datasets. The resulting dataset and collection system provide foundational infrastructure for early identification of mental health concerns and for the development of personalized intervention strategies, with increasing utility anticipated as the system is applied in larger and longitudinal studies.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076261415916 - Supplemental material for Development and validation of a multimodal data collection system for adolescent mental health management

Supplemental material, sj-pdf-1-dhj-10.1177_20552076261415916 for Development and validation of a multimodal data collection system for adolescent mental health management by Siyeon Ko, Kyoungsu Oh, Uhyeong Won, Jung-A Oh, Nak-Jung Kwon, Hyun-sook Park, Young-A Ji, Sungjin Kim, Yonghwan Moon, Nayoung Park, Dohyoung Kim, Euijun Yang, Kyungmin Na, Yeonju Kim, Youngho Lee and Hyekyung Woo in DIGITAL HEALTH

Supplemental Material

sj-xlsx-2-dhj-10.1177_20552076261415916 - Supplemental material for Development and validation of a multimodal data collection system for adolescent mental health management

Supplemental material, sj-xlsx-2-dhj-10.1177_20552076261415916 for Development and validation of a multimodal data collection system for adolescent mental health management by Siyeon Ko, Kyoungsu Oh, Uhyeong Won, Jung-A Oh, Nak-Jung Kwon, Hyun-sook Park, Young-A Ji, Sungjin Kim, Yonghwan Moon, Nayoung Park, Dohyoung Kim, Euijun Yang, Kyungmin Na, Yeonju Kim, Youngho Lee and Hyekyung Woo in DIGITAL HEALTH