A simple and convenient sampling method was adopted to send 1321 questionnaires online to middle school students in Shandong Province of China. During the test, the subjects answered in the classroom. All questionnaires were self-reported. Additionally, we excluded those participants who took an unreasonably short or long time to complete the questionnaires, and who had missing data, and finally obtained 1208 valid data, with the recovery efficiency of 91.44%. Among them, the total sample of valid data included 605 males and 603 females, 562 students in Grade One, 311 students in grade two and 335 students in grade three. The age distribution was between 11 and 15 years old, with an average age of 13.02 ± 0.89 years old. After an interval of 2 weeks, 323 people from the initial sample were selected to retest the scale at random. 301 people were valid, and the recovery rate was 93.18%, including 153 boys, 148 girls, 115 students in grade one, 104 students in grade two, and 82 students in grade three. The demographic characteristics of the participants are shown in Table 1.

The researchers reached out to the authors who developed the original UMHL-A questionnaire and communicated with them about the project. Two English-Chinese bilingual researchers who specialized in psychology translated the original questionnaire into Chinese. Then, seven graduate students in the team expressed and corrected the questionnaire to avoid ambiguity, lengthy, and difficult to understand. In response to their feedback, the newly translated version was revised and subsequently sent to two bilingual individuals fluent in English and Chinese, who were then invited to independently perform back-translation. After discussion with nine subjects, the language development of students beyond the comprehension of junior high school students or difficult-to-understand parts were replaced, and the Chinese version of the scale was finally formed. The study was approved by Ethics Committee of School of Education and Psychology at the University of Jinan (IRB number: 202301). All participants signed the informed consent in this study.

Universal Mental Health Literacy Scale for Adolescents (UMHL-A). The UMHL-A, developed by Kagstrom et al. in 2023 [13], is based on the definition of mental health literacy by Kutcher et al. It comprises an attitudes subscale and a knowledge subscale, featuring four dimensions: help-seeking attitude, stigmatization attitude, general mental health knowledge, and specific mental illness knowledge, totaling 17 items. The attitudes subscale is evaluated using a 5-point Likert scale with an additional “don’t know” option (1 = strongly agree, 5 = strongly disagree, don’t know = 3 = uncertain). The Knowledge subscale uses the T/F scale, which means subscale uses a “yes” or “no” format, with “don’t know” as an option. A “yes” response scores 1 point, while “no” or “don’t know” responses score 0 points. A higher score on this scale indicates a greater level of mental health literacy.

Self-Stigma of Seeking Help (SSOSH) Scale. Developed by Vogel et al. in 2006 [20] and revised in China [21], the SSOSH Scale consists of 10 items rated on a 5-point Likert scale (1 = strongly agree, 5 = strongly disagree). A higher score indicates more severe self-stigma among research participants regarding seeking professional psychological help. In this study, Cronbach’s alpha was 0.75.

Seek Professional Psychological Help Attitude Scale. This scale, originally compiled by Fischer et al. in 1970 [22] and later revised in China [23], contains 29 items across four dimensions: trust in professionals, societal and individual perceptions, level of self-disclosure, and recognition of the need for psychological help. It utilizes a 5-point Likert scale (1 = strongly agree, 5 = strongly disagree), with higher scores reflecting a more positive attitude towards seeking psychological help. The scale’s Cronbach’s alpha in this study was 0.83.

Mental Health Knowledge Questionnaire. Issued by the General Office of the Ministry of Health of China in 2010 as part of a mental health indicators survey and evaluation program, this questionnaire includes 20 questions [24]. Questions 1 to 16 score 1 point for each correct answer and 0 points for incorrect ones. For questions 17 to 20, selecting “Know” scores 1 point, while “Don’t know” scores 0 points. The total score is the sum of all question scores, with higher scores indicating better knowledge. Cronbach’s alpha in this study was 0.78.

Data processing and analysis were conducted using SPSS 25.0, which included item analysis, reliability analysis, and exploratory factor analysis (EFA) for the UMHL-A. Additionally, confirmatory factor analysis (CFA) was performed using AMOS 24.0.

To explore the factor structure of the UMHL-A, the total sample was randomly divided into two equal parts: one half (n = 604) was used for EFA, and the other half (n = 604) for CFA. For EFA, Kaiser-Meyer-Olkin (KMO) and Bartlett’s test were conducted. Then, principal component analysis, employing the varimax rotation method, was selected to extract common factors and to ascertain the scale’s factor structure. CFA was conducted to validate the factors identified through EFA. The maximum likelihood estimation method was used to assess the fit of the measurement model [25], with indicators including the root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker–Lewis index (TLI) used to evaluate model fit [26].

Reliability analysis involved Cronbach’s alpha, split-half reliability, and test-retest reliability. Subsequently, 323 students from the initial sample were retested two weeks later, and the data were analyzed for test-retest reliability.

Construct validity of the sample data was assessed by analyzing the Pearson correlation between the scores of UMHL-A subscales and each dimension’s scores, along with other scale qualities such as help-seeking attitude and stigma. This analysis provided insights into the construct validity of the scale. A threshold p-value of 0.05 was used to determine statistical significance.

Since the scale consists of two subscales with differing scoring methods, exploratory factor analysis was conducted separately on the attitude subscale and the knowledge subscale. The exploratory factor analysis of 604 samples revealed that the KMO measure of sampling adequacy for the attitude subscale was 0.83, and the Bartlett’s test of sphericity showed a significant difference (χ² = 1484.82, p < 0.001). Utilizing principal component analysis with a maximum variance rotation method, two common factors were identified: help-seeking attitude and stigmatization attitude. The cumulative variance explained by these factors was 60.01%, aligning with the original scale structure (Table 2). For the knowledge subscale, the KMO value was 0.84, with Bartlett’s test yielding (χ² = 1315.98, p < 0.001). Exploratory factor analysis extracted two factors—mental health knowledge and mental illness knowledge—with a cumulative variance contribution of 51.28%, also consistent with the original scale structure (Table 3).

Given the distinct scoring methods of the two subscales, confirmatory factor analysis was performed separately on the attitude subscale and the knowledge subscale. Following the exploratory factor analysis confirmatory factor analysis, confirmatory factor analysis was conducted on the subscale items using the other half of the sample data, according to the determined two-factor structure, and maximum likelihood estimation was employed to evaluate the fit of the measurement model. The results of the CFA indicated that the two-factor model had an acceptable model fit. The standard load range for each item on the attitude subscale varied from 0.52 to 0.80 (Fig. 1), while the standardized load range for each item on the knowledge subscale ranged from 0.39 to 0.76 (Fig. 2). The fit indices, presented in Table 4, indicate that the model satisfactorily met the evaluative criteria and demonstrated a good fit.

Critical ratio method and item-total correlation were employed to analyze the items. The sample was sorted by subscale total scores, with the top 27% and bottom 27% forming the “high” and “low” subgroups, respectively. An independent sample t-test was performed for each item score on the scale, comparing the high and low subgroups. The analyses revealed significant differences between high and low subgroups for all items; the specific data are presented in Table 5. The correlation coefficients between the scores of each item and the subscales total score ranged from 0.55 to 0.74 (all p < 0.01), as shown in Table 5. This confirms the scale’s effective differentiation and discrimination capabilities.

Reliability analysis (Table 6) reported Cronbach’s alpha values of 0.82 and 0.79 for the attitude and knowledge subscales, respectively. Cronbach’s alpha for each dimension varied between 0.71 and 0.80, indicating the scale’s high reliability. Split-half reliability of the two subscales is 0.87 and 0.82, respectively. And Split-half reliability of each dimension is between 0.70 and 0.84. Test-retest reliability for the subscales was found to be 0.76 and 0.80, respectively, over a two-week period, with dimension-specific reliability ranging from 0.69 to 0.84. All correlation coefficients were significant (p < 0.01), underscoring the scale’s consistent reliability.

The construct validity of the scale was confirmed through significant positive correlations between the help-seeking dimension, attitude subscale, and the scale for seeking professional psychological help (r = 0.51, 0.43, p < 0.01), and negative correlations between the stigmatization dimension, attitude subscale, and the self-stigma scale for seeking psychological help (r = 0.45, 0.51, p < 0.05). Additionally, the mental health knowledge dimension, mental illness knowledge dimension, and knowledge subscale were all positively correlated with the mental health knowledge questionnaire (r = 0.59, 0.55, 0.65, p < 0.01), further validating the scale’s construct validity.

First and foremost, this study’s geographical scope was limited to a single region, with all participants hailing from Jinan City, Shandong Province (China), resulting in relatively high sample homogeneity. To enhance the applicability of our findings, further research involving more nationally representative samples and larger population sizes is recommended. Additionally, the use of a cross-sectional study design in this research does not allow for the examination of the dynamic changes in adolescent mental health literacy over time. Consequently, there is a pressing need for longitudinal studies that delve into the evolution of mental health literacy among adolescents, including its related factors, and potential impacts at various stages of development. Lastly, the reliance on self-reported data in this study introduces certain limitations. Future research should employ a more diverse methodology, incorporating evaluations from family members, teachers, and other relevant individuals to obtain a more comprehensive understanding of the subject matter.