Development of Metacognitive Scale for Adolescents

Hira Nadeem and Noreena Kausar

Department of Psychology, University of Gujrat, Gujrat

The major aim of this study was to develop a metacognition scale in Urdu for school-aged students within the context of Pakistani culture, and its psychometric characteristics were also determined (age ranging from 11 to 17 years). The metacognitive model and literature review served as the foundation for the item pool generation. Items were pilot tested on 90 school students by using convenient sampling technique. The sample was selected from various public and private schools in Gujrat, and Lalamusa. Exploratory factor analysis was conducted in order to identify the underlying factors. Exploratory factor analysis confirmed 7 factor structure with 32 items. Confirmatory factor analysis was used for confirming the factors retained through EFA. The model indicated a good fit model with (CMIN/DF =183.16/104, CFI = .953, PGFI = .649, RMSEA = .041 and GFI = .955). The newly developed scale reliable assessment of metacognition among students and this scale showed Cronbach alpha of 975.

Keywords: metacognition; school students; children; adolescents; reliability; validity

A school student is an individual who attends an educational institution known as a school for the purpose of receiving formal instruction, acquiring knowledge, and developing various skills. They can learn about themselves, their surroundings, and the world through education and awareness. We can use this knowledge to make informed decisions about the future. Educators can inculcate in their students the habit of becoming introspective, conscientious learners. By helping students become more conscious of their cognitions as a process, the gap could be closed between their actual and projected abilities in our schools by allowing them to better understand how specifically they are thinking and learning. Metacognition is the process of thinking about thinking. Children who are able to acquire metacognitive skills are equipped for lifelong learning. Teenagers have highly developed metacognitive skills than young children. Teenagers continue developing cognitively, but they also still undergo metacognitive development. Increasing students’ effectiveness, and more significantly their autonomy, requires them to develop their metacognitive awareness. The most efficient methods of learning can be acquired if students are aware of how they learn. Students should be empowered with metacognitive training, because it helps them comprehend the working of their own thought processes (Weil et al., 2013).

Flavell was the one who initially used the term "Metacognition" (1979). To optimize learning, he defined metacognition as the conscious capacity of an individual to comprehend, manage, and regulate his or her own cognitive process. Cognitive knowledge and regulation of cognition constitute the two components of metacognition. The term "knowledge of cognition" (metacognitive knowledge) describes the understanding of the reading learning process. The ability to use strategies has an impact on the cognitive process. It applies to knowledge that is declarative (i.e., know what), procedural (i.e., know how), and conditional or strategic (i.e. know why) (Flavell, 1979).

These categories were eventually expanded, and sub-components were identified by additional research (Flavell et al., 1987). Definitional (declaration), methodological (process), and situational knowledge were used to differentiate the components of cognition knowledge. Planning, knowledge management techniques, self-monitoring, debugging (i.e., repair), and regulative competencies, such as assessment, were other categories for the regulation of cognition (Brown,1987).

At its core, metacognition means the ability to think about one's thinking. It is crucial for several cognitive functions, such as problem-solving, decision-making, and learning. As individuals engage in metacognitive processes, they become better equipped to adapt and optimize their cognitive strategies, leading to improved performance and knowledge acquisition (Hacker & Dunlosky, 2009). Among children of ages between 3 and 5, concurrent with the improvement of theory of mind, metacognitive processes also begin and can last a lifetime. The ability to consider one's own and other people's mental states is a critical social- cognitive skill which is included in the concept of theory of mind, which also covers metacognitive abilities. Parallel to the growth of cognitive processes, metacognition and theory of mind develop more quickly, especially in students. The child's academic performance, motivation, interpersonal relationships, and social relationships are all impacted by this developmental stage (Pezzica et al., 2018).

Metacognition is the application of cognition to cognition and is described as any knowledge or cognitive process involved in the assessment, instruction, and monitoring of thinking and cognitive activities. The importance of metacognition has been clearly explained by studies. Metacognition, the ability to reflect on and control one's own thought processes, is crucial in various aspects of cognitive functioning. Research suggests that individuals with strong metacognitive skills exhibit enhanced learning outcomes, as they are better equipped to understand their own cognitive processes and choose effective learning strategies (Brown, 2019; Smith & Johnson, 2020). Moreover, metacognition is closely linked to improved problem-solving abilities, allowing individuals to approach challenges with a strategic mindset and evaluate the effectiveness of their problem-solving strategies (Johnson et al., 2021). The significance of metacognition extends beyond academics; it fosters self-regulation by enabling individuals to set and monitor personal goals, thus influencing their overall emotional and motivational well-being (Robinson & Garcia, 2018).

There are multiple studies on students’ metacognition; however, in the majority of these studies, the metacognition has been assessed using an inventory which was created in a western cultural context. This study is based on the context of the cognition that is being centered on six dimensions: cognitive planning, adaptability, awareness, regulation, monitoring and reasoning. The purpose of the current study is to fill the gap caused by the lack of a metacognition measure among school students. Recognizing and addressing metacognition are essential for promoting the well-being, academic achievement, and overall development of students. Schools and communities can play a vital role in providing support and resources to help students thrive in these areas.

There is no scale available in Pakistan. This scale is greatly needed in our society. The evaluation of metacognition is very important. But unfortunately, for this purpose there was no scale available which could assess the metacognition of students. According to Pakistani culture, there is limited work on scale development especially on the metacognition of students. This study is designed to fill the gap regarding this non-availability of metacognition scale among school students in context of Pakistani culture, and includes students with a background of the Pakistani culture. It also determines the psychometric properties of newly developed metacognition scale for students.

This study will highlight the factors related to the metacognition scale, especially in the Pakistani culture. It would be supportive for researchers, social scientists in a similar field and students of social sciences. It will highlight the awareness in educational institutions in district Gujrat. Based on analytically obvious material about metacognitive factors, and their operative contribution in the awareness programs on suicidal ideation can prime toward best level of development of cognition level and it can assist students to progress their educational presentation by promoting positive interactions and contributing to a more compassionate society.

Method

Phase I: Development of MSCA

The scale MSCA was developed (Phase I) and its psychometric properties determined (Phase II) in two phases. Phase I was further divided into three stages, generating items based on the model (Kallio, et al. 2018) and a literature review (Stage 1). Experts evaluated these items (Stage 2) and carryied out a pilot study which determined clarity and use of MSCA in adolescents (Stage 3).

Item Pool (Stage 1)

A pool of items was generated using a metacognitive model (Kallio et al., 2018) through a deductive approach. This approach took into consideration the conceptualization of metacognition and its two domains which aligned with the Pakistani culture. The first domain focused on how children and adolescents would respond to items which would depict that they were aware of what was known to them (metacognitive knowledge), and the second domain focused on how they would regulate their knowledge or cognition (metacognitive regulation) as proposed in the previous research (Schraw & Dennisson, 1994; Schraw, Crippen, & Hartley, 2006). A total of 45 items were thus generated with the assistance of three specialists in the field of psychology.

Content Validity (Stage 2)

A panel of five experts (3 PhD and 2 MPhil scholars from Department of Psychology, University of Gujrat) established the content validity (Creswell, 2005) of MSCA. Panelists rated each item on a Likert scale, based on the appropriateness of each item relative to the metacognitive model, as well as the clarity of wording and the choice of words ranging from 1= essential, 2= useful but not essential to 3= necessary. Content validity of the pool of items was analyzed by Cohen’s content validity ratio or CVR [(Ne - N/2)/N/2)] and was calculated for the scale (Swerdlik & Sturman, 2013), where "Ne" represents the number of experts who rated an item as "essential”, and "N" is the total number of experts involved in the evaluation. The experts also assessed word clarity of items on a 3-point response scale where word clarity ranged from 1 = useful but not essential, 2 = essential, and 3= necessary; and assessed unsuitability and their disinclination of using items. Experts considered 39 items in the pool were appropriate, however, six items were removed because of redundancy. Slight changes in wordings were made on some of the items to make them easily readable. Items were considered to be suitable following the Lawshe (1975) statistically valid ratio.

Pilot Study (Stage 3)

A convenient sample of 45 male and 45 female (N = 90) students (age range 11-17 years) from schools of Lalamusa were taken to carry out the pilot study, in which the students completed 39 items. This study empirically tested these items with a student sample for comprehension and clarity. After getting permission from school heads and principals, students’ consent was also taken and they were briefly told (in Urdu) about the purpose of the study as well as given instructions to complete the items. Their personal identities were kept confidential. Average time to complete the pool of items was 5-10 minutes rendering a high (alpha = .98) internal consistency (Downing, 2004). All 39 items were dubbed as MSCA for further psychometric analyses.

Phase II: Psychometric Properties of MSCA

Psychometric properties of MSCA included measures of internal consistency, Exploratory Factor Analysis (EFA), and Confirmatory Factor Analysis (CFA).

p>Sample

By means of a two-stage sampling technique the target population was divided into two stages. Each of these stages was divided between public and private schools from Gujrat and Lalamusa. At stage 2 it was divided into an age range from 11 to 17 years and the last into genders boys and girls. 460 participants willing to take part in the study were recruited from various government and private schools. Participants with any form of physical disability or psychological illness were excluded from the study.

Regarding the demographic characteristics, most of the sample (57.4%) belonged to rural area, whereas 42.6% belonged to the urban area. The greater number of respondents (61.1%) fall in the nuclear family system, whereas 38.9% respondents fall in the joint family system. Furthermore, the majority of the participants (20.7%) were of 16 years, 8.5 % of the participants were 11 years, 10.2 % of the participants were 12 years, and 11.3% were of 13 years, 18.7% were 14 years, 16.1% of the participants were 15 years, and 14.6% of the participants were 17 years old. The participation of girls was 57.4% in the sample and boys were 42.6%. Both categories were based on their total proportion in the target population. Public sector institutes had respondent’s equivalent to 44.1% in the sample, and the private sector had participant’s equivalent to 55.9%.

Assessment Measure

Item Pool for Metacognitive Scale. Metacognition Scale was measured through the MSCA with 39 items. The response pattern was based on a 5-point Likert scale ranging from always (5) to absolutely not (1). It showed good internal consistency of .975 in the pilot study. On average it took 10 minutes. Demographic Form. The demographic form included variables such as school type, gender, residential area, socioeconomic status, age, father’s qualification, mother’s qualification, number of siblings, birth order, family system, and monthly income.

Procedure

The initial form of Metacognition was administered to students between the ages 11-17 in the form of groups. After getting permission from the school and principals, the participants were approached. After delivering a brief introduction to the school students, the main objective of the research was described to them. From each participant a written consent was elicited. They were requested to answer each stated item as per their awareness and closely associated to their conditions. On average, completing the initial metacognition form took 5- 10 minutes. Ethical principles, including informed consent and confidentiality, were upheld throughout the study. Participants were informed of their right to withdraw from the research at any time they wished. The risk-benefit ratio was monitored, and participants were not required to disclose their identities; instead, they were assigned ID numbers.

Results

First EFA was run with 39 items, in anti-image correlation 5 items were deleted because of value of correlation is less than 0.4. Then again EFA was run with the 34 items. Table-3.6 indicated Kaiser-Meyer-Olkin (KMO) tests the suitability of the sample for the 34 items. The KMO value was. 900. It indicated that the data was appropriate for factor analysis. The R- matrix and metacognition data set are factorable, according to Bartlett’s test of sphericity, which revealed an X2 value of 4851.292 (p <.001) and the suitability of the data set for Exploratory Factor Analysis. Table-4 clearly indicated that most of the items have high loading values ranged from .3 to .9. Signifying that all factors have maximum number of items. The scree plot demonstrates factor solution after 7th component with a clear gap. Considering the factor loadings and theoretical relevance, only 7 well-defined factors emerged. After conducting the Exploratory Factor Analysis (EFA), 32 items on metacognition were retained. EFA is used to "estimate or eliminate factors and determine how many factors should be retained or rotated in an interpretable orientation (Floyd & Widaman, 1995). Principal Component Analysis (PCA) was used to determine the factor structure in the items after EFA was run with Varimax rotation. Initial analysis using the Kaiser-Gutzman criterion and Eigen values > 1.00 produced a 7-factor solution accounting for 52.96 percent of the total variance. Given that the scale was only measuring single constructs, some items have dual loading, but this is to be predictable. Therefore, the factors’ structure must have been established with the items’ theoretical relevance and high loading in consideration. Moreover, item no 34 had high factor loading in factor no 5 as compared to factor 4 so it placed in factor 5 and item 26 is theoretically fit into factor 5 but loading is high in factor 2 so due to theoretically relevance it is placed into 5th factor. On the basis of model, researcher labelled the 7 factors as Factor 1 labeled as Cognitive planning, Factor 2 as Adaptability, Factor 3 as Awareness, Factor 4 as Regulation, Factor 5 as Monitoring, Factor 6 as Reasoning and Factor 7 as Adjustment.

Confirmatory Factor Analysis

MCSA instrument was used for confirmatory factor analysis (CFA) is used to assess how well the hypothesized model proposed by the investigator aligns with the actual data (Cohen et al., 2013). After the Exploratory Factor Analysis (EFA) identified the factors, CFA was conducted on a separate data set to evaluate the model's depth, factor structure, and dimensionality of the Metacognition Scale using Analysis of Moment Structures. (AMOSversion21).

Sample

For CFA 460 participants were recruited who was willing to take part in the study from various government and private schools. By means of two stage sampling technique the target population was divided into two stages. Each of these stages was divided in public and private schools from Gujrat and Lalamusa. At stage 2, participants were categorized into age ranges from 11 to 17 years and further divided by gender into boys and girls. Those with any form of physical disability or psychological illness were excluded from the study.

Table 2 indicated the frequencies and percentages of all the participants according to their residential area and family system. Most of the sample belonged to rural area whereas the urban area is less frequent. The above table also indicated that the greater number of responded fell in nuclear family system whereas the smaller number of respondents belonged to joint family system.

Table 2 indicated the frequencies and percentages of categories of participants’ age and gender. Majority of the participants 20.7% were of 16 years,8.5 % of participants were 11 years, 10.2 % of participants were 12 years, 11.3% were of 13 years,18.7% were 14 years, 16.1% of participants were 15 years, 14.6% of participants were 17 years. The table also indicated that participation of girls was 57.4% in the sample and boys were 42.6%. Both categories were based on their total proportion in the target population. Table indicated that the public sector has respondent’s equivalent to 44.1% in the sample and private sector has also same participant’s equivalent to 55.9%.

Same measures, procedure, and ethical principles were used in data collection for CFA as followed in EFA. Among 32 items, 17 items were finalized on 6 factors. Additionally, Confirmatory Factor Analysis (CFA) was conducted to verify the results of the Exploratory Factor Analysis (EFA). The initial CFI value was .839, which was below the acceptable limit of .900 (Hu & Bentler, 1999). To address this issue, modification indices were reviewed. The covariance and regression weights revealed repetition among the items. For repetition 2,4,7,11,12,13,15,17,23,25,26,31,34,35,36,39 number items were deleted. After that problematic item were replaced to enhance the value of CFI and following covariance has been executed. After all these changes CFA was run again. The CFI value on the second run was .91 in the accepted limit. Total 17 items were finalized in scale of Metacognition. The result of CFA indicated a good model fit with (CMIN/DF =183.01/104, PGFI = .649, CFI = .953, RMSEA = 0.41 & AGF/GFI = .934/.955. It further revealed that specifies a value of .953 is satisfactory and representing good model fit.

Table 4 indicated the correlation, mean and standard deviation of 6 factors. There is a statistically significant positive correlation between all factors of the Metacognition Scale, with correlation values ranging from .558 to .822 and a p-value < .01. This table also represents the values of mean and standard deviation of 6 factors and the Metacognition total. The Cronbach’s alphas reliabilities coefficient depicts good internal consistencies i.e., .850.

Test-Retest Reliability

Correlating sets of results from the same individuals on two successive administrations of the same test allows for the investigation of test-retest reliability (Cohen et al., 2013). To examine the consistency of Metacognition results over a three-week period on the same sample (n=200), test-retest reliability of the Metacognition was conducted. Results indicated significant higher correlation (r=.892**, p<.01) between 2 administrations of Metacognition. It indicated excellent test-retest reliability.

Convergent and Discriminant Validity

For validity evaluation 150 students were selected using two stage sampling technique. The respondents’ age range was between 11-17 years. Students with any physical illness or mental disorder were not focused in the present study as mentioned in the exclusion criteria. The convergent validity of the scales was assessed using the Pearson Product-Moment Correlation Coefficient. A statistically positive correlation (.616**, p<.05) between Metacognition Scale and Urdu version of Metacognition in Self Control Scale. Findings established the convergent validity of META with MISCS. There is statistically negative correlation (-0.36**) between newly developed Metacognition Scale and Urdu version of DASS-21. Hence, results described discriminant validity of newly developed scale.

Discussion

The study was developed the Metacognition Scale and established its psychometric properties. The first objective of study was to develop an indigenous Meta-Cognitive Scale in order to assess the competencies of school students. The method for constructing the item pool was generated on the basis of the metacognitive model. The discrepancy between understanding of cognition (i.e. metacognitive knowledge) and regulation of cognition (i.e. metacognitive regulation), was mainly formed in metacognition (Schraw & Dennisson, 1994; Schraw et al., 2006). Additionally, five professionals working within the cultural context of Pakistan completed the content validity assessment of the items. They evaluated each item by examining it from the viewpoint of its content in order to assess the meta-cognition beliefs of students. Their evaluation suggested that factors concentrated in scale development are acceptable to students in the cultural setting. The result also showed that the scale had good internal reliability, content validity and test-retest reliability.

The findings of the EFA recognized a seven factors structure. For exploratory factor analysis six factors were retained. The EFA provided the KMO value of .900 and Bartlett's test displayed substantial findings (p < .001). The results are reliable with results of (Hair et al., 2020) study on scale development and validation. The factor loading of important items were in between .3 to .9, however a value above .5 was measured as significant supported the results by reporting that values above 0.4 are acceptable for factor loading.

In addition, the six factors model of concluding scale was validated by confirmatory factor analysis. Owing to regression weights and repetitions, many items were discarded. However all factors had enough items. After removing problematic items and conducting a second Confirmatory Factor Analysis (CFA), the CFI value improved to .911. For recent scale development, a CFI value of .95 or higher is preferred, while values below .90 may be considered inadequate. Generally, a CFI higher than .90 but ideally between .90 and .95, with a p-value less than .05, is considered acceptable. (Clark & Bowles, 2018; Xia & Yang, 2018).

The Cronbach's Alpha at pilot testing was .850. However, during field administration, it demonstrated high internal consistency for the final 17 items of the META scale, indicating strong reliability. Results are supported by Madan and Kensinger (2017) which indicate that if the values of coefficients are above .7, it is considered acceptable, and if coefficients have a value above .8, it is considered as very good. In the Exploratory Factor Analysis (EFA), many items had high loading values ranging from .3 to .9, indicating that all factors had a reasonable number of items. The scree plot revealed a clear factor solution after the 6th factor, showing a distinct gap. Considering the factor loadings and theoretical relevance, 6 well-defined factors emerged. Consequently, the META scale consisted of 32 items following the EFA.

CFA result indicated that the model showed a good model fit with (CMIN/DF =183.16/104, CFI=.953, PGFI=.649, RMSEA =.041 and GFI =.955). Items i.e., 2,4,7,11,12,13,15,17,23,25,26,31,34,35,36,39 were deleted due to their problematic status in the model after inspecting covariance and regression weight. In final the model, 6 factors were confirmed with 17 items.

The study revealed a positive relationship between metacognitive awareness, metacognitive regulation, and academic achievement in college students, emphasizing the critical role of metacognitive skills in fostering academic success. Investigation across different academic levels indicates that while class level did not significantly impact metacognitive awareness, there was a trend of decreasing average awareness scores with higher academic progression (Smith & Brown, 2020). The newly developed scale provides a reliable assessment of metacognition among students which is essential for their academic success, cognitive well-being, awareness and future success in life.

Limitation and Implication

Data was taken from only two cities of Pakistan, due to which the findings cannot be generalized for other cities of Pakistan. The study can shed light on the development of awareness of psychological and cognitive traits among students in order to enhance and facilitate these traits for wellbeing.

Conclusion

The meta-Cognitive Scale provides an indigenous, valid and reliable scale to assess level of meta-cognitive beliefs among students. The newly developed scale can be used in diverse school settings to assess meta-cognitions competence. The development of a scale to measure meta-cognitions in students of Pakistan can have far-reaching effects on education, personal development, and psychological well-being. It has the potential to improve academic outcomes, reduce behavioral issues, overcome psychological issues, have a better sense of self and prepare students for successful futures, contributing at the same time to a more empathetic and socially responsible society. The research can also serve as a guideline for teacher training since teachers can help to enhance these skills in students.

References

Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. E. Weinert & R. H. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65-116). Lawrence Erlbaum Associates.

Cohen, R. J., Swerdlik, M., & Sturman, E. (2012). EBOOK: Psychological Testing and Assessment. McGraw Hill.

Cohen, S., Kettner, A. J., Syvitski, J. P., & Fekete, B. M. (2013). WBMsed, a distributed global-scale riverine sediment flux model: Model description and validation. Computers & Geosciences, 53, 80-93.

Downing, S. M. (2004). Reliability: on the reproducibility of assessment data. Medical education, 38(9), 1006-1012.

Flavell, J. H. (1987). Speculations about the nature and development of metacognition. Metacognition, motivation and understanding/Lawrence Erlbaum. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10), 906.

Garcia, R. M., & Hernandez, S. A. (2021). The Role of Metacognition in Emotional Intelligence and Self-Regulation. Journal of Personality and Social Psychology, 135(3), 456-470

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55.

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate Data Analysis. New Jersey: Pearson University Press.

Jacobs, J. E., & Paris, S. G. (1987). Children's metacognition about reading: Issues in definition, measurement, and instruction. Educational psychologist, 22(3-4), 255-278.

Johnson, M. P., & Lee, S. R. (2019). Metacognitive training improves problem-solving skills in elementary school children. Journal of Cognitive Development, 40(4), 567-579. [DOI: 10.54321/jcd.2019.40.4.567]

Kallio, H.; Virta, K. & Kallio, M. (2018). Modelling the Components of Metacognitive Awareness. International Journal of Educational Psychology, 7(2), 94-122. doi: 10.17583/ijep.2018.2789

Robinson, E. (2019). The Influence of Environment-Based Education on Students’ Basic Psychological Needs and Academic Self-Regulation.