On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests

Journal of Crosscultural Research

Volume: 49 issue: 5, page(s): 713-734

Article first published online: May 21, 2018; Issue published: June 1, 2018

ESEM

ESEM (Asparouhov & Muthén, 2009; Marsh, Morin, Parker, & Kaur, 2014) is especially helpful when dealing with many groups where strong levels of invariance are often unattainable, and goodness-of-fit indices are weak. ESEM is an extension of the CFA approach. According to Marsh and colleagues (2010), ESEM integrates the best aspects of the CFA/SEM and the EFA approaches. The use of SEM in the exploratory approach means that it makes use of confirmatory tests of a priori factor structures as well as associations with latent factors, and multigroup tests of full measurement invariance (Marsh et al., 2014). The advantage of the ESEM is that it is not restricted because, in a conventional CFA approach, each item only loads on one factor. Items are allowed to load on all factors in ESEM. ESEM can be used for interrelated or independent factors by modeling oblique or orthogonal factor structures (Bowden, Saklofske, van de Vijver, Sudarshan, & Eysenck, 2016). ESEM is a fairly novel procedure that has not yet been often applied in content-related research; yet, it appears promising, and we would welcome further testing of its applicability in future research.

Bayesian approximate invariance

Bayesian approximate invariance testing is another promising approach to remedy some of the problems of conventional MGCFA. Instead of constraining the parameters of loadings and/or intercepts to be exactly the same across groups, this approach allows small differences in these parameters across groups (Muthén & Asparouhov, 2012; van de Schoot et al., 2013). The underlying rationale is that absolute invariance is unattainable and slight variations may not severely hinder comparability. Hence, a valid comparison can still be achieved. In two simulation studies (Muthen & Asparouhov, 2012; van de Schoot et al., 2013) involving a two-group and a 10-group comparison, respectively, appropriate model specifications (also called priors) that admit a certain degree of flexibility were proposed. In operational terms, pairwise differences in each parameter (loadings and/or intercepts) across groups can be modeled to follow a normal distribution with a mean of zero and a very small variance (.01 or .05).

Several applications with such prior specifications have been reported in multiple group comparisons (see, for example, Bujacz, Vittersø, Huta, & Kaczmarek, 2014; Cieciuch, Davidov, Schmidt, Algesheimer, & Schwartz, 2014; Davidov et al., 2015; He & Kubacka, 2015; Zercher, Schmidt, Cieciuch, & Davidov, 2015). We know that ignoring the lack of invariance may lead to biased comparative research results (Guenole & Brown, 2014). We do not know yet in the case of approximate invariance, whether the relaxed constraints would bias the parameters and distort the findings. In a few studies, it was shown that factor scores derived from a conventional CFA and the approximate approach were very similar (Davidov et al., 2015; Zercher et al., 2015). Simulation studies that examine the impact of different priors and the consequences of demonstrating approximate invariance are in much need (van de Schoot et al., 2013).

Alignment

Alignment is a third promising approach to estimate group-specific factor means and variances without requiring full measurement invariance (Asparouhov & Muthén, 2014). In a sense, alignment can be viewed as exploratory, where it incorporates a simplicity function similar to the rotation criteria used in EFA to discover the most optimal measurement invariance pattern (i.e., the simplest model with the fewest noninvariant parameters) and to estimate the factor mean and variance parameters in each group. This approach has been tested in multigroup CFA models with maximum likelihood as well as in Bayesian estimation, and it has been extended to IRT modeling (Muthén & Asparouhov, 2014).

Given its very recent development, only a few applications are available to date. De Bondt and Van Petegem (2015) used the Bayesian approximate invariance test with alignment optimization in evaluating the psychometric quality of an overexcitability scale, in which they showed the superiority of this combined approach compared with a conventional CFA with modification indices when comparing male and female students. Desa and Carstens (2015) proposed to apply this approach in the future in large-scale assessment contexts such as the Programme for International Student Assessment (PISA) and the Teaching and Learning International Survey (TALIS), as it legitimates mean comparisons in dozens of groups without requiring full measurement invariance. Weziak-Bialowolska (2014) tested gender ideology in the World Value Survey with CFA with and without alignment. She reported different patterns of country factor means from these two methods, and suggested that comparisons of the country rankings were valid provided that a correction for noninvariance of certain factor loadings and/or intercepts is applied in the alignment framework. Similar to the Bayesian approximate invariance tests, the implications of using alignment on the validity of cross-cultural comparisons await further investigation.

In sum, various older and newer techniques are available to assess measurement invariance and to distinguish error from cultural variance. Given the importance of establishing ground of comparability for drawing conclusions about how and in which form culture influences psychological states and processes, we further investigate to what extent invariance tests have been implemented in cross-cultural research.

T32-1A: Whatnot Shelf for Psychiatric Epidemiologists

Friday, October 4, 2019

Should you rely solely on Google Scholar for your literature reviews.

Wednesday, October 2, 2019

Emerging drug trends website

Tuesday, October 1, 2019

Coevidence for systematic reviews

Monday, September 23, 2019

More on Measurement Equivalence Modeling: MGCFA

On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests

ESEM

Bayesian approximate invariance

Alignment

Thursday, September 19, 2019

Analyzing the past to prepare for the future: Writing a literature review

Tuesday, September 17, 2019

Online Probing (OP) To Complement Measurement Equivalence Modeling