Effect Size Reporting and Interpreting Practices in Published Higher Education Journal Articles (open access)

Effect Size Reporting and Interpreting Practices in Published Higher Education Journal Articles

Data-driven decision making is an integral part of higher education and it needs to be rooted in strong methodological and statistical practices. Key practices include the use and interpretation of effect sizes as well as a correct understanding of null hypothesis significance testing (NHST). Therefore, effect size reporting and interpreting practices in higher education journal articles represent an important area of inquiry. This study examined effect size reporting and interpretation practices of published quantitative studies in three core higher education journals: Journal of Higher Education, Review of Higher Education, and Research in Higher Education. The review covered a three-year publication period between 2013 and 2015. Over the three-year span, a total of 249 articles were published by the three journals. The number of articles published across the three years did not vary appreciably. The majority of studies employed quantitative methods (71.1%), about a quarter of them used qualitative methods (25.7%), and the remaining 3.2% used mixed methods. Seventy-three studies were removed from further analysis because they did not feature any quantitative analyses. The remaining 176 quantitative articles represented the sample pool. Overall, 52.8% of the 176 studies in the final analysis reported effect size measures as part of their major …
Date: August 2016
Creator: Stafford, Mehary T.
System: The UNT Digital Library
A Multilevel Multitrait-Multimethod Analysis of the Child Behavior Checklist (open access)

A Multilevel Multitrait-Multimethod Analysis of the Child Behavior Checklist

Behavioral and emotional problems (BEPs) are known to affect children's ability to shape and maintain effective social relationships. BEPs are typically categorized into two main factors: internalizing and externalizing behaviors. Internalizing behaviors represent introverted problems, directed inwardly to the individual. While externalizing behavior patterns represent behaviors that are directed outwardly. Behaviors, emotions and thoughts are experienced by all people but on a continuum rather than in terms of absence versus presence of the behavior. The child behavior checklist (CBCL) is used to measure BEPs. The system of CBCL (parent form) measures also includes a teacher rating form and a youth self-report. Using 62 teachers and 311 students, the present study assessed convergent and discriminant validity using a correlated trait, correlated method minus one [CT-C(M-1)] model. The results showed low to moderate teacher-student agreement on the traits. To extend the theoretical structure of the teacher and self-report forms, the present study assessed the nested structure of the data using a multilevel model. Results revealed the nested structure of the data should not be ignored.
Date: August 2016
Creator: Powell, Marvin
System: The UNT Digital Library
Stereotypical Science: Exploring High School Occupational Preferences for Science by Sex, Personality, and Cognitive Ability (open access)

Stereotypical Science: Exploring High School Occupational Preferences for Science by Sex, Personality, and Cognitive Ability

Circumscription and Compromise theory suggests self-concept and sex stereotype explain occupational preferences, including preferences for science, technology, engineering and mathematics (STEM). Support exists for sex differences between males and females in both science degrees and science careers. The main thrust of observed sex differences in science lies in the development of occupational interest, as it has been suggested females are encouraged away from science due to stereotypes and social pressure. The present study evaluates high school juniors and seniors (n = 295) to explore their preference for science as indicated by science motivation, attitude, academic experience, and interest. Latent Profile Analysis was used to model profiles of preferences for science with a person-centered approach. Then, the impact of self-concept variables was explored and four profiles of science interest were identified. Sex differences were identified based on science interest, but were not always in favor of males. Covariate analysis indicates vocabulary ability and personality as significantly different for students in the high science interest profile. Implications of these results and future research directions are discussed.
Date: May 2016
Creator: Ferguson, Sarah Lynn
System: The UNT Digital Library
Comparing Three Effect Sizes for Latent Class Analysis (open access)

Comparing Three Effect Sizes for Latent Class Analysis

Traditional latent class analysis (LCA) considers entropy R2 as the only measure of effect size. However, entropy may not always be reliable, a low boundary is not agreed upon, and good separation is limited to values of greater than .80. As applications of LCA grow in popularity, it is imperative to use additional sources to quantify LCA classification accuracy. Greater classification accuracy helps to ensure that the profile of the latent classes reflect the profile of the true underlying subgroups. This Monte Carlo study compared the quantification of classification accuracy and confidence intervals of three effect sizes, entropy R2, I-index, and Cohen’s d. Study conditions included total sample size, number of dichotomous indicators, latent class membership probabilities (γ), conditional item-response probabilities (ρ), variance ratio, sample size ratio, and distribution types for a 2-class model. Overall, entropy R2 and I-index showed the best accuracy and standard error, along with the smallest confidence interval widths. Results showed that I-index only performed well for a few cases.
Date: December 2015
Creator: Granado, Elvalicia A.
System: The UNT Digital Library
Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice (open access)

Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice

Reliability generalization (RG) is a method for meta-analysis of reliability coefficients to estimate average score reliability across studies, determine variation in reliability, and identify study-level moderator variables influencing score reliability. A total of 107 peer-reviewed RG studies published from 1998 to 2013 were systematically reviewed to characterize the meta-analytic methods employed and to evaluate quality of reporting practice against standards for transparency in meta-analysis reporting. Most commonly, RG studies meta-analyzed alpha coefficients, which were synthesized using an unweighted, fixed-effects model applied to untransformed coefficients. Moderator analyses most frequently included multiple regression and bivariate correlations employing a fixed-effects model on untransformed, unweighted coefficients. Based on a unit-weighted scoring system, mean reporting quality for RG studies was statistically less than that for a comparison study of 198 meta-analyses in the organizational sciences across 42 indicators; however, means were not statistically significantly different between the two studies when evaluating reporting quality on 18 indicators deemed essential to ethical reporting practice in meta-analyses. Since its inception a wide variety of statistical methods have been applied to RG, and meta-analysis of reliability coefficients has extended to fields outside of psychological measurement, such as medicine and business. A set of guidelines for conducting and reporting RG …
Date: December 2015
Creator: Holland, David F.
System: The UNT Digital Library
Time Series Data Analysis of Single Subject Experimental Designs Using Bayesian Estimation (open access)

Time Series Data Analysis of Single Subject Experimental Designs Using Bayesian Estimation

This study presents a set of data analysis approaches for single subject designs (SSDs). The primary purpose is to establish a series of statistical models to supplement visual analysis in single subject research using Bayesian estimation. Linear modeling approach has been used to study level and trend changes. I propose an alternate approach that treats the phase change-point between the baseline and intervention conditions as an unknown parameter. Similar to some existing approaches, the models take into account changes in slopes and intercepts in the presence of serial dependency. The Bayesian procedure used to estimate the parameters and analyze the data is described. Researchers use a variety of statistical analysis methods to analyze different single subject research designs. This dissertation presents a series of statistical models to model data from various conditions: the baseline phase, A-B design, A-B-A-B design, multiple baseline design, alternating treatments design, and changing criterion design. The change-point evaluation method can provide additional confirmation of causal effect of the treatment on target behavior. Software codes are provided as supplemental materials in the appendices. The applicability for the analyses is demonstrated using five examples from the SSD literature.
Date: August 2015
Creator: Aerts, Xing Qin
System: The UNT Digital Library
Construct Validation of the Social-Emotional Character Development Scale in Belize: Measurement Invariance Through Exploratory Structural Equation Modeling (open access)

Construct Validation of the Social-Emotional Character Development Scale in Belize: Measurement Invariance Through Exploratory Structural Equation Modeling

Social-emotional learning (SEL) measures assessing social-emotional learning and character development across a broad array of constructs have been developed but lack construct validity. Determining the efficacy of educational interventions requires structurally valid measures which are generalizable across settings, gender, and time. Utilizing recent factor analytic methods, the present study extends validity literature for SEL measures by investigating the structural validity and generalizability of the Social-Emotional and Character Development Scale (SECDS) with a large sample of children from schools in Belize (n = 1877, ages 8 to13). The SECDS exhibited structural and generalizability evidence of construct validity when examined under exploratory structural equation modeling (ESEM). While a higher order confirmatory factor structure with six secondary factors provided acceptable fit, the ESEM six-factor structure provided both substantive and methodological advantages. The ESEM structural model situates the SECDS into the larger body of SEL literature while also exhibiting generalizability evidence over both gender and time.
Date: August 2014
Creator: Hinerman, Krystal M.
System: The UNT Digital Library
Criterion Validity of Common Career Interest Inventories: Relative Efficacy with High School Seniors (open access)

Criterion Validity of Common Career Interest Inventories: Relative Efficacy with High School Seniors

Professional school counselors frequently use career interest inventories as part of a comprehensive guidance program to help students create a post-secondary school plan. The present study evaluates the validity of three commonly used interest inventories, the Myers-Briggs Type Indicator, Self-Directed Search, and Strong Interest Inventory on field of study choice for graduating high school seniors (N = 616) from a large, suburban high school in Texas. Students identified their intended postsecondary field of study category, were randomly assigned using stratification to three groups, and each group completed a different inventory. Group membership was evaluated to establish covariate balance on a wide variety of indicators. Data from each group was evaluated to determine the extent to which the inventory predicted the chosen field of study, as well as Other and Undeclared categories using logistic regression models. None of the inventory models suggest that the inventory accurately predicts Other or Undeclared outcomes. For students selecting intended postsecondary fields of study, the Self Directed Search predicts such outcomes better than other measures. Professional school and career counselors should consider the SDS in addition to narrative counseling strategies to add greater precision with career decision making among clients and students.
Date: August 2014
Creator: Martin, Summer M.G.
System: The UNT Digital Library
A Structural and Psychometric Evaluation of a Situational Judgment Test: The Workplace Skills Survey (open access)

A Structural and Psychometric Evaluation of a Situational Judgment Test: The Workplace Skills Survey

Some basic but desirable employability skills are antecedents of job performance. The Workplace Skills Survey (WSS) is a 48-item situational judgment test (SJT) used to assess non-technical workplace skills for both entry-level and experienced workers. Unfortunately, the psychometric evidence for use of its scores is far from adequate. The purpose of current study was two-fold: (a) to examine the proposed structure of WSS scores using confirmatory factor analysis (CFA), and (b) to explore the WSS item functioning and performance using item response theory (IRT). A sample of 1,018 Jamaican unattached youth completed the WSS instrument as part of a longitudinal study on the efficacy of a youth development program in Jamaica. Three CFA models were tested for the construct validity of WSS scores. Parameter estimations of item difficulty, item discrimination, and examinee’s proficiency estimations were obtained with item response theory (IRT) and plotted in item characteristics curves (ICCs) and item information curves (IICs). Results showed that the WSS performed quite well as a whole and provided precise measurement especially for respondents at latent trait levels of -0.5 and +1.5. However, some modifications of some items were recommended. CFA analyses showed supportive evidence of the one-factor construct model, while the six-factor …
Date: August 2014
Creator: Wei, Min
System: The UNT Digital Library
Eastern Work Ethic: Structural Validity, Measurement Invariance, and Generational Differences (open access)

Eastern Work Ethic: Structural Validity, Measurement Invariance, and Generational Differences

This present study examined the structural validity of a Chinese version of Multidimensional Work Ethic Profile (MWEP-C), using a large sample of Chinese parents and their young adult children (N = 1047). Confirmatory factor analysis (CFA) was applied to evaluate the model fit of sample data on three competing models using two randomly split stratified subsamples. Measurement invariance for these two generational respondents was checked using differential item functioning (DIF) analysis. The results indicated that MWEP-C provided a reasonable fit for the sample data and the majority of survey items produced similar item-level responses for individuals that do not differ on the attributes of work ethic across these two generations. DIF items were detected based on advanced and successive iterations. Monte Carlo simulations were also conducted for creating threshold values and for chi-square probabilities based on 1,000 replications. After identifying the DIF items, model fit improved and generational differences and similarities in work ethic between parents and their young adult children were also identified. The results suggested that the younger Chinese generations have higher work ethic mean scores on the dimensions of work centrality and morality/ethics while they have similarities on time concept, self-reliance, delay of gratification, and hard work …
Date: May 2014
Creator: Chen, Danxia
System: The UNT Digital Library
Spatial Ability in Registered Nurses (open access)

Spatial Ability in Registered Nurses

Spatial ability is the skill associated with mental relations among objects, the process of maintaining the physical aspects of an object after mentally rotating it in space. Many studies report a strong association of spatial ability with success in various areas of health care, especially surgery, radiology and dentistry. To date, similar investigations in professional nursing could not be located. Registered nurses, employed in an acute care multi-hospital setting, were surveyed using the Shipley-2Block Pattern Test, the Group Embedded Figures Test, and a newly created test of general nursing knowledge. The sample size of 123 nurses was composed of 31 male nurses and 92 female nurses. Data was collected between May and August of 2013 and analyzed using R, version 2.15.2. The present study did not demonstrate a statistically significant effect for gender differences on two measures of spatial ability. However, Cohen’s d effect sizes for mean gender differences in the present study are consistent with prior studies. This may suggest the nursing profession is comparable with other professions where males perform higher than females on spatial ability. The present study should be considered an initial step toward evaluating the relevance of spatial ability in the performance of nursing care.
Date: May 2014
Creator: Gardner, Janet E.
System: The UNT Digital Library
Is It More Advantageous to Administer Libqual+® Lite Over Libqual+®? an Analysis of Confidence Intervals, Root Mean Square Errors, and Bias (open access)

Is It More Advantageous to Administer Libqual+® Lite Over Libqual+®? an Analysis of Confidence Intervals, Root Mean Square Errors, and Bias

The Association of Research Libraries (ARL) provides an option for librarians to administer a combination of LibQUAL+® and LibQUAL+® Lite to measure users' perceptions of library service quality. LibQUAL+® Lite is a shorter version of LibQUAL+® that uses planned missing data in its design. The present study investigates the loss of information in commonly administered proportions of LibQUAL+® and LibQUAL+® Lite when compared to administering LibQUAL+® alone. Data from previous administrations of LibQUAL+® protocol (2005, N = 525; 2007, N = 3,261; and 2009, N = 2,103) were used to create simulated datasets representing various proportions of LibQUAL+® versus LibQUAL+® Lite administration (0.2:0.8, 0.4:0.6. 0.5:0.5, 0.6:0.4, and 0.8:0.2). Statistics (i.e., means, adequacy and superiority gaps, standard deviations, Pearson product-moment correlation coefficients, and polychoric correlation coefficients) from simulated and real data were compared. Confidence intervals captured the original values. Root mean square errors and absolute and relative biases of correlations showed that accuracy in the estimates decreased with increase in percentage of planned missing data. The recommendation is to avoid using combinations with more than 20% planned missing data.
Date: August 2013
Creator: Ponce, Hector F.
System: The UNT Digital Library