A laboratory study on the reliability estimations of the mini-CEX
Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent...
Guardado en:
| Publicado: |
2013
|
|---|---|
| Materias: | |
| Acceso en línea: | https://bibliotecadigital.exactas.uba.ar/collection/paper/document/paper_13824996_v18_n1_p5_AlvesdeLima http://hdl.handle.net/20.500.12110/paper_13824996_v18_n1_p5_AlvesdeLima |
| Aporte de: |
| id |
paper:paper_13824996_v18_n1_p5_AlvesdeLima |
|---|---|
| record_format |
dspace |
| spelling |
paper:paper_13824996_v18_n1_p5_AlvesdeLima2023-06-08T16:12:27Z A laboratory study on the reliability estimations of the mini-CEX Full crossed design Laboratory study Mini-CEX Post-graduate training Reliability Residents Argentina cardiology clinical competence education female human male medical education medical staff procedures reproducibility standards teaching videorecording Argentina Cardiology Clinical Competence Education, Medical, Graduate Educational Measurement Female Humans Male Medical Staff, Hospital Patient Simulation Reproducibility of Results Videotape Recording Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent. This is difficult to achieve. Furthermore, the variance caused by the case/patient or by assessor is completely confounded. We have no idea how much each of these factors contribute to the noise in the measurement. The aim of this study was to use a controlled setup that overcomes these difficulties and to estimate the reproducibility of the mini-CEX. Three encounters were videotaped from 21 residents. The patients were the same for all residents. Each encounter was assessed by 3 assessors who assessed all encounters for all residents. This delivered a fully crossed (all random) two-facet generalizability design. A quarter of the total variance was associated with universe score variance (28%). The largest source of variance was the general error term (34%) followed by the main effect of assessors (18%). Generalizability coefficients indicated that an approximate sample of 9 encounters was needed assuming a single different assessor per encounter and assuming different cases per encounter (the usual situation in real practice), 4 encounters when 2 raters were used and 3 encounters when 3 raters are used. Unexplained general error and the leniency/stringency of assessors are the major causes for unreliability in mini-CEX. To optimize reliability rater training might have an effect. © 2011 The Author(s). 2013 https://bibliotecadigital.exactas.uba.ar/collection/paper/document/paper_13824996_v18_n1_p5_AlvesdeLima http://hdl.handle.net/20.500.12110/paper_13824996_v18_n1_p5_AlvesdeLima |
| institution |
Universidad de Buenos Aires |
| institution_str |
I-28 |
| repository_str |
R-134 |
| collection |
Biblioteca Digital - Facultad de Ciencias Exactas y Naturales (UBA) |
| topic |
Full crossed design Laboratory study Mini-CEX Post-graduate training Reliability Residents Argentina cardiology clinical competence education female human male medical education medical staff procedures reproducibility standards teaching videorecording Argentina Cardiology Clinical Competence Education, Medical, Graduate Educational Measurement Female Humans Male Medical Staff, Hospital Patient Simulation Reproducibility of Results Videotape Recording |
| spellingShingle |
Full crossed design Laboratory study Mini-CEX Post-graduate training Reliability Residents Argentina cardiology clinical competence education female human male medical education medical staff procedures reproducibility standards teaching videorecording Argentina Cardiology Clinical Competence Education, Medical, Graduate Educational Measurement Female Humans Male Medical Staff, Hospital Patient Simulation Reproducibility of Results Videotape Recording A laboratory study on the reliability estimations of the mini-CEX |
| topic_facet |
Full crossed design Laboratory study Mini-CEX Post-graduate training Reliability Residents Argentina cardiology clinical competence education female human male medical education medical staff procedures reproducibility standards teaching videorecording Argentina Cardiology Clinical Competence Education, Medical, Graduate Educational Measurement Female Humans Male Medical Staff, Hospital Patient Simulation Reproducibility of Results Videotape Recording |
| description |
Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent. This is difficult to achieve. Furthermore, the variance caused by the case/patient or by assessor is completely confounded. We have no idea how much each of these factors contribute to the noise in the measurement. The aim of this study was to use a controlled setup that overcomes these difficulties and to estimate the reproducibility of the mini-CEX. Three encounters were videotaped from 21 residents. The patients were the same for all residents. Each encounter was assessed by 3 assessors who assessed all encounters for all residents. This delivered a fully crossed (all random) two-facet generalizability design. A quarter of the total variance was associated with universe score variance (28%). The largest source of variance was the general error term (34%) followed by the main effect of assessors (18%). Generalizability coefficients indicated that an approximate sample of 9 encounters was needed assuming a single different assessor per encounter and assuming different cases per encounter (the usual situation in real practice), 4 encounters when 2 raters were used and 3 encounters when 3 raters are used. Unexplained general error and the leniency/stringency of assessors are the major causes for unreliability in mini-CEX. To optimize reliability rater training might have an effect. © 2011 The Author(s). |
| title |
A laboratory study on the reliability estimations of the mini-CEX |
| title_short |
A laboratory study on the reliability estimations of the mini-CEX |
| title_full |
A laboratory study on the reliability estimations of the mini-CEX |
| title_fullStr |
A laboratory study on the reliability estimations of the mini-CEX |
| title_full_unstemmed |
A laboratory study on the reliability estimations of the mini-CEX |
| title_sort |
laboratory study on the reliability estimations of the mini-cex |
| publishDate |
2013 |
| url |
https://bibliotecadigital.exactas.uba.ar/collection/paper/document/paper_13824996_v18_n1_p5_AlvesdeLima http://hdl.handle.net/20.500.12110/paper_13824996_v18_n1_p5_AlvesdeLima |
| _version_ |
1768543718815563776 |