Correlation of Mock Board Examination Scores During Anatomic Pathology Residency Training with Performance on the Certifying Examination
Publication: Journal of Veterinary Medical Education
Volume 47, Number 1
Abstract
Mock board exams are common in residency programs across many disciplines. However, the value of mock board results in predicting success on the actual certifying examination is largely anecdotal and undocumented. The University of Tennessee anatomic pathology residency program has a long history of giving mock board exams twice a year during the course of the 3-year diagnostic training program. The mock exams give residents a sense of the types of questions that may appear on the actual certifying examination. The resulting scores serve to help identify improvement areas to focus additional study. In addition, by providing residents the mental and physical experiences designed to mimic the test day, we hope to better prepare these trainees for optimal performance on the certifying examination. This study correlated mock board results of 16 anatomic pathology residents, from July 2006 through January 2016, with their subsequent performance on the certifying exam. The results of these biannual exams were significantly correlated (p < .001) with results for the American College of Veterinary Pathologists Certifying Examination.
Introduction
The certifying exam for veterinary specialties is a source of incredible stress for residents. In an effort to better prepare residents for the mental and physical stresses of the certifying exam, many institutions implement mock exams. However, the validity of mock board results in predicting success on the actual certifying examination has not been well established. The purpose of this study was to investigate the correlation of mock board results (deliberate practice) with performance on the American College of Veterinary Pathology (ACVP) certifying examination.
Deliberate practice theory encompasses the idea that repetition of a task and feedback fosters improved performance.1 Associated with this theory is the concept of test-enhanced learning, in which repeated retrieval practice and subsequent feedback promote retention and further learning (testing effect).2 The University of Tennessee College of Veterinary Medicine’s (UTCVM) Anatomic Pathology residency program has a long history of giving mock board exams twice a year during the 3-year diagnostic training program. To try to mimic the actual certifying exam experience as much as possible, the mock exam questions are divided into the same four sections that trainees will encounter with the certifying exam: gross pathology, microscopic pathology, veterinary pathology, and general pathology.
The broad goals of the residency are to prepare trainees for careers in veterinary anatomic pathology and for eligibility for board certification by the ACVP. Residents enter the UTCVM anatomic pathology training program in July, and the first mock exam is administered in mid to late July or August to provide a reference point for each resident. Additional mock exams are created and implemented approximately every 6 months after that to allow for identification of progression of knowledge and skills throughout residency training (Figure 1). Specifically, the mock exams help trainees formatively evaluate what they do not know to guide them in future study.2 Resident training aims to include and encourage the elements outlined by the ACVP Training Program Development Task Force.3 This training encompasses continuous, active mentorship by credentialed pathologists, including team mock exam review and individualized written feedback after each mock exam.
Until 2011, the biannual mock exams were created solely by the UTCVM anatomic pathologists and represented about a third of the volume of the certifying exam. In 2011, we began coordinating a collaborative mock board exam and using it for the July mock exam. For this collaborative exam, pathology faculty from three to five other academic institutions with pathology residency training programs contribute materials for each section of the exam. The collaborative nature of the exam is intended to create a greater diversity of questions. The exam is collated at UTCVM and then sent out to each of the contributing institutions to implement as they wish. This larger collaborative exam is a full-day experience and represents approximately half of the full certifying exam. Administration of this larger exam is intended to mimic the physical experience of the actual exam. This exam is now given each summer, including at the start of the residency program, and therefore is the first (baseline) exam the residents take. The shorter exam created solely within UTCVM is given in January or February of each subsequent year. Residents complete the training program at the end of June, after having taken six mock exams (Figure 1).
Until 2015, the entire ACVP certifying exam occurred in September, approximately 2 months after completion of the residency program, and had four sections, each of which was scored separately. Test-takers’ passing score for each section was traditionally at or around 60%, and trainees were required to eventually pass all four sections to become board certified. Beginning in 2015, however, the general pathology portion of the exam was moved to March to serve as a qualifying exam (Phase I) for the remaining sections. Residents are now able to take this qualifying general pathology exam in March of their second or third year of training.
Based on the theory of deliberate practice,1 we hypothesized that success on the mock board exams would be correlated with success on the ACVP certifying exam.
Methods
The mock exam results from 16 anatomic pathology residents completing the 36-month training program at the UTCVM from July 2006 through January 2016 were analyzed and correlated with the results of residents’ first attempt on each section of the certifying exam. Certifying exam results through September 2017 are included for comparison. The data from 2 additional residents who had completed the Phase I exam, but had not completed the residency and had not attempted Phase II were included in the general pathology analyses. Five of the residents took and passed the Phase I general pathology portion of the exam after their fourth mock board attempt. Therefore, only 13 data points are included for the fifth and sixth attempts for the general pathology section. One resident took and passed the Phase I exam after the sixth mock board attempt; all data from this individual are included.
The mock exams were administered as part of the regular training program; therefore, this study was approved for exempt review by the University of Tennessee’s Institutional Review Board 45 CFR 46.101, Category 4.
Statistical analysis
The effect of mock exams on passing the certifying exam was evaluated using mixed model analysis, with mock test scores as the response variable, while section and certifying exam results were the within-subject, independent variables. The time trend of mock exam scores was evaluated using repeated-measures analysis with mock test score as the response variable and the number of attempts as the independent variable. Post hoc multiple comparisons were performed with Tukey’s adjustment. Statistical significance was set at the level of .05. Analyses were conducted using PROC MIXED in SAS 9.4 TS1M3 for Windows x64 statistical software.a
Results
For each section of the mock exam, there was a significant positive linear trend of increasing scores over time: general pathology (gen path), t(76) = 5.46; gross pathology (gross), t(76) = 11.91; histopathology (histo), t(75) = 10.22; veterinary pathology (vet path), t(76) = 5.66; p < .001 (Figure 2). When the sections were combined, the overall mock exam scores also showed a significant positive linear trend, with a 2.76% mean increase in exam scores for each consecutive test: t(348) = 4.33, p < .001 (Figure 3).
Figure 2: Positive linear trend of increasing mock exam scores for each section across the six attempts by certifying exam pass status
Note: “Missed” indicates that those individuals did not pass that section of the certifying exam on their first attempt. “Passed” indicates that they did pass that section of the certifying exam on their first attempt. “All” reflects the data of both those that missed and passed the corresponding section of the certifying exam on their first attempt
N, Gen path = 95, Gross = 93, Histo = 92, Vet path = 93; p < .001.
N, Gen path = 95, Gross = 93, Histo = 92, Vet path = 93; p < .001.
Figure 3: Positive linear trend of mean overall mock exam scores across the six attempts, by certifying exam pass status
Note: “Missed” includes those individuals that did not pass all parts of the certifying exam on the first attempt. “Passed” refers to individuals that passed all parts of the certifying exam on the first attempt. “All” reflects the data of both those that missed or passed all sections of the certifying exam on their first attempt
N = 373, p < .001.
N = 373, p < .001.
Those passing the certifying exam had significantly higher overall mean mock board scores than those who did not pass: F(1, 348) = 19.3, p < .001. A statistically significant difference existed in mean final attempt mock test scores between those who missed and passed the veterinary pathology section, but not the general and gross pathology sections, nor the histopathology section, F(1, 37) = 12.52, p < .001 (Figure 4; Table 1).
Figure 4: Mean sixth-attempt (last) mock exam score by certifying exam pass status
Note: “Missed” indicates that those individuals did not pass that section of the certifying exam on their first attempt. “Passed” indicates that they did pass that section of the certifying exam on their first attempt
N = 60, *p < .05.
N = 60, *p < .05.
| Mean ± SE | |||
|---|---|---|---|
| Board exam result | Section | All attempts (N = 373) | Sixth attempt (N = 60) |
| Missed | Gen path | 29.50 ± 3.37 | 46.00 ± 14.00 |
| Gross | 48.64 ± 4.01 | 61.90 ± 8.06 | |
| Histo | 57.93 ± 2.78 | 71.83 ± 3.66 | |
| Vet path | 36.81 ± 1.61 | 37.17 ± 4.95 | |
| Passed | Gen path | 45.23 ± 1.51 | 54.55 ± 3.33 |
| Gross | 62.83 ± 1.88 | 77.63 ± 1.98 | |
| Histo | 61.36 ± 1.78 | 73.55 ± 2.10 | |
| Vet path | 48.06 ± 1.23 | 57.36 ± 2.98 | |
Gen path = general pathology; Gross = gross pathology; Histo = histopathology; Vet path = veterinary pathology
In an additional analysis using the first mock test score as the covariate, to control for the initial aptitude of each resident, baseline scores exhibited a positive effect on subsequent mock test performance, with an additional 0.23% increase in the exam scores for each additional 1% on mock test score in the first attempt: t(255) = 3.49, p < .001 (data not shown).
Discussion
The mock exam for first-year residents serves as an introduction to what will be expected of them on the certifying exam. These practice exams serve to help demystify the testing process, enable trainees to better identify their weak areas, and generate long-term retention through repeated retrieval practice.2 We have long believed this experience to be essential in helping guide and shape residents’ study habits; however, there is very little objective data supporting this theory. One report describes similar strategies on the development and implementation of mock board exams in two different clinical pathology residency training programs, but that report provides no conclusive data confirming the validity of mock boards as effective preparatory exercises.4 In pathology residencies for physicians, formal mock board exams (resident in-service examinations) are given in all years of the residency training program, and residents passing the American Board of Pathology certifying examination had significantly higher scores on the mock exams in their final year of training.5
In keeping with the deliberate practice framework,1 the mock exams give residents a sense of the types of questions that may appear on the actual certifying examination. In addition to receiving a percentage score on each section of the exam, the residents are also provided with a range of percentage scores achieved by previous residents at the same point in training. These data serve to help residents identify improvement areas on which to focus additional study. In addition to this analytical feedback, answer keys and references for each question are provided for the multiple-choice sections (general and veterinary pathology). For the histopathology and gross sections, explanations of the answers and rationale for grading are provided, in a group setting, for each question. When appropriate, organization strategies for descriptive answers, time efficiency, and clarity are provided.
By exposing the residents to mental and physical experiences designed to mimic the demands of the test day, we believe we are better preparing our residents for optimal performance on the certifying examination. This study documents the progressive improvement in scores throughout the training period (Figure 3) and correlates the results of these twice-yearly mock exams with success on the ACVP certifying examination. Overall, higher mock scores correlated with passing the certifying exam.
The scores of most residents on the final (sixth) mock exam were within the passing range for the histopathology and gross pathology portions of the exam, but not for the general pathology and veterinary pathology sections (Figures 2 and 4). The histopathology and gross pathology portions of the exam reflect the day-to-day learning and experience that come from cases and rounds sessions and which are constantly reinforced throughout the training program. Alternatively, the general pathology and veterinary pathology sections are based on information from textbooks and recent literature and require more focused attempts at information retention. Much of the early years of the residency are spent accumulating and organizing the information for these sections, and then most residents spend the final months leading up to the exam studying to retain this information. This is possibly the reason why residents do not typically pass these sections on the mock exams. More focused study plans throughout the program may help improve mock exam scores on these sections, but cognitive science research indicates that information retrieval from memory, such as what occurs with mock exams, facilitates retention and understanding over time.6 Those residents who performed well on the final veterinary pathology mock exam were significantly more likely to pass this portion of the certifying exam (Figure 4), emphasizing the importance of reinforcement of this information retrieval via repeated mock exams throughout the training period.
Higher scores on the mock histopathology exam (overall or on the last attempt) were not associated with passing this portion of the certifying exam. This result could suggest that this section of the mock exam consistently lacked the rigor needed to discriminate between those who are poised to pass the certifying exam and those who are not. In addition, as pointed out earlier, the histopathology skills are used and reinforced daily; therefore, all UTCVM residents tend to do well on this section. Consequently, the mean mock scores on this section were higher than for other sections, and the sample size might have been too small to detect a significant effect.
Interestingly, higher scores on the first mock exam contributed to greater increases in the mock exam scores over time, which would ultimately predict greater success on the certifying exam. This finding also suggests that the acceptance of high-quality individuals who enter the training program with a solid knowledge base in pathology might also be a predictor of overall success on the certifying exam.
Because a different exam was created for each testing event, there was variability in the content and rigor of each exam; this is true for the certifying exam as well. Each exam aimed to have an appropriate and similar distribution of species, tissue type, pathogeneses, and rigor. The perception of the difficulty of the questions by the residents and the intended difficulty of the questions by pathologists was not always in agreement. The content of the exams reflects the literature of the preceding 5 years; therefore, it is not possible to implement a standard exam that is given to all residents in the program over the course of 10 years, for example.
A clear limitation to this study is that it was conducted at only one institution. In addition, other research suggests that such repeated assessment might cause trainees to adapt their learning to the assessment itself rather than to longer-term goals.7 Nonetheless, the authors believe the benefits of preparing residents for the rigors of the certifying exam in a safe setting outweighs any deleterious effects. The results of this study, in support of deliberate practice theory,1 suggest that mock boards are an important part of anatomic pathology resident training and help prepare residents for success on the certifying exam.
Footnote
a
SAS 9.4 TS1M3, Cary, NC: SAS Institute Inc, 2015.
References
1.
Gauthier S, Cavalcanti R, Goguen J, et al. Deliberate practice as a framework for evaluating feedback in residency training. Med Teach. 2015;37(6):551–7. https://doi.org/10.3109/0142159X.2014.956059. Medline:25511982
2.
Larsen DP, Butler AC. Test-enhanced learning. In: Walsh K, editor. Oxford textbook of medical education. Oxford: Oxford University Press; 2013. p. 443–452xx. https://doi.org/10.1093/med/9780199652679.003.0038.
3.
Munson L, Craig LE, Miller MA, et al, and the American College of Veterinary Pathologists Training Program Development Task Force. Elements of good training in anatomic pathology. Vet Pathol. 2010;47(5):995–1002. https://doi.org/10.1177/0300985810377725. Medline:20807827
4.
Raskin RE. Residency training programs in veterinary clinical pathology: a comparison of experiences at two institutions. J Vet Med Educ. 2007;34(4):478–84. https://doi.org/10.3138/jvme.34.4.478. Medline:18287476
5.
Rinder HM, Grimes MM, Wagner J, et al, and the RISE Committee, American Society for Clinical Pathology and the American Board of Pathology. Senior pathology resident in-service examination scores correlate with outcomes of the American Board of Pathology certifying examinations. Am J Clin Pathol. 2011;136(4):499–506. https://doi.org/10.1309/AJCPA7O4BBUGLSWW. Medline:21917671
6.
Roediger HL III, Butler AC. The critical role of retrieval practice in long-term retention. Trends Cogn Sci. 2011;15(1):20–7. https://doi.org/10.1016/j.tics.2010.09.003. Medline:20951630
7.
Cilliers FJ, Schuwirth LW, Adendorff HJ, et al. The mechanism of impact of summative assessment on medical students’ learning. Adv Health Sci Educ Theory Pract. 2010;15(5):695–715. https://doi.org/10.1007/s10459-010-9232-9. Medline:20455078
Information & Authors
Information
Published In
Journal of Veterinary Medical Education
Volume 47 • Number 1 • February 2020
Pages: 39 - 43
PubMed: 31951799
Copyright
2019 AAVMC.
History
Published in print: February 2020
Published online: 17 January 2020
Key Words:
Authors
Metrics & Citations
Metrics
Related Content
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Download article citation data for:
Journal of Veterinary Medical Education 2020 47:1, 39-43
Journal of Veterinary Medical Education 2020 47:1, 39-43