The UCAT was created to be a universally applicable, entrustment-based, POCUS competency assessment tool . An early argument for validity of the UCAT for the assessment of FAST competency has been previously published as part of its derivation . This current study sought to expand on previous work and explore the use of the UCAT with a more technically difficult application, the four-view cardiac POCUS. Using Messick’s validity framework , we articulate how this study provides further validity evidence for the UCAT in the domains of internal structure validity (internal consistency, inter-rater reliability) and relations with other variables. We then propose how individuals and programs engaging in POCUS training and assessment can utilize the UCAT in their programs of assessment.
Internal structure validity
Our results suggest that the UCAT maintains high internal consistency ((alpha =0.90)) when used to assess cardiac POCUS, and that all four domains (preparation, image acquisition, image optimization, clinical integration) as well as the entrustment rating, captured relevant, but not redundant, information on POCUS competency. As per Koo and Li’s (2016) Intra-Class Correlation (ICC) guidelines , the results of this study also demonstrate that the UCAT domains have moderate-to-excellent inter-rater reliability (ICCs = 0.61–0.91). This level of agreement was achieved with one local rater who was aware of trainees’ level and general ability, and one external rater who was blinded to the trainee’s experience, which speaks to the strength of the tool. The inter-rater reliability and internal consistency findings are consistent with the UCAT results found when assessing a FAST , suggesting that the UCAT is a reliable POCUS assessment tool across multiple POCUS applications.
Consistent with previous literature [17, 18], the entrustment category had the highest Intra-class Correlation Coefficients (ICC = 0.91). Global assessment of observable competencies using entrustment scales has become commonplace in postgraduate education, where the typical goal is to capture the assessors tacit capacity for judgement of the need for supervision of a trainee . The UCAT both facilitates this global assessment of competency via an overall entrustment score and also provides individual domain scoring which may serve to facilitate feedback and coaching for improvement (or assessment for learning) . The use of domain specific items and overall entrustment aligns with modern programs of assessment in North America, including the structure of entrustable professional activities and milestones in the Canadian Competency by Design framework , and milestones and sub-competencies in the ACGME Milestones Project . Additionally, as our work identified similar validity evidence for the domain specific scores and the overall entrustment score, we believe both are critical and reliable features of the UCAT.
Correlations with other variables
The argument for the validity of the UCAT is further bolstered by its correlation with other variables that are traditionally used as indicators of POCUS competency, such as PGY, previous experience, intent to pursue POCUS certification, and self-rated entrustment or confidence. The results of this study suggest that together, these variables predicted UCAT scores [F (5,16) = 4.06, p = 0.01; R2 = 0.56]. Although no variables were statistically significant predictors on their own, there were notable differences in mean UCAT scores based on PGY. Despite the PGY2-4 s having similar experience with POCUS scans (Table 1), the results of this study suggest that the PGY4 cohort outperformed the other trainees on the cardiac POCUS exam with a median entrustment score of ≥ 4 out of 5 on the UCAT. These results align with previous literature suggesting the number of previous scans and PGY year are not sufficiently reliable indicators of competency in isolation [12, 23,24,25], reinforcing the need for an assessment tool like the UCAT.
Moreover, in typical competency-based medical education frameworks, it is becoming widely recognized that a score of 4 (e.g., I needed to be there just in case) or above is often considered an acceptable indicator of competency for trainees. In this study, only the PGY4s had an average UCAT performance at 4 or higher. This is in sharp contrast to the initial FAST UCAT where all trainees who had previously completed the Canadian Association of Emergency Physicians POCUS curriculum  achieved a level of 4 or higher. This difference in entrustment scores between FAST and four-view cardiac assessments speaks to the discriminatory capacity of the tool. First, it aligns with the general relationship between expected trajectories of competency in FAST versus four-view cardiac. The difference in UCAT scores for FAST and four-view cardiac POCUS also aligns with prior research which suggests trainees are able to rapidly achieve competency with the FAST  and four-view cardiac POCUS requires more practice and training before competency is achieved [12, 27]. The results of the pre-session questionnaire in this study also support this, as trainees reported a high degree of FAST experience and self-rated entrustment, and a lower self-rated entrustment and experience with four-view cardiac POCUS. Taken together, these findings reinforce that a four-view cardiac POCUS is more difficult than FAST, and that the UCAT is able to capture differences in task difficulty.
UCAT implementation moving forward
POCUS is a core competency for graduating EM residents [22, 28], and practicing physicians. At present there is no consensus on which assessment tools are best to measure POCUS competency [3, 29]. This study shows that the UCAT can be used to effectively assess POCUS competency in a consistent manner, and, taken with previous work, that it is accurate across more than one POCUS study type. The UCAT’s high inter-rater reliability and internal consistency means that it can be used as a direct observation tool for both low- and high-stakes assessments. Broad usage of the simple and adaptable UCAT would standardize the assessment of POCUS competency across EM and other specialties. This may simplify credentialling across institutions and professional associations.
There are a number of limitations that must be mentioned with respect to the current study. This study occurred at a single center and was used with a small population of trainees who had largely undergone core ultrasound training emphasizing the subxiphoid cardiac view . There were very few learners who had not had some exposure to the four-view cardiac POCUS, although most had only completed less than 30 studies. While this may be a limited exposure for this POCUS type, it is aligned with the findings on trainee experience from the previous UCAT study . While all available trainees participated in the study, a post hoc power analysis suggested this study was underpowered (0.40), limiting our ability to identify individual predictors . Moreover, the factors evaluated in the regression were treated as independent factors, and while there is some variability within and between PGY, number of studies completed, and entrustment, it is difficult to truly disentangle these measures. All background digital survey data were self-reported. Due to the charting and billing structure in this residency program, there was no way of verifying the quantity of POCUS studies previously performed by each resident. Both raters in this study were involved in the derivation and the FAST validation of the UCAT, thus further work is needed to confirm inter-rater reliability with assessors less familiar with the tool. Lastly, at present, assessments of the UCAT have taken place in a simulated environment. As a result, the documentation and comprehension components of the UCAT were assessed through the trainee’s verbal articulation of their subsequent plan based on the findings from their POCUS study in the context of the stem rather than in a written medical record. Previously derived similar tools such as the Resuscitation Assessment Tool have shown a smooth transition from the simulated to the clinical environment , the UCAT should be trialed and evaluated in the clinical environment to ensure that it remains an effective evaluation tool for the assessment of POCUS competency.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.