In a before-after quasi-clinical trial study, conducted in Mashhad University of Medical Sciences (MUMS), orthopedic interns who were participating in the orthopedic course within MUMS educational hospitals were recruited in 2019 over a period of 10 months (divided into two time spans of four months and 2 months of washout period between them). All the experiment protocol of this study was in accordance to Declaration of Helsinki. The study was approved by MUMS Research Ethics Committee (IR.MUMS.REC.1398.065).
Using a convenience sampling method, students who entered the orthopedic department for their internship course were recruited. The required sample size was calculated by G*power 126.96.36.199 software. The smallest effect of interest in this study is 5/100. The SD for final exam score was 8/100 which was achieved in a pilot study of six courses before the current study enrollment (effect size of 0.62). Considering type I error of 0.05, test power of 0.90 and ratio allocation of 1, the needed sample size was 55 individuals in each group.
Over the span of the first four months of the research (February 3 to June 5 2019), a total of 65 students were expected to pass the orthopedic course without using the Orthobox application who comprise the control group. Each internship course lasts two weeks and includes an average of nine students; during the four months, a total of seven internship courses took place which means a total of around 65 students pass the orthopedic course. Accordingly, fourteen different standard exams were designed by the orthopedic standard exam board before study recruitment began, and seven of them were randomly chosen for our seven groups of 9 students. Then, each group was given one of these seven exams randomly at the end of the two-week course. In a similar manner, after two months of study pause (this will be more clarified in the limitation section), 65 students were expected to participate in the orthopedics course over the span of the second four months of the study (August 5 to December 18 2019) who are considered as the case groups. However, this time, the Orthobox application, available in iOs®- or Android®-based operating systems, was installed on the students’ smartphones. As such, they used the application during the course, and were then tested by the same seven final exam tests given to the control groups. The application panel provided the executors with case group activity reports for further correlational analysis. Both the case and the control group participants were asked to fill out an attitude Visual Analogue Scale (VAS) questionnaire at the end of each course before receiving their scores. Data were also collected from students’ scores on the final exams specifically designed for this research. These data were then gathered and compared with each other in order to identify the impact of the Orthobox on students’ knowledge of orthopedics.
As mentioned before, the interventional tool used for this study was an application specifically designed for teaching orthopedics named Orthobox. The Orthobox application is free of charge. The app consists of five main parts of educational information (Fig. 1A): 1) frequently used medication in orthopedic wards, 2) order samples for common orthopedic hospitalized patients, 3) prescriptions for common orthopedic ambulatory patients, 4) cast and splint types, and 5) educational movies. Here is the detailed description of each part:
1) The first section is mainly a list of several groups of medication including antibiotics, anticoagulants, analgesics, fluid therapy, vaccines, and casting materials. In each group, we have introduced those medicines which are most frequently ordered or prescribed in orthopedic wards or clinics. The available information about each medicine in this application includes name and classification, forms, indications, contraindications, caution, side effects, and group in pregnancy (Fig. 1B). 2) In the second section, we have selected eight common order notes in orthopedic wards, which include orders for the following orthopedic conditions: active bleeding wound, amputated finger, fat embolism, open fracture, laceration without active bleeding, supracondylar fracture, trochanteric fracture, and deep vein thrombosis (DVT). For each order, a typical history of the patient is provided along with their X-ray or real limb image, and also the appropriate order is written at the end. Certain words in the texts of this section are highlighted, which will provide additional information if you click on them (Fig. 1C). 3) The third section presents several frequently used prescriptions for patients with orthopedic conditions such as electromyography, knee physiotherapy, radiology, and serologic tests (Fig. 1D). 4) The section about casts and splints provides specific descriptions along with their related images about different types of casts and splints, which are used in different parts of the body (Fig. 1E). 5) In an attempt to familiarize students with the casting and splinting skills, useful educational movies about upper and lower limb casting and splinting are provided in the last section of the application. This part of the application requires internet connection to work (Fig. 1F). We are going to make the app publicly available once the investigation is complete. An English-language demonstration video of the application is provided as appendix to the paper (Additional file-1). The development of a new platform enables executors to update and customize the content of the app in order to meet the demands of each specific center.
Application activity report
Each student of the case group is submitted to the application with a defined username and password which would be expired after two weeks at the end of the course. The Orthobox panel enables the executors to have access to the number of times each user visits each part of the application. At the end of the study, we had an activity report from students in case group so that we could analyze the correlation between student’s exam/ VAS score and his/her activity on application (Additional file-2).
Both the teaching plan and final exams for orthopedics internship courses are based on 4 divisions: Order, Prescription, Medication and Skill. The final exams are designed by two designated professors from the orthopedics department. The exam material for both the control and case groups was the Textbook of Orthopedics and Fractures by Dr B. Aalami Harandi, et al., which students are supposed to self-study. For the purposes of this study, the case groups were also given access to the app as a supplementary source of study. The exam consists of four questions requiring the students to write down a complete order for a certain patient history provided on the exam sheet, write a clinic patient prescription for a presented patient, answer questions regarding a certain medicine, and finally explain the techniques of casting or splinting. Before recruitment, the orthopedics exam board prepared fourteen different, but equally difficult (the level of difficulty was carefully balanced by the in-charge professors), questions for each of the four divisions. These questions were then randomly put together to design 14 written question sheets seven of which were finally selected for students’ final exam in both groups. The exam papers were corrected based on a system of differentiating between major and minor faults. First, the writers carefully defined the standard answer, major fault and minor fault which are reported in Table 1 in detail. This table is based on similar definitions used in driving tests. The term ‘major fault’ is defined here as those answers which miss a necessary point or are completely wrong, whereas the term ‘minor fault’ refers to those answers in which the required point is mentioned but they are not the standard answer. Then, one professor (who was not part of the research team but was involved in the education program routinely) corrected the exam papers based on the determined guideline. Therefore, there was no bias in this regard. This method enabled us to compare the major and minor faults on different exam questions between case and control students, and to analyze their correlation with the number of visits case students had made to each corresponding part of the application.
In order to determine the reliability of exam questions, we asked the previously designated professor to correct the papers for a second time (test-retest) and also one other staff of the exam board to correct half of the exam papers randomly (interrater). We then analyzed the intra and inter observer reliability between the resultant scores and the prior correction scores. The test-retest and interrater reliabilities were inter-class correlations of > 0.9 and > 0.8, respectively, in all four question types and final score (Table 2).
A short VAS scale questionnaire (consisting six question) was designed (Additional file-3). The questions required participants to rate how much they had learned about the different branches of orthopedics knowledge, five questions about medicine, order, prescription, casting and splint skills, so that the amount of learning could be measured quantitatively. Moreover, the last question on this questionnaire asked students about their satisfaction regarding the overall educational programs of the orthopedics course. The ratings given to each question on the questionnaire were then compared between case and control groups to determine their relationship with the application usage. Moreover, the correlation of each VAS question score with number of visits made to the corresponding part in the application within case group students were investigated.
Using a convenience sampling method, the orthopedic intern students attending orthopedic departments were recruited for this study. Unfortunately, because of the Covoid-19 pandemic hence changes in educational programs, the last case group was not recruited and we could not reach to the expected 65 students of case group. As a result, a total of 123 students – 66 students in control group and 57 students in case group – participated in this research. Students who did not own a smartphone, who were a guest student, or those who were passing the course for the second time were excluded from the experiment. Two interns in control and one in case group were then excluded according to the criteria of second time participation. Therefore, the final 120 students were studied. Table 3 provides an overview of the study population demographic data. It can be seen from the data that none of the demographic differences were statistically significant between two groups of study. The average score in demographic data refers to the average score of students from the beginning of their studies in medical school. The pre-internship exam is taken twice a year from those students who have passed ten semesters and want to become interns; the score of this exam is also included in the demographic data.
Data management and analysis were performed using the software SPSS 19.0 (2018). Significance levels were set at the 0.05 level using all tests. Case and control groups demographic data are analyzed using T test except for gender and marital status which are based on Chi square test. Students’ quiz scores and VAS scores are analyzed using T test in order to compare the scores between two groups. Pearson’s and Spearman Correlations was used to analyze the association of factors (application visit numbers, final exam score and VAS questionnaire score). We analyzed the intra observer reliability of the exam scores (two correction scores by one rater) and inter observer reliability (two correction scores by two different raters) using ICC tests.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.