Title: Fundamental Surgery achieves validation for face validity and product value in a pilot study at the British Orthopaedic Training Association conference

 

Author: Peter Rainger, Chief Learning Officer

 

Keywords: Simulation, Surgery, Orthopedics, Face Validity

 

Abstract

A pilot study was undertaken of Fundamental Surgery’s orthopedics simulation at the British Orthopaedic Training Association conference. 40 participants were asked to a trial of the platform to assess the validity of the simulations and provide feedback to aid in improving the product.

 

At the conclusion of the sessions, participants completed a short questionnaire which assessed their view of the simulation in terms of face validity and product value. These results have been analysed and demonstrate an overwhelmingly positive result for the face validity and product value of orthopedics simulation on the Fundamental Surgery platform.

 

Background

 

Fundamental Surgery launched in May 2018 as the world’s first Software-As-A-Service (SaaS) surgical simulator using virtual reality and haptics. As part of a commitment to the importance of developing a safe and quality product for surgical training Fundamental Surgery are working with a range of independent research institutes to evaluate our simulations and platform. To support this Fundamental Surgery are also undertaking collaborations with leading surgical training conferences and training events to run pilot studies to initiate the collection of data and analysis to provide early evidence towards the validity of the simulation.

 

A pilot study for face validity and product value was held at the British Orthopaedic Training Association (BOTA) annual conference on the 14-16th November 2018. As the UK’s leading and largest conference for orthopedics trainees, the event provided an excellent opportunity to study the use of the simulation platform by real target users. The study created significant interest from attendees and we exceeded our initial aim in recruiting sufficient participants to facilitate a statistically significant study.

 

Method

Participants for the study were recruited from exhibition space at the British Orthopaedic Training Association (BOTA) annual conference on the 14-16th November 2018, in Newcastle Upon Time in the UK.

 

Each participant was given a brief general introduction on the purpose of the trials and instructions on how to use the simulator and the ask to undertake a series of surgical tasks from the Posterior Approach to Total Hip Replacement (P-THR n=38) or the Total Knee Arthroplasty (TKA n=2) repeating their simulation trials of each surgical steps at least twice.

 

Participants were asked to fill out our face validity questionnaire using a standardised scale on their agreement with each statement 1 to 7 from strongly agree to strongly disagree.

 

  1. Participants received no incentives for participation and all results collected at the event have been used in the analysis.

 

Results

There were 40 participants from covering a broad range of medical and surgical training from with a median level of year 6 of orthopedics surgical training. The diversity of training levels of the participants in this study represents a realistic range of targets users who could derive benefit from our simulations.

 

Level of training No.
Medical Student 1
FY2 3
CT1 1
CT2 7
ST1 1
ST4 4
ST3 6
ST4 4
ST5 6
ST6 2
ST7 3
ST8 2
Total 40

 

Simulation value and utility

92.5% of participants agreed or strongly agreed with a statement ‘The simulator would be a useful tool for rehearsal prior to an operation’ whilst 90% agreed or strongly agreed with ‘The simulator represents a useful training and assessment tool’ – this shows overwhelmingly that the simulations and Fundamental Surgery platform provide real tangible product value to participants.

 

Though it is early days in terms of looking at transferability of knowledge and skills into the operating room it is worth noting that 77.5% of participants agreed or strongly agreed with ‘Regularly using the simulator would help me avoid surgical errors’ and 70% ‘Regularly using the simulator would improve my patient outcomes’.

 

Confidence and improved performance

The development of surgical trainees’ confidence is an important factor in becoming efficient and improving surgical performance. To support this concept 90% of participants agree with or strongly agree with the statement ‘Regularly using the simulator would improve my confidence in active contributions in surgery’ and 85% agreed or strongly agreed with the statements ‘Regularly using the simulator would improve my surgical performance’.

 

Simulation face validity

Evaluating the face validity of simulations is an important first step in the validation process. Face validity can be defined as the extent of a simulator’s realism and appropriateness when compared to the actual task. As such the participants were asked to evaluate face validity in terms of visual quality, experience, anatomy, and procedural flow.

 

85% of participants agreed or strongly agreed with the statements ‘The simulator presented a high-quality visual representation’ and ‘The operating room and simulated surgery provides an authentic experience’. In addition, 80% of participants agreed or strongly agreed with ‘The simulator presented a realistic representation of the patient anatomy’ and 82.5% ‘The flow of the surgical procedure(s) was realistic’.

 

With an average of 83% for face validity metrics, the results of this study provide evidence that Fundamental Surgery’s Orthopedics P-THR has shown sufficiently positive results from real users in this pilot study to conclude that the simulation demonstrates face validity.

 

Table of results

 

Statement % A or SA
The simulator would be a useful tool for rehearsal prior to an operation’ 92.5
Regularly using the simulator would improve my confidence in active contributions in surgery 90
Regularly using the simulator would improve my surgical performance 85
The simulator presented a high-quality visual representation 85
‘The operating room and simulated surgery provides an authentic experience 85
The flow of the surgical procedure(s) was realistic’. 82.5
The simulator presented a realistic representation of the patient anatomy 80
Regularly using the simulator would help me avoid surgical errors 77.5
Regularly using the simulator would improve my patient outcomes’. 70

 

  1. More sophisticated statistical analysis has been undertaken to look for correlations between years of training for individual statements but no significant results were found. For simplicity, this report documents the results from the BOTA trial as a whole but they are representative for the sample as whole and sub-groups.

 

Discussion

These results provide confidence in the quality and validity of the Fundamental Surgery’s orthopedics simulations in presenting to participants a valid simulated representation of the surgical procedure and tasks.

 

It should also be noted that as Fundamental Surgery is a SAAS product continual improvements are being made to both the simulations and the platform. As such with significant improvements made to P-THR in the last release we expect to achieve even better results in future studies.

 

Conclusion

In summary, this pilot study confirms that the Fundamental Surgery orthopedics simulation for P-THR has achieved face validity and confirmation of product value.

 

It is noted this is an internally resourced pilot study and as such further research is needed with independent research partners to confirm the results of this pilot study.

 

It is recommended that the platform should be subject to ongoing validation research to study the validity of the platform for other orthopedics procedures in equal depth. Further research is also recommended to evaluate construct and transfer validity.

 

Acknowledgments

Special thanks to the British Orthopaedic Training Association for facilitating our study at their annual conference. Our raw questionnaire data was independently reviewed by Dr Phil Pucher from Imperial College. We are also thankful to all the participants for sharing their experience and valuable feedback.

 

Notes:

 

Fundamental Surgery is a software as a service platform developed by FundamentalVR delivering surgical simulation in VR with haptics.

 

BOTA is an independent professional body promoting surgical excellence in trauma & orthopedic surgical training within the UK and it is an organisation run by trainees, for trainees, for the benefit of patients.