After Apple launched the iPad (1st generation, 2010; Apple Inc., Cupertino, CA, USA) in January 2010, different types of ‘tablet devices’ have been introduced in the market. This new-generation device has an appearance similar to previous products (such as notebooks, netbooks, and tablet personal computers [PCs]); however, it is clearly different in terms of user experience. This new device works on a ‘mobile operating system’ designed only for mobile use and has a touch screen instead of conventional input devices, such as a keyboard and mouse. Additionally, it has a long battery life, a high graphic performance, a user-friendly interface, and low weight.

Medical personnel have tried to apply such new devices in clinical practice [1,2]. Recently, several hospitals have started using tablet devices for purposes such as maintaining electronic medical records, referring to laboratory results and radiological images, prescribing medications, and calculating the dosage of medication.

Here, this study focused on the three-dimensional (3D) graphics power and touch-based input method of tablet devices. The 3D graphical objects were created on a PC-based DICOM 3D reconstruction program and then transferred to an iPad. Subsequently, surgeries were performed with 3D graphical guidance using an iPad.

DICOM raw images were extracted from patients’ abdominal computed tomography (CT) scans. This was followed by 3D object modeling using a PC-based 3D reconstruction software (OsiriX Imaging Software ver. 5.7–5.8.5: Pixmeo Sarl, Geneva, Switzerland, http://www.osirix-viewer.com; InVesalius ver. 3.0.0 Beta 4: InVesalius, Campinas, Brazil, http://www.cti.gov.br/invesalius/; open sourced free software only used in this study). A range based on the radiodensity (Hounsfield units) in individual sections of a series was selected using the semi-automated ‘region of interest’ function [3] (Fig. 1). Subsequently, the 3D object processed with 3D surface rendering was exported to a conventional 3D file format (*.stl). Finally, this 3D image file was transferred on to an iPad 3D viewer application (STL Viewer ver. 2.2–2.21: Micro Systems Engineering Inc., Lake Oswego, OR, USA, https://itunes.apple.com/kr/app/stl-viewer/id351268929?mt=8; Meshlab for iOS ver. 1.0: 2011–2013, Visual Computing Lab., Apple Inc., Cupertino, CA, USA, https://itunes.apple.com/kr/app/meshlab-for-ios/id451944013?mt=8). Surgery was performed with reference to the 3D object loaded onto the app by zooming, flipping, and rotating the image using fingers (Figs. 2, 3).

This study was approved by the Institutional Review Board of Soonchunhyang University Seoul Hospital (IRB approval no., 2018-05-006).

From March 2014 to August 2014, 3D image-guided surgeries were performed using an iPad in five patients who underwent laparoscopic colectomy for colorectal diseases. All 3D images were recorded by the operator. A 3D graphical reconstruction was performed for several abdominal organs, such as bone structure, back bone, artery, portal vein, ureter, hollow viscus (limited in air substance not soft tissue), pancreas, and tumor.

All surgeries were performed as routine laparoscopic procedures, and the 3D images were only used for obtaining additional information on the intra-abdominal structure. The time taken for the 3D reconstruction was steadily reduced with an increase in the number of cases but the results are not statistically significant. After each surgery, the benefits and proposals for 3D image-guided surgery were recorded.

Intra-operative direct handling of 3D-rendered images with a tablet device for target structure visualization has several major benefits, some of which have been listed below. The operator can easily interpret the 3D structure of the target organ without the conflict associated with secondary processing of two-dimensional (2D) images in the brain. The tablet device is lightweight and small enough for portable hand-held work, and it is technically powerful enough for 3D imaging. Surgeons can easily and quickly enter the process as ‘intra-operative 3D work’ without any complex and burden of prior preparation process because almost pre-step has already finished before surgery. In terms of the cost, these processes are relatively inexpensive compared to the costs associated with previously commercialized large 3D navigation machines.

Referring to 3D images during surgeries has several advantages. A 3D structure can be grasped intuitively without reconstructing 2D images from virtual to virtual platforms. If 3D images are directly placed on the abovementioned tablet devices and can easily be used during surgery, their advantages can be further increased and the accessibility be improved. On the basis of the patient’s abdominal CT images, we constructed a 3D object, using a 3D reconstruction software that could easily be used at the patient’s bedside; transferred it to the 3D viewer application on an iPad; and used it before, during, and after surgery. Especially during the surgery, this object was helpful in understanding the 3D relationship of the target organ with the surrounding structures as it could be rotated to various angles and zoomed in.

The 3D graphics have several benefits in structural fields, such as architecture and mechanics. Understanding hundreds of cross-sliced magnetic resonance and CT images and creating 3D structural objects in mentally is a highly complex cognitive process involving ‘mental representation’ and ‘mental rotation.’ Therefore, it is more convenient to refer to a visual 3D model directly rather than to refer to 2D images in the operative field. However, it is very difficult to statistically evaluate how clinically helpful this new technology is.

We believe there are several advantages of 3D graphics for abdominal surgery. (1) Interpretation of the relationship between two closely located (adjacent, artifact, or real invasion) intra-abdominal organs is easier in 3D. (2) Interpretation of an array or pathway of thin and complex branched structures is easier in 3D: it was more helpful for viewing complex and variable tubular structures like the arteries and ureter compared with solid organs. (3) It serves critical information for surgical decision-making. In fifth case, the surgeon could easily confirm a multiple banded and long sigmoid colon for anastomosis with assistance from the 3D graphic image before surgery, and decided not to perform splenic flexure takedown (this was not clear with a conventional CT scan).

‘Navigation surgery,’ which is a well-known concept that uses a 3D image reconstructed from a patient’s CT or magnetic resonance imaging scan during surgery, has been described in the literature [4]. In addition, expensive commercialized devices are already in use in this field. Assessing studies on the use of mobile devices during or before surgery (as some parts of ‘navigation surgery’ or ‘image-guided surgery’), Sugimoto [5] conducted a study on the use of mobile devices in the medical field in 2011. They generated a 3D-reconstructed image based on a patient’s CT scan through ‘OsiriX’ and referred to the image on an iPad during surgery. Although this was a simple study, it had an influence on further studies such as that using an iPad with a sterile plastic bag and surgical gloves [6]. However, all these studies only used 2D still images or movie clips instead of a real 3D object generated on a PC.

A limitation in this pilot study was ‘medical validation.’ The tablet application used in the study is not certified for medical use. for even it is not invasive. Who does make 3D image. In this study, all rendering processes were performed by the surgeon who performed the surgery, but the graphical processes were performed by a professional operator.

Eguchi et al. [7] performed a clinical study on using a 3D-rendered image on an iPad during lung segmentectomy, and used a surface-rendered model like the one used by us. Müller et al. [8] conducted a study on ex vivo percutaneous kidney biopsy using the augmented reality technique. In their study, the 3D object was overlapped onto the target organ using the visual geometric mapping technique; virtual rotation of a target object using this technique has clinical application.

Several major and minor medical imaging software providers have already commercialized various solutions. Most of them represent full through solutions for a picture archiving and communication system (PACS) server: PC–mobile. However, there is no product that can be used for clinical purposes at the mobile level yet, and such use is limited for most research purposes. In recent years, a system using a cloud image server has emerged. This system can easily refer to a 3D image from a mobile acting as a conventional PACS viewer; however, such a system has several limiting factors including the effect of network performance and problems such as safety, privacy, and legal issues.

To improve the 3D reconstruction process possible with the existing workstation through the cloud, the ‘Visible Patient Project’ model of IRCAD Institute, France, suggested that each end-clinician should upload the patient’s CT image via the web and the rest of the process should be performed by IRCAD [9]. Each clinician will be able to retrieve 3D images using a tablet device on the next day. Although there are many legal and ethical issues to be solved, this end-user model may prove helpful to small- and medium-sized hospitals with limited manpower. In addition, clinicians are offering alternatives uses of 3D imaging.

The safety of the iPad in an operating room can be a critical point in this trial. There is no evidence that supports safety against infection for long and continuous physical contact with the device [10].

The source DICOM for the abdominal CT for 3D reconstruction can make a precise model as the interval between sections becomes shorter. The clinically used 5-mm sections are generally too rough for 3D reconstruction. However, the 1-mm sections used for raw DICOM files are often not available due to the RAM capacity limitations of OsiriX (32 bit free OsiriX has limitation for RAM using as 3.5 gigabyte). Thus, if the detail of the 3D model is not so important, a 5-mm section should be used, and if the detail is important, a 1-mm section with limited sections should be used.

Once surface rendering is completed, it is not easy to modify the result. Therefore, it is better to finish the target work in the working space before rendering as much as possible. Further, it is impossible to transfer the 3D file to an iPad if the file size is too big; therefore, one should limit the ‘resolution’ bar to ‘low’ in the quality option. (If the created 3D file size is too large, you cannot move the file to the iPad, so you should limit the resolution bar to ‘low’ close to the ‘quality option.’)

Unlike 3D reconstruction for bones, that for intra-abdominal organs is difficult and time-consuming. In the original CT scan, it is easy to obtain the 3D image of the desired structure by paying attention to the selection of the shooting point according to the contrast injection. The ability of OsiriX to selectively remove bones from the selection is also helpful. We used 3D reconstruction for each structure in pre-enhanced CT images at various phases using OsiriX and later combined them.

The ‘.stl’ file is a file format used by 3D printers and is increasingly being used as the 3D printer market share grows. Thus, this poses the disadvantage of second processing becoming impossible; however, it may be an advantage from a security standpoint. Another advantage is that printing with a 3D printer is possible with only minor modifications.

In conclusion, although there are many limitations to our study, this study is presents the possibility of using mobile devices as end terminals for navigation surgery and augmented reality. This field appears promising, and we expect medical researchers to contribute to the rapid development of this field.