Abstract
The development of new medical imaging methods contributes to advancements in diagnosis and therapy. To assist the surgeon 3D stereotactic navigation systems for surgical instruments and open MRI for intraoperative real time imaging are being developed. But the exponential increase of knowledge in medicine makes it necessary to develop new concepts for visualization of medical information.
Usually telemedicine is associated with transfer of radiological [1] or histological [2] data, although decision support systems have been described [3]. The remote expert consultation [4, 5] will reduce the cost of high level medical expertise to rural communities [6].
In contrast to these activities that communicate or attach knowledge referring to medical imaging data we have taken an entirely different approach. The information to be transferred is first extracted from radiological data to a more abstract definition of tumor size or surgery simulation, a three dimensional virtual data structure [7]. This information is then visualized directly at the patient as 3D annotation in the form of image fusion of these virtual computer graphics with the live video of the patient or the operating field [8]. When used during surgery this visualization method results in image guided stereotactic surgery.
Although this technology now makes it possible to visualize at the patient information relating to imaging data simultaneously with surgical simulations from a consulting expert we had to overcome another barrier. In the field of telerobotics the operating environment is classified as structured or unstructured [9]. A structured environment means that the remote operating volume can be modeled predictably in the computer. In medicine this assumption is only valid for special cases like hip replacement surgery [10]. More likely the operating field will resemble an unstructured environment. Therefore studies about the accuracy of navigation systems [11] are based on partly inaccurate data like obsolete information from CT. The local surgeon has additional sensory input and can compensate for this inconsistent data.
In the EURODOC project we present the teleconsultation of a remote expert during surgery. Therefore we had to develop a method, Interventional Video Tomography [12], that models the operating environment for the expert at the remote site as accurately as possible. As readily available real-time imaging method we transmit over the Internet the video output from a microscope or endoscope, simultaneously with the local visualization at the operating theater. This enables the remote expert to verify any recommendation based on available medical imaging data also as graphical overlay on the video.
With stereoscopic video virtual structures can be accurately positioned in 3D [13], using mainly the binocular parallax as depth cue. As we transmit monoscopic video data we have to use an alternative method to define the depth of virtual graphical structures relative to the objects visible in the video. We implement an unique approach, we use the motion parallax in monoscopic video to compensate for the missing stereoscopic depth cue.
The position of objects relative to the video source is continously tracked by 3D sensors, therefore defining the projection parameters of the virtual structures for any number of objects tracked in a single video frame. The analysis or adjustment of the depth of the virtual structures is an interactive process that involves moving the imaging device relative to the object or vice versa. This manipulation is not accessible to the remote expert. Therefore we also record an image sequences for retrospective analysis of the video sequence based on the motion parallax depth cue [14].
The expert in a remote location receives this video data almost in real time over standard transmission protocols (LAN, WAN, ISDN, Internet). But in addition also stereotactic navigation data is sent over the network as rigid body coordinates. The actual graphic overlay structures are computed on the remote computer, thus dramatically reducing the bandwidth necessary for transmission. By teleconsulting, the composite images and overlapping graphics - instrument, target structure, landmark, contour - can be seen in connected clinics with the possibility of interactive graphical assistance.
For interactive telesurgical consultation
- the expert modifies the surgical simulation on the remote computer
- the modified projection matrix definition is sent back to the operating site
- the surgeon in the operating theater has now available an alternative graphic overlay defining the changes proposed by the expert
With intraoperative teleconsultation considerations, advises and remarks by teleconsulting surgeons are possible by interactive graphical planning on video-, CT-, or other radiographic images, displaying the intraoperative situs. This enables an expert to participate in the surgical procedure from any remote location.
Within the EURODOC project this technology will be made available free of charge to all leading research institutions in Europe by the Forschungsinstitut für Telepräsenz (Telepresence Research Institute).
Bibliography
[1]
T. Coons, "Teleradiology: the practice of radiology enters Cyberspace," Radiol Technol, vol. 67, pp. 125-40; quiz 141-4, 1995.
[2]
K. Kayser, P. Fritz, M. Drlicek, and W. Rahn, "Expert consultation by use of telepathology--the Heidelberg experiences," Anal Cell Pathol, vol. 9, pp. 53-60, 1995.
[3]
E. K. Tsibulkin, S. I. Goldberg, A. O. Makhanek, A. B. Blokhin, and S. M. Galperin, "Creating a computer-aided intellectual environment for a consultative center of intensive care," Medinfo, vol. 8 Pt 1, pp. 611, 1995.
[4]
F. H. Binkhuysen, F. P. Ottes, J. Valk, C. de Vries, and P. R. Algra, "Remote expert consultation for MRI procedures by means of teleradiology," Eur J Radiol, vol. 19, pp. 147-50, 1995.
[5]
E. De Sutter, P. Duveau, and J. C. Hache, "Remote diagnosis in ophthalmology," Bull Soc Belge Ophtalmol, vol. 254, pp. 67-70, 1994.
[6]
P. A. Halvorsen and I. S. Kristiansen, "Radiology services for remote communities: cost minimisation study of telemedicine," Bmj, vol. 312, pp. 1333-6, 1996.
[7]
M. Truppe, "Interventional video tomography in computer assisted surgery concepts," J Craniomaxillofac Surg, vol. 24, Suppl. 1, 1996.
[8]
A. Wagner, O. Ploder, G. Enislidis, M. Truppe, and R. Ewers, "Virtual image guided navigation in tumor surgery--technical innovation," J Craniomaxillofac Surg, vol. 23, pp. 217-3, 1995.
[9]
P. Milgram, A. Rastogi, and J. Grodski, "Telerobotic Control Using Augmented Reality," presented at 4th IEEE International Workshop on Robot and Human Communication, Tokyo, Japan, 1995.
[10]
E. H. Spencer, "The ROBODOC clinical trial: a robotic assistant for total hip arthroplasty," Orthop Nurs, vol. 15, pp. 9-14, 1996.
[11]
D. A. Simon, R. V. O'Toole, M. Blackwell, F. Morgan, A. M. DiGioia, and T. Kanade, "Accuracy Validation in Image-Guided Orthopaedic Surgery," presented at MRCAS 95, Pittsburgh, 1995.
[12]
M. Truppe, "Process for imaging the interior of bodies," European Patent Office 1993, patent WO94/03100.
[13]
D. Drascic and P. Milgram, "Positioning Accuracy of a Virtual Stereographic Pointer in a Real Stereoscopic Video World," presented at Stereoscopic Displays and Applications II, San Jose, California, 1991.
[14]
M. Truppe, F. Pongracz, W. Freysinger, A. Gunkel, and W. Thumfart, "Interventional Video Tomography," presented at Computer Assisted Radiology, Berlin, 1995.