Dynamic concision for three-dimensional reconstruction of human organ built with virtual reality modelling language (VRML)

Dynamic concision for three-dimensional reconstruction of human organ built with virtual reality modelling language (VRML) 1 first-author †E-mail:yulinlin@sohu.comyuraining@sohu.com †E-mail:yulinlin@sohu.comyuraining@sohu.com yulinlin@sohu.com 1 2 2 2 Key Lab of Combined Multi-Organ Transplantation, First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou 310003, China Key Lab of Combined Multi-Organ Transplantation, First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou 310003, China Department of Virtual Reality, School of Computer Science, University of Electronic Science and Technology of China, Chengdu 610054, China Department of Virtual Reality, School of Computer Science, University of Electronic Science and Technology of China, Chengdu 610054, China This research studies the process of 3D reconstruction and dynamic concision based on 2D medical digital images using virtual reality modelling language (VRML) and JavaScript language, with a focus on how to realize the dynamic concision of 3D medical model with script node and sensor node in VRML. The 3D reconstruction and concision of body internal organs can be built with such high quality that they are better than those obtained from the traditional methods. With the function of dynamic concision, the VRML browser can offer better windows for man-computer interaction in real-time environment than ever before. 3D reconstruction and dynamic concision with VRML can be used to meet the requirement for the medical observation of 3D reconstruction and have a promising prospect in the fields of medical imaging. Virtual Reality Modelling Language (VRML) Direct texture mapping Three-dimensional reconstruction Dynamic concision INTRODUCTION 1 Surgeons usually rely on their own understanding of pathological changes based on the two-dimensional (2D) medical images (computerized tomography (CT), magnetic resonance imaging (MRI), digital subtraction angiography (DSA), etc.) to create three-dimensional (3D) operation scenes in their imagination. The creation of traditional operation plan is deeply associated with personal experiences, and therefore difficult to share with each other in the operation group. When the internal organs and tissues are complicated, just depending on the human eyes and hands in the operation is not enough and it is difficult even for skillful operators to perceive the internal three-dimensional dissection relationship before the structures are exposed. The visualization and interaction of 3D medical images are currently available and helpful for solving the above problems and hopefully will be more widely used for patient diagnosis, therapy, and surgery in the near future (Yu et al., 2003 2003 ). Traditional 3D reconstruction methods from serial section images such as surfaces and volume reconstruction always adopt the process that starts with mesh reconstruction or isosurfaces extraction, and then moves to data simplification and interpolation (Sweet and Ware, 2004 2004 ; Li et al., 2004 2004 ). Surface rendering usually only offers the appearance of human organs, while volume rendering always provides clear observation of several limited inner soft tissues. Moreover, most existing algorithms whether for surface or volume rendering are complicated to a certain extent in that they cannot be used away from special workstations and the attached software platforms which usually have high demand in workstations and thus hamper all kinds of 3D interaction of models and their further application in PC platforms (Dietrich et al., 2004 2004 ; Huang and Dony, 2004 2004 ). Furthermore, most existing algorithms involve the interpolation method to get equal sample spacing in all directions, as the spacing between slices is greater than the spacing between points on a slice. The traditional method usually does interpolation before rendering, but this procedure is always blind and unselective for clinicians. The interpolation method is sometimes necessary for computer generated images, and so, are well accepted and understood by computer programmers, but it may not be welcomed by clinic doctors as many useful and valuable information of focus detail may get lost when a sequence of cross-sectional slices is thin and adequate enough (the slice width may be at most 0.1~0.2 mm/layer in some advanced high resolution thin-section helical CT, high resolution MRI, etc. at present). In fact, one of the main reasons for the general application of traditional interpolation method in the past was that slice width and slice collimation were also very limited in traditional CT, MRI, etc. If a sequence of cross-sectional slices is thin and adequate enough for the 3D reconstruction, approximately human organs reconstruction can be simplified to layer by layer on the whole in direct texture mapping without any interpolation (Okuda et al., 2003 2003 ; Christodoulou et al., 2003 2003 ; Aubourg et al., 2004 2004 ; Wu et al., 2004 2004 ). Direct texture mapping and 3D texture mapping are widely used in computer 3D games to build gorgeous surfaces and scenes of models. Apart from the traditional surfaces and volume reconstruction, the direct texture mapping and 3D texture mapping, a “new” way of 3D medical reconstruction in our opinion, are nearly forgotten in the medical field by most computer programmers who think the algorithm is very simple and limited and then disdain to adopt them. But the simple is not always bad and useless, especially to those surgeons who have not majored in computer science and are not good at manipulating expensive special workstations and their complex attached software. PRINCIPLES AND METHOD 1 We study this “new” process of 3D reconstruction and dynamic concision based on 2D medical digital images (We attempt to adopt 4 mm/layer and 200 pieces thin-section helical CT images in all in the case which would be thought enough and acceptable for focusing details for the moment) using VRML and JavaScript language, and focus on how to realize the dynamic concision of 3D medical model with script node and sensor node in VRML. Because memories used are relatively rational and acceptable (slices of thin-section helical CT images is 16.8 M, the average of physical memories used are approximately ≤73.8 M in 3D reconstruction and ≤284.6 M in dynamic concision at the beginning by 10 testings through task manager in Windows XP Service Pack1), 3D reconstruction from large datasets can work well on PC using the VRML. And users can control model interaction through different VRML browsers such as Cosmo Player, Cortona VRML Client, etc. with great convenience. The medical digital images firstly and necessarily should be modified by 2D medical image software and batch program. It is noteworthy and inevitably that direct texture mapping algorithm has a preprocessing step for the moment which is different from the volume rendering which constructs the polygonal meshes or isosurfaces. The algorithm of 3D texture mapping for 3D reconstruction, which does not need the preprocessing step for 2D medical images at all or can greatly simplify that step in our prediction, was conceived and confirmed by our research group. 3D reconstruction of human organ built with VRML and JavaScript language is simply based on direct texture mapping. The model is made up of some shape modelling nodes, to establish a square geometry plane for setting up the corresponding texture map with IndexedFaceSet node. The final 3D model like this has completely kept all the information of the original 2D images. After programming in JavaScript to control the 3D model, the function of dynamic concision can be realized by script node and sensor node in VRML. The algorithm of dynamic concision: The concision face is confirmed by three input points that do not lie on the same straight line. The relationship between concision face and geometry plane is judged by expression of beeline relationship on the concision face intersect with sides of the geometry plane. If concision happens, the intersect point of the concision face with the geometry plane is calculated, the relative position coordinate of the geometry plane and texture after concision can be obtained, and then the geometry sculpt of the new model after concision can be obtained. Suppose that the three points passing through the input are respectively: P1 ( x 1, y 1, z 1), P2 ( x 2, y 2, z 2), P3 ( x 3, y 3, z 3), and that the concision face is confirmed by the three points is: Ax + By + Cz + D =0. Set up four summits of the square separately as: ( x 01, y 01, z 0), ( x 02, y 01, z 0), ( x 02, y 02, z 0), ( x 01, y 02, z 0). Then the four sides of the square’s beeline expression is: L1: y = y 01+ t ( z = z 0); L3: y = y 02+ t ( z = z 0); L2: x = x 02+ t ( z = z 0); L4: x = x 01+ t ( z = z 0). Beeline L1 runs parallel to L3, L2 runs parallel to L4. So if A =0, the concision face runs parallel to L1, L3, only crossing with L2, L4. If B =0, the concision face runs parallel to L2, L4, only crossing with L1, L3 (Fig. 1a 1a ). Otherwise, the concision face will cross with L1 and L3, L2 and L4 respectively, which has two possibilities. One is crossing outside the square area; the other is in the square. When concision happens, the concision face divides the geometry plane into two parts resulting in six possibilities (Fig. 1b 1b ). According to the normal direction of the concision face, two parts after concision can be judged as to whether it belongs to the right or left sculpt node in order to get the corresponding texture map at the same time. The flow chart of dynamic concision (Fig. 2 2 ): RESULTS AND DISCUSSION 1 Based on VRML 3D model from a sequence of thoracic thin-section helical CT images (Siemens Sensation Workstation 4 mm/layer), visualization of interval changes of pulmonary nodules evolution is very clear. Combined with evaluation of 3D pathological changes as a whole, the inner nodular size and location in left hilus pulmonis, local white inflammation exudation in left apex pulmonis and left basis pulmonis, inner white inflammation changes in both bronchi in details, etc. can be well observed after dynamic concision. This gives doctors a vivid and direct impression of real double bronchopneumonia, which is usually hard to be seen and understood fully in 2D high-resolution CT images and even in normal 3D reconstruction. This computer-aided diagnosis and analysis for pulmonary nodules and structure are useful to doctors. Unlike algorithms of the volume rendering and surface rendering emphasizing the light and transparency of image pixels which usually make color of 3D model turns white, the whole boundary and region information of pulmonary structure keep the original gray and contrast well after 3D reconstruction and dynamic concision without any distortion (Figs. 3a~3e 3a~3e ). After successful reconstruction of 3D medical images, the precise control and delicate manipulation of these 3D model interactions becomes important for doctor’s further diagnosis and therapy. It is especially essential and useful to surgeons. Most of traditional 3D concision algorithms from VRML are only involved with stable concision or static concision (Law and Heng, 2000 2000 ; Huang et al., 2002 2002 ). And the concision scheme should be predefined by manual work first and then images are modified or calculated to rebuild the final concision model. Moreover, most traditional algorithms can only do limited regular 3D concision and the complexity of the algorithm limited the interaction speed of the model. While our algorithm of dynamic concision can freely select the position and direction, the 3D interaction speed of our model is faster, the concision is more convenient. After the concision, two parts of the internal organs lie beside the whole organ, which makes it convenient and useful to choose which inner part to be carefully observed at different angles. This will help doctors to make the correct judgment on the local pathological changes and devise better preoperation schemes. By setting of light source on the 3D model and transparency on images through node in VRML such as PointLight, DirectionalLight, SpotLight, etc., the light can easily radiate through small gaps of slices, so that the 3D model becomes transparent to some extent. It is specially noteworthy and interesting that original slices, outer surface and inner structure of 3D model can be well observed at the same time at some angles in the process of the continually moving model after concision (Fig. 3f 3f ). This is very helpful for clinic doctors to understand and then to be familiar with the internal 3D dissection relationship among important structures. As the organs are reconstructed layer by layer in direct texture mapping, all important 3D inner structures can keep original image’s continuity, gray and contrast well with surroundings, important tract and tree structures are more easily identified and orientated as a whole by clinic doctors without needing any extra complicated manual and automated preprocessing operation to obtain the extracted and segmented area. Simply and practically, without involving any interpolation, 3D reconstruction and concision of body internal organs can be done in this way with such high quality that they are better than those obtained from the traditional methods. Moreover, with the function of dynamic concision, VRML browser can offer better windows of man-computer interaction in real-time environment than before. Although there are many researchers working on PC-based real-time volume rendering recently, such as the shear-warp transform and its variations, as a possible, useful and “new” supplement for traditional methods, in our opinion, direct texture mapping and 3D texture mapping are potentially important ways for medical 3D reconstruction. The involved algorithms will be well and quickly developed in the near future (Frueh et al., 2005 2005 ). 3D reconstruction and dynamic concision with VRML can be used to meet some basic requirement for medical 3D reconstruction, for observation, demonstration, teaching and training, and have promising prospect in the field of medical imaging. References J Aubourg , G Moreau , P Fuchs . Three-dimensional reconstruction and texturation of museographic objects using multiple images and stereoscopic depth map fusion . Proceedings of SPIE–The International Society for Optical Engineering , 2004 . 5302 136 - 147 . . Three-dimensional reconstruction and texturation of museographic objects using multiple images and stereoscopic depth map fusion C Christodoulou , C Pattichis , M Pantziaris , A Nicolaides . Texture-based classification of atherosclerotic carotid plaques . IEEE Transactions on Medical Imaging , 2003 . 22 902 - 912 . . Texture-based classification of atherosclerotic carotid plaques C Dietrich , L Nedel , S Olabarriaga , J Comba , D Zanchet , A Silva , S de Souza Montero . Real-time interactive visualization and manipulation of the volumetric data using GPU-based methods . Proceedings of SPIE–The International Society for Optical Engineering , 2004 . 5367 181 - 192 . . Real-time interactive visualization and manipulation of the volumetric data using GPU-based methods C Frueh , S Jain , A Zakhor . Data processing algorithms for generating textured 3D building facade meshes from laser scans and camera images . International Journal of Computer Vision , 2005 . 61 159 - 184 . . Data processing algorithms for generating textured 3D building facade meshes from laser scans and camera images Q Huang , R Dony . Neural network texture segmentation in equine leg ultrasound images . Canadian Conference on Electrical and Computer Engineering , 2004 . 3 1269 - 1272 . . Neural network texture segmentation in equine leg ultrasound images X Huang , B Wang , S Huang . Three-dimensional reconstruction of human head CT slices using VRML language . Journal of Xiamen University (Nature Science) , 2002 . 46 740 - 743 . (in Chinese) . Three-dimensional reconstruction of human head CT slices using VRML language T Law , P Heng . Automated extraction of bronchus from 3D CT images of lung based on genetic algorithm and 3D region growing . Proceedings of SPIE–The International Society for Optical Engineering , 2000 . 3979 905 - 916 . . Automated extraction of bronchus from 3D CT images of lung based on genetic algorithm and 3D region growing B Li , Z Wang , E Smouha , D Chen , Z Liang . Accelerating virtual surgery simulation for congenital aural atresia . Proceedings of SPIE–The International Society for Optical Engineering , 2004 . 5367 654 - 660 . . Accelerating virtual surgery simulation for congenital aural atresia M Okuda , M Ikehara , S Takahashi . Compression of 3D models by remesh on texture images . IEICE Transactions on Information and Systems , 2003 . 86 1110 - 1115 . . Compression of 3D models by remesh on texture images G Sweet , C Ware . View direction, surface orientation and texture orientation for perception of surface shape . Proceedings–Graphics Interface , 2004 . 2004 97 - 106 . . View direction, surface orientation and texture orientation for perception of surface shape C Wu , Y Chen , C Liu , C Chang , Y Sun . Automatic extraction and visualization of human inner structures from endoscopic image sequences . Proceedings of SPIE–The International Society for Optical Engineering , 2004 . 5369 464 - 473 . . Automatic extraction and visualization of human inner structures from endoscopic image sequences Z Yu , S He , Q Xiong . The application of virtual reality modelling language (VRML) in the fields of medical digital image . Journal of Biomedical Engineering , 2003 . 20 222 - 228 . . The application of virtual reality modelling language (VRML) in the fields of medical digital image 10.1631/jzus.2005.B0611 Project supported by Postdoctoral Fund of China (No. 2003034518), Fund of Health Bureau of Zhejiang Province (No. 2004B042), China 2005-01-27 2005-04-10 2005-06-23 2022-01-05 Copyright © 2005, Journal of Zhejiang University Science 2005 Journal of Zhejiang University-SCIENCE B (Biomedicine and Biotechnology) 07 6B