Prediction of shelled shrimp weight by machine vision

Prediction of shelled shrimp weight by machine vision first-author corresp ‡Corresponding Author ‡Corresponding Author †E-mail:jpli@zju.edu.cn †E-mail:jpli@zju.edu.cn jpli@zju.edu.cn Department of Biosystems Engineering, School of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou 310029, China Department of Biosystems Engineering, School of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou 310029, China The weight of shelled shrimp is an important parameter for grading process. The weight prediction of shelled shrimp by contour area is not accurate enough because of the ignorance of the shrimp thickness. In this paper, a multivariate prediction model containing area, perimeter, length, and width was established. A new calibration algorithm for extracting length of shelled shrimp was proposed, which contains binary image thinning, branch recognition and elimination, and length reconstruction, while its width was calculated during the process of length extracting. The model was further validated with another set of images from 30 shelled shrimps. For a comparison purpose, artificial neural network (ANN) was used for the shrimp weight predication. The ANN model resulted in a better prediction accuracy (with the average relative error at 2.67%), but took a tenfold increase in calculation time compared with the weight-area-perimeter (WAP) model (with the average relative error at 3.02%). We thus conclude that the WAP model is a better method for the prediction of the weight of shelled red shrimp. Shelled shrimp Image Feature Length extracting Weight prediction Weight-area-perimeter (WAP) model INTRODUCTION 1 Shrimp is very popular seafood in the world. Shelled shrimp is a main form of shrimp products for trading. Grading process is very important for increasing the value of shelled shrimp. Traditionally, grading process is conducted manually, which may cause problems such as low efficiency and even bacterial and chemical contamination. Machine vision has been widely applied for size and weight evaluation because it is nondestructive and high efficient. In agriculture and food industries, this technology has been used to determine the sizes of seeds (Granitto et al., 2002 2002 ; 2005 2005 ), pears (Ying, 2000 2000 ) and beans (Kılıç et al., 2007 2007 ), and to grade apples (Throop et al., 2005 2005 ; Menesatti et al., 2009 2009 ), eggs (Chen et al., 2004 2004 ), oranges and peaches (Blasco et al., 2003 2003 ). Scallops were automatically graded by the area and length by computer vision (Lin et al., 2006 2006 ). Fish features were recognized based on image analysis, including fish length (Dunbrack, 2006 2006 ; Harvey, 2003 2003 ; Costa et al., 2006 2006 ), species (White et al., 2006 2006 ), and behavior (Xu et al., 2005 2005 ). Morphological and spectral methods were compared to find the optimum location for shrimp beheading (Ling and Searcy, 1991 1991 ). Correlation between prawn’s size and weight was analyzed and a vision-robotic system was developed for prawn grading and packing (Kassler et al., 1993 1993 ). The weights of different forms of white shrimp and visual quality were determined by machine vision (Luzuriaga et al., 1997 1997 ). Image analysis was also utilized to evaluate carapace length of shrimp (Harbitz, 2007 2007 ). However, up to now, few machine systems have been developed for shrimp industrial application. Luzuriaga et al.( 1997 1997 ) and Kassler et al.( 1993 1993 ) demonstrated that there was a strong correlation between shrimp area and weight, but the samples they used were intact. Nevertheless, shrimps in real grading and packing situation usually contain both unbroken and broken ones. If samples are taken randomly from a real production site, the accuracy of the results will be limited. Moreover, shrimp weight prediction by vision may be feasible, but area univariate model is unreasonable when a wider and shorter shrimp is probably heavier than a thinner and longer one while their dimensions are the same. To overcome the limitation of the existing methods for shrimp weight prediction, an improved method was proposed in this paper. The objectives of this study were: (1) to determine morphological features including area, perimeter, length and width, (2) to establish weight predication models with different combination of features and tests, and (3) to compare the time cost of each model. MATERIALS AND METHODS 1 Computer vision system 2 A diagram of a computer shrimp image capture system is shown in Fig. 1 1 . The system consisted of a lighting box and a computer. The size of lighting box was 660 mm×680 mm×1300 mm. A sample platform (15 mm×20 mm) was placed at the bottom of the box and covered with dark blue background (fabric). A charge coupled device (CCD) camera (TMC-7DSP, PULNIX) was installed at the center of the box, about 40 cm above the sample platform. Six lamps (F40BX/840, GE) were placed behind the white dousers in both sides of the box. Images were acquired by a Matrox Meteor II/Standard image grabber (Matrox Inc., Canada). The resolution of these images acquired was 3.876 pixels/mm. All images were processed and analyzed using Matlab 6.5 (Mathworks 2002, Natick, MA). Weight measurement 2 Shelled red shrimps, originally produced in Zhoushan, Zhejiang Province, China, were purchased from a local market. The weights of shelled shrimps were measured by an electronic balance (Jinnuo Instruments Ltd., China) with accuracy of 0.01 g. Before weighting, water on the shrimp surface was cleaned by absorbent paper. A total of 150 shrimps were used in this study, including 120 intact and 30 broken ones. The weights of these shrimps ranged from 0.99 to 9.18 g, with 52 shrimps in the range of 0.99~2.00 g, 47 in 2.01~4.00 g, 38 in 4.01~6.00 g, and 13 in 6.01~9.18 g. These shrimps were randomly divided into two groups. One group containing 100 intact and 20 broken shrimps was used for model validation, and the other one was for predication. Background segmentation 2 Although by analyzing values of R (red), G (green) and B (blue) channels, background components can be identified in RGB color image (Kim et al., 2008 2008 ), it is unnecessary to keep color information in order to reduce data processing time. Therefore, gray-scale maps were used for image analysis, instead of color ones. To get a clear object, all images were processed as follows: (1) changing RGB images (Fig. 2a 2a ) to gray-scale maps (Fig. 2b 2b ) by MATLAB function rgb2gray ; (2) obtaining an optimal threshold value from histogram by taking the middle value between the two main peaks in the histogram (150 shrimps were determined after analysis); (3) threshold segmentation by MATLAB function im2bw (Fig. 2c 2c ). (a) Original image; (b) Gray scale map; (c) Threshold segmentation; (d) Morphological operation; (e) Edge extracting; (f) Image thinning Morphological operation 2 Images after segmentation may contain some flaws such as holes in the background and black area in the object, which could cause errors in area measurement and other calculations. Furthermore, if the edge of object is very rough, the length of perimeter will be amplified. To solve these problems, morphological open ( imopen ) operation was applied to filter the segmented image (Fig. 2d 2d ). The structure element ( se ) of filtering was a disk with a radius of 4 pixels. The reasons for one-time morphological opening operation were: (1) holes were all filled after erosion operation; (2) perimeter length decreased significantly (2.65% on average), showing that the edge became smooth; (3) no obvious change of edge length occurred while a second morphological open was done; (4) the operation was beneficial to the next step of thinning process. Feature extracting 2 Features of a shelled shrimp image were area, perimeter, length and width. Area and perimeter 2 After image pretreated, the area of shrimp could be obtained by MATLAB function bwarea . The edge of shrimp was acquired by MATLAB function bwperim (Fig. 2e 2e ). Shrimp length 2 The length of shelled shrimp was determined with following procedures: Shrimp width 2 For a shrimp, the width decreased gradually from the connection of head and body (point a in Fig. 3c 3c ) to tail (point d in Fig. 3c 3c ). Added a line fg that was perpendicular to line ab , and points f and g were the crosses between line fg and the edge of the shrimp. The width at point a was determined as the distance of fg . Similarly, the width of shrimp at point d could be evaluated. The width of shrimp was then calculated as the average of the width results at points a and d . Correlation of weight and image parameters 2 Although weight-area (WA) model was not comprehensive, the high correlation between weight and area was proved. Based on this result, four patterns of multivariable prediction models were designed as: weight-area-perimeter (WAP), weight-area-length (WAL), weight-area-width (WAW) and weight-area-perimeter-length-width (WAPLW). By fitting with all equations contained in TableCurve 3D software (Systat Software Inc., 2002a 2002a ), a relationship of w = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + c 4 x 4 + c 5 y , where w is weight, x is area, y is the perimeter for WAP, length for WAL, or width for WAW, c 0 , c 1 , c 2 , c 3 , c 4 and c 5 are coefficients, was found to have high correlation coefficient for WAP, WAL and WAW models (Table 1 1 ). WAPLW method was operated by artificial neural network (ANN) (Costa et al., 2006 2006 ). In order to test the necessity of morphological operation, a control group was utilized as WAP1 (weight-area-perimeter model with data not processed by morphological operation). For a comparison purpose, WA model (Systat Software Inc., 2002b 2002b ) was also used by fitting data to the equation of z = c 0 + c 1 x 1.5 . Error was the average relative error. c 0 , c 1 , c 2 , c 3 , c 4 and c 5 were constants in each model c 0 c 1 c 2 c 3 c 4 c 5 R 2 Error (%) Timing (s) WA 0.100 3.753×10 −6 − − − − 0.980 7.34 0.1700 WAP 1.674 −6.094×10 −4 1.677×10 −7 −1.032×10 −11 2.364×10 −16 6.627×10 −4 0.989 3.02 0.2340 WAP1 1.286 −3.947×10 −4 1.343×10 −7 −8.063×10 −12 1.824×10 −16 8.635×10 −4 0.988 4.15 0.1840 WAL 1.415 −5.874×10 −4 1.602×10 −7 −9.801×10 −12 2.240×10 −16 −3.323×10 −4 0.990 2.79 2.8568 WAW 1.381 −6.109×10 −4 1.709×10 −7 −1.162×10 −11 3.514×10 −16 −2.917×10 −4 0.989 3.01 2.9270 ANN − − − − − − − 2.67 3.1131 RESULTS AND DISCUSSION 1 Considering calculation time as an important factor for the practical application of models, the time cost of each model was measured by recording CPU time under Intel® core (TM) 2 E6550 2.33 GHz, 1 GB memory and 160 GB hard disk. The function in MATLAB was: St =cputime; //Here add the program to be tested. Et =cuptime− St ; Et ; // Et is the time cost in the program calculation which does not contain image acquiring process. Table 1 1 summarizes the results obtained for all models established in this study in comparison with WA model. As anticipated, the prediction accuracy was significantly improved after combining image area with other data (perimeter, length and width). In previous weight prediction researches, shrimps or shelled shrimp samples were all intact, and their weights could be calculated exactly only by analyzing their areas. However, in commercial processing of shrimp meat, goods cannot be as perfect as the samples in labs, which means that the methods used in the previous researches will lead to a low accuracy of weight prediction. It is a common situation that shrimps with the same area have different weights or dissimilar areas may have an identical weight, if shrimp samples are composed of both intact and fragmentary ones. To solve this problem, perimeter, length and width were used in combination with shrimp area. As a result, multivariable forecast models such as WAP, WAL and WAW models provided better prediction accuracies than WA model. It is easy to prove that a wider and shorter shrimp has a smaller perimeter value than a thinner and longer one while their areas are the same. Because of their distinct volumes, the wider and shorter shrimp has a heavier weight than the thinner and longer one. This difference could be reflected correctly by all the models except WA, indicating that WA model is not as practically applicable as the others. On the other hand, when the length data were used for prediction in both WAL and ANN, the average relative error was decreased, but predicted time increased by tenfold. For the first step of length extraction, the MATLAB cost more than 2 s to function, indicating that the two models are not applicable in shelled shrimp producing line. Table 1 1 shows that the maximal error of WAP model was 0.34 g (relatively 11.41%), while WAL and ANN models had 0.33 g (relatively 11.18%) and 0.13 g (relatively 8.55%), respectively. Although the results of ANN and WAL models were better than that of WAP, the error of WAP was still considered to be acceptable, and the accuracy could be compared favorably with human eyes. If more precise target image is required in the future, the length of shrimp can be considered for weight prediction, then a rapid length calculation algorithm or better computer hardware will be required. By comparing the results between WAP and WAP1, image pretreatment was proved to be necessary. It is obvious that shrimp edge was not so smooth especially in the leg part, which might be caused by peeling process. Then the perimeter value is amplified so that it fails to reflect the real dimension of shelled shrimp. After morphological operation, small concave and salient parts of the shrimp edge are erased, so the perimeter value decreases and becomes more useful in demonstrating shrimp size. Therefore, an optimum prediction method can be obtained by the WAP model. CONCLUSION 1 A weight prediction method, the WAP model, for shelled shrimp was designed, which includes pretreatment, image feature extraction, and prediction model analysis. This model was tested to have an accuracy rate of greater than 96.9%. In this model, the pretreatment (morphologically open process) significantly improves the prediction accuracy. The feature extracting can remove braches by erasing end points and a slop algorithm can compensate the missing length. The model analysis determines prediction accuracy and running time. Our results demonstrate that it is feasible to use a machine vision system, coupled with image processing methods, to measure the weight of shelled red shrimps. It is more accurate and faster than experienced shrimp grading workers. References J Blasco , N Aleixos , E Moltó . Machine vision system for automatic quality grading of fruit . Biosystems Engineering , 2003 . 85 ( 4 ): 415 - 423 . DOI: 10.1016/S1537-5110(03)00088-6 http://doi.org/10.1016/S1537-5110(03)00088-6 . Machine vision system for automatic quality grading of fruit H Chen , YC Ding , LR Xiong , YX Wen . Study on automatic non-destructive detection and grading system of duck egg . Transactions of the Chinese Society for Agricultural Machinery , 2004 . 35 ( 6 ): 127 - 129 . (in Chinese) . Study on automatic non-destructive detection and grading system of duck egg C Costa , A Loy , S Cataudella , D Davis , M Scardi . Extracting fish size using dual underwater cameras . Aquacultural Engineering , 2006 . 35 ( 3 ): 218 - 227 . DOI: 10.1016/j.aquaeng.2006.02.003 http://doi.org/10.1016/j.aquaeng.2006.02.003 . Extracting fish size using dual underwater cameras RL Dunbrack . In situ measurement of fish body length using perspective-based remote stereo-video . Fisheries Research , 2006 . 82 ( 1-3 ): 327 - 331 . DOI: 10.1016/j.fishres.2006.08.017 http://doi.org/10.1016/j.fishres.2006.08.017 . In situ measurement of fish body length using perspective-based remote stereo-video PM Granitto , HD Navone , PF Verdes , HA Ceccatto . Weed seeds identification by machine vision . Computers and Electronics in Agriculture , 2002 . 33 ( 2 ): 91 - 103 . DOI: 10.1016/S0168-1699(02)00004-2 http://doi.org/10.1016/S0168-1699(02)00004-2 . Weed seeds identification by machine vision PM Granitto , PF Verdes , HA Ceccatto . Large-scale investigation of weed seed identification by machine vision . Computers and Electronics in Agriculture , 2005 . 47 ( 1 ): 15 - 24 . DOI: 10.1016/j.compag.2004.10.003 http://doi.org/10.1016/j.compag.2004.10.003 . Large-scale investigation of weed seed identification by machine vision A Harbitz . Estimation of shrimp ( Pandalus borealis ) carapace length by image analysis . ICES Journal of Marine Science , 2007 . 64 ( 5 ): 939 - 944 . DOI: 10.1093/icesjms/fsm047 http://doi.org/10.1093/icesjms/fsm047 . Estimation of shrimp (Pandalus borealis) carapace length by image analysis E Harvey . The accuracy and precision of underwater measurements of length and maximum body depth of southern bluefin tuna ( Thunnus maccoyii ) with a stereo-video camera system . Fisheries Research , 2003 . 63 ( 3 ): 315 - 326 . DOI: 10.1016/S0165-7836(03)00080-8 http://doi.org/10.1016/S0165-7836(03)00080-8 . The accuracy and precision of underwater measurements of length and maximum body depth of southern bluefin tuna (Thunnus maccoyii) with a stereo-video camera system M Kassler , P Corke , P Wong . Automatic grading and packing of prawns . Computers and Electronics in Agriculture , 1993 . 9 ( 4 ): 319 - 333 . DOI: 10.1016/0168-1699(93)90049-7 http://doi.org/10.1016/0168-1699(93)90049-7 . Automatic grading and packing of prawns Y Kim , JF Reid , Q Zhang . Fuzzy logic control of a multispectral imaging sensor for in-field plant sensing . Computers and Electronics in Agriculture , 2008 . 60 ( 2 ): 279 - 288 . DOI: 10.1016/j.compag.2007.09.008 http://doi.org/10.1016/j.compag.2007.09.008 . Fuzzy logic control of a multispectral imaging sensor for in-field plant sensing K Kılıç , İK Boyacı , H Köksel , İ Küsmenoğlu . A classification system for beans using computer vision system and artificial neural networks . Journal of Food Engineering , 2007 . 78 ( 3 ): 897 - 904 . DOI: 10.1016/j.jfoodeng.2005.11.030 http://doi.org/10.1016/j.jfoodeng.2005.11.030 . A classification system for beans using computer vision system and artificial neural networks AG Lin , BY Sun , S Yada . Studies on the detecting method of scallop grading based on machine vision . Journal of Fisheries of China , 2006 . 30 ( 3 ): 397 - 403 . (in Chinese) . Studies on the detecting method of scallop grading based on machine vision PP Ling , SW Searcy . Feature extraction for a machine-vision-based shrimp deheader . American Society of Agricultural Engineers , 1991 . 34 ( 6 ): 2631 - 2636 . . Feature extraction for a machine-vision-based shrimp deheader DA Luzuriaga , MO Balaban , S Yeralan . Analysis of visual quality attributes of white shrimp by machine vision . Journal of Food Science , 1997 . 62 ( 1 ): 113 - 118 . DOI: 10.1111/j.1365-2621.1997.tb04379.x http://doi.org/10.1111/j.1365-2621.1997.tb04379.x . Analysis of visual quality attributes of white shrimp by machine vision P Menesatti , A Zanella , S D′Andrea , C Costa , G Paglia , F Pallottino . Supervised multivariate analysis of hyperspectral NIR images to evaluate the starch index of apples . Food and Bioprocess Technology , 2009 . 2 ( 3 ): 308 - 314 . DOI: 10.1007/s11947-008-0120-8 http://doi.org/10.1007/s11947-008-0120-8 . Supervised multivariate analysis of hyperspectral NIR images to evaluate the starch index of apples Systat Software Inc. TableCurve3D for Windows , 2002 . Version 4.0 : San Jose, CA . TableCurve3D for Windows Systat Software Inc. TableCurve2D for Windows , 2002 . Version 5.01 : San Jose, CA . TableCurve2D for Windows JA Throop , DJ Aneshansley , WC Anger , DL Peterson . Quality evaluation of apples based on surface defects: development of an automated inspection system . Postharvest Biology and Technology , 2005 . 36 ( 3 ): 281 - 290 . . Quality evaluation of apples based on surface defects: development of an automated inspection system DJ White , C Svellingen , NJC Strachan . Automated measurement of species and length of fish by computer vision . Fisheries Research , 2006 . 80 ( 2-3 ): 203 - 210 . DOI: 10.1016/j.fishres.2006.04.009 http://doi.org/10.1016/j.fishres.2006.04.009 . Automated measurement of species and length of fish by computer vision JY Xu , XW Miao , Y Liu , SR Cui . Behavioral response of tilapia ( Oreochromis niloticus ) to acute ammonia stress monitored by computer vision . Journal of Zhejiang University SCIENCE B , 2005 . 6 ( 8 ): 812 - 816 . 16052716 DOI: 10.1631/jzus.2005.B812 http://doi.org/10.1631/jzus.2005.B812 . Behavioral response of tilapia (Oreochromis niloticus) to acute ammonia stress monitored by computer vision YB Ying . Research in method to detect size and area of fruits by machine vision . Journal of Zhejiang University (Agric & Life Sci) , 2000 . 26 ( 3 ): 229 - 232 . (in Chinese) . Research in method to detect size and area of fruits by machine vision 10.1631/jzus.B0820364 2008-11-14 2009-06-26 2009-05-31 2022-01-05 Copyright © 2009, Journal of Zhejiang University Science 2009 Journal of Zhejiang University-SCIENCE B (Biomedicine and Biotechnology) 08 10