Application of least squares vector machines in modelling water vapor and carbon dioxide fluxes over a cropland

Application of least squares vector machines in modelling water vapor and carbon dioxide fluxes over a cropland 1 first-author †E-mail:q_breeze@126.com; bmhu@mail.hz.zj.com †E-mail:q_breeze@126.com; bmhu@mail.hz.zj.com q_breeze@126.com 2 2 3 4 †E-mail:q_breeze@126.com; bmhu@mail.hz.zj.com †E-mail:q_breeze@126.com; bmhu@mail.hz.zj.com q_breeze@126.com Institute of Ecology, School of Life Science, Zhejiang University, Hangzhou 310029, China Institute of Ecology, School of Life Science, Zhejiang University, Hangzhou 310029, China Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing 100101, China Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing 100101, China Institute of Applied Entomology, School of Agriculture and Biotechnology, Zhejiang University, Hangzhou 310029, China Institute of Applied Entomology, School of Agriculture and Biotechnology, Zhejiang University, Hangzhou 310029, China School of Science, Zhejiang University, Hangzhou 310029, China School of Science, Zhejiang University, Hangzhou 310029, China Least squares support vector machines (LS-SVMs), a nonlinear kemel based machine was introduced to investigate the prospects of application of this approach in modelling water vapor and carbon dioxide fluxes above a summer maize field using the dataset obtained in the North China Plain with eddy covariance technique. The performances of the LS-SVMs were compared to the corresponding models obtained with radial basis function (RBF) neural networks. The results indicated the trained LS-SVMs with a radial basis function kernel had satisfactory performance in modelling surface fluxes; its excellent approximation and generalization property shed new light on the study on complex processes in ecosystem. Least squares support vector machines (LS-SVMs) Water vapor and carbon dioxide fluxes exchange Radial basis function (RBF) neural networks INTRODUCTION 1 Modelling the variation in surface-atmospheric exchange of water vapor and carbon dioxide fluxes and how they are influenced by a complex combination of environment variables and plant physiology is crucial for assessing the annual water and carbon budget for cropland. Biophysical or process-based models such as soil-vegetation-atmosphere transfer (SVAT) (Franks et al., 1997 1997 ; Franks and Beven, 1999 1999 ; Mo and Beven, 2004 2004 ), Simultaneous Heat and Water (SHAW) (Flerchinger et al., 1996 1996 ; Flerchinger and Pierson, 1991 1991 ) has been developed to quantify these fluxes in different time scales and their interrelationship with biotic and abiotic factors based on field experiments. In their simulation, detailed processes of photosynthesis, respiration from vegetation and soil carbon components, evapotranspiration, hydrological cycling, etc. are required to be explicitly modelled and more realistic parameters such as soil, plant and vegetation characteristics, surface resistance are required to be specified (Unland et al., 1996 1996 ; Arora, 2003 2003 ), which makes fluxes estimation complex and carbon or water budget construction relatively inconvenient. Artificial neural network (ANN), a tool used to process information in a non-liner manner, enables completely unconstrained optimization and estimation of input-output responses without a predefined mathematical model (Kosko, 1992 1992 ; Demuth and Beale, 1994 1994 ; Schulz and Härtling, 2003 2003 ), has recently been applied to model fluxes exchanges at the land-atmosphere interface recently and been proved to have a higher accuracy compared to classical methods (Huntingford and Cox, 1997 1997 ; Van Wijk and Bouten, 1999 1999 ), though physical processes and parameters are introduced into the model structure. Despite their advantages, a number of drawbacks still remain like non-convex training problem with multiple local minima, dependence on quantity and quality of training dataset, choice of the number of hidden units, etc. (Suykens, 2001 2001 ). A breakthrough was obtained at this point when a new powerful machine learning method−support vector machines (SVMs), was developed on the basis of statistical learning theory in the last decade. Many successful applications in nonlinear classification and function estimation have shown that SVMs can handle higher dimensional data better even with relatively fewer training samples and that they exhibit very good generalization ability for complex models (Vapnik, 1995 1995 ; 1998 1998 ). The standard SVM is solved using complicated quadratic programming methods, which are often time consuming and difficult to implement adaptively, whilst LS-SVM is solved by a set of linear equations amenable to solution via on-line adaptive methods (Suykens, 2001 2001 ), so we hereby use LS-SVMs to model surface water vapor and carbon dioxide fluxes. The goal of this paper is to investigate applicability of LS-SVMs in modelling the dynamics of water vapor and carbon dioxide fluxes over a cropland. THEORY 1 Given a training set { x k , y k } k =1 N with input data x k ∈R n and output data y k ∈R, the LS-SVM model for function estimation has the following representation in feature space, y ( x )= w T φ ( x )+ b Here the nonlinear function φ (·): R n →R n k maps the input space to a higher dimension feature space. The dimension n k of this space is only defined in an implicit way; b is a bias term; w ∈R n k is weight vector; e k ∈R is error vector; γ is the regularization parameter. The optimization problem is defined as: $$\mathop {\min }\limits_{w,b,e} J(w,e) = \frac{1}{2}w^{^{\text{T}} } w + \gamma \frac{1}{2}\sum\limits_{i = 1}^N {e_i^2 }$$ Subject to the equality constraints: $$y_i = w^{^{\text{T}} } \varphi (x_i ) + b + e_i {\text{ }}i = 1, \cdots ,N$$ The solution is obtained after constructing the Lagrangian, $$L(w,b,e;\alpha ) = J(w,e) - \sum\limits_{i = 1}^N {\alpha _i \left\{ {w^{^{\text{T}} } \varphi (x_i ) + b + e_i - y_i } \right\}}$$ where α i are Lagrangian multipliers. Application of the conditions for optimality yields the following linear system ( 5 5 ): $$\left[ \begin{gathered}0{\text{ }}1^{^{\text{T}} } \\1{\text{ }}\Omega + \gamma ^{ - 1} I \\\end{gathered} \right]\left[ \begin{gathered}b \\\alpha \\\end{gathered} \right] = \left[ \begin{gathered}0 \\y \\\end{gathered} \right]$$ where y =[ y 1 ,…, y N ], 1 =[1,…,1], α =[ α 1 ,…, α N ], Mercer’s condition is applied in the Ω matrix $$\Omega _{il} = \psi (x_i ,x_l ) = \varphi (x_i )^{\text{T}} \varphi (x_l ),{\text{ }}i,l = 1, \cdots ,N$$ The resulting LS-SVM model for function estimation becomes, $$y(x) = \sum\limits_{i = 1}^N {\alpha _i \psi (x,x_i ) + b}$$ where α i , b comprise the solution to the linear system. In Eq.( 6 6 ), ψ ( x i , x l ) is the so-called kernel function with which the input vector can be mapped implicitly into a high-dimension feature space. The most usual kernel functions are polynomial, Gaussian-like or some particular sigmoids (Suykens, 2001 2001 ). SIMULATION AND RESULTS 1 Collection of the dataset 2 The dataset used in this work was obtained after continuous measurements of carbon dioxide and water vapor fluxes above a winter wheat and summer maize rotation field with eddy covariance technique from Nov. 2002 to Oct. 2003 at Yucheng Comprehensive Experiment Station (36°57′ N, 116°36′ E, 20 m a.s.l) in the North China Plain. Only data collected during the summer maize growth stage from the day of sowing [day of year (DOY165)] to harvest (DOY275) was applied for this study. Of 105 days for fluxes and meteorological measurements on half hourly basis, the missing 5.67% data in observed fluxes was due to instrument maintenance, calibration, malfunction of the sensors and supporting equipment. The 0.06%, 0.48% and 0.73% of unreasonable data for carbon dioxide flux (F c ), water vapor flux (LE) and sensible heat flux (H s ) respectively were rejected; about 3.06% and 3.41% of soil heat flux data recorded at depth of 0.03 m and 0.05 m were eliminated from the dataset for the same reason. Measurements within 24 h after a rain event were also removed from the database. Nighttime fluxes have been reported to be underestimated by the eddy covariance approach during stable condition because of CO 2 storage in the layer below the eddy flux system. A u * threshold ( u * > 0.12 ms −1 ) was determined and data with values below this threshold were removed from the dataset (Falge et al., 2001 2001 ; Anthoni et al., 2004 2004 ). Consequently, there were 1951 half hourly complete data records available for modelling. Determination of input variables 2 Land-atmosphere exchange of water vapor and carbon dioxide fluxes associated with complex ecophysiological processes including assimilation and transpiration, which are driven by external factors such as solar radiation, atmospheric conditions, soil water status and internal factors such as plant physiological and biometrical conditions (Bosveld and Bouten, 2001 2001 ). Many previous studies on assessing the environmental constraints on carbon and water vapour exchange in forest, grassland or crop field (Huntingford and Cox, 1997 1997 ; Kelliher et al., 1997 1997 ; Valentini et al., 1996 1996 ; Granier et al., 2000a 2000a ; 2000b 2000b ; Van Wijk and Bouten, 1999 1999 ; Baldocchi and Wilson, 2001 2001 ; Bosveld and Bouten, 2001 2001 ) proved this very well. Based on the results of study with artificial neural networks using the same dataset, photosynthetically active radiation (PAR), vapor pressure deficit (VPD), air temperature ( T ), and leaf area index (LAI) were taken as the inputs in modelling both carbon dioxide and water vapor fluxes exchange. Besides these variables, wind velocity (U) and soil volumetric water content (W) were selected for computing F c and LE respectively. Preparation of training dataset 2 One thousand and nine hundred fifty-one data records were randomly divided into two subsets, one used exclusively for training and the other exclusively for testing, were applied to the development of SVMs. All the variables were rescaled to be included within the interval [−1, 1] by using the following equation: $$x_{{\text{scale}}} = \frac{{x - x_{\min } }}{{x - x_{\max } }}$$ where x and x scale were the old and new value of the variable for a sampling point respectively, x min and x max were the minimum and maximum values of that variable in the original dataset. SVM training 2 In this study, we take radial basis function (RBF) kernals with LS-SVMs, $$\psi (x,x_i ) = \exp ( - \frac{{\left\| {x - x_i } \right\|^2 }}{{\sigma ^2 }})$$ where σ > 0 is a constant defining the kernel width. It should be noted that predetermined parameters in LS-SVMs algorithms with RBF kernel are γ and σ 2 , which are less than those in standard SVMs (Jemwa and Aldrich, 2003 2003 ). Moreover, ε -insensitive formulation in Vapnik’s standard SVMs are modified by introducing a squared error term and equality constraints so that one solves a linear system instead of a quadratic programming problem, thus greatly reduce the computing complexity. More detailed information on SVMs can be found (Vapnik, 1995 1995 ; 1998 1998 ; 1999 1999 ). A 5-fold cross-validation procedure (Duan et al., 2001 2001 ; Witten and Frank, 2000 2000 ) was used to train and test the LS-SVMs under various model and parametric settings. Neural network models were implemented with software package LS-SVMlab1.5. (Pelckmans et al., 2002 2002 ). Parameters in LS-SVMs for F c and LE are given in Table 1 1 . Since the results of modelling water vapor and carbon dioxide fluxes were similar, we only showed the former in this paper (Fig. 1 1 , Table 2 2 ). As shown in Table 2 2 , LS-SVM with RBF kernel for modelling water flux had greater learning mean squared errors (LMSE) and less generalization means squared errors (GMSE) compared with those in RBF neural network no matter what the proportion of dataset for analysis was. The GMSEs of the two algorithms increased with the training dataset size at different rate with the RBF neural network having the larger. LS-SVM was less dependent on the size of the training dataset and having better generalization ability than that of the RBF neural network. Outputs γ σ ² F c 115.342 3.254 LE 279.568 4.513 SVs are support vectors 2 Training set 2 Testing set 3 LS-SVM 2 RBF neural network LMSE GMSE SVs * * LMSE GMSE 390 1561 0.1749 0.2753 272 0.1256 0.2875 585 1366 0.1786 0.2684 472 0.1367 0.2742 780 1171 0.1843 0.2576 638 0.1534 0.2615 976 975 0.1735 0.2357 829 0.1610 0.2386 1171 780 0.1701 0.2415 948 0.1583 0.2274 1366 585 0.1773 0.2364 1026 0.1642 0.2382 DISCUSSION AND CONCLUSION 1 Support vector machines (SVMs) have been proposed as a powerful machine learning approach. An improved SVMs model, least squares support vector machines (LS-SVMs), which enables solution of highly nonlinear and noisy black-box modelling problems was presented and applied to modelling the dataset obtained from the field measurements in this study. The results indicated that LS-SVMs could be used to model surface fluxes exchange without restrictive assumptions and parameters specification required by other models. Compared with RBF neural network, it has stronger learning ability, better generalization ability and is less dependent on the size of the training dataset. Besides, SVM’s ability to handle high-dimension and incomplete data allows successful extraction of information even when part of the data records was missing or unreasonable owing to the problems of instrument malfunction or maintenance, calibration and climate influences, so SVMs method is suitable to simulate land-atmosphere interaction in an efficient and stable way. Although the proposed LS-SVM-based model may be superior to other modelling methods in some aspects, it has some potential drawbacks such as the underlying Gaussian assumptions related to a least squares cost function. Some researchers have made some efforts to overcome these by applying an adapted form called weighted LS-SVM (Thissen et al., 2003 2003 ). We intend to continue the studies on the application of LS-SVMs in modelling energy and mass exchanges at the cropland-atmosphere interface using larger dataset obtained from different sites. Possibilities to model these fluxes independently of crop species or sites specification will be investigated. Furthermore, LS-SVMs application in prediction or gap-filling the missing data as well as its algorithm optimization will be the focus in our future work. References PM Anthoni , A Freibauer , O Kolle , ED Schulze . Winter wheat carbon exchange in Thuringia, Germany . Agricultural and Forest Meteorology , 2004 . 121 55 - 67 . . Winter wheat carbon exchange in Thuringia, Germany VK Arora . Simulating energy and carbon fluxes over winter wheat using coupled land surface and terrestrial ecosystem models . Agricultural and Forest Meteorology , 2003 . 118 21 - 47 . . Simulating energy and carbon fluxes over winter wheat using coupled land surface and terrestrial ecosystem models DD Baldocchi , KB Wilson . Modelling CO 2 and water vapor exchange of a temperate broadleaved forest across hourly to decadal time scales . Ecological Modelling , 2001 . 142 155 - 184 . . Modelling CO2 and water vapor exchange of a temperate broadleaved forest across hourly to decadal time scales FC Bosveld , W Bouten . Evaluation of transpiration models with observations over a Douglas-fir forest . Agricultural and Forest Meteorology , 2001 . 108 247 - 264 . . Evaluation of transpiration models with observations over a Douglas-fir forest H Demuth , M Beale . Neural Network Toolbox for Use with MATLAB , 1994 . : Natick The MathWorks, Inc , . Neural Network Toolbox for Use with MATLAB K Duan , S Keerthi , A Poo . Evaluation of Simple Performance Measures for Tuning SVM Hyperparameters (Tech. Rep. No. Control Division Technical Report CD-01-11) . 2001 . Department of Mechanical Engineering, National University of Singapore , . Evaluation of Simple Performance Measures for Tuning SVM Hyperparameters (Tech. Rep. No. Control Division Technical Report CD-01-11) E Falge , D Baldocchi , R Olson , P Anthoni , M Aubinet , C Bernhofer , G Burba , R Ceulemans , R Clement , H Dolman , 等 . Gap filling strategies for defensible annual sums of net ecosystem exchange . Agricultural and Forest Meteorology , 2001 . 107 43 - 69 . . Gap filling strategies for defensible annual sums of net ecosystem exchange GN Flerchinger , FB Pierson . Modelling plant canopy effects on variability of soil temperature and water . Agric and Forest Meteor , 1991 . 56 227 - 246 . . Modelling plant canopy effects on variability of soil temperature and water GN Flerchinger , CL Hanson , JR Wight . Modelling of evapotranspiration and surface energy budgets across a watershed . Water Resour Res , 1996 . 32 ( 8 ): 2539 - 2548 . . Modelling of evapotranspiration and surface energy budgets across a watershed SW Franks , K Beven . Conditioning a multiple patch SVAT model using uncertain time-space estimates of latent heat fluxes as inferred from remotely sensed data . Water Resour Res , 1999 . 35 2751 - 2761 . . Conditioning a multiple patch SVAT model using uncertain time-space estimates of latent heat fluxes as inferred from remotely sensed data SW Franks , KJ Beven , PF Quinn , IR Wright . On the sensitivity of soil-vegetation-atmosphere transfer (SVAT) schemes: Equifinality and the problem of robust calibration . Agric Forest Meteor , 1997 . 86 63 - 75 . . On the sensitivity of soil-vegetation-atmosphere transfer (SVAT) schemes: Equifinality and the problem of robust calibration A Granier , E Ceschia , C Damesin , E Dufrêne , D Epron , P Gross , S Lebaube , V Le Dantec , N Le Goff , D Lemoine , 等 . The carbon balance of a young beech forest . Funct Ecol , 2000 . 14 312 - 325 . . The carbon balance of a young beech forest A Granier , P Biron , D Lemoine . Water balance, transpiration and canopy conductance in two beech stands . Agric Forest Meteor , 2000 . 100 291 - 308 . . Water balance, transpiration and canopy conductance in two beech stands C Huntingford , PM Cox . Use of statistical and neural network techniques to detect how stomatal conductance responds to changes in the local environment . Ecol Model , 1997 . 97 217 - 246 . . Use of statistical and neural network techniques to detect how stomatal conductance responds to changes in the local environment GT Jemwa , C Aldrich . Identification of Chaotic Process Systems with Least Squares Support Vector Machines. Neural Networks . Proceedings of the International Joint Conference , 2003 . 3 20 - 24 . . Identification of Chaotic Process Systems with Least Squares Support Vector Machines. Neural Networks FM Kelliher , DY Hollinger , ED Schulze , NN Vygodskaya , JN Byers , JE Hunt , TM McSeveny , I Milukova , AF Sogachev , AV Varlagin , 等 . Evaporation from an eastern Siberian larch forest . Agric Forest Meteor , 1997 . 85 135 - 147 . . Evaporation from an eastern Siberian larch forest B Kosko . Neural Networks and Fuzzy Systems. A Dynamical Systems Approach to Machine Intelligence , 1992 . : Englewood Cliffs, New Jersey Prentice-Hall, Inc , 449 . Neural Networks and Fuzzy Systems. A Dynamical Systems Approach to Machine Intelligence XG Mo , K Beven . Multi-objective parameter conditioning of a three-source wheat canopy model . Agricultural and Forest Meteorology , 2004 . 122 39 - 63 . . Multi-objective parameter conditioning of a three-source wheat canopy model K Pelckmans , JAK Suykens , T Van Gestel , J De Brabanter , L Lukas , B Hamers , B Moor , J Vandewalle . A Matlab/C Toolbox for Least Squares Support Vector Machines . 2002 . : Leuven, Belgium Internal Report 02-44, ESAT-SISTA and K.U. . A Matlab/C Toolbox for Least Squares Support Vector Machines H Schulz , S Härtling . Vitality analysis of Scots pines using a multivariate approach . Forest Ecology and Management , 2003 . 186 73 - 84 . . Vitality analysis of Scots pines using a multivariate approach JAK Suykens . Nonlinear Modelling and Support Vector Machines . 2001 . IEEE Instruments and Measurement Technology Conference . Budapest, Hungary . Nonlinear Modelling and Support Vector Machines U Thissen , R van Brakel , AP de Weijer , WJ Melssen , LMC Buydens . Using support vector machines for time series prediction . Chemometrics and Intelligent Laboratory Systems , 2003 . 69 35 - 49 . . Using support vector machines for time series prediction HE Unland , PR Houser , WJ Shuttleworth , ZL Yang . Surface flux measurement and modelling at a semi-arid Sonoran Desert site . Agricultural and Forest Meteorology , 1996 . 82 119 - 153 . . Surface flux measurement and modelling at a semi-arid Sonoran Desert site R Valentini , P Deangelis , G Matteucci , R Monaco , S Dore , GES Mugnozza . Seasonal net carbon dioxide exchange of a beech forest with the atmosphere . Global Change Biol , 1996 . 2 199 - 208 . . Seasonal net carbon dioxide exchange of a beech forest with the atmosphere MT Van Wijk , W Bouten . Water and carbon fluxes above European coniferous forests modeled with artificial neural networks . Ecological Modelling , 1999 . 20 181 - 197 . . Water and carbon fluxes above European coniferous forests modeled with artificial neural networks V Vapnik . The Nature of Statistical Learning Theory , 1995 . : New York Sringer-Verlag , 311 . The Nature of Statistical Learning Theory V Vapnik . Statistical Learning Theory , 1998 . : New York John Wiley and Sons , . Statistical Learning Theory V Vapnik . The Nature of Statistical Learning Theory , 1999 . 2nd Ed : New York Springer-Verlag , . The Nature of Statistical Learning Theory I H Witten , E Frank . Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations , 2000 . : San Diego, CA: Morgan Kaufmann . Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations 10.1631/jzus.2005.B0491 Project supported by the National Science Fund for Outstanding Youth Overseas (No. 40328001) and the Key Research Plan of the Knowledge Innovation Project of the Institute of Geographic Sciences and Natural Resources, Chinese Academy of Sciences (No. KZCXI-SW-01) 2004-10-07 2005-02-28 2005-05-22 2022-01-05 Copyright © 2005, Journal of Zhejiang University Science 2005 Journal of Zhejiang University-SCIENCE B (Biomedicine and Biotechnology) 06 6B