SVM (support vector machine) with libsvm

1 min read

I am learning svm lately and tried libsvm. It’s a good package.

Linear kernel example (support vectors are in circles):

Linear
Linear

Nonlinear example (radial basis)

Nonlinear, circle
Nonlinear, circle
Nonlinear, two circles
Nonlinear, two circles
Nonlinear, quadrant
Nonlinear, quadrant

3-class example

Linear, 3 classes
Linear, 3 classes

Basic procedure to use libsvm:

  1. Preprocess your data. This including normalization (make all values between 0 and 1) and transform non-numeric values to numeric. You can use the following code to normalize (from libsvm webpage):
    (data - repmat(min(data,[],1),size(data,1),1))*spdiags(1./(max(data,[],1)-min(data,[],1))',0,size(data,2),size(data,2))
  2. Find optimal parameter values. For linear kernel, you have 1 parameter C (penalize parameter). For commonly used radial kernel, you have two parameters (C and gamma). Different parameter values will yield different accuracy rate. To avoid over fitting, you use n-fold cross validation. For example, a 5-fold cross validation is to use 4/5 of the data to train the svm model and the rest 1/5 to test. The option -c, -g, and -v controls parameter C, gamma and n-fold cross validation. A piece of code from libsvm website is:
    bestcv = 0;
    for log2c = -1:3,
      for log2g = -4:1,
        cmd = ['-v 5 -c ', num2str(2^log2c), ' -g ', num2str(2^log2g)];
        cv = svmtrain(heart_scale_label, heart_scale_inst, cmd);
        if (cv >= bestcv),
          bestcv = cv; bestc = 2^log2c; bestg = 2^log2g;
        end
        fprintf('%g %g %g (best c=%g, g=%g, rate=%g)\n', log2c, log2g, cv, bestc, bestg, bestcv);
      end
    end
  3. You may have to run the above code several times with different range of parameter values to find the optimal values. For example, you might want to start from a bigger range with coarse resolution; then fine tune to smaller regions with higher resolution.
  4. After finding the optimal parameter values, use all data to train your model with your optimal parameter values.
    cmd = ['-t 2 -c ', num2str(bestc), ' -g ', num2str(bestg)];
    model = svmtrain(l, d, cmd);
  5. If you have new data, you may use this model to classify the new data.
    [predicted_label, accuracy, decision_values] = svmpredict(zeros(size(dd,1),1), dd, model);

Commonly used options

  • -v n: n-fold cross validation
  • -t 0: linear kernel
  • -t 2: radial basis (default)
  • -s 0: SVC type = C-SVC
  • -C: C parameter value, default 1
  • -g: gamma parameter value

libsvm performance

I tested on different data size and record the time spent (in second).

Computer: Processor: 2×2.66G, memory: 12G, OS: Windows XP installed in VMWare in Mac OS 10.5

data size    # features    svmtrain    svmpredict
100    2    0.00    0.00
100    6    0.00    0.00
100    10    0.00    0.00
100    20    0.00    0.00
100    50    0.01    0.00
100    100    0.02    0.01
500    2    0.02    0.01
500    6    0.03    0.02
500    10    0.05    0.03
500    20    0.08    0.03
500    50    0.46    0.07
500    100    0.56    0.12
1000    2    0.07    0.04
1000    6    0.10    0.06
1000    10    0.15    0.10
1000    20    0.36    0.14
1000    50    1.09    0.30
1000    100    3.07    0.50

It’s fairly fast.

Resources:

MatLab code to generate the plots above:cuixu_test_svm1

SVM basics: http://en.wikipedia.org/wiki/Support_vector_machine

Download libsvm for matlab at: http://www.csie.ntu.edu.tw/~cjlin/libsvm/#matlab

The meaning of libsvm output is at: http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f804

采用基于频率簇(Cluster)的置换检验(Permutation)方法选取感兴趣频段

作者:北京师范大学 龙宇航,longyuhangwork@163.com代码来源(见本页底部):周思远 在使用wtc计算脑间神经同步后,我们需要在多个频率段、多个通道组合上对神经同步值进行统计检验,因
Xu Cui
1 min read

Calculate phase difference between two general signals (e.g. HbO…

In a recent fNIRS journal club (vedio recorded here), Dr. Tong talked about their work on the phase difference between oxy and deoxy Hb, and its relationship with participants’ age. This article is a demo of how to use Hilbert transform to calc
Xu Cui
1 min read

Functional Near Infrared Spectroscopy (fNIRS): List of manufactures

Contents Company Products Artinis Brite, OxyMon, OctaMon, PortaMon, PortaLite, BIOPAC fNIR 100, fNIR
Xiaodan Feng
11 min read

58 Replies to “SVM (support vector machine) with libsvm”

  1. Hi Xu Cui,

    I am working also with LibSVM and I am very interested in your work 🙂
    I have some questions and therefore it would be great if we can discuss it via e-mail.

    Please write me your e-mail-address.

    Many thanks in advance!
    Best regards
    Florian

  2. Hi Xu Cui,
    I am working with libsvm newely and I want to write a simple c++ program for one againts rest svm.I will be happy if you help me.
    thanks,
    erol

  3. Hi Xu Cui,

    I have estimated SVM parameter thrugh some of the papers using model selection criteria, now I want to test my prediction model using cross-validation as it gives generalized error. will In svmtrain option if I give -v 5 my code shall work, kindly let me know. I’m using MATLAB version n quite poor in C++, so help me out

  4. Hi Xu Cui,

    Your SVM plotting code for “Linear – 3 classes” only seems to plot a 2-dimensional vector. If I had a matrix of 3 or 4-dimensional vectors, how would the code be modified so that it can represent a +2-dimensional vector accurately in the output plot?

  5. Dear Xu Cui,
    I found the Matlab code for LIBSVM is very helpful. However, I would like to comment on one point you mentioned about the testing data. In your code, in the linear part, you claimed that half of the data is for training and the other half for testing. I can’t see that reflected in the published code. As a matter of fact the testing data you use is only coming from the use of “meshgrid” and it does not resemble half of the data.

    If you have another version of the code then that will be very good.

    Thanks

    Raied

  6. Dear Xu Cui,
    I have a very interesting and weird question about SVM. When I use a different set (9 dimensions) of image and a reference image that doesn’t belong to it, the result of poly -SVM is giving %51 percent of accuracy.
    How does this possible? Is taht mean I am doing smthg wrong or Lib-SVM is doing wrong.I can send the code if u want.
    Thanks

  7. @serra
    I wonder why is it not possible to get 51%. Yes, code would be helpful.

    BTW, what is poly-SVM?

  8. thanks for your effort, but I get an error saying;
    ”??? Error using ==> svmtrain at 172
    Group must be a vector.”
    in the cv = svmtrain(l, d, cmd); line,
    and when i want to drow SVs it draws only 2 circles in irrelevant places? what is wrong i didn’t understand?
    I need help, tahnk you in advace
    Regards,

  9. dear Xu Cui
    i have to use LibSvm for a multiclass classification so actually i’m using libsvm 3.0 for matlab. i want to know if the code above ( which generate the plots) support a multiclass classification?? if not how can i change it to support a multiclass classification
    Thank you
    yours

  10. could you give me some example for libsvm matlab for classification and espicially regression?
    i really need matlab mfile of regression using libsvm

    thanks

  11. Hello there, You’ve done a fantastic job. I will definitely digg it and personally suggest to my friends. I am confident they’ll be benefited from this website.

  12. Hi Xu Cui

    I used your SVM code (RBF) for my data. I have two classes, coded 0.1 and 0.9
    I scaled the data in a rank [0.1 0.9], 10 variables and 1368 observations

    Accuracy of The train data is good. But the testing data (3080 observation) is 0% or very low

    http://ifile.it/d0svzmt/Data.rar

    I need you help to check.

    Thanks

  13. Hey, nice article. Was a great help. I want to use multi-class SVM (one-vs-rest approach). I have 6 labels and each label has 1000 feature. Can you please guide me in the right direction.
    Thanks

  14. Dear Xu Cui,

    While I was searching for probability estimates for SVR as an output of
    LIBSVM, I have seen your webpage. I would like to ask you if you know what
    can be reason to have empty output evenif I active -b 1 in both svmtrain
    and svmpredict functions? Evenif I created the data randomly, I do not
    know why but the output is empty. Do you have any idea?

  15. @Demir
    Demir,

    Unfortunately I am not sure what happened. I tested on my side and it works. Please note the last output variable of svmpredict is probability.

  16. Hi Xu Cui and other users,

    This is great. Thanks for sharing it. In the last few months has anyone developed any code for 2 class plots bigger than 2D?

    Would be great.

    Also has anyone worked out code for accuracy confidence intervals (such as by bootstrapping?)

    Thanks,

    Tom

  17. i have 6 (labels: 1 2 3 4 5 6) classes and the data dimension is 262 i use libsvm to classify the data labels
    first is this labeling is correct ?
    how i can the classes result from svmpredict model

  18. Dear Xu Cui
    I want to use LIBSVM through MATLAB. I have a MS Visual C++ 2008 compiler.But the LIBSVM package is not opening in MATLAB. Please help.

  19. I get an error saying;
    ”??? Error using ==> svmtrain at 172
    Group must be a vector.”
    in the cv = svmtrain(l, d, cmd); line,
    what’s wrong? i’m trying to understand this svm code.
    Regards,

  20. Hi Xu,

    I’m pretty new in Libsvm (Matlab interface) and SVM in general.I was able to write a little code but I’m not sure if it works correctly and I have some other questions, too. Is there a possibility to send an e-mail to you?

    Thank you in advance

  21. Hi,

    I have a question regarding non linear SVM. does the LibSVM package provide the Support Vectors in the higher dimensional space or not? I need to know what is the coordinates of samples when transferred to higher dimension space by SVM.

    Thank you

  22. If I have only one data set how can I determine how much my model is good? Is “non sense” try to predict the model I’ve built on the same dataset that has been use for the training?

  23. Hi Xu,

    How to determine the values of the for loops :
    for log2c = -1:3,
    for log2g = -4:1,

    Thank you

  24. @Suresh Gorakala
    Usually you need a few steps. First step you will try big ranges, e.g. -5:5, then after you find peak, narrow down the range with higher resolution around the peak, and repeat.

  25. Hi
    I want to use libsvm with iris dataset witch have 4 features and 3 classes, I want to do 1 vs 1 and 1 vs all. I find W and b for each 2 class in 1 vs all but I cant plot the result, please help me ?!

  26. Dear Xu
    I want to built an SVR for solar cell modeling where I want to predict the “voltage” in terms of “current” and “temperature”, namely, V=f(I,T). I have worked with neural networks and I know the meaning of training and testing. I used your provided code and faced with some problems. For example, suppose that x_train=[1 2;3 4;5 6] and y_train=[5;6;8] (target). how can I built a SVR model to predict the voltage for the point of x_test=[1.4 3]. please give me the matlab codes.
    I will be very thankful in advance.
    Ali

  27. Hi Xu

    I have some questions regarding svm and I need your help and expertise.

    1) Based on the first image (linear shown above) which is obtained via training/testing data using libsvm, May I know what does the axes of the graph represent?

    2) Is it alright to use a linear kernel for a two-class classfication that has a total of 69 features/dimensions?

    As I am still new to svm, I am not sure if i am asking the right question. I apologise if I am not accurate or specific enough. I sincerely thank you for reading this comment and your help will be deeply appreciated.

    Marcus

  28. @Marcus Fwu
    1) they are the two dimensions of the data.
    2) It depends on the nature of your data. Usually when the # of features is big, non-linear kernel may overfit.

  29. Hello Xu Cui, I need to make a classification of satellite images, I’m use LIBSVM in MATLAB, but I have 2 question:

    1. How can i use a classification “1 vs 1” and “1 vs rest”? ¿what commands can i do use?
    2. In my case I have got 3 classes, How can i use cross validation ?

    I will be waiting for your answer, Thanks a lot, Regards

  30. Hello Sir,
    I have developed svm_model for regression, now i want to test it for a given input’x’. after reading the readme file i got that i should use
    Function: double svm_predict(const struct svm_model *model,
    const struct svm_node *x);
    but i do not know how to use? please help so that i can use it in matlab.
    thanks and regards
    Vishal mishra

  31. hello sir,
    i am using libsvm for the classification of human actions. i have selected to actions and want to classify those actions so it is a binary classification.
    i am facing the problem that libsvm shows 100% accuracy but the decision values are constant and hyperplane is not shown on graph
    it gives me the warning.
    Warning: Contour not rendered for constant ZData

    please help me to continue my work

    Thanks

  32. hello
    I would like to ask you what are the best svmtrain parameters for phoneme recognition
    thank you for your reply

  33. Hello XU Cui!

    Can you please guide me how to create test and train files for libsvm use I have image data set, do not know how to divide it into test and train sets?

  34. Hello Xu Cui,

    I really dont have a question but I just wanna say thank you. you made a great explanation and very helpful. I couldnt find any perfect examples as yours

  35. Hello Xu Cui,

    I have two classes that have 300×200, 300 instances and 200 is the dimensionality of the feature. I hope you help me figure out how to represent this in a two linear class(decision boundary and # support vectors in each class.

    Thank u

  36. Dear Xu Cui,
    I am very confused about one point. Until now I was always using test_label in SVMpredict command,where you have used zeros. Is it wrong what I am doing? I always get 0% accuracy, when used zeros or any double values.
    I checked read_me file it also says if you dont have test labels you can use any double value. But when I use different values I get 0 % accuracy.
    Thanks a lot.
    Regards,

  37. Thank you so much for this priceless information. I was wondering why do I get zero accuracy when I used the half of the training data in testing with zero labels as new data? I would be more than happy if you could kindly reply to my massage. Thank you

  38. I just realized, I have asked the same question twice, I am really wondering why I get the same problem and is there a particular reason why it is happening:( thank you in advance.

  39. Dear Xu Cui,
    Can you please guide for inputting satellite images into matlab for libsvm usage

  40. In my classification problem, I have 6 classes. When I use libsvm and use predict function, it will give 15 decision values for each input. From these values how can I decide predict label class?
    Also, is these values representing the values of decision funtion?

  41. Hello Xu Cui,
    I have learned cross-validation to learn optimal values for c and gamma for radial basis function SVM using LIBSVM and the code available at LIBSVM site. But if i want to use the same code to obtain value of only C for a linear SVM it does not work.Some matrix dimension mismatch is the error message. Can you help in this regards. Tnx.

  42. @Xu Cui

    How to determine the values of the for loops :
    for log2c = -1:3,
    for log2g = -4:1,

    How to know the optimum range for the value of c and g?
    when i tried with different range i get different accuracy and different cross validation accuracy.(low cross validation like 52.4 )but accuracy 90 and sometime even 100% . how to know whether i have overfitting in my data?

    thanks in advance

  43. Hello
    your code is really helpful.
    Just a question, do you know how to draw a line as a decision boundary among the support vectors?
    Regards
    Amin

  44. Please, I need your help. How can I construct an routines in Octave, to give k random groups automatically? For example, I have n = 161, k = 5 and I want to construct an routines to give 5 random group.

    Jailson Lopes

  45. Dear Xu Cui,I am using epsilon SVR to my research problem. I scaled the input and output both the variable in between 0 to 1. Now i wanted to use grid search optimization technique to get the optimized value of cost (c), radial basis(gamma) and the margin(epsilon)parameter. Literature suggested that, the search range for C is in between 0 to 32768. Could you please suggest me for the scaled dependent data also, this range will works. Whether C is any dependency over the range of dependent variable or not. Or i should change the search space from 0 to 3 for the scaled dependent data.

Leave a Reply

Your email address will not be published. Required fields are marked *