How to do comparisons between machine learning schemes

Nice paper comparing 16 model selection and weighting schemes. Includes 58 benchmark datasets. The data analysis was done in the following way - for each dataset, rank the schemes. Then average the ranks. - use the Friedman test to test whether ranks are all equal ( - if ranks are not all equal, use the Nemenyi test (covered in papers by Demsar, Garcia et al  http://jmlr.csail.mit.edu/papers/volume9/garcia08a/ Ying Yang, Geoffrey I. Webb, Jesús Cerquides, Kevin B. Korb, Janice R. Boughton, Kai Ming Ting: To Select or To Weigh: A Comparative Study of Linear Combination Schemes for SuperParent-One-Dependence Estimators. IEEE Trans. Knowl. Data Eng. 19(12): 1652-1665 (2007), ISSN: 1041-4347 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.134.8561