How to do comparisons between machine learning schemes

Nice paper comparing 16 model selection and weighting schemes. Includes 58 benchmark datasets. The data analysis was done in the following way - for each dataset, rank the schemes. Then average the ranks. - use the Friedman test to test whether ranks are all equal ( - if ranks are not all equal, use the Nemenyi test (covered in papers by Demsar, Garcia et al  http://jmlr.csail.mit.edu/papers/volume9/garcia08a/ Ying Yang, Geoffrey I. Webb, Jesús Cerquides, Kevin B. Korb, Janice R. Boughton, Kai Ming Ting: To Select or To Weigh: A Comparative Study of Linear Combination Schemes for SuperParent-One-Dependence Estimators. IEEE Trans. Knowl. Data Eng. 19(12): 1652-1665 (2007), ISSN: 1041-4347 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.134.8561

What probability is about

Just as formal logic is a tool for capturing intuitions about effective ways of reasoning probability is a tool for capturing effective ways of reasoning about uncertainty.  In logic, we are taught to proceed from premises to conclusions. We conclude  that "If every CSE graduate takes the exit exam  before graduating and Emily is a CSE graduate, then Emily must have taken the exam" because this matches the Modus ponens pattern .  The good thing about this is that if we trust Modus ponens and we believe that we have correctly mapped the English description of the conclusion onto the terms of the formula, we know that the argument is valid. Notice that this claim of validity is not affected by the truth or falsity of the premises. Even if Emily never was a CSE student, or even if some CSE students are in fact able to weasel out of taking the exit exam, the argument is still OK. Not, however, especially useful as a way of working out stuff about Emily: if we aren't sure about the assumptions, we can't be sure about the conclusion either. Aristotle hit on the idea of laying out a taxonomy of plausible ways of reasoning, then thinking, in the abstract, about which of these patterns of reasoning actually deliver valid arguments. This is a wonderful idea, because you can rely on the insights about the patterns of reasoning without having to be concerned about the truth or falsity of particular facts.

In the same way that logic is the study of patterns of true/false reasoning, probability is the study of patterns of reasoning in the face of uncertainty.

Why statisticians shouldn't write movie titles

Never Give a Sucker an Asymptotically Even Flip
The Variational Enigma
The Fisher King
Independence Day
Return to Monte Casino
The Man Who Measured the Bank at Monte Carlo
Between 99 and 103 Dalmatians
8.5 +/- 0.2
The Metropolis Method (void where prohibited by law)
Improper Priors go wild on Cancun