The *learning tags

Question

We have no less than 13 tags containing "learning". Some of these are certainly fine, e.g. machine-learning, deep-learning, unsupervised-learning, supervised-learning, reinforcement-learning, manifold-learning, active-learning or multitask-learning. My issues are with the following one:

• learning looks completely vacuous to me. Sometimes another of the *learning tags would be far better, and sometimes it seems to be used as a synonym for references. We currently have 63 questions tagged learning. I'm afraid that if we simply remove the tag, it will pop up again. Should it be burninated?

• statistical-learning: the wiki excerpt says that "SL refers to the statistical perspective on machine learning." To be honest, I am less than convinced. Shouldn't all questions on CrossValidated that carry the machine-learning tag have at least some "statistical perspective"? Then again, the posters of 133 questions seem to disagree with me. Thoughts, anyone?

• q-learning (25 questions), transfer-learning (3 questions), representation-learning (3 questions): all these have no tag wiki. Can someone create one?

Answer 1

1. The learning tag is in the process of being removed, see the answer in Understanding the use of the [education] tag.

2. I wrote wiki excerpts to q-learning and transfer-learning. The former is IMHO a valid & useful tag, about the latter I am not sure as it only has 5 threads, but the concept seems clear enough so I'd say let it be.

3. I removed [representation-learning] from all threads as too vague and unspecific.

4. statistical-learning should become a synonym of machine-learning, see also the discussion in the comments above where everybody agrees with that.

   Update: The synonym has been created, thanks to @Scortchi. Case closed.

Answer 2

EDIT BELOW (August 25th, 2017)

I concur with what @amoeba proposed, except point 4.

4. statistical-learning should be merged into machine-learning, see also the discussion in the comments above where everybody agrees with that.

---

@amoeba changed point 4, so now we are in agreement

4. statistical-learning should become a synonym of machine-learning, see also the discussion in the comments above where everybody agrees with that.

---

@gung had already said in the comments

Making [statistical-learning] a synonym of [machine-learning] seems like a good idea. I think in principle SL could be a valid tag, but it's very unlikely to work out--it'll just create more fragmentation instead. – gung Jan 5 at 13:22

For now I'm against the merging and in favor of proposing as synonym. But I can't still pinpoint if SL and ML warrant two tags. As @gung commented in this answer perhaps this warrants a separate question as well.

Below, I collected some evidence SL might not be simply ML.

---

Alright, found a somewhat (blurry) contrast between statistical-learning and machine-learning. While I was searching for it in Elements, it was actually in An Introduction.

Right at Chapter 1 - Introduction ^{(emphasis mine)}:

Statistical learning refers to a set of tools for modeling and understanding complex datasets. It is a recently developed area in statistics and blends with parallel developments in computer science and, in particular, machine learning. The field encompasses many methods such as the lasso and sparse regression, classification and regression trees, and boosting and support vector machines.

In "A Brief History of Statistical Learning" (sorry, long quote)

By the end of the 1970s, many more techniques for learning from data were available. However, they were almost exclusively linear methods, be-cause fitting non-linear relationships was computationally infeasible at the time. By the 1980s, computing technology had finally improved sufficiently that non-linear methods were no longer computationally prohibitive. In mid 1980s Breiman, Friedman, Olshen and Stone introduced classification and regression trees, and were among the first to demonstrate the power of a detailed practical implementation of a method, including cross-validation for model selection. Hastie and Tibshirani coined the term generalized additive models in 1986 for a class of non-linear extensions to generalized linear models, and also provided a practical software implementation.

Since that time, inspired by the advent of machine learning and other disciplines, statistical learning has emerged as a new subfield in statistics, focused on supervised and unsupervised modeling and prediction. In recent years, progress in statistical learning has been marked by the increasing availability of powerful and relatively user-friendly software, such as the popular and freely available R system. This has the potential to continue the transformation of the field from a set of techniques used and developed by statisticians and computer scientists to an essential toolkit for a much broader community.

In Chapter 2 - Statistical Learning there's also some definition of the term.

Following the next chapters, you also have staples in Machine Learning: Linear Regression, Classification (LR, LDA, QDA, KNN), Resampling, Linear model selection and Regularization (subset selection, shrinkage, PCR, PLS), Non-linear regression (regression splines, GAM), Trees (CART, Random Forest), Support Vector Machines (SVC, SVR), Unsupervised Learning.

Sadly, nowhere SL and ML are directly compared one against the other.

I'd like to foment some discussion on the term, because it's not sure in what does it deviate from machine learning, if in anything at all.

*Now I'm under the impression it's a synonym (i.e. the ML framework under the statistics culture and jargon), but why not use the more vendible term then? Though in the scientific literature SL is a really popular term.

Edit:

Perphaps the difference is simply cultural, like many discussions in the main site pointed. Consider Stanford, where two courses are taught: Stats 315a/315b - Statistical Learning and CS 229 - Machine Learning. Apart from being named different and being in different concentrations areas, they also attract different students.

Tibshirani even shares his views in his page comparing both courses and then both terms:

Machine learning research focusses more on low noise situations, eg engineering applications like robotics and physical sciences

Statistical learning focusses more on high noise, observational data like medicine and genomics, and problems where interpretation of the fitted model is important

But more and more overlap in application areas!

---

EDIT:

I've come to the conclusion Statistical Learning is the application of Learning algorithms to classical statistical problems (I think the small phrase at the Machine Learning article in Wiki and the ISLR book description corroborate this notion). The distinction to Machine Learning is better shown with examples:

• Machine Learning is concerned with optimizing generalized predictive power. So the focus is mostly on loss functions. Eg.: Studies trying to predict if a person has Alzheimer from neuroimaging, thus producing biomarkers of Alzheimer, but not focused on the biological meaning on the features, just on performance.

• Statistical Learning on the other hand wants to make inference over this scenario. Eg.: "How do learning algorithms trained to predict the biological aging from neuroimages of healthy people perform on the presence of Alzheimer? Why?"

Another possible scenario for Statistical Learning is predicting states, such as task paradigms, from neuroimaging using linear models with shrinkage, such as SVMs, producing interpretable weight maps. Yet another scenario is in the introduction of a new imaging technique, where Statistical Learning can help the scientific community to uncover if said technique improves the diagnosis of a disorder.

*I'm mostly talking about neuroimaging because that's my area of expertise.

Said all that, I'm of the opinion the tag would be mostly useless here on CV, and wouldn't be used for it's true meaning.

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 6). New York: Springer.