Resources

Extract, Transform & Load

The first, and probably the most labour intensive part of the data science process is to prepare and structure datasets to facilitate analysis, specifically importing, tidying and cleaning data. Hadley Wickham wrote about it in The Journal of Statistical Software, vol. 59, 2014.

Transform

The result of the transform step in the data science process is to:

• reshape data, which could be used to produce tidy data,
• transform data, like rescaling⁵ of numeric values, or reducing dimensions of categorical values using Principle Component Analysis
• create new features, also known as ‘Feature Engineering’,
• or a combination of the above that typically results in aggregation.
  
  

Visualisation & Modelling

Visualisation and Modelling is used to detect patterns in data.

In Feburary 2013 Hadley Wickham gave a talk where he described the interaction between visualisation and modelling very well.

Visualisation can surprise you, but it doesn’t scale well. Modelling scales well, but it can’t surprise you.

Visualisation can show you something in your data that you didn’t expect. But some things are hard to see, and visualisation is a slow, human process.

Modelling might tell you something slightly unexpected, but your choice of model restricts what you’re going to find once you’ve fit it.

So you iterate. Visualisation suggests a model, and then you use your model to factor out some feature of the data. Then you visualise again.

Communicate

The result and product of the data science process is to communicate findings, typically to non-technical and busy recipients. It is the most important deliverable of the process, even if not the first thing that springs to mind when considering data science. Fantastic insights are of no use if the intended audience doesn’t understand or trust it. It is therefore vital to take care when presenting findings.

Visualisation is a significant deliverable.

Visualisation is an essential ingredient of communication. It summarises and simplifies information, with the following guidelines applying:

• Intuitive and straightforward with a low learning curve (KISS)
• Focus on the salient points, keeping noise | clutter to a minimum
• Content is explicit and not implicit
• Promoting shared understanding, including producing underlying information supporting layered and increasing levels of detail and complexity to align operations and various levels of management
  
  

Domain Knowledge

It is challenging to deliver good analytics and associated visualisations without domain knowledge or a procedure that extracts and distils required information instead.

In practice, clients often have very little time or cannot provide a decent brief (unconscious competence for example), leading to a thin and vague specification. Domain knowledge allows one to anticipate and prepare for potential questions in advance, especially if the brief is inadequate.

---

Statistics and Mathematics are used to solve a variety of problems. The aspects listed below exposes a few uses but are in no means exhaustive.

Statistics

The most widely used are Descriptive Statistics, which simplifies and quantitatively describes or summarises features of a collection of information, including central tendency, spread and shape.

Statistical inference assumes an underlying probability distribution that best describes an observed population, enabling one to deduce properties from it.

Probability

Probability underpins statistics and deals with the likelihood of an event occurring.

Mathematics