~~CLOSETOC~~ 
|<100% 25% - >|
^  \\ DATA ANALYTICS REFERENCE DOCUMENT\\ \\   ^^
^  Document Title:|52465 - Programming for Data Analytics module summary|
^  Document No.:|1540154135|
^  Author(s):| |
^  Contributor(s):|Rita Raher, Gerhard van der Linde, Gearoid O'Gcrannaoil |



**REVISION HISTORY**
|< 100% 10% - - 10% 17% 10% >|
^  \\  Revision\\  \\  ^\\ Details of Modification(s)^\\ Reason for modification^  \\ Date  ^  \\ By  ^
|  [[:doku.php?id=module:52465&do=revisions|0]]  |Draft release|Create a printable reference documented summary of module highlights|  2018/10/21 20:35  |  Gearoid O'Gcrannaoil  |


----

====== 52465 - Programming for Data Analysis ======

===== Learning outcomes =====

On completion of this module((https://learnonline.gmit.ie/course/view.php?id=515)) the learner will/should be able to:
  - Perform exploratory analysis on data.
  - Programmatically create plots and other visual outputs from data.
  - Design computer algorithms to solve numerical problems.
  - Create software that incorporates and utilises standard numerical libraries.
  - Employ appropriate data structures when programming for data-intensive applications.
  - Model real-world, data-intensive problems as computing problems.

===== Indicative syllabus =====

The following is a list of topics that will likely be covered in this module. Note that the topics might not be presented in this order and might not be explicitly referenced in course materials.
==== Data ====
Two-dimensional arrays, matrices, data frames, time series data structures, dictionaries, sets, vectors, slicing, indexing
==== Programming ====
Reshaping data structures, unzipping arrays, slicing, calculating descriptive statistics.
==== Analytics ====
Exploratory data analysis, scatterplots, histograms, boxplots, principal component analysis.

===== Python review =====

{{:wiki:python.png?nolink&200|}}

  * A review of how to get set up with a repository for starting a new software project.
  * A review of if statements in Python.
  * A review of while loops in Python.
  * A review of for loops in Python.
  * A review of functions in Python.

===== Plotting basics (matplotlib) =====

{{:wiki:matplotlib.png?nolink&200|}}

This week we will look at the pyplot plotting library for Python.

  * An introduction to matplotlib with pyplot.
  * Creating a simple plot with pyplot.((https://matplotlib.org/tutorials/introductory/pyplot.html#sphx-glr-tutorials-introductory-pyplot-py))
  * Plotting two plots on one set of axes.
  * Adding extras to plots.
  * Creating histograms in pyplot.
  * Creating two plots side by side.
  * Having fun with pyplot.
  * Repository containing examples((https://github.com/ianmcloughlin/pyplot-examples)).

===== Browser workflows (jupyter) =====

{{:wiki:jupyter.png?nolink&200|}}

This week we will look at the jupyter package for creating visual workflows to tell a data analytics story.\\

  * Starting jupyter
  * Renaming notebooks
  * Cells in jupyter
  * Jupyter keyboard shortcuts
  * Code and markdown cells in jupyter
  * Jupyter kernel
  * Plotting in jupyter
  * Jupyter lab
  * A gallery of interesting Jupyter Notebooks((https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks))
===== Generating random data (numpy) =====

{{:wiki:numpy.png?nolink&200|}}

This week we will look at the numpy.random package for generating random data in Python.

  * An introduction to the numpy.random package.((https://docs.scipy.org/doc/numpy-1.15.1/reference/routines.random.html))
  * An introduction to the numpy package.((https://docs.scipy.org/doc/numpy-1.15.1/user/whatisnumpy.html))  ((https://docs.scipy.org/doc/numpy-1.15.1/user/quickstart.html))
  * Getting started with a numpy.random notebook.
  * The numpy.random documentation.
  * About the rand function in the numpy.random package.
  * Figuring out what the numpy.random [[help:statistics:distributions|Distributions]]((https://docs.scipy.org/doc/numpy-1.15.1/reference/routines.random.html#distributions)) ((http://mathworld.wolfram.com/UniformDistribution.html))functions do. FIXME
  * Seeds in the numpy.random package.((https://docs.scipy.org/doc/numpy-1.15.1/reference/routines.random.html#random-generator))

{{:modules:42465:random_seeds_pi_example.png?nolink&400|}}
**//An example of a random "seed//"**

===== Exploring data sets (pandas) =====

{{:wiki:pandas.png?nolink&300|}}

  * Introduction to pandas 
  * Introduction to pandas notebook((http://pandas.pydata.org/))
  * Reading CSV files in pandas
  * <color #ff7f27>Using loc and iloc</color>
  * <color #ff7f27>Boolean selects</color>
  * Summarising datasets with pandas and seaborn
  * [[https://github.com/ianmcloughlin/jupyter-teaching-notebooks/raw/master/pandas-with-iris.ipynb|Pandas and iris dataset notebook]] (right click and 'save as')
  * [[https://nbviewer.jupyter.org/github/ianmcloughlin/jupyter-teaching-notebooks/blob/master/pandas-with-iris.ipynb|View in NBViewer]]

==== Highligts from the series ====

Use pandas to read as csv file into a dataframe and run some basic analytics on it.
<code python pandas_samples.py>
import pandas as pd
df = pd.read_csv("https://raw.githubusercontent.com/uiuc-cse/data-fa14/gh-pages/data/iris.csv")

# show the data type in the dataframe and the names of the columns imported if any
df.dtypes
#show the headings with data
df.head()
# a list of columns
df.columns
# a list of statistics per column
df.describe()
# the mean or median only
df.mean()
df.median()
</code>

and show it all in graphs using Seabborn and Pandas

<code python seaborn_samples.py>
import seaborn as sns
import matplotlib.pyplot as pl
sns.pairplot(df, hue='species')
pl.show()
</code>


===== Machine learning (sklearn) =====

{{:modules:320px-scikit_learn_logo_small.svg.png?400|}}

This week we will look at the sklearn package for machine learning in Python.

  * Intro to sklearn. Machine learning in Python - a very Powerful package((http://scikit-learn.org/stable/)) 
  * Build up a case for a machine learning - visualise the data in a plot before deciding to use K-nearest neighbours. We look at how to split a data set into inputs and outputs for supervised learning.
  * How to train a knn classifier in sklearn. 

<code python>
# classifier - takes 5 nearest neighbours
knn = nei.KNeighborsClassifier(n_neighbors=5)

# fit
knn.fit (inputs, outputs)

</code>

  * How to make predictions from a trained classifier.
<code python>
knn.predict([[5.1, 3.5, 1.4, 0.2]])
</code>
  * How to test a classifier. Even on the data values you trained the classifiers, if might not get them correct.
<code>
# To Evaluate, you can check and compare to the dataframe using:
knn.predict(inputs) == outputs

# converts all the trues to 1s and false to 0 and adds them
(knn.predict(inputs) == outputs).sum()

#Split arrays or matrices into random train and test subsets

import sklearn.model_selection as mod

inputs_train, inputs_test, outputs_train, outputs_test = mod.train_test_split(inputs, outputs, test_size=0.33)
</code>

  * Notebook [[https://raw.githubusercontent.com/ianmcloughlin/jupyter-teaching-notebooks/master/knn-iris.ipynb]]
  * NBViewer [[https://nbviewer.jupyter.org/github/ianmcloughlin/jupyter-teaching-notebooks/blob/master/knn-iris.ipynb]]


**References:**
[[http://scikit-learn.org/]]

**Tutorials**
[[http://scikit-learn.org/stable/tutorial/basic/tutorial.html
]]

**Other Machine Learning frameworks**
[[https://www.tensorflow.org/]]

[[http://deeplearning.net/software/theano/]]


===== Exploring time series (yet more pandas) =====

{{:modules:wide_data_lineplot.png?nolink|}}

<code python time-series.py>
import pandas as pd
df = pd.read_csv("http://cli.met.ie/cli/climate_data/webdata/hly4935.csv", skiprows=23, low_memory=False, nrows=1000)
# Change the date column to a Pythonic datetime.
df['datetime'] = pd.to_datetime(df['date'])
</code>

  * [[https://github.com/ianmcloughlin/jupyter-teaching-notebooks/raw/master/time-series.ipynb]]
  * [[https://nbviewer.jupyter.org/github/ianmcloughlin/jupyter-teaching-notebooks/blob/master/time-series.ipynb|View in NBViewer]]

===== Statistical bias (numpy.random) =====

{{:modules:640px-normal_distribution_pdf.svg.png?nolink|}}

==== Bias ====

A statistic is biased if, in the long run, it consistently over or underestimates the parameter it is estimating. More technically it is biased if its expected value is not equal to the parameter. A stop watch that is a little bit fast gives biased estimates of elapsed time. Bias in this sense is different from the notion of a biased sample. A statistic is positively biased if it tends to overestimate the parameter; a statistic is negatively biased if it tends to underestimate the parameter. An unbiased statistic is not necessarily an accurate statistic. If a statistic is sometimes much too high and sometimes much too low, it can still be unbiased. It would be very imprecise, however. A slightly biased statistic that systematically results in very small overestimates of a parameter could be quite efficient. ((http://localhost:8888/notebooks/52446_playground/bias.ipynb))

{{:modules:untitled.png?nolink|}}


===== Databases(sqlite) =====

{{page>modules:52446#databases_sqlite3&noheader}} 

====== IPython ======

{{:modules:ipy_header.png?400|}}


https://ipython.org/

closely linKed to jupyter. Ipython runs on the command line. ipython comes by default with the anaconda distribution. use the up arrows to navigate previous commands. It works as a python interpreter. You can import the usual packages. 

Run ipython in the command line

<code python>
import numpy as np
np.arange(0.0, 10.0, 0.1)
</code>

**python: python statements**

<code python>
import numpy as np
np.linspace(0.0, 10.0, 100)

import matplotlib.pyplot as plt

plt.plot(np.linspace(0.0, 10.0, 100)
plt.show()
</code>

and a new window shows up! code the window to get a new prompt from ipython

It remembers all the variables you have created.
type “exit” to get out of ipython


===== ipython: %paste =====


paste magin command

https://ipython.readthedocs.io/en/stable/interactive/magics.html

<code python>
for i in range(10):
	print(i, i**2)
</code>

Paste in python code from python tutorials and ipython will run it 
https://docs.python.org/3/tutorial/controlflow.html
<code python>
>>> for n in range(2, 10):
...     for x in range(2, n):
...         if n % x == 0:
...             print(n, 'equals', x, '*', n//x)
...             break
...     else:
...         # loop fell through without finding a factor
...         print(n, 'is a prime number’)
</code>
**Best practice** is to copy the code and then type %paste into the command line and this will run the script
%paste

===== ipython: %timeit =====

<code python>
%timeit [i**2 for i in range(100)]
</code>
30.4 µs ± 668 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

<code python>
%timeit np.arange(100)**2
</code>
1.17 µs ± 13.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

===== ipython: %run =====

  * %pwd - present working directory
  * %cd - change directory
  * %cd Desktop/
  * %ls - lists files
  * %cls - clears everything
  * %run filename.py
  * %reset - removes variables