Python Machine Learning Course

The image above of a picture of a wave with the style of the The Great Wave off Kanagawa by Katsushika Hokusai imposed on it is an example of the power of machine learning. It is possible to use machine learning to do tasks like this without knowing how it works. You can do tons of cool stuff with it all day long by treating it like an appliance. But, in my opinion it is interesting and worth examining how it actually works by starting from the very basics.

Preface: Machine Learning from the basics

This course is inspired by other great resources. The flavor of the course has been influenced by Grant Sanderson from Three Blue One Brown, Python Crash Course book by Eric Matthes, No Starch Press and the online book “Neural Networks and Deep Learning” by Micheal A. Neilsen, Determination Press, 2015

The idea in the course is to approach machine learning (ML) from the basics as if a time machine could transport the reader back to a time before machine learning existed. How would one stumble upon the basic principles of it.

This idea is not far fetched as I have been programming long enough to have had brushes with the basics of machine learning without even knowing what it was. The idea here is to go back to what in Buddhism is referred to as beginners mind. Forget all that you have seen, heard and read about machine learning and start from scratch.

The idea behind this came from my initial exploration into machine learning, the modern form of it at least. Trying to dig into it and put it to use without really understanding it and jumping into texts that either moved to far ahead too fast or were quite frankly over my head at the time gave me a sort of half understanding of it. Videos online although helpful, did not seem to be deep enough on the basics to solidify my understanding.

So I took a page figuratively out of the Python Crash Course book and decided to start at the very beginning. The Python Crash Course book assumes no knowledge of Python or programming for that matter. In this case I assumed knowledge of Python but no knowledge of machine learning and started with a thought experiment…

What is the most basic thing that can be done that could be considered machine learning? What is the elemental particle of machine learning to put it in physics parlance?

I could have said Perceptrons are the most basic elemental particle but, from my point of view and experience years ago early in life I decided to start with solvers. A solver being defined as an algorithm that can solve a simple mathematical equation such as y = mx which is about as simple as it can get. This is inspired by a memory from when I was young and learned Newton’s method for solving roots. It could be done easily by iterating through inputs on a calculator and you can come up with the square root of a number fairly quickly.

Solvers

From the standpoint of a basic solver it is possible to show a few different types from the brute force search, to a step ahead and jump back model, a split-search and finally warm up to something that looks a lot like backpropagation. All of the this will be linear and in some cases proportional to the error. Finally this can be extended to linear regression with backpropagation both single variable and multivariate in a later section (4).

Perceptrons

On a second but related path perceptrons can be looked at. Long before I knew what a perceptron was, I remember using something a lot like a perceptron with some measure of random input to control the movement of characters in video games that I was coding at the time. It was a natural outcome of the problem at hand and not a far reach to see that someone would have natural discovered/invented the perceptron at some point in the history of computer science. It is one of those basic building blocks that seems obvious in hindsight. The trick in machine learning is the barrier that was crossed in the past by using layers of perceptrons with the important element of a non-linear activation function that has a continuous derivative. In some ways this at least from the theoretical point of view is the threshold that is crossed when entering modern machine learning with multiple layers that can solve problems beyond the linear domain. The device that sits on the threshold that will be examined is the XOR gate which cannot be created in a one layer perceptron, normally. I say normally for as we will see in this course the XOR problem can be solved using a single layer in conjunction with one added feature. This feature engineering concept can be handy to keep in mind when designing a system, sometimes adding a feature can cause the design to remain relatively simple. Such as solving the XOR linearly and with one layer.

Self Learning Perceptron

What do you get when you fuse what has been learned about solvers with a regular pretuned perceptron? A perceptron that self learns. All of the logic gates will be revisted using a self learning approach for perceptrons. Activation functions that add a non-linearity beyond using the Heaviside step function will also be explored.

Code

The code is shown inline in each chapter and the entire repository is available on Github at https://github.com/erickclasen/PMLC

Outline/Menu

The outline will follow this ordering:

1. Solvers – Basic Math Via Algorithm
   1. Brute Force
   2. Step Up and Jump Down
   3. Split Search
   4. Linear Proportional Solver – backpropagation
   5. Differential Evolution – to show a different way of doing it
2. Perceptrons – Logic Gate Examples
   1. AND to XOR
   2. Solving XOR with 2 layers and non-linearity
   3. Solving XOR with  feature engineering
3. Self Learning Perceptron, Fusion of Perceptron and Solver.
   1. Gates revisited using self learning via backpropagation
   2. Activation functions and non-linearity.
4. Solver to Linear Regression
   1. Single Variable
   2. Dual Variable
   3. General Multivariate
5. Feedforward Networks
   1. Decimal to Binary 10 in 4 out Single Feed Forward Network
   2. Image Recognition – Simplified Digit Recognition

 

FYI/Disclaimer

This course covers the basics and there are many more resources available that either go into more details or take a different view of machine learning. In this course my attempt was to provide a simple introduction without too much overlap of other resources that are readily available.

Resources

1. TBD – Organized
2. TBD – Mega Dump from Notestack
3. The code for this project is on Github at https://github.com/erickclasen/PMLC