Math in Machine Learning

Linear Algebra

• mathematics of data: multivariate, least square, variance, covariance, PCA
• equotion: y = , where A is a matrix, b is a vector of depency variable
• application in ML
1. Dataset and Data Files
2. Images and Photographs
3. One Hot Encoding: A one hot encoding is a representation of categorical variables as binary vectors. encoded = to_categorical(data)
4. Linear Regression. L1 and L2
5. Regularization
6. Principal Component Analysis. PCA
7. Singular-Value Decomposition. SVD. M=U*S*V
8. Latent Semantic Analysis. LSA typically, we use tf-idf rather than number of terms. Through SVD, we know the different docments with same topic or the different terms with same topic
9. Recommender Systems.
10. Deep Learning

Numpy

1. add a scalar or one dimension matrix to another matrix. where b is broadcated.
2. it oly works when when the shape of each dimension in the arrays are equal or one has the dimension size of 1.
3. The dimensions are considered in reverse order, starting with the trailing dimension;

Matrice

• Vector
1. lower letter. 3. Multiplication, Divsion(Same length) a*b or 4. Dot product: • Vector Norm
1. Defination: the length of vector
2. L1. Manhattan Norm. python: norm(vector, 1) . Keep coeffiencents of model samll
3. L2. Euclidean Norm. python: norm(vector)
4. Max Norm. python: norm(vector, inf)
• Matrices
1. upper letter. 2. Addtion, substruction(same dimension)
3. Multiplication, Divsion( same dimension)
4. Matrix dot product. If , A’s column(n) need to be same size to B’s row(m). python: A.dot(B) or A@B
5. Matrix-Vector dot product. 6. Matrix-Scalar. element-wise multiplication
7. Type of Matrix
1. square matrix. m=n. readily to add, mulitpy, rotate
2. symmetric matrix. 3. triangular matrix. python: tril(vector) or triu(vector) lower tri or upper tri matrix
4. Diagonal matrix. only diagonal line has value, doesnot have to be square matrix. python: diag(vector)
5. identity matrix. Do not change vector when multiply to it. notatoin as python: identity(dimension)
6. orthogonal matrix. Two vectors are orthogonal when dot product is zeor. or . which means the project of to is zero. An orthogonal matrix is a matrix which 8. Matrix Operation
1. Transpose. number of rows and columns filpped. python: A.T
2. Inverse. where python: inv(A)
3. Trace. the sum of the values on the main diagonal of matrix. python: trace(A)
4. Determinant. a square matrix is a scalar representation of the volume of the matrix. It tell the matrix is invertable. or . python: det(A) .
5. Rank. Number of linear indepent row or column(which is less). The number of dimesions spanned by all vectors in the matrix. python: rank(A)
9. Sparse matrix
1. sparsity score = 2. example: word2vector
3. space and time complexity
4. Data and preperation
1. record count of activity: match movie, listen a song, buy a product. It usually be encoded as : one hot, count encoding, TF-IDF
5. Area: NLP, Recomand system, Computer vision with lots of black pixel.
6. Solution to represent sparse matrix. reference
1. Dictionary of keys:  (row, column)-pairs to the value of the elements.
2. List of Lists: stores one list per row, with each entry containing the column index and the value.
3. Coordinate List: a list of (row, column, value) tuples.
4. Compressed Sparse Row: three (one-dimensional) arrays (A, IA, JA).
5. Compressed Sparse Column: same as SCR
7. example
1. covert to sparse matrix python: csr_matrix(dense_matrix)
2. covert to dense matrix python: sparse_matrix.todense()
3. sparsity = 1.0 – count_nonzero(A) / A.size
10. Tensor
1. multidimensional array.
2. algriothm is similar to matrix
3. dot product: python: tensordot()

Factorization

• Matrix Decompositions
1. LU Decomposition
1. square matrix
2. , L is lower triangle matrix, U is upper triangle matrix. P matrix is used to permute the result or return result to the orignal order.
3. python: lu(square_matrix)
2. QR Decomposition
1. n*m matrix
2. where Q a matrix with the size mm, and R is an upper triangle matrix with the size mn.
3. python: qr(matrix)
3. Cholesky Decomposition
1. square symmtric matrix where values are greater than zero
2. , L is lower triangle matrix, U is upper triangle matrix.
3. twice faster than LU decomposition.
4. python: cholesky(matrix)
4. EigenDecomposition
1. eigenvector: , is matrix we want to decomposite, is eigenvector, is eigenvalue(scalar)
2. a matrix could have one eigenvector and eigenvalue for each dimension. So the matrix can be shown as prodcut of eigenvalues and eigenvectors. where Q is the matrix of eigenvectors, is the matrix of eigenvalue. This equotion also mean if we know eigenvalues and eigenvectors we can construct the orignal matrix.
3. python: eig(matrix)
5. SVD(singluar value decomposition)
1. , where A is m*n, U is m*m matrix, is m*m diagonal matrix also known as singluar value, is n*n matrix.
2. python: svd(matrix)
3. reduce dimension
1. select top largest singluar values in 2. , where column select from , row selected from , B is approximate of the orignal matrix A.
3. python: TruncatedSVD(n_components=2)

Stats

• Multivari stats
1. variance: , python: var(vector, ddof=1)
2. standard deviation: , python:std(M, ddof=1, axis=0)
3. covariance: , python: cov(x,y)[0,1]
4. coralation: , normorlized to the value between -1 to 1. python: corrcoef(x,y)[0,1]
5. PCA
1. project high dimensions to subdimesnion
2. steps:
1. 2. 3. 4. 5. , which order by eigenvalue
3. scikit learn

pca = PCA(2) # get two components
pca.fit(A)
print(pca.componnets_) # values
print(pca.explained_variance_) # vectors
B = pca.transform(A) # transform to new matrix
• Linear Regression
1. , where b is coeffcient and unkown
2. linear least squares( similar to MSE) , then . Issue: very slow
3. MSE with SDG

Reference: Basics of Linear Algebra for Machine Learning, jason brownlee, https://machinelearningmastery.com/linear_algebra_for_machine_learning/

Types of Convolution(Translation)

For most of us who learned CNN, we already knew the convolutional operation is used for feature extraction in the spatial relationship. Compared with the full connection NN, it is good for weights sharing and translation invariant. There are many different convolutions. Recently, I found a very good article which summarized this topic. I translated it to English combined with my understanding. If you want to read the original one, you can go here.

1. Standard Convolution

1.1 Single channel It’s element-wise multiply then sum together. The Convolutional filter moves forward each element in the picture. Here we set padding = 0, stride = 1. This is very useful for the gray picture. 1.2 multi channels For the color pictures, they are made of 3 layers: Red, Green and Yellow. we create a 333 convolution which contains 3 convolutional kernels. Then we sum the three results togher to one channel 2D array. First Running! Deep learning to CFD

This is our first apply ConvLSTM to CFD successfully! although the case is simple and under control of lots of factors. The ground factor is generated by Openfoam, and the custom model is predNet from coxlab. We trained three models in this time.

1.Training: use Nth frame to predict (N+1)th frame
Prediction: use 1-10th frame to predict 2-11th frame, then combined 11th frame in the predicted output with 1-10th frame. With new input(2-10th are ground truth, 11th is predicted), we can keep predicting 3-12th frame. In this experience, we predict the frames until 20th where sliding window = 1 frame. ( only first few frame are good, since we use the predicting frames to do the prediction) 2.Training: Nth frame to predict (N+10)th frame
Prediction: use 1-10 frame to predict 11-20 frame, no sliding window. ( this is very good since all input are ground truth) 3.Training: use Nth frame to predict (N+1)th frame
Prediction: use one frame to predict next frame, like driving prediction (animation has a little problem. right side is prediction, left side is ground truth) Convolutional LSTM

For some reason, there is a request to predict video frames. We need that video is a combination of spatial and temporal dimensions. FCN and LSTM are good for them respectively. But for both of them, we need to use ConvLSTM. Since I just start to learn it, so I write down some of notes for good understanding.

1.First thing first, let’s see what LSTM looks like: From left to right, we can see forget gate, input gate, input modulation gate and output gate. On the top side is memory pipe. It simulates the manner that human remember things. For more information, how the LSTM works please click here.
In keras, there are already three kinds of RNN: simpleRNN, LSTM and GRU. They are all easy to use.

2. What is ConvLSTM

Since LSTM is not good for spatial vector as input, which is only one dimension, they created ConvLSTM to allowed multidimensional data coming with convolutional operations in each gate.  We can find the basic formulas are as same as LSTM, they just use convolutional operations instead of one dimension for input, previous output and memory. Keras needs a new component which called ConvLSTM2D to wrap this ConvLSTM.

3. Where we use it?

As I said in the beginning, it is used for prediction with time and space. The already done in academic inculds: predict precipitation, video frame prediction, some physic movement activities. You can find more in my reference. Reference:
1. the bounce ball. https://www.youtube.com/watch?v=RjZ1VKYyHhs
2. weather forecast. https://papers.nips.cc/paper/5955-convolutional-lstm-network-a-machine-learning-approach-for-precipitation-nowcasting.pdf
3. some video prediction. https://www.youtube.com/watch?v=MjFpgyWH-pk

Incremental Load DW by using CDC in SSIS

To load data from OLTP system to DW, we have to face a problem: how to balance time and cost. Since data raises faster and faster, we need to increase our hardware ability to match the time requirement. So, incremental load coming out to reduce the data transmission significantly. There are three way to achieve it: 1. use datetime column 2. Changed data capture 3. changed data tracking. For a long time, I use the first way to capture the changed data manual, it is good, but too many works in development and testing. Here I want to introduce the CDC. Basically, CDC just a feature to utilize LSN(Log Sequence Number) and log tables to capture changed data, while SSIS itself provide native components to easy work with this new feature.

Let’s build a CDC workflow in SSIS for example:

Enable CDC feature

• enable CDC by executing sp_cdc_enable_db(disable by sp_cdc_disable_db). Then check it with select name from sys.databaseswhere is_cdc_enabled=1
• enable CDC for spec table with sp_cdc_enable_table then we can find the CDC table in systemtable folder or using select namefrom sys.tables tabwhere is_tracked_by_cdc=1 to check.
exec sys.sp_cdc_enable_table
@source_schema = N'Person'
, @source_name = N'Address'
, @role_name = N'cdc_Admin'
, @capture_column_list = N'column1, column2'; //can track spec columns, rather than the whole table

Control flow setting

• set CDC control operation to mark cdc start and set the cdc states for saving cdc states
• run this control task, it will create a record in tablecdc_states
• create two CDC control tasks , one set operation to Get Processing Range , another for Mark process range, they will get changed data and update CDC states respectively.
• put a dataflow which is response for ETL operation, between two CDC control tasks.

Data flow setting into staging table

• add CDC source which points to the table enabled CDC and choose the correct cdc_states table as well.
• choose Net CDC processing mode in CDC source.
• add CDC splinter after CDC source, create three Derived Column transformation for insert(0), update(2) and delete(1) data.
• create a Union All transformation to union all data and export to stage database.
• if necessary, we need to add a truncate script before all control flow to delete everything in stage database. Update fact table through staging tables

• create a oledb source to connect to stage database
• use conditional split to split insert and update+delete
• for insert, we directly export; for update+delete we need to delete from fact table by identifier by OLE DB Command transformation, and use conditional split to export update data.
• if necessary, use lookup to replace some dimensional columns
• export to fact database.

Do we really know how water moves?

Close your eyes, think about what would happen when the spray beats the shore, or the water from faucet comes into your body.
This is a simplest animation showing how water changes its speed after it counters a wall. The color is speed of water and the water is from left side with 1m/s. Can anyone has ability to simulate it in our brain? I guess its super hard unless you see thousands of similar pictures like this.
So that if the simulation becomes much complex than this one like I mentioned before the spray beats the shore, I guess there will be a big gap between our imaginations and real situations.

Learn Django with me(part 3)

Handle view and templates

View consist of a set of functions which handle the different url request with the specific url pettarns.
And it returns either of HttpResponse or Http404.

Firstly, let’s update webapp/views.py:

from django.shortcuts import render
from .models import Question

def index(request):
# get the lastest 5 questions
latest_question_list = Question.objects.order_by('-pub_date')[:5]
# create context
context = {'latest_question_list': latest_question_list}
# a shortcuts for render request by template
return render(request, 'webapp/index.html', context)

def detail(request, question_id):
question = get_object_or_404(Question, pk=question_id)
return render(request, 'webapp/detail.html', {'question': question})

def results(request, question_id):
response = "You're looking at the results of question %s."
return HttpResponse(response % question_id)

def vote(request, question_id):
return HttpResponse("You're voting on question %s." % question_id)

Here we used template webapp/index.html, which locates in webapp/tempaltes/webapp/index.html. So let’s create a folder templates and its subfolder webapp, the code of index.html:

# list of all items from question object
{% if latest_question_list %}
<ul>
{% for question in latest_question_list %}
# webapp is the namespace, detail is the name
<li><a href="{% url 'webapp:detail' question.id %}">{{ question.question_text }}</a></li>
{% endfor %}</ul>
{% else %}

No polls are available.

{% endif %}

The trick point is when we refer to the details, we use {% url 'webapp:detail' question.id %} instand of absolute path. Here webapp is the name space, detail is the name, all can be found in updated webapp/urls.py:

from django.urls import path

from . import views

# namespace
app_name = 'webapp'
urlpatterns = [
# ex: /webapp/
path('', views.index, name='index'),
# ex: /webapp/5/
path('<int:question_id>/', views.detail, name='detail'),
# ex: /webapp/5/results/
path('<int:question_id>/results/', views.results, name='results'),
# ex: /webapp/5/vote/
path('<int:question_id>/vote/', views.vote, name='vote'),
]
</int:question_id></int:question_id></int:question_id>

Similar to the template index.html, we should add the template webapp/tempaltes/webapp/detail.html:
 html
<h1>{{ question.question_text }}</h1>
<ul>
{% for choice in question.choice_set.all %}
<li>{{ choice.choice_text }}</li>
{% endfor %}</ul>

All the dynamic codes in html are easy to understand, I wouldn’t waste time to explain them.

Now, you can access http://localhost:8000/webapp/ to display the reuslts.The whole process can be described like this:
1. send request to server
2. Djongo pastes the url by ROOT_URLCONF = 'mysite.urls' in mysite/settings.py, which points to mysite.urls.
3. In term of urlpatterns in mysite/urls.py, the request will be transfered to webapp folder.
4. The request can be handled by webapp/urls.py, which points to the different functions in webapp/views.py. Here the second param in the function results is from request pattern.
5. View pastes and handle the request, then retrieves template in template/webapp/.
6. HttpResponse or Http404 back to client

In a nutshell, urls.py handles url patterns and sends request to views.py, views.py calls model.py and templates to send response back.

Get Rid of ETL , Move to Spark.

ETL is the most common tool in the process of building EDW, of course the first step in data integration. As big data emerging, we would find more and more customer starting using hadoop and spark. Personally, I agree the idea that spark will replace most ETL tools.

Background

• Business Intelligence -> big data
• Data warehouse -> data lake
• Applications -> Micro services

ETL hell

• Data getting out of sync, each copy is a risk.
• Performance issues and waste of server resource(peek Performance), although ETL can do limited parallel work.
• Plain-text code in hidden stages(VB or java typical)
• CSV files are not type safe
• all or nothing approach in batch jobs.
• legacy code

Spark for ETL

• parallel processing in build in
• using steaming to parallel ETL
• Hadoop which is data source, we don’t need copy and reduce risk
• just one code(scala or python)
• Machine learning included
• security, unit testing, Performance measurement , excepting handling, monitoring

Code Demo

1. Simple one
.filter(...)   # Transform and blew
.agg(...)
.parquet("/outputpath")

2.Steam

# @param1: master
# @param2: appname
sc = SparkContext("local", "NetworkWordCount")
# @param1: spark context
# @param2: seconds
ssc = StreamingContext(sc, 1)
steam = ssc.textFileStream("path")
# do transform
ssc.start()
ssc.awaitTermination()

reference:
1. https://spark.apache.org/docs/2.2.0/streaming-programming-guide.html
3.https://www.slideshare.net/databricks/building-robust-etl-pipelines-with-apache-spark

Learn Django with me(part 2)

Change Database Setting

Open up mysite/settings.py, find snippet as blew:

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}

Here you can change your database to others if needed
* ENGINE – Either ‘django.db.backends.sqlite3’, ‘django.db.backends.postgresql’, ‘django.db.backends.mysql’, or ‘django.db.backends.oracle’.
* NAME – Name of database

We can aslo change the time zone in the setting file

TIME_ZONE = 'America/Chicago'

To create related tables in the database, we need to execute
python manage.py migrate. It will create tables following by INSTALLED_APPS in setting.py.

To read the recetly created tables in sqlite:

python manage.py dbshell
# into sqlite shell
.table

Thre result will like blew:

sqlite&gt; .tables
auth_group                  auth_user_user_permissions
auth_permission             django_content_type
auth_user                   django_migrations
auth_user_groups            django_session

Create a model

In offical defination, model is the single, definitive source of truth about your data.. In my option, model is only data model in single place, rather than in database as well as in you codes.

Let copy this into webapp/models.py. We create two classes which are also two tables in the database. Each variable is a filename with its data type, such as models.CharField is type char, and models.DataTimeField is datatime. Here we can also figure out a ForeignKey in class Choice which points to Question.

from django.db import models

class Question(models.Model):
question_text = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
def __str__(self):
return self.question_text

class Choice(models.Model):
question = models.ForeignKey(Question, on_delete=models.CASCADE)
choice_text = models.CharField(max_length=200)
def __str__(self):
return self.choice_text

To active model, we need to add config file into INSTALLED_APPS. webapp.apps.WebappConfig means calling WebappConfig in apps file in webapp folder.

INSTALLED_APPS = [
'webapp.apps.WebappConfig',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]

Then we run makemigrations to create migration files.

python manage.py makemigrations webapp
# then you will see somehitng like the following:
Migrations for 'webapp':
webapp/migrations/0001_initial.py
- Create model Choice
- Create model Question
- Add field question to choice

We can find the migration opertaion in webapp/migrations/, we run python manage.py migrate, then we can find two new table in the database already(.schema {tablename})

Summary 3 steps for making model changes:

• Change your models (in models.py).
• Run python manage.py makemigrations to create migrations for those changes
• Run python manage.py migrate to apply those changes to the database.

Play with Shell to add some records into db

To get into the shell, we need to execute

python manage.py shell

Then add some questions and choice

from webapp.models import Choice, Question
# show all question
Question.objects.all()

# add a new question
from django.utils import timezone
q = Question(question_text="What's new?", pub_date=timezone.now())

# save into database
q.save()
q.id()

# search in database, similar to where in sql
Question.objects.filter(id=1) # id=1
Question.objects.filter(question_text__startswith='What')
Question.objects.get(pk=1) # filter with pk

# add some choices, here Django creates a set to hold the "other side" of ForeignKey relation

# delete records
d = q.choice_set.filter(choice_text__startswith='The moon')
d.delete()

To create a admin with python manage.py createsuperuser, then system will ask you enter the username, email and password. After this, we can access admin website http://localhost:8000/admin/

If admin account want to add new question in the website, we need to add the follow snippet into admin.py

from django.contrib import admin

from .models import Question

Some issue

When I tried to save question in the admin webpage, there poped out a issue like no such table: main.auth_user__old. Just marked here waiting to find the reason later.

Learn Django with me(part 1)

Although I touched python for a while, most of time I use it only for data analysis with panda or some machine learning packages. Django as one of most famous webframeworks has been existing for over 12 years. So, I decide to learn it step by step with the official tutorial and share my experience with you.

Prepare Django

# install django
sudo pip install Django
# build project
django-admin startproject {name of site}
# run test
# goto project folder, you will find manage.py, run
python manage.py runserver {port}

Fast explain some of files:

• mange.py: A command-line utility that lets you interact with this Django project in various way.
• {name of site}: python package for the project
• mysite/__init__.py: An empty file that tells Python that this directory should be considered a Python package. If you’re a Python beginner, read more about packages in the official Python docs.
• mysite/settings.py: Settings/configuration for this Django project. Django settings will tell you all about how settings work.
mysite/urls.py: The URL declarations for this Django project; a “table of contents” of your Django-powered site. You can read more about URLs in URL dispatcher.
• mysite/wsgi.py: An entry-point for WSGI-compatible web servers to serve your project. See How to deploy with WSGI for more details.

Create a new app

# add a new app
python manage.py startapp {name of app}
• In each App folder, there are three important python files
• urls.py: controls what is served based on url patterns
• models.py: database structures and metadata
• views.py: handles what the end-user “views” or interacts with

Then we need add the app into setting.py under the site folder {name of site}

INSTALLED_APPS = [
'{name of app}',
]

And update the urls.py under the same folder

from django.contrib import admin
from django.urls import path,include

urlpatterns = [
path('webapp/', include('webapp.urls')),
]

We have Completed all files modification under the site folder. Then we go to app folder to create urls.py and change view.py.

# create a file name urls.py` under app folder
touch urls.py

# copy this to the file, which directs the request to views.py
# path(route, view, kwargs,name)
# @route: URL pattern
# @view: function name to be called
# @kwargs: argument to be passed in a dictionary
# @name: refer URL with the name
from django.urls import path
from . import views
urlpatterns = [
path('', views.index, name='index'),
]

# change view.py as
from django.http import HttpResponse
def index(request):
return HttpResponse("Hello, world. You're at the polls index.")

After all these done, we can access webapp by http://localhost:8000/{name of app}/