# Eiganvectors from Eiganvalues

This is a unbelievable discovery from PETER B. DENTON, STEPHEN J. PARKE, TERENCE TAO, AND XINING ZHANG. The original paper you can find here. In a short nut, we can get eiganvector through eiganvalues only. This magic formular is like blew: We only need to know eiganvalues in original matrix then we can calculate its eiganvectors through sub-matrix.

Currently, I don’t know or image what will effect on the road, but the eiganX is the basic for AI, dimension reduction, feature extraction, etc. It also may help us improve the speed to get eiganvector if we need to incrementally add data on a known matrix.

I wrote a very simple python script to prove this formula(surely, it is correct). I think it can archive by GPU as well.

import numpy as np
from numpy import linalg as LA

matrix_size = 6
org_x = np.diag(range(1,matrix_size+1))
w, v = LA.eig(org_x)

print("orgnal matriax: \n %s \n" % org_x)
print("eigan values: \n %s \n" % w)
print("normalized eigenvectors: \n %s \n" % v)

print("START NEW ALGORITHM \n")

result=[]
for _ in range(matrix_size):
result.append(0)

for n in range(matrix_size):
for j in range(matrix_size):
sub_x = np.delete(org_x,j,axis=0)
sub_x = np.delete(sub_x,j,axis=1)
w1,v1 = LA.eig(sub_x)

# in term of new formula to get orignal matrix eigenvecotr through eiganvalue
numberator = 1
denominator = 1
for i in range(matrix_size-1):
temp_n = w[n] - w1[i]
numberator = numberator*temp_n

for i in range(matrix_size):
if(i!=n):
temp_d = w[n] - w[i]
denominator = denominator*temp_d
result[j] = numberator/denominator

print("%s \n" % result)

result:

orgnal matriax:
[[1 0 0 0 0 0]
[0 2 0 0 0 0]
[0 0 3 0 0 0]
[0 0 0 4 0 0]
[0 0 0 0 5 0]
[0 0 0 0 0 6]]

eigan values:
[1. 2. 3. 4. 5. 6.]

normalized eigenvectors:
[[1. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0.]
[0. 0. 1. 0. 0. 0.]
[0. 0. 0. 1. 0. 0.]
[0. 0. 0. 0. 1. 0.]
[0. 0. 0. 0. 0. 1.]]

START NEW ALGORITHM

[1.0, -0.0, -0.0, -0.0, -0.0, -0.0]

[0.0, 1.0, -0.0, -0.0, -0.0, -0.0]

[0.0, 0.0, 1.0, -0.0, -0.0, -0.0]

[0.0, 0.0, 0.0, 1.0, -0.0, -0.0]

[0.0, 0.0, 0.0, 0.0, 1.0, -0.0]

[0.0, 0.0, 0.0, 0.0, 0.0, 1.0]

Reference:

EIGENVECTORS FROM EIGENVALUES, https://arxiv.org/pdf/1908.03795.pdf

Tao’s Blog, https://terrytao.wordpress.com/2019/08/13/eigenvectors-from-eigenvalues/

# Math in Machine Learning

### Linear Algebra

• mathematics of data: multivariate, least square, variance, covariance, PCA
• equotion: y = , where A is a matrix, b is a vector of depency variable
• application in ML
1. Dataset and Data Files
2. Images and Photographs
3. One Hot Encoding: A one hot encoding is a representation of categorical variables as binary vectors. encoded = to_categorical(data)
4. Linear Regression. L1 and L2
5. Regularization
6. Principal Component Analysis. PCA
7. Singular-Value Decomposition. SVD. M=U*S*V
8. Latent Semantic Analysis. LSA typically, we use tf-idf rather than number of terms. Through SVD, we know the different docments with same topic or the different terms with same topic
9. Recommender Systems.
10. Deep Learning

### Numpy

1. add a scalar or one dimension matrix to another matrix. where b is broadcated.
2. it oly works when when the shape of each dimension in the arrays are equal or one has the dimension size of 1.
3. The dimensions are considered in reverse order, starting with the trailing dimension;

### Matrice

• Vector
1. lower letter. 3. Multiplication, Divsion(Same length) a*b or 4. Dot product: • Vector Norm
1. Defination: the length of vector
2. L1. Manhattan Norm. python: norm(vector, 1) . Keep coeffiencents of model samll
3. L2. Euclidean Norm. python: norm(vector)
4. Max Norm. python: norm(vector, inf)
• Matrices
1. upper letter. 3. Multiplication, Divsion( same dimension)
4. Matrix dot product. If , A’s column(n) need to be same size to B’s row(m). python: A.dot(B) or A@B
5. Matrix-Vector dot product. 6. Matrix-Scalar. element-wise multiplication
7. Type of Matrix
2. symmetric matrix. 3. triangular matrix. python: tril(vector) or triu(vector) lower tri or upper tri matrix
4. Diagonal matrix. only diagonal line has value, doesnot have to be square matrix. python: diag(vector)
5. identity matrix. Do not change vector when multiply to it. notatoin as python: identity(dimension)
6. orthogonal matrix. Two vectors are orthogonal when dot product is zeor. or . which means the project of to is zero. An orthogonal matrix is a matrix which 8. Matrix Operation
1. Transpose. number of rows and columns filpped. python: A.T
2. Inverse. where python: inv(A)
3. Trace. the sum of the values on the main diagonal of matrix. python: trace(A)
4. Determinant. a square matrix is a scalar representation of the volume of the matrix. It tell the matrix is invertable. or . python: det(A) .
5. Rank. Number of linear indepent row or column(which is less). The number of dimesions spanned by all vectors in the matrix. python: rank(A)
9. Sparse matrix
1. sparsity score = 2. example: word2vector
3. space and time complexity
4. Data and preperation
1. record count of activity: match movie, listen a song, buy a product. It usually be encoded as : one hot, count encoding, TF-IDF
5. Area: NLP, Recomand system, Computer vision with lots of black pixel.
6. Solution to represent sparse matrix. reference
1. Dictionary of keys:  (row, column)-pairs to the value of the elements.
2. List of Lists: stores one list per row, with each entry containing the column index and the value.
3. Coordinate List: a list of (row, column, value) tuples.
4. Compressed Sparse Row: three (one-dimensional) arrays (A, IA, JA).
5. Compressed Sparse Column: same as SCR
7. example
1. covert to sparse matrix python: csr_matrix(dense_matrix)
2. covert to dense matrix python: sparse_matrix.todense()
3. sparsity = 1.0 – count_nonzero(A) / A.size
10. Tensor
1. multidimensional array.
2. algriothm is similar to matrix
3. dot product: python: tensordot()

### Factorization

• Matrix Decompositions
1. LU Decomposition
1. square matrix
2. , L is lower triangle matrix, U is upper triangle matrix. P matrix is used to permute the result or return result to the orignal order.
3. python: lu(square_matrix)
2. QR Decomposition
1. n*m matrix
2. where Q a matrix with the size mm, and R is an upper triangle matrix with the size mn.
3. python: qr(matrix)
3. Cholesky Decomposition
1. square symmtric matrix where values are greater than zero
2. , L is lower triangle matrix, U is upper triangle matrix.
3. twice faster than LU decomposition.
4. python: cholesky(matrix)
4. EigenDecomposition
1. eigenvector: , is matrix we want to decomposite, is eigenvector, is eigenvalue(scalar)
2. a matrix could have one eigenvector and eigenvalue for each dimension. So the matrix can be shown as prodcut of eigenvalues and eigenvectors. where Q is the matrix of eigenvectors, is the matrix of eigenvalue. This equotion also mean if we know eigenvalues and eigenvectors we can construct the orignal matrix.
3. python: eig(matrix)
5. SVD(singluar value decomposition)
1. , where A is m*n, U is m*m matrix, is m*m diagonal matrix also known as singluar value, is n*n matrix.
2. python: svd(matrix)
3. reduce dimension
1. select top largest singluar values in 2. , where column select from , row selected from , B is approximate of the orignal matrix A.
3. python: TruncatedSVD(n_components=2)

### Stats

• Multivari stats
1. variance: , python: var(vector, ddof=1)
2. standard deviation: , python:std(M, ddof=1, axis=0)
3. covariance: , python: cov(x,y)[0,1]
4. coralation: , normorlized to the value between -1 to 1. python: corrcoef(x,y)[0,1]
5. PCA
1. project high dimensions to subdimesnion
2. steps:
1. 2. 3. 4. 5. , which order by eigenvalue
3. scikit learn
 pca = PCA(2) # get two components pca.fit(A) print(pca.componnets_) # values print(pca.explained_variance_) # vectors B = pca.transform(A) # transform to new matrix `
• Linear Regression
1. , where b is coeffcient and unkown
2. linear least squares( similar to MSE) , then . Issue: very slow
3. MSE with SDG

Reference: Basics of Linear Algebra for Machine Learning, jason brownlee, https://machinelearningmastery.com/linear_algebra_for_machine_learning/