As an amateur photographer, I was believing DSL is better than phone camera since it has a much larger CMOS so that it can receive more photons, until I installed Google Camera on my Galaxy S8. The results surprised me a lot. I mean, it performed better than my Conan 5DIII in the most of case without editing in the software like photoshop.
Except HDR+ and portrait mode, Google camera provides a magic mode called Night Sight. Unfortunately, this can only work on Pixel series (Pixel 3/3a is the best) phone with hardware support. You can find the A/B Testing here.
How does Night Sight improve the quality of shots in the night?
HDR+. HDR+ is the foundational function for Night Sight. It is a computational photography technology that improves this situation by capturing a burst of frames, aligning the frames in software, and merging them together. Since each frame is short enough to prevent the blur caused by hand shake, the result turns out to be sharper and wilder dynamic range than without HDR+.
Positive-shutter-lag (PSL) . In the regular mode, Google camera uses zero-shutter-lag (ZSL) protocol which limits exposures to at most 66ms no matter how dim the scene is, and allows our viewfinder to keep up a display rate of at least 15 frames per second. Using PSL means you need to hold still for a short time after pressing the shutter, but it allows the use of longer exposures, thereby improving SNR at much lower brightness levels.
Motion metering. Optical image stabilization (OIS) is widely used in many devices to prevent hand shake. But it doesn’t help for long exposure and motion object. Pixel 3 adds motion metering to detect the motion object and adjust the exposure time for each frame. For example, if it detects a dog moving in the frame or we are using the tripod, it will increase exposure time.
Super Res Zoom. HDR essentially uses algorithm to aliment and merge the frames to increase the SNR( signal to noise ratio). Pixel 3 provides a new algorithm called Super Res Zoom for super-resolution and reduce noise, since it averages multiple images together. Super Res Zoom produces better results for some nighttime scenes than HDR+, but it requires the faster processor of the Pixel 3.
Learning-based AWB algorithm. When it is dim, AWB( auto white balance) is not functional well. And it is an ill-posed problem, which means we cannot inverse the problem (find out the real color of object in the dark). In this case, Google camera utilizes machine learning to “guess” what is the true color and shift the white balance.
S-curve into our tone mapping. As we know, if we exposure a picture for a long time, all the objects become brighter so that we can not figure out when this picture takes. Google uses sigmoid function to adjust the object brightness ( dark objects become darker, light objects become brighter).
YOLO also know as You Only Look Once. Not like R-CNN, YOLO uses single CNN to do the object detection as well as localization which makes it super faster than R-CNN with only losing a little accuracy. From 2016 to 2018, YOLO has been imporved from v1 to v3. In this article, I will use a simple way to explain how YOLO works.
What tasks we need to solve in object detection problem?
Yolo use the same method as human to detect the object. There are three major steps: 1. is it an object? 2. what object is it? 3. where is the position and size of this object. BUT! Through CNN, YOLO can do these three things all together.
How YOLO solve this problem?
First, Let’s introduce Grid Cell in YOLO. The whole input image is divided into grid. Each grid cell predicts only one objects with fixed boundary boxes( say #B). For each boundary box has its own box confidence score to reflet the possibility of object. For each grid cell it predicts C conditional class probabilities( one per class). so that we will get predictions. Here 5 means central location(x,y), size( h,w) and confidence score of each boundary box.
Then you will find so many boundary box from output. How we choose of them? Here we need to do Non-max suppression. The step is as blew:
discard all boxes with box confidence less then a threshold. ( say 0.65)
While there are any renaming box(overlapping):
pick the box with the largest confidence that as a prediction
discard any remaining boxes with IoU(intersection over union: you can see it as overlap size between two boundary box) greater than a threshold(say 0.5)
After Non-max suppression, we need to calculate class confidence score , which equals to box confidence score * conditional class probability. Finally, we get the object with probability and its localization. (see Figure 1)
YOLO Network Design
Let’s see how YOLO v1 looks like. Input = 448*448 image, output = . There are 24 convolutional layers followed by 2 full connected layer for localization. It use sum-squared error between the predictions and ground truth to calculate loss which is consist of classification loss, localization loss and confidence loss.
Classification loss:
Localization loss:
Confidence loss:
The final loss add three components together.
YOLO V2
YOLO v2 improves accuracy compared with YOLO v1.
Add batch normalization on all of the convolutional layers. It get more than 2% improvement in accuracy.
High-resolution classifier. First fine tune the classification network at the full resolution for 10 epochs on IMageNet. This gives network time to adjust tis filters to work better on higher resolution input.
Convolutional with Anchor Boxes. YOLO v1 can only predicts 98 boxes per images and it makes arbitrary guesses on the boundary boxes which leads to bad generalization, but with anchor boxes, YOLO v2 predicts more than a thousand. Then it use dimension cluster and direct location prediction to get the boundary box.
Dimension Cluster. use K-mean to get the boundary boxes patterns. Figure 5 might be the most common boundary boxes in spec dataset.
5. Direct location prediction. Since we use anchor boxes, we have to predict on the offsets to these anchors.
6. Multi-Scale Training. Every 10 batches, YOLOv2 randomly selects another image size to train the model. This acts as data augmentation and forces the network to predict well for different input image dimension and scale.
YOLO v3
Detection at three scales. YOLOv3 predicts boxes at 3 different scales. Then features are extracted from each scale by using a method similar to that of feature pyramid networks
Bounding box predictions. YOLO v3 predicts the object score using logistic regression.
Class prediction. Use independent logistic classifiers instead of softmax. This is done to make the classification multi-able classification.
Spark provides spark MLlib for machine learning in a scalable environment. MLlib includes three major parts: Transformer, Estimator and Pipeline. Essentially, transformer takes a dataframe as an input and returns a new data frame with more columns. Most featurization tasks are transformer. Estimator takes a dataframes as an input and returns a model(transformer), as we know the ML algorithms.. Pipeline combines transformer and estimator together. Additionally, data frame becomes the primary API for MLlib. There is not any more new features for RDD based API in Spark MLib.
If you already understood or used high level machine learning or deep learning frameworks, like scikit-learning, keras, tersorflow, you will find everything is so familiar with. But when you use spark MLlib in practice, you still need third library’s help. I will talk about it in the end.
Basic Stats
# Corrlation
from pyspark.ml.stat import Correlation
r1 = Correlation.corr(df, "features").head()
print("Pearson correlation matrix:\n" + str(r1[0]))
# Summarizer
from pyspark.ml.stat import Summarizer
# compute statistics for multiple metrics without weight
df.select(summarizer.summary(df.features)).show(truncate=False)
# ChiSquare
r = ChiSquareTest.test(df, "features", "label").head()
print("pValues: " + str(r.pValues))
print("degreesOfFreedom: " + str(r.degreesOfFreedom))
print("statistics: " + str(r.statistics))
Featurization
# TF-IDF
# stop word remove
remover = StopWordsRemover(inputCol="raw", outputCol="filtered")
remover.transform(sentenceData)
# tokenize
tokenizer = Tokenizer(inputCol="sentence", outputCol="words")
wordsData = tokenizer.transform(sentenceData)
# n grame
ngram = NGram(n=2, inputCol="wordsData", outputCol="ngrams")
ngramDataFrame = ngram.transform(wordDataFrame)
# word frequence
hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=20)
featurizedData = hashingTF.transform(wordsData)
# idf
idf = IDF(inputCol="rawFeatures", outputCol="features")
idfModel = idf.fit(featurizedData)
rescaledData = idfModel.transform(featurizedData)
# word2vec
word2Vec = Word2Vec(vectorSize=N, minCount=0, inputCol="text", outputCol="result")
model = word2Vec.fit(documentDF)
# binarizer
binarizer = Binarizer(threshold=0.5, inputCol="feature", outputCol="binarized_feature")
binarizedDataFrame = binarizer.transform(continuousDataFrame)
# PCA
# reduce dimension to 3
pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures")
model = pca.fit(df)
# StringIndex
# encodes a string column of labels to a column of label of indices order by frequency or alphabet
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
indexed = indexer.fit(df).transform(df)
# OneHotEstimator
# we need to use StringIndex first if apply to categorical feature
encoder = OneHotEncoderEstimator(inputCols=["categoryIndex1", "categoryIndex2"],
outputCols=["categoryVec1", "categoryVec2"])
model = encoder.fit(df)encoded = model.transform(df)
# Normalize & Scaler
normalizer = Normalizer(inputCol="features", outputCol="normFeatures", p=1.0)
lInfNormData = normalizer.transform(dataFrame, {normalizer.p: float("inf")})
l1NormData = normalizer.transform(dataFrame)
# standard scaler
# withMean=false: standard deviation, withMean=true: mean
scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures",
withStd=True, withMean=False)
# maxmin scaler
scaler = MinMaxScaler(inputCol="features", outputCol="scaledFeatures")
# max abs scaler
scaler = MaxAbsScaler(inputCol="features", outputCol="scaledFeatures")
# bin
from pyspark.ml.feature import Bucketizer
splits = [-float("inf"), -0.5, 0.0, 0.5, float("inf")]
bucketizer = Bucketizer(splits=splits, inputCol="features", outputCol="bucketedFeatures")
# QuantileDiscretizer
discretizer = QuantileDiscretizer(numBuckets=3, inputCol="hour", outputCol="result")
# ElementwiseProduct
transformer = ElementwiseProduct(scalingVec=Vectors.dense([0.0, 1.0, 2.0]),
inputCol="vector", outputCol="transformedVector")
# SQL Transformer
sqlTrans = SQLTransformer(
statement="SELECT *, (v1 + v2) AS v3, (v1 * v2) AS v4 FROM __THIS__")
# VectorAssembler
# combine vector together for future model inputs
assembler = VectorAssembler(
inputCols=["hour", "mobile", "userFeatures"], # the columns we need to combine
outputCol="features") # output column
# Imputer
# handle missing value
imputer = Imputer(inputCols=["a", "b"], outputCols=["out_a", "out_b"])
imputer.setMissingValue(custom_value)
# slice vector
slicer = VectorSlicer(inputCol="userFeatures", outputCol="features", indices=[1])
# ChiSqSelector
# use Chisqare to select the features
selector = ChiSqSelector(numTopFeatures=1, featuresCol="features",
outputCol="selectedFeatures", labelCol="clicked")
Clarification and Regression
# Linear regression
lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
# logistic regression
from pyspark.ml.classification import LogisticRegression
lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
mlr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8, family="multinomial") # multinomial
# decision tree
# classification
labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel").fit(data)
featureIndexer =\
VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)
(trainingData, testData) = data.randomSplit([0.7, 0.3])
dt = DecisionTreeClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures")
# regression
dt = DecisionTreeRegressor(featuresCol="indexedFeatures")
# Random forest
# classification
rf = RandomForestClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures", numTrees=10)
# regeression
rf = RandomForestRegressor(featuresCol="indexedFeatures")
# gradient-boosted
gbt = GBTRegressor(featuresCol="indexedFeatures", maxIter=10)
# preceptron
trainer = MultilayerPerceptronClassifier(maxIter=100, layers=layers, blockSize=128, seed=1234)
# SVM
# Linear SVM, there is no kernel SVM like RBF
lsvc = LinearSVC(maxIter=10, regParam=0.1)
# Naive Bayes
nb = NaiveBayes(smoothing=1.0, modelType="multinomial")
# KNN
knn = KNNClassifier().setTopTreeSize(training.count().toInt / 500).setK(10)
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="movieId", ratingCol="rating",
coldStartStrategy="drop")
model = als.fit(training)
# Evaluate the model by computing the RMSE on the test datapredictions = model.transform(test)evaluator = RegressionEvaluator(metricName="rmse", labelCol="rating",predictionCol="prediction")
rmse = evaluator.evaluate(predictions)
Validation
# split train and test
train, test = data.randomSplit([0.9, 0.1], seed=12345)
# cross validation
crossval = CrossValidator(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=BinaryClassificationEvaluator(),
numFolds=2) # use 3+ folds in practice
You might be already found the problem. The ecosystem of Spark MLlib is not as rich as scikit learning, and it is lack of deep learning (of course, its name is machine learning). According to Databricks documents, We still have the solutions.
Use scikit learning on single node. Very simple solution. but since scikit leanring load the data still in memory. If the note is faster enough(driver), we can get a good performance as well.
To solve deep learning problem. we have two work around methods.
Apply keras, tensorflow on single node with GPU acceleration(Recommend by databricks).
Distribute Training. It might be slower than on the single node because of communication overhead. There are two frameworks used for distribute training. Horovod and Apache SystemML. I’ve never use Horovod, but you can find information here. As to SystemML, it is more like a wrapper for high level API and provide cluster optimizer which parses the code into spark RDD(live variable analysis, propagate stats, rewrite by matrix decomposition and runtime instruction). From the official website, we know it is much faster than MLLib and native R. The problem is it didn’t update anymore since 2017.
# Create and save a single-hidden-layer Keras model for binary classification# NOTE: In a typical workflow, we'd train the model before exporting it to disk,# but we skip that step here for brevity
model = Sequential()
model.add(Dense(units=20, input_shape=[num_features], activation='relu'))
model.add(Dense(units=1, activation='sigmoid'))
model_path = "/tmp/simple-binary-classification"
model.save(model_path)
transformer = KerasTransformer(inputCol="features", outputCol="predictions", modelFile=model_path)
It seems no perfect solution for machine learning in spark, right? Don’t forget we have other time costing jobs: hyper parameters configuration and validation. We can run same model with different hyper parameters on different nodes using paramMap which similar to grid search or random search.
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
paramGrid = ParamGridBuilder().addGrid(lr.maxIter, [10, 100, 1000]).addGrid(lr.regParam, [0.1, 0.01]).build()
crossval = CrossValidator(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=RegressionEvaluator(),
numFolds=2) # use 3+ folds in practice
cvModel = crossval.fit(training)
There might be someone saying: why we don’t use MPI? The answer is simple, too complex. Although it can gain the perfect performance, and you can do whatever you want even running distributed GPU + CPU codes, there are too many things we need to manually configuration on low level API without fail tolerance.
In conclusion, we can utilize spark for ELT and training/ validation model to maximize the performance(it did really well for these works). But until now, we still need third frameworks to help us do deep learning or machine learning tasks on single strong node.
Today I tried a text classification task where the data is about the message on the flights and labeled into 5 levels. Observably, it is a supervised problem. And I though there are bunch of solutions already for this kind of problem. So that I started with full of confidence. But…. the result was so bad, no more than 35% accurate for 5 classification. Only a little bit better than guess.
I am considering following reasons leading this failure:
The module is easy to over fitting. For example, when GRU model’s training loss decreasing, the invalidation loss was decreasing in the beginning, but after 40 epochs, it started to increasing or jumped up/down.
Since I used trained embedding model(FastText) which is based on wiki but the dataset is in civil aviation. The words and word vectors may far away.
In the data source, there might be lack of significant or clear rules to classify them to 5 categories. If we just label it to binary “attention/no worry”. The result will be better.
@20190402: I change the category from 5 to 2, hopefully the result would be better. But NO improvement. Only 63.5% for 2 categories.
Last week, my wife told me she logged into my netflix’s account, then she found it was not hers immediately since the items did not match her tastes. This activated my interesting in the recommendation system of Netflix & Youtube which are the most watched channels in US. (maybe spotfiy will be the same way). Here I want to give a brief analysis how they work.
Basic
Before we quickly look how many different manners(that I knew) used in the recommendation systems.
Popularity. This is the simplest way in term of PV. It works very good for new users and avoid the “cold start” problem. However, the downside is this method can not provide the personalized recommendation. The way to optimize it is adding some categories at the beginning so that users can filter the categories by themselves. Collaborative filtering (CF). The Collaborative Filtering (CF) algorithms are based on the idea that if two clients have similar rating history then they will behave similarly in the future (Breese,Heckerman, and Kadie, 1998). It can also split into two subcategories, one is Memory-based, another is Model-based.
Memory-based approach can be divided into User-based and Item-based. They find the similar users or similar items respectively in term of Pearson Correlation.
User-based.
Build correlation matrix which is symmetric.
select top k users who has the largest scores.
identify items that similar users like but the prediction user has not seem before. The prediction of a recommendation is based on the wighted combination of the selected neighbor’s rating.
pick up top N of movies based on the predicted rating.
Item-based.
Build correlation matrix based on items. (similar to user-based)
Get the top n movies that prediction user watched and rated before.
return the movies that mostly related to these n movies and the prediction user has never watched.
In the real word. The size of user are growing faster than item, and they are easy to be changed. So item-based are most frequency used.
Model-based approach are based on matrix factorization which is popular in dimension reduction. Here we use Singular value decomposition(SVD) to explain.
, where U represents the freature vectors corresponding to the users in the latent space with dimension r, V represents the feature vectors corresponding to the items in the latent space with dimension r. Once we find U and V, we can calculate any by .
CF is based on historical data, it has “cold start” problem. and the accuracy of prediction is based on the mount of data since the CF matrix has sparsity problem, e.g, few mistake rating will effect the prediction seriously.
Contented-based(CB). This approach is based on the information of item itself rather than only rating in CF approach. We need to create meta data for the items. These meta data can be tagged manual or use TF-IDF tech to automatically extra keywords. Then build the connection between the item that prediction user liked and the items with similar meta data. CB avoid of “cold start” and “over recommend” problems, however, it is hard to metain and keep accuracy of meta data.
Hybrid. It combined CF and CB. We can merge the prediction together or set the weights in different scenarios.
Deep Learning. In the large scale dataset, it is hard to use traditional recommendation system because of 4V(volume, variety, velocity, and veracity). Deep learning model are good at solving complex problem( A review on deep learning for recommender systems: challenges and remedies). We will introduce deep learning model used by YouTube in the next section.
Netflix
I firstly log into the Netflix to find some information provided by the official website. Fortunately, there was a topic How Netflix’s Recommendations System Works. They didn’t give much detail about algorithms but the provides the clues which information they are using for predict users’ choices. Blew is their explanation:
We estimate the likelihood that you will watch a particular title in our catalog based on a number of factors including:
your interactions with our service (such as your viewing history and how you rated other titles),
other members with similar tastes and preferences on our service (more info here), and
information about the titles, such as their genre, categories, actors, release year, etc.
So, we can guess it is a hybrid approach combined with CF(item-base and user-based) and CB approaches. But we don’t know how they design it at this moment. Let keep reading from the official website.
In addition to knowing what you have watched on Netflix, to best personalize the recommendations we also look at things like:
the time of day you watch,
the devices you are watching Netflix on, and
how long you watch.
These actives are not mentioned in the basic section. They are all used as input vector for the deep learning model which we will see in YouTube section.
It also mentioned “Cold start” problem:
When you create your Netflix account, or add a new profile in your account, we ask you to choose a few titles that you like. We use these titles to “jump start” your recommendations. Choosing a few titles you like is optional. If you choose to forego this step then we will start you off with a diverse and popular set of titles to get you going.
It’s clear they use popularity approach with categories to solve “cold start” problem. As user has more historical information, Netflix will use another approaches to replace the initial one.
They also personalized row and title inside:
In addition to choosing which titles to include in the rows on your Netflix homepage, our system also ranks each title within the row, and then ranks the rows themselves, using algorithms and complex systems to provide a personalized experience. …. In each row there are three layers of personalization:
the choice of row (e.g. Continue Watching, Trending Now, Award-Winning Comedies, etc.)
which titles appear in the row, and
the ranking of those titles.
They calculate the score for each item for each users, then sum up these scores into each category to decide the order of rows. As I said, we don’t know how they mix CB and CF to get the score of each item yet. But they are mixed for sure.
YouTube
As Google’s product, it is not surprised that YouTube uses Deep learning as a solution for recommendation system. It is too large both in user and item aspects. A simple stats model can not handle it well. In the paper “Deep Neural Networks for YouTube Recommendations“, they explained how they use DL to YouTube.
It has two parts: Candidate Generation and Ranking. One for filtering hundred candidates from millions, second for sorting by adding more scenario or video features information. Let’s see how they work:
Candidate generation. For candidate generation, it filters from millions videos, so it only uses user activities and scenario information. The basic idea is getting probabilities of watching specific video V through user U and context C.
. The key point is to get user vector and . To get user vector , author embeds the video watches and search tokens, then average them into watch vector and search vector, then combined with other geogrphic , video ages and gender vectors to get through 3 connected ReLU layer. The output is user vector . To get video vectors , we need to use to predict probabilities for all through softmax. After training, the video vector is what we want. In the serving processing, we only need to put and together to calculate the top N highest probability vectors.
Ranking.Compared with Candidate Generation, the number of videos is much less. So we can put more video features into the embedding vectors. These features are mostly focus on scenario, like topic of video, how many videos the user watched under each topic and time since last watch. It embeds categorical features with shared embeddings and continuous features with powers of normalization.
Reference:
Deep Neural Networks for YouTube Recommendations, Paul Covington, Jay Adams, Emre Sargin, https://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/45530.pdf
How Netflix’s Recommendations System Works, n/a, https://help.netflix.com/en/node/100639
Finding the Latent Factors | Stanford University, https://www.youtube.com/watch?v=GGWBMg0i9d4&index=56&list=PLLssT5z_DsK9JDLcT8T62VtzwyW9LNepV
Recommendation System for Netflix, Leidy Esperanza MOLINA FERNÁNDEZ, https://beta.vu.nl/nl/Images/werkstuk-fernandez_tcm235-874624.pdf
Recently I was given a topic to research a manner to summary the text automatically. So I shared some my search results, hope it is helpful.
Summarization Methods
we can classify summarization methods into different types by input type, the purpose and output type. Typically, extractive and abstractive are the most common ways.
Here, we would like introduce two methods for Extractive. One is Stats-based , another is Deep Learning-based.
Stats-based
Idea: for each word, we would give a weight frequency. For each sentence, we summary the weight frequency for the words inside. Then pick up the sentences ordered by the sum of weight frequency.
Steps
2.1 Preprocessing: replace extra whitespace characters or delete some parts we do not need to analysis.
stopwords = nltk.corpus.stopwords.words('english')
word_frequencies = {}
for word in nltk.word_tokenize(formatted_article_text):
if word not in stopwords: if word not in word_frequencies.keys():
word_frequencies[word] = 1
else:
word_frequencies[word] += 1
2.4 Weighted frequency of occurrence
maximum_frequncy = max(word_frequencies.values())
for word in word_frequencies.keys():
word_frequencies[word] = (word_frequencies[word]/maximum_frequncy)
2.5 Calculate the sum of weight frequency for each sentence
sentence_scores = {}
for sent in sentence_list:
for word in nltk.word_tokenize(sent.lower()):
if word in word_frequencies.keys():
if len(sent.split(' ')) < 30:
if sent not in sentence_scores.keys():
sentence_scores[sent] = word_frequencies[word]
else:
sentence_scores[sent] += word_frequencies[word]
Idea: vectorizing each sentence into a high dimension space, then cluster the vector using kmean, pick up the sentences which mostly close to the center of each cluster to form the summery of text.
steps:
2.1 prepossessing and tokenizing the sentence( same as stats-based method)
2.2 Skip-Thought Encoder
Encoder Network: The encoder is typically a GRU-RNN which generates a fixed length vector representation h(i) for each sentence S(i) in the input. Decoder Network: The decoder network takes this vector representation h(i) as input and tries to generate two sentences — S(i-1) and S(i+1), which could occur before and after the input sentence respectively.
These learned representations h(i) are such that embeddings of semantically similar sentences are closer to each other in vector space, and therefore are suitable for clustering.
Skip-Thoughts Architecture
import skipthoughts
# You would need to download pre-trained models first
model = skipthoughts.load_model()
encoder = skipthoughts.Encoder(model)
encoded = encoder.encode(sentences)
2.3 Clustering
import numpy as np
from sklearn.cluster import KMeans
n_clusters = np.ceil(len(encoded)**0.5)
kmeans = KMeans(n_clusters=n_clusters)
kmeans = kmeans.fit(encoded)
2.4 Summerization
from sklearn.metrics import pairwise_distances_argmin_min
avg = []
for j in range(n_clusters):
idx = np.where(kmeans.labels_ == j)[0]
avg.append(np.mean(idx))
closest, _ = pairwise_distances_argmin_min(kmeans.cluster_centers_, encoded)
ordering = sorted(range(n_clusters), key=lambda k: avg[k])
summary = ' '.join([email[closest[idx]] for idx in ordering])
Reference:
Unsupervised Text Summarization using Sentence Embeddings,https://medium.com/jatana/unsupervised-text-summarization-using-sentence-embeddings-adb15ce83db1
mathematics of data: multivariate, least square, variance, covariance, PCA
equotion: y = , where A is a matrix, b is a vector of depency variable
application in ML
Dataset and Data Files
Images and Photographs
One Hot Encoding: A one hot encoding is a representation of categorical variables as binary vectors. encoded = to_categorical(data)
Linear Regression. L1 and L2
Regularization
Principal Component Analysis. PCA
Singular-Value Decomposition. SVD. M=U*S*V
Latent Semantic Analysis. LSA typically, we use tf-idf rather than number of terms. Through SVD, we know the different docments with same topic or the different terms with same topic
Recommender Systems.
Deep Learning
Numpy
array broadcasting
add a scalar or one dimension matrix to another matrix. where b is broadcated.
it oly works when when the shape of each dimension in the arrays are equal or one has the dimension size of 1.
The dimensions are considered in reverse order, starting with the trailing dimension;
Matrice
Vector
lower letter.
Addtion, Substruction
Multiplication, Divsion(Same length) a*b or
Dot product:
Vector Norm
Defination: the length of vector
L1. Manhattan Norm. python: norm(vector, 1) . Keep coeffiencents of model samll
L2. Euclidean Norm. python: norm(vector)
Max Norm. python: norm(vector, inf)
Matrices
upper letter.
Addtion, substruction(same dimension)
Multiplication, Divsion( same dimension)
Matrix dot product. If , A’s column(n) need to be same size to B’s row(m). python: A.dot(B) or A@B
Matrix-Vector dot product.
Matrix-Scalar. element-wise multiplication
Type of Matrix
square matrix. m=n. readily to add, mulitpy, rotate
symmetric matrix.
triangular matrix. python: tril(vector) or triu(vector) lower tri or upper tri matrix
Diagonal matrix. only diagonal line has value, doesnot have to be square matrix. python: diag(vector)
identity matrix. Do not change vector when multiply to it. notatoin as python: identity(dimension)
orthogonal matrix. Two vectors are orthogonal when dot product is zeor. or . which means the project of to is zero. An orthogonal matrix is a matrix which
Matrix Operation
Transpose. number of rows and columns filpped. python: A.T
Inverse. where python: inv(A)
Trace. the sum of the values on the main diagonal of matrix. python: trace(A)
Determinant. a square matrix is a scalar representation of the volume of the matrix. It tell the matrix is invertable. or . python: det(A) .
Rank. Number of linear indepent row or column(which is less). The number of dimesions spanned by all vectors in the matrix. python: rank(A)
Sparse matrix
sparsity score =
example: word2vector
space and time complexity
Data and preperation
record count of activity: match movie, listen a song, buy a product. It usually be encoded as : one hot, count encoding, TF-IDF
Area: NLP, Recomand system, Computer vision with lots of black pixel.
Dictionary of keys: (row, column)-pairs to the value of the elements.
List of Lists: stores one list per row, with each entry containing the column index and the value.
Coordinate List: a list of (row, column, value) tuples.
Compressed Sparse Row: three (one-dimensional) arrays (A, IA, JA).
Compressed Sparse Column: same as SCR
example
covert to sparse matrix python: csr_matrix(dense_matrix)
covert to dense matrix python: sparse_matrix.todense()
sparsity = 1.0 – count_nonzero(A) / A.size
Tensor
multidimensional array.
algriothm is similar to matrix
dot product: python: tensordot()
Factorization
Matrix Decompositions
LU Decomposition
square matrix
, L is lower triangle matrix, U is upper triangle matrix. P matrix is used to permute the result or return result to the orignal order.
python: lu(square_matrix)
QR Decomposition
n*m matrix
where Q a matrix with the size mm, and R is an upper triangle matrix with the size mn.
python: qr(matrix)
Cholesky Decomposition
square symmtric matrix where values are greater than zero
, L is lower triangle matrix, U is upper triangle matrix.
twice faster than LU decomposition.
python: cholesky(matrix)
EigenDecomposition
eigenvector: , is matrix we want to decomposite, is eigenvector, is eigenvalue(scalar)
a matrix could have one eigenvector and eigenvalue for each dimension. So the matrix can be shown as prodcut of eigenvalues and eigenvectors. where Q is the matrix of eigenvectors, is the matrix of eigenvalue. This equotion also mean if we know eigenvalues and eigenvectors we can construct the orignal matrix.
python: eig(matrix)
SVD(singluar value decomposition)
, where A is m*n, U is m*m matrix, is m*m diagonal matrix also known as singluar value, is n*n matrix.
python: svd(matrix)
reduce dimension
select top largest singluar values in
, where column select from , row selected from , B is approximate of the orignal matrix A.
`python: TruncatedSVD(n_components=2)
Stats
Multivari stats
variance: , python: var(vector, ddof=1)
standard deviation: , python:std(M, ddof=1, axis=0)
covariance: , python: cov(x,y)[0,1]
coralation: , normorlized to the value between -1 to 1. python: corrcoef(x,y)[0,1]
PCA
project high dimensions to subdimesnion
steps:
, which order by eigenvalue
scikit learn
pca = PCA(2) # get two components
pca.fit(A)
print(pca.componnets_) # values
print(pca.explained_variance_) # vectors
B = pca.transform(A) # transform to new matrix
Linear Regression
, where b is coeffcient and unkown
linear least squares( similar to MSE) , then . Issue: very slow
MSE with SDG
Reference: Basics of Linear Algebra for Machine Learning, jason brownlee, https://machinelearningmastery.com/linear_algebra_for_machine_learning/
For most of us who learned CNN, we already knew the convolutional operation is used for feature extraction in the spatial relationship. Compared with the full connection NN, it is good for weights sharing and translation invariant. There are many different convolutions. Recently, I found a very good article which summarized this topic. I translated it to English combined with my understanding. If you want to read the original one, you can go here.
1. Standard Convolution
1.1 Single channel
It’s element-wise multiply then sum together. The Convolutional filter moves forward each element in the picture. Here we set padding = 0, stride = 1. This is very useful for the gray picture.
1.2 multi channels
For the color pictures, they are made of 3 layers: Red, Green and Yellow. we create a 333 convolution which contains 3 convolutional kernels. Then we sum the three results togher to one channel 2D array.
This is our first apply ConvLSTM to CFD successfully! although the case is simple and under control of lots of factors. The ground factor is generated by Openfoam, and the custom model is predNet from coxlab. We trained three models in this time.
1.Training: use Nth frame to predict (N+1)th frame
Prediction: use 1-10th frame to predict 2-11th frame, then combined 11th frame in the predicted output with 1-10th frame. With new input(2-10th are ground truth, 11th is predicted), we can keep predicting 3-12th frame. In this experience, we predict the frames until 20th where sliding window = 1 frame. ( only first few frame are good, since we use the predicting frames to do the prediction)
2.Training: Nth frame to predict (N+10)th frame
Prediction: use 1-10 frame to predict 11-20 frame, no sliding window. ( this is very good since all input are ground truth)
3.Training: use Nth frame to predict (N+1)th frame
Prediction: use one frame to predict next frame, like driving prediction (animation has a little problem. right side is prediction, left side is ground truth)
For some reason, there is a request to predict video frames. We need that video is a combination of spatial and temporal dimensions. FCN and LSTM are good for them respectively. But for both of them, we need to use ConvLSTM. Since I just start to learn it, so I write down some of notes for good understanding.
1.First thing first, let’s see what LSTM looks like:
From left to right, we can see forget gate, input gate, input modulation gate and output gate. On the top side is memory pipe. It simulates the manner that human remember things. For more information, how the LSTM works please click here.
In keras, there are already three kinds of RNN: simpleRNN, LSTM and GRU. They are all easy to use.
2. What is ConvLSTM
Since LSTM is not good for spatial vector as input, which is only one dimension, they created ConvLSTM to allowed multidimensional data coming with convolutional operations in each gate.
We can find the basic formulas are as same as LSTM, they just use convolutional operations instead of one dimension for input, previous output and memory. Keras needs a new component which called ConvLSTM2D to wrap this ConvLSTM.
3. Where we use it?
As I said in the beginning, it is used for prediction with time and space. The already done in academic inculds: predict precipitation, video frame prediction, some physic movement activities. You can find more in my reference.
Reference:
1. the bounce ball. https://www.youtube.com/watch?v=RjZ1VKYyHhs
2. weather forecast. https://papers.nips.cc/paper/5955-convolutional-lstm-network-a-machine-learning-approach-for-precipitation-nowcasting.pdf
3. some video prediction. https://www.youtube.com/watch?v=MjFpgyWH-pk