Artificial Intelligence for Natural Language Processing (NLP) Part II – From Word to Numerical Analysis

4. Cleaning process: Regular Expression (re)

Examples: @ali, @ahmed, #, ‘e’, ‘A12’, ‘A13’, … Can not be removed using NLTK

functions

It will process the text shared on web or on social media as String

• \d : Matches any decimal digit; this is equivalent to the class [0-9].

• \D: Matches any non-digit character; this is equivalent to the class [^0-9].

• \s: Matches any whitespace character; this is equivalent to the class [ \t\n\r\f\v].

• \S: Matches any non-whitespace character; this is equivalent to the class [^ \t\n\r\f\v].

• \w: Matches any alphanumeric character; this is equivalent to the class [a-zA-Z0-9_].

• \W: Matches any non-alphanumeric character; this is equivalent to the class [^a-zA-Z0-

9_].

• Exemple: Re.sub(r’,[^@],’ ‘) => @ @ @ @

Wael Ouarda - CRNS

7

4. Cleaning process: Regular Expression (re)

Pattern

Description

^ Matches beginning of line (^ab means that it starts with ab)

$ Matches end of line. ($a means that it ends with a)

. Matches any single character except newline. Using m option allows it to match newline as well. (Etc …)

[...] Matches any single character in brackets.

[^...] Matches any single character not in brackets

Wael Ouarda - CRNS

8

4. Cleaning process

Hi? How are you, I am very content to see you today :)!

1.

2.

3.

4.

5.

6.

7.

8.

9.

Tokenization: Split document into list of

words

Lower casing: Transform Upper case to

lower case

Stop words removal: Stop words is a list

of words=[‘When”, “I”, “How”, … ] (It

can be modified by removing some

words by adding other ones

Special Character removal: @#’” etc

Punctuation removal: :,;-,?! etc

Stemming: take the basic of the word:

player players plaied plays -> play

Lemmatization: have and had will be

considered have plays and played will

be considered as play

Spell check

Translation

[very,happy,see,today]

[Hi,?,,How,are,you,,,I,am,very,content,to,see,you,today, :,),!]

Tokenization

Punctuation Removal

[Hi,How,are,you,I,am,very,content,to,see,you,today,)]

Special Character Removal

[Hi,How,are,you,I,am,very,content,to,see,you,today]

[hy,how,are,you,i,am,very,content,to,see,you,today]

Lower case

Translation & Spell check

[hi,how,are,you,i,am,very,happy,to,see,you,today]

Stop words removal

Wael Ouarda - CRNS

Stop words removal

[very,happiness,see,today]

9

5. Sample of NLP Libraries for sentiment analysis

Sentiment = is a tuple of (Polarity, Subjectivity)

• Polarity in [-1 (Negative),1(Positive)]: The orientation of opinion behind the text;

• Subjectivity in [0,1]: Weight of Subjectivity of the text.

Data Collection

Data Cleaning

Data

Representation

Data

Classification

Wael Ouarda - CRNS

10

6. Word Embedding Techniques (TF-IDF)

TF-IDF: Term Frequency – Inverse Document Frequency

Terminology

• t — term (word)

• d — document (set of words)

• N — count of corpus

• Corpus — the total document set

TF(t,d) = count of t in d / number of words in d

DF(t) = occurrence of t in documents (IDF=N/df)

TF-IDF(t, d) = tf(t, d) * log(N/(df + 1))

user

Tweets

Label

Id1

Id1

Id2

Tweet 11 = [« word 111 », « word 112 »] -> TF

= [0,5, 0,5]

Tweet 12

Tweet 21

+

+

-

11

Wael Ouarda - CRNS

6. Word Embedding Techniques (TF-IDF)

Activity

TF(t,d) = count of t in d / number of words in d

DF(t) = occurrence of t in documents (IDF=N/df)

TF-IDF(t, d) = tf(t, d) * log(N/(df + 1))

user

Id1

Id1

Id2

Tweets

Label

[bonjour, ali, bienvenue, leaders]

[bonsoir, ahmed, leaders, souhaite, bienvenue,

ahmed]

[bonsoir, ali, ahmed]

+

+

-

N-gram to include context (N=3)

TF-IDF(‘bonjour’,id1) = tf(bonjour,id1) log (N/1)= 1 log(7/2)

TF-IDF(‘Ali’,id1) = tf(‘ali’, id1) log (7/df(‘ali)) = 1 log (7/3)

TF-IDF(‘Ali’,id2) = tf(‘ali’, id2) log (7/df(‘ali)) = 1 log (7/3)

TF-IDF(‘Ahmed,id1’) = 2 * log (7/4)

TF-IDF(‘Ahmed,id2’)

TF-IDF(‘bonsoir’)

TF-IDF(‘leaders’) = 1*log(7/3)

TF-IDF(‘souhaite’)

TF-IDF(‘bienvenue’) = 1* log(7/3)

[bonjour, ali, bienvenue] [ali, bienvenue, leaders]

[log(7/2), log (7/3), log(7/3)]

[log(7/3), log(7/3), , log(7/3)]

[bonjour, ali, bienvenue] [ali, bienvenue, leaders]

[bonsoir, ahmed, leaders] [ahmed, leaders, souhaite] [leaders, souhaite, bienvenue]

[bonsoir, ali, ahmed]

Wael Ouarda - CRNS

12

6. Word Embedding Techniques (Word2Vec)

Term = “machine”

Word Identification in

the Vocabulary

Yes/No

Features Vector

Bag of Words

Neural Network Training

WordNet is the dictionary (N) in default

prediction

Neural Network

prediction

Error: Out of vocabulary

“machine”

Wael Ouarda - CRNS

0

0

…

1

0

0

W

V

0

0

…

1

0

0

13

13

6. Word Embedding Techniques (Word2Vec)

Some facts about the Autoencoder:

●

●

●

Image Representation in Low Dimensional Level

It is an unsupervised learning algorithm (like PCA)

It minimizes the same objective function as PCA

It is a neural network

●

● The neural network’s target output is its input

z = f(Wx)

y = g(Vz)

X=Input Vector

X’: Output Vector

X=X’

Possible Derivatives of Autoencoder

W

V

Stacked Autoencoder

Sparse Autoencoder

Wael Ouarda - CRNS

14

6. Word Embedding Techniques (Word2Vec)

Activity

N=4 size of vocabulary

W is the size of features vector

[bonjour, ali, bienvenue]

Bonjour

ali

bienvenue

0

1

0

0

0

0

1

0

0

0

0

1

Input Weight Matrix

(4,W)

v11

V1w

V21

V2w

v31

V3w

user

Tweets

Id1

Id1

Id2

[bonjour, ali, bienvenue, leaders]

[bonsoir, ahmed, leaders, souhaite, bienvenue, ahmed]

[bonsoir, ali, ahmed]

Label

+

+

-

N-gram to include context (N=3)

(V11+V21+V31)/3

(V1W+V2W+V3W)/

3

Final Features

Vectors

Wael Ouarda - CRNS

15

7. Features Selection, Analysis and Transformation

• Transformation

• Linear Transformation: Principal Component Analysis (PCA)

• Non-Linear Transformation: Auto encoder

• Selection

• Heuristic Methods: Genetic Algorithm, Particle Swarm Optimization, Ant Colony

Optimization, etc.

• Statistical Methods: Correlation Matrix

Wael Ouarda - CRNS

16

7. Features Selection, Analysis and Transformation

A given dataset of size N features and M samples

Correlation Matrix is based on Pearson moment

Correlation Matrix

M(feature I, feature J) = covariance (I,J) / Variance(I) * Variance (J)

Example: N=3

N=2 (Feature I & III) or (Features II & III)

Feature I

Feature II

Features III

Feature I

M(I,I) = 1

M(II,I)

M(III,I)

Feature II

M(I,II)

M(II,II) = 1

M(III,II)

Feature III

M(I,III)

M(II,III)

M(III,III) = 1

M is in [-1;1]

[-1;-0,5]

]-0,5;0]

]0;0,5]

]0,5;1]

Features I & II are high correlated. So we

Can drop one among it

Feature I

Feature II

Features III

Feature I

M(I,I) = 1

0,6

-0,2

M(I,J)

I & J are

inversely

high

correlated

I & J are not

High inversely

correlated

I & J are not

high

correlated

I & J are high

correlated

Feature II

Feature III

0,6

-0,2

Wael Ouarda - CRNS

M(II,II) = 1

0,001

0,001

M(III,III) = 1

17

7. Features Selection, Analysis and Transformation

Compute Average

Vector

A = 1/N * Sum(Vi)

Adjustment of the Dataset

For i=1:N

Va = Vi - A

Principal Component Analysis

Dataset {Vi}

Sort for the proper

vector

85%

3/8

8/8

2/8

7/8

V1

V2

V3

V4

Vn

Adjustment of

the Dataset

Adjustment of

the Dataset

Example: Vector1= a1v1 + a2v2 + … an*vn

Dataset {Vai} adjusted

Compute the N proper vectors (vi)

Each vector from the old dataset

can be described as a weighted

sum of the proper vectors

Singular Value

Decomposition

Transform the dataset into

matrix N*N (N features)

Wael Ouarda - CRNS

18

8. NLP Applications

• NLP Classification

• Spam & Ham Detector

• Fake News Detecor

• Sentiment Analysis

• NLP Topic Modeling

• Word Cloud Visualisation

• Clustering data/ User -> Communities

• Chatbot

• Natural Lanagage Processing (NLP): to process the natural lanagege input by human

• Natural Lanagage Generation (NLG): to generate response to human

Wael Ouarda - CRNS

19