Sentiment Analysis witth TextBlob – Part I

Create a TextBlob

First, the import.

Let’s create our first TextBlob.

>>> from textblob import TextBlob

>>> wiki = TextBlob("Python is a high-level, general-purpose programming language.")

Part-of-speech Tagging

Part-of-speech tags can be accessed through the tags property.

>>> wiki.tags

[('Python', 'NNP'), ('is', 'VBZ'), ('a', 'DT'), ('high-level', 'JJ'), ('general-purpose', 'JJ'),

('programming', 'NN'), ('language', 'NN')]

Noun Phrase Extraction

Similarly, noun phrases are accessed through the noun_phrases property.

>>> wiki.noun_phrases

WordList(['python'])

Sentiment Analysis

The sentiment property returns a namedtuple of the form Sentiment(polarity, subjectivity). The polarity score is a float

within the range [-1.0, 1.0]. The subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and

1.0 is very subjective.

>>> testimonial = TextBlob("Textblob is amazingly simple to use. What great fun!")

>>> testimonial.sentiment

Sentiment(polarity=0.39166666666666666, subjectivity=0.4357142857142857)

Tokenization

>>> testimonial.sentiment.polarity

0.39166666666666666

You can break TextBlobs into words or sentences.

>>> zen = TextBlob("Beautiful is better than ugly. "

... "Explicit is better than implicit. "

... "Simple is better than complex.")

>>> zen.words

WordList(['Beautiful', 'is', 'better', 'than', 'ugly', 'Explicit', 'is', 'better', 'than', 'implicit',

'Simple', 'is', 'better', 'than', 'complex'])

>>> zen.sentences

[Sentence("Beautiful is better than ugly."), Sentence("Explicit is better than implicit."),

Sentence("Simple is better than complex.")]

Sentence objects have the same properties and methods as TextBlobs.

>>> for sentence in zen.sentences:

... print(sentence.sentiment)

Words Inflection and Lemmatization

Each word in TextBlob.words or Sentence.words is a Word object (a subclass of unicode) with useful methods, e.g. for word

inflection.

>>> sentence = TextBlob('Use 4 spaces per indentation level.')

>>> sentence.words

WordList(['Use', '4', 'spaces', 'per', 'indentation', 'level'])

>>> sentence.words[2].singularize()

'space'

>>> sentence.words[-1].pluralize()

'levels'

WordLists

A WordList is just a Python list with additional methods.

>>> animals = TextBlob("cat dog octopus")

>>> animals.words

WordList(['cat', 'dog', 'octopus'])

>>> animals.words.pluralize()

WordList(['cats', 'dogs', 'octopodes'])

Spelling Correction

Use the correct() method to attempt spelling correction.

>>> b = TextBlob("I havv goood speling!")

>>> print(b.correct())

I have good spelling!

Word objects have a spellcheck() Word.spellcheck() method that returns a list of (word, confidence) tuples with spelling

suggestions.

>>> from textblob import Word

>>> w = Word('falibility')

>>> w.spellcheck()

[('fallibility', 1.0)]

Get Word and Noun Phrase Frequencies

There are two ways to get the frequency of a word or noun phrase in a TextBlob.

The first is through the word_counts dictionary.

>>> monty = TextBlob("We are no longer the Knights who say Ni. "

... "We are now the Knights who say Ekki ekki ekki PTANG.")

>>> monty.word_counts['ekki']

If you access the frequencies this way, the search will not be case sensitive, and words that are not found will

have a frequency of 0.

The second way is to use the count() method.

>>> monty.words.count('ekki')

You can specify whether or not the search should be case-sensitive (default is False).

>>> monty.words.count('ekki', case_sensitive=True)

Each of these methods can also be used with noun phrases.

>>> wiki.noun_phrases.count('python')

Translation and Language Detection

TextBlobs can be translated between languages.

>>> en_blob = TextBlob(u'Simple is better than complex.')

>>> en_blob.translate(to='es')

TextBlob("Lo simple es mejor que lo complejo.")

If no source language is specified, TextBlob will attempt to detect the language. You can specify the source

language explicitly, like so. Raises TranslatorError if the TextBlob cannot be translated into the requested language

or NotTranslated if the translated result is the same as the input string.

>>> chinese_blob = TextBlob(u"美丽优于丑陋")

>>> chinese_blob.translate(from_lang="zh-CN", to='en')

TextBlob("Beauty is better than ugly")

You can also attempt to detect a TextBlob’s language using TextBlob.detect_language().

>>> b = TextBlob(u"عمجم نم لضفأ وه طيسب")

>>> b.detect_language()

'ar'

As a reference, language codes can be found here.

Language translation and detection is powered by the Google Translate API.

TextBlobs Are Like Python Strings!

You can use Python’s substring syntax.

>>> zen[0:19]

TextBlob("Beautiful is better")

You can use common string methods.

>>> zen.upper()

TextBlob("BEAUTIFUL IS BETTER THAN UGLY. EXPLICIT IS BETTER THAN

IMPLICIT. SIMPLE IS BETTER THAN COMPLEX.")

>>> zen.find("Simple")

You can make comparisons between TextBlobs and strings.

>>> apple_blob = TextBlob('apples')

>>> banana_blob = TextBlob('bananas')

>>> apple_blob < banana_blob

True

>>> apple_blob == 'apples'

True

You can concatenate and interpolate TextBlobs and strings.

>>> apple_blob + ' and ' + banana_blob

TextBlob("apples and bananas")

>>> "{0} and {1}".format(apple_blob, banana_blob)

'apples and bananas'

n-grams

The TextBlob.ngrams() method returns a list of tuples of n successive words.

>>> blob = TextBlob("Now is better than never.")

>>> blob.ngrams(n=3)

[WordList(['Now', 'is', 'better']), WordList(['is', 'better', 'than']), WordList(['better', 'than',

'never'])]

Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

1/7

123456import numpy as np import pandas as pd import reimport nltk import matplotlib.pyplot as plt%matplotlib inline12data_source_url = "https://raw.githubusercontent.com/kolaveridi/kaggle-Twitter-US-Airline-Sentiment-/master/Tweets.csv"airline_tweets = pd.read_csv(data_source_url)tweet_idairline_sentimentairline_sentiment_confidencenegativereasonnegativereason_confidenceairlineairl0570306133677760513neutral1.0000NaNNaNVirginAmerica1570301130888122368positive0.3486NaN0.0000VirginAmerica2570301083672813571neutral0.6837NaNNaNVirginAmerica3570301031407624196negative1.0000Bad Flight0.7033VirginAmerica4570300817074462722negative1.0000Can't Tell1.0000VirginAmerica1airline_tweets.head()Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

2/7

6.0 4.0 123456plot_size = plt.rcParams["figure.figsize"] print(plot_size[0]) print(plot_size[1])plot_size[0] = 8plot_size[1] = 6plt.rcParams["figure.figsize"] = plot_size <matplotlib.axes._subplots.AxesSubplot at 0x7fe41fdf19e8>1airline_tweets.airline.value_counts().plot(kind='pie', autopct='%1.0f%%')1airline_tweets.airline_sentiment.value_counts().plot(kind='pie', autopct='%1.0f%%', colors=["red", "yellow", "green"])Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

3/7

<matplotlib.axes._subplots.AxesSubplot at 0x7fe41fd2ec18>12airline_sentiment = airline_tweets.groupby(['airline', 'airline_sentiment']).airline_sentiment.count().unstack()airline_sentiment.plot(kind='bar')Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

4/7

<matplotlib.axes._subplots.AxesSubplot at 0x7fe41f83c160>12import seaborn as snssns.barplot(x='airline_sentiment', y='airline_sentiment_confidence' , data=airline_tweets)Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

5/7

<matplotlib.axes._subplots.AxesSubplot at 0x7fe41f016e48>12features = airline_tweets.iloc[:, 10].valueslabels = airline_tweets.iloc[:, 1].values1678101195432import nltk for sentence in range(0, len(features)): # Remove all the special characters processed_feature = re.sub(r'\W', ' ', str(features[sentence])) processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature) # Remove single characters from the start # remove all single charactersprocessed_features = []from sklearn.feature_extraction.text import TfidfVectorizerfrom nltk.corpus import stopwordsnltk.download('stopwords')Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

6/7

[nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Unzipping corpora/stopwords.zip. 12131415161718192021 processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature) # Substituting multiple spaces with single space processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I) # Removing prefixed 'b' processed_feature = re.sub(r'^b\s+', '', processed_feature) # Converting to Lowercase processed_feature = processed_feature.lower() processed_features.append(processed_feature)vectorizer = TfidfVectorizer (max_features=2500, min_df=7, max_df=0.8, stop_words=stopwords.words('english'))processed_features = vectorizer.fit_transform(processed_features).toarray()12from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(processed_features, labels, test_size=0.2, random_state=0)1predictions = text_classifier.predict(X_test)12345from sklearn.metrics import classification_report, confusion_matrix, accuracy_score print(confusion_matrix(y_test,predictions))print(classification_report(y_test,predictions))print(accuracy_score(y_test, predictions))RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=None, max_features='auto', max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=200, n_jobs=None, oob_score=False, random_state=0, verbose=0, warm_start=False)123from sklearn.ensemble import RandomForestClassifiertext_classifier = RandomForestClassifier(n_estimators=200, random_state=0)text_classifier.fit(X_train, y_train)Python for NLP: Sentiment Analysis with Scikit-Learn.ipynb - Colaboratory

7/7

14/02/2020

Word2Vec Training.ipynb - Colaboratory

https://colab.research.google.com/drive/1wjHN_1NmDSjWd9y656tSOHAY6f9HwECC#scrollTo=O_WMPIU1ApyE&uniqifier=1&printMode=true

1/5

Collecting gensim Downloading https://files.pythonhosted.org/packages/d1/dd/112bd4258cee11e0baaaba064060eb156475a42362e59e3ff28e7ca2d29d/gensim |████████████████████████████████| 24.2MB 147kB/s Requirement already satisfied, skipping upgrade: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.4.1) Requirement already satisfied, skipping upgrade: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.17.5) Requirement already satisfied, skipping upgrade: smart-open>=1.8.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.9Requirement already satisfied, skipping upgrade: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.12.0) Requirement already satisfied, skipping upgrade: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.8.1->gensimRequirement already satisfied, skipping upgrade: boto>=2.32 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.8.1->gRequirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.8.1->genRequirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->sRequirement already satisfied, skipping upgrade: botocore<1.15.0,>=1.14.14 in /usr/local/lib/python3.6/dist-packages (from boto3Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requestsRequirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requestsRequirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->smartRequirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->smRequirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botRequirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1Installing collected packages: gensim Found existing installation: gensim 3.6.0 Uninstalling gensim-3.6.0: Successfully uninstalled gensim-3.6.0 Successfully installed gensim-3.8.1 1!pip install --upgrade gensim!p123import pandas as pddf = pd.read_csv('data.csv')df.head()14/02/2020

Word2Vec Training.ipynb - Colaboratory

https://colab.research.google.com/drive/1wjHN_1NmDSjWd9y656tSOHAY6f9HwECC#scrollTo=O_WMPIU1ApyE&uniqifier=1&printMode=true

2/5

Word2Vec Training.ipynb - Colaboratory

https://colab.research.google.com/drive/1wjHN_1NmDSjWd9y656tSOHAY6f9HwECC#scrollTo=O_WMPIU1ApyE&uniqifier=1&printMode=true

3/5

MakeModelYearEngineFuelTypeEngineHPEngineCylindersTransmissionTypeDriven_WheelsNumberofDoorsMarket CategoryVehicleSizeVehicleStyleh0BMW1SeriesM2011premiumunleaded(required)335.06.0MANUALrear wheel drive2.0FactoryTuner,Luxury,High-PerformanceCompactCoupe1BMW1Series2011premiumunleaded(required)300.06.0MANUALrear wheel drive2.0Luxury,PerformanceCompactConvertible2BMW1Series2011premiumunleaded(required)300.06.0MANUALrear wheel drive2.0Luxury,High-PerformanceCompactCoupe3BMW1Series2011premiumunleaded(required)230.06.0MANUALrear wheel drive2.0Luxury,PerformanceCompactCoupe4BMW1Series2011premiumunleaded(required)230.06.0MANUALrear wheel drive2.0LuxuryCompactConvertible12345678910# Select features from original dataset to form a new dataframe df1 = df[['Engine Fuel Type','Transmission Type','Driven_Wheels','Market Category','Vehicle Size', 'Vehicle Style', 'Maker_Model']# For each row, combine all the columns into one columndf2 = df1.apply(lambda x: ','.join(x.astype(str)), axis=1)# Store them in a pandas dataframedf_clean = pd.DataFrame({'clean': df2})# Create the list of list format of the custom corpus for gensim modeling sent = [row.split(',') for row in df_clean['clean']]# show the example of list of list format of the custom corpus for gensim modeling sent[:2]14/02/2020

Word2Vec Training.ipynb - Colaboratory

https://colab.research.google.com/drive/1wjHN_1NmDSjWd9y656tSOHAY6f9HwECC#scrollTo=O_WMPIU1ApyE&uniqifier=1&printMode=true

4/5

[['premium unleaded (required)', 'MANUAL', 'rear wheel drive', 'Factory Tuner', 'Luxury', 'High-Performance', 'Compact', 'Coupe', 'BMW 1 Series M'], ['premium unleaded (required)', 'MANUAL', 'rear wheel drive', 'Luxury', 'Performance', 'Compact', 'Convertible', 'BMW 1 Series']]<gensim.models.word2vec.Word2Vec at 0x7f769bd08e48>12from gensim.models import Word2Vecmodel = Word2Vec(sent, min_count=1,size= 50,workers=3, window =3, sg = 1)1model['Toyota Camry']14/02/2020

Word2Vec Training.ipynb - Colaboratory

https://colab.research.google.com/drive/1wjHN_1NmDSjWd9y656tSOHAY6f9HwECC#scrollTo=O_WMPIU1ApyE&uniqifier=1&printMode=true

5/5

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated __getitem__ (Method wil """Entry point for launching an IPython kernel. array([-0.06417835, -0.09275129, 0.05958268, -0.02864562, -0.0538294 , -0.14630224, 0.07459477, -0.1708039 , -0.0107823 , -0.20295398, -0.03064371, 0.07940909, 0.23251899, -0.0032685 , 0.18781912, -0.07640981, -0.06949015, -0.17593247, 0.1422481 , -0.05508935, -0.09981751, -0.10790525, -0.084803 , -0.0601706 , 0.09693231, 0.05144079, -0.28539822, 0.14235567, 0.08585091, -0.10648351, 0.38443646, -0.10456032, 0.03479529, 0.03759629, -0.13798805, 0.00146567, -0.02155977, -0.09146847, -0.02380568, -0.02360898, 0.12138011, -0.13577844, -0.00547892, -0.22444133, 0.21260163, 0.09178843, -0.01907091, 0.03841864, -0.07984174, 0.41191557], dtype=float32)[-0.08366955 -0.05964565 0.07203643 -0.04613186 -0.09290247 -0.1433193 0.02276208 -0.12569097 -0.00580759 -0.16596414 -0.06013276 0.10451026 0.24728008 0.00752487 0.17998965 -0.06541821 -0.1521897 -0.1394007 0.11704888 -0.04366257 -0.09647141 -0.08375981 -0.08183265 -0.04778878 0.08877282 0.08159016 -0.27344024 0.165536 0.08464111 -0.10411307 0.35802853 -0.12511638 0.01041985 0.06723206 -0.08702769 0.01015439 -0.00365196 -0.06646202 0.00494511 -0.00929783 0.08671172 -0.15059675 -0.01291043 -0.22367048 0.16581357 0.08654065 -0.01826465 0.05042068 -0.09550156 0.3554561 ] 50123vector = model.wv['Toyota Camry']print(vector)vector.size14/02/2020

NLP with FastText.ipynb - Colaboratory

https://colab.research.google.com/drive/1fT1wXEJQ9ZgKJJDInSbzThKPzJhsRZiT#scrollTo=zzZpLuX5TZ24&printMode=true

1/4

Collecting wikipedia Downloading https://files.pythonhosted.org/packages/67/35/25e68fbc99e672127cc6fbb14b8ec1ba3dfef035bf1e4c90f78f24a80b7d/wikipedRequirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from wikipedia) (4.6.3) Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from wikipedia) (2.21.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipeRequirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikRequirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikRequirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia)Building wheels for collected packages: wikipedia Building wheel for wikipedia (setup.py) ... done Created wheel for wikipedia: filename=wikipedia-1.4.0-cp36-none-any.whl size=11686 sha256=a34bcb0113f1c8c797bcb65cf61aa9d83a61 Stored in directory: /root/.cache/pip/wheels/87/2a/18/4e471fd96d12114d16fe4a446d00c3b38fb9efcb744bd31f4a Successfully built wikipedia Installing collected packages: wikipedia Successfully installed wikipedia-1.4.0 1!pip install wikipedia!p123456789101112131415161718from keras.preprocessing.text import Tokenizerfrom gensim.models.fasttext import FastTextimport numpy as npimport matplotlib.pyplot as pltimport nltkfrom string import punctuationfrom nltk.corpus import stopwordsfrom nltk.tokenize import word_tokenizefrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import sent_tokenizefrom nltk import WordPunctTokenizerimport wikipediaimport nltknltk.download('punkt')nltk.download('wordnet')nltk.download('stopwords')en_stop = set(nltk.corpus.stopwords.words('english'))%matplotlib inline14/02/2020

NLP with FastText.ipynb - Colaboratory

https://colab.research.google.com/drive/1fT1wXEJQ9ZgKJJDInSbzThKPzJhsRZiT#scrollTo=zzZpLuX5TZ24&printMode=true

2/4

Using TensorFlow backend. The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.We recommend you upgrade now or ensure your notebook will continue to use TensorFlow 1.x via the %tensorflow_version 1.x magic: more info.[nltk_data] Downloading package punkt to /root/nltk_data... [nltk_data] Unzipping tokenizers/punkt.zip. [nltk_data] Downloading package wordnet to /root/nltk_data... [nltk_data] Unzipping corpora/wordnet.zip. [nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Unzipping corpora/stopwords.zip. 1234567891011artificial_intelligence = wikipedia.page("Artificial Intelligence").contentmachine_learning = wikipedia.page("Machine Learning").contentdeep_learning = wikipedia.page("Deep Learning").contentneural_network = wikipedia.page("Neural Network").contentartificial_intelligence = sent_tokenize(artificial_intelligence)machine_learning = sent_tokenize(machine_learning)deep_learning = sent_tokenize(deep_learning)neural_network = sent_tokenize(neural_network)artificial_intelligence.extend(machine_learning)artificial_intelligence.extend(deep_learning)artificial_intelligence.extend(neural_network)1234567891011121314import refrom nltk.stem import WordNetLemmatizerstemmer = WordNetLemmatizer()def preprocess_text(document): # Remove all the special characters document = re.sub(r'\W', ' ', str(document)) # remove all single characters document = re.sub(r'\s+[a-zA-Z]\s+', ' ', document) # Remove single characters from the start document = re.sub(r'\^[a-zA-Z]\s+', ' ', document) # Substituting multiple spaces with single space document = re.sub(r'\s+', ' ', document, flags=re.I) # Removing prefixed 'b' document = re.sub(r'^b\s+', '', document)14/02/2020

NLP with FastText.ipynb - Colaboratory

https://colab.research.google.com/drive/1fT1wXEJQ9ZgKJJDInSbzThKPzJhsRZiT#scrollTo=zzZpLuX5TZ24&printMode=true

3/4

151617181920212223 # Converting to Lowercase document = document.lower() # Lemmatization tokens = document.split() tokens = [stemmer.lemmatize(word) for word in tokens] tokens = [word for word in tokens if word not in en_stop] tokens = [word for word in tokens if len(word) > 3] preprocessed_text = ' '.join(tokens) return preprocessed_textartificial intelligence advanced technology present 12345sent = preprocess_text("Artificial intelligence, is the most advanced technology of the present era")print(sent)final_corpus = [preprocess_text(sentence) for sentence in artificial_intelligence if sentence.strip() !='']word_punctuation_tokenizer = nltk.WordPunctTokenizer()word_tokenized_corpus = [word_punctuation_tokenizer.tokenize(sent) for sent in final_corpus]1234embedding_size = 60window_size = 40min_word = 5down_sampling = 1e-2CPU times: user 1min 39s, sys: 318 ms, total: 1min 40s Wall time: 51.3 s 12345678%%timeft_model = FastText(word_tokenized_corpus, size=embedding_size, window=window_size, min_count=min_word, sample=down_sampling, sg=1, iter=100)1print(ft_model.wv['artificial'])14/02/2020

NLP with FastText.ipynb - Colaboratory

https://colab.research.google.com/drive/1fT1wXEJQ9ZgKJJDInSbzThKPzJhsRZiT#scrollTo=zzZpLuX5TZ24&printMode=true

4/4

[-0.30679983 0.14952238 -0.21810903 -0.7395383 -0.23074621 0.10807946 0.04915254 0.08504375 0.0450432 -0.04488863 -0.15089108 -0.31847388 0.30588818 -0.19040418 0.05474224 0.05302594 -0.10397033 0.3067033 0.09131836 0.03521951 0.46956265 0.11327234 0.26022822 -0.03658391 -0.2489354 -0.22652984 0.2772911 -0.23342757 0.6781643 0.08554471 -0.05705503 0.09560576 -0.2070933 0.01806752 -0.42353478 -0.44353613 0.06448416 0.5410614 0.08621874 0.05771734 0.08044741 0.14682104 0.0913202 -0.12521906 0.2854835 -0.30264926 0.04221692 -0.14606497 -0.13468763 0.21106029 -0.2290192 -0.45784584 -0.289197 -0.16205801 -0.645687 0.02343115 0.2587803 -0.24801745 -0.23193733 0.03140956] Social Media Sentiment Analysis II.ipynb - Colaboratory

1/8

12345678910import reimport pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as snsimport stringimport nltkimport warnings warnings.filterwarnings("ignore", category=DeprecationWarning)%matplotlib inlineidlabeltweet010@user when a father is dysfunctional and is s...120@user @user thanks for #lyft credit i can't us...230bihday your majesty340#model i love u take with u all the time in ...450factsguide: society now #motivation123train = pd.read_csv('https://raw.githubusercontent.com/dD2405/Twitter_Sentiment_Analysis/master/train.csv')train_original=train.copy()train_original.head()123test = pd.read_csv('https://raw.githubusercontent.com/dD2405/Twitter_Sentiment_Analysis/master/test.csv')test_original=test.copy()test_original.head()Social Media Sentiment Analysis II.ipynb - Colaboratory

2/8

idtweet031963#studiolife #aislife #requires #passion #dedic...131964@user #white #supremacists want everyone to s...231965safe ways to heal your #acne!! #altwaystohe...331966is the hp and the cursed child book up for res...4319673rd #bihday to my amazing, hilarious #nephew...idlabeltweet010.0@user when a father is dysfunctional and is s...120.0@user @user thanks for #lyft credit i can't us...230.0bihday your majesty340.0#model i love u take with u all the time in ...450.0factsguide: society now #motivation12combine = train.append(test,ignore_index=True,sort=True)combine.head()1combine.tail()Social Media Sentiment Analysis II.ipynb - Colaboratory

3/8

idlabeltweet4915449155NaNthought factory: left-right polarisation! #tru...4915549156NaNfeeling like a mermaid ð(cid:0)(cid:0)(cid:0) #hairflip #neverre...4915649157NaN#hillary #campaigned today in #ohio((omg)) &am...4915749158NaNhappy, at work conference: right mindset leads...4915849159NaNmy song "so glad" free download! #shoegaze ...idlabeltweetTidy_Tweets010.0@user when a father is dysfunctional and is s...when a father is dysfunctional and is so sel...120.0@user @user thanks for #lyft credit i can't us...thanks for #lyft credit i can't use cause th...230.0bihday your majestybihday your majesty340.0#model i love u take with u all the time in ...#model i love u take with u all the time in ...450.0factsguide: society now #motivationfactsguide: society now #motivation12345678910def remove_pattern(text,pattern): # re.findall() finds the pattern i.e @user and puts it in a list for further task r = re.findall(pattern,text) # re.sub() removes @user from the sentences in the dataset for i in r: text = re.sub(i,"",text) return text combine['Tidy_Tweets'] = np.vectorize(remove_pattern)(combine['tweet'], "@[\w]*")combine.head()12combine['Tidy_Tweets'] = combine['Tidy_Tweets'].str.replace("[^a-zA-Z#]", " ")combine.head(10)Social Media Sentiment Analysis II.ipynb - Colaboratory

4/8

idlabeltweetTidy_Tweets010.0@user when a father is dysfunctional and is s...when a father is dysfunctional and is so sel...120.0@user @user thanks for #lyft credit i can't us...thanks for #lyft credit i can t use cause th...230.0bihday your majestybihday your majesty340.0#model i love u take with u all the time in ...#model i love u take with u all the time in ...450.0factsguide: society now #motivationfactsguide society now #motivation560.0[2/2] huge fan fare and big talking before the...huge fan fare and big talking before the...670.0@user camping tomorrow @user @user @user @use...camping tomorrow danny780.0the next school year is the year for exams.ð(cid:0)(cid:0)...the next school year is the year for exams ...890.0we won!!! love the land!!! #allin #cavs #champ...we won love the land #allin #cavs #champ...9100.0@user @user welcome here ! i'm it's so #gr...welcome here i m it s so #gr12combine['Tidy_Tweets'] = combine['Tidy_Tweets'].apply(lambda x: ' '.join([w for w in x.split() if len(w)>3]))combine.head(10)Social Media Sentiment Analysis II.ipynb - Colaboratory

5/8

idlabeltweetTidy_Tweets010.0@user when a father is dysfunctional and is s...when father dysfunctional selfish drags kids i...120.0@user @user thanks for #lyft credit i can't us...thanks #lyft credit cause they offer wheelchai...230.0bihday your majestybihday your majesty340.0#model i love u take with u all the time in ...#model love take with time450.0factsguide: society now #motivationfactsguide society #motivation560.0[2/2] huge fan fare and big talking before the...huge fare talking before they leave chaos disp...670.0@user camping tomorrow @user @user @user@use...camping tomorrow danny780.0the next school year is the year for exams.ð(cid:0)(cid:0)...next school year year exams think about that #...890.0we won!!! love the land!!! #allin #cavs #champ...love land #allin #cavs #champions #cleveland #...0 [when, father, dysfunctional, selfish, drags, ... 1 [thanks, #lyft, credit, cause, they, offer, wh... 2 [bihday, your, majesty] 3 [#model, love, take, with, time] 4 [factsguide, society, #motivation] Name: Tidy_Tweets, dtype: object12tokenized_tweet = combine['Tidy_Tweets'].apply(lambda x: x.split())tokenized_tweet.head()1234from nltk import PorterStemmerps = PorterStemmer()tokenized_tweet = tokenized_tweet.apply(lambda x: [ps.stem(i) for i in x])tokenized_tweet.head()13/02/2020

Social Media Sentiment Analysis II.ipynb - Colaboratory

6/8

0 [when, father, dysfunct, selfish, drag, kid, i... 1 [thank, #lyft, credit, caus, they, offer, whee... 2 [bihday, your, majesti] 3 [#model, love, take, with, time] 4 [factsguid, societi, #motiv] Name: Tidy_Tweets, dtype: objectidlabeltweetTidy_Tweets010.0@user when a father is dysfunctional and is s...when father dysfunct selfish drag kid into dys...120.0@user @user thanks for #lyft credit i can't us...thank #lyft credit caus they offer wheelchair ...230.0bihday your majestybihday your majesti340.0#model i love u take with u all the time in ...#model love take with time450.0factsguide: society now #motivationfactsguid societi #motiv1234for i in range(len(tokenized_tweet)): tokenized_tweet[i] = ' '.join(tokenized_tweet[i])combine['Tidy_Tweets'] = tokenized_tweetcombine.head()1all_words_positive = ' '.join(text for text in combine['Tidy_Tweets'][combine['label']==0])123456Mask = np.array(Image.open(requests.get('http://clipart-library.com/image_gallery2/Twitter-PNG-Image.png', stream=True).raw))# We use the ImageColorGenerator library from Wordcloud # Here we take the color of the image and impose it over our wordcloudimage_colors = ImageColorGenerator(Mask)# Now we use the WordCloud function from the wordcloud library wc = WordCloud(background_color='black', height=1500, width=4000,mask=Mask).generate(all_words_positive)1234from wordcloud import WordCloud,ImageColorGeneratorfrom PIL import Imageimport urllibimport requestsSocial Media Sentiment Analysis II.ipynb - Colaboratory

7/8

12345678# Size of the image generated plt.figure(figsize=(10,20))# Here we recolor the words from the dataset to the image's color# recolor just recolors the default colors to the image's blue color# interpolation is used to smooth the image generated plt.imshow(wc.recolor(color_func=image_colors),interpolation="hamming")plt.axis('off')plt.show()1 Social Media Sentiment Analysis II.ipynb - Colaboratory

https://colab.research.google.com/drive/1UCnu0fEoO-75aMH_U9wahCioOfxaR2f2#scrollTo=3qw7f2usx0ZJ&printMode=true

8/8

15/02/2020

Topic Modeling with NLP.ipynb - Colaboratory

https://colab.research.google.com/drive/1GwTaDT5qygw_O38LC3wAHXpDU2zK6GeI#scrollTo=FvKI8_b1BnG4&printMode=true

1/12

[nltk_data] Downloading package stopwords to /root/nltk_data... [nltk_data] Unzipping corpora/stopwords.zip. Requirement already satisfied: en_core_web_sm==2.1.0 from https://github.com/explosion/spacy-models/releases/download/en_core_we✔ Download and installation successful You can now load the model via spacy.load('en_core_web_sm') ✔ Linking successful /usr/local/lib/python3.6/dist-packages/en_core_web_sm --> /usr/local/lib/python3.6/dist-packages/spacy/data/en You can now load the model via spacy.load('en') 1234# Run in python consoleimport nltk; nltk.download('stopwords')# Run in terminal or command prompt!python3 -m spacy download en123456789101112131415171819202116import reimport numpy as npimport pandas as pdfrom pprint import pprint # Gensimimport gensimimport gensim.corpora as corporafrom gensim.utils import simple_preprocessfrom gensim.models import CoherenceModel # spacy for lemmatizationimport spacy # Plotting toolsimport pyLDAvisimport pyLDAvis.gensim # don't skip thisimport matplotlib.pyplot as plt%matplotlib inline !pip install pyLDAvis15/02/2020

Topic Modeling with NLP.ipynb - Colaboratory

https://colab.research.google.com/drive/1GwTaDT5qygw_O38LC3wAHXpDU2zK6GeI#scrollTo=FvKI8_b1BnG4&printMode=true

2/12

222324252627# Enable logging for gensim - optionalimport logginglogging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.ERROR) import warningswarnings.filterwarnings("ignore",category=DeprecationWarning)15/02/2020

Topic Modeling with NLP.ipynb - Colaboratory

https://colab.research.google.com/drive/1GwTaDT5qygw_O38LC3wAHXpDU2zK6GeI#scrollTo=FvKI8_b1BnG4&printMode=true

3/12

Collecting pyLDAvis Downloading https://files.pythonhosted.org/packages/a5/3a/af82e070a8a96e13217c8f362f9a73e82d61ac8fff3a2561946a97f96266/pyLDAvi |████████████████████████████████| 1.6MB 3.3MB/s Requirement already satisfied: wheel>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (0.34.2) Requirement already satisfied: numpy>=1.9.2 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (1.17.5) Requirement already satisfied: scipy>=0.18.0 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (1.4.1) Requirement already satisfied: pandas>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (0.25.3) Requirement already satisfied: joblib>=0.8.4 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (0.14.1) Requirement already satisfied: jinja2>=2.7.2 in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (2.11.1) Requirement already satisfied: numexpr in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (2.7.1) Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (3.6.4) Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyLDAvis) (0.16.0) Collecting funcy Downloading https://files.pythonhosted.org/packages/ce/4b/6ffa76544e46614123de31574ad95758c421aae391a1764921b8a81e1eae/funcy-1 |████████████████████████████████| 552kB 24.5MB/s Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.0->pyLDAvis) (2018.9) Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.0->pyLDAvis) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2>=2.7.2->pyLDAvis) (1.1.1Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (45.1.0) Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (1.12.0) Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (0.7.1) Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (19.3.0) Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (1.8.1) Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (1.3.0) Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->pyLDAvis) (8.2.0)...

Sentiment Analysis with TextBlob – Part I