Sentiment Analysis is a procedure used to determine if a chunk of text is positive, negative or neutral. In text analytics, natural language processing (NLP) and machine learning (ML) techniques are combined to assign sentiment scores to the topics, categories or entities within a phrase.

Table of Contents:

• What is sentiment Analysis?
• How does sentiment analysis work?
• What is a sentiment library?
• What is the role of Part of Speech tagging in sentiment analysis?
• A combination of ML and NLP for Sentiment Analysis
• How is sentiment analysis used?
  • Sentiment Analysis for Customer Experience
  • Sentiment Analysis for Employee Experience
• Further Reading

What is Sentiment Analysis?

Data analysts make use of sentiment analysis to extract information for market research, monitor brand and product reputation purposes. This procedure is also quite useful to get what customers think and act upon this enhancing the so-called customer experience.

What’s more, companies involved in analyzing data generally integrate third-party sentiment analysis APIs into their own infrastructures to obtain useful insights and provide their own customers with them.

Keep reading and we’ll make it clear for you how sentiment analysis works and which are the pros and cons of a rule-based sentiment analysis process, outlining, at the same time, the role of NLP and machine learning techniques.

How does Sentiment Analysis Work?

Opposite to general sentence-based sentiment analysis systems, the Bitext Sentiment Analysis tool applies sentiment to each topic in a single sentence. The internal process follows these steps:

• First, a text is split into its basic components: tokens, sentences, phrases, entities…
• Then, each topic and related words are identified.
• Finally, a sentiment score is assigned to each topic (-1, +4, 0…)

This is best explained using examples:

What is a Sentiment library?

An efficient sentiment analysis system must rely on a proper sentiment library to correctly detect sentiments and scores in words or phrases.

Sentiment libraries are composed by a collection of dictionaries including adjectives and phrases which have been manually scored beforehand. This scoring process must be done carefully so that the sentiment analysis system can differentiate, afterwards, between words like ‘bad and ‘horrible’, holding ‘horrible’ a more negative meaning.

Apart from that, if you have a multilingual sentiment analysis engine, there must be libraries for each language supported. Every library can be personalized and modified as needed: adding or removing phrases/words, fine-tuning scores…

Once those dictionaries are prepared, a series of rules must be written in the software so the computer is able to properly detect the sentiment expressed towards a particular topic, based on its nearness to other positive and negative words. A clear example is the adjective phrase ‘a bit disappointed’ in comparison with ‘totally disappointed’ which bears a more negative sentiment.

POS-Tagging in Sentiment Analysis

To properly analyze a sentence for sentiment, there is a need to break it down into pieces involving, as briefly seen above, several sub-processes, including POS-tagging. Part of Speech tagging consists of an identification of the basic elements of a text, such as verbs, nouns, adjectives, and adverbs.

Many languages follow some rules and patterns in terms of word formation that can be translated for a computer program to develop a basic POS-tagger. Nevertheless, a trustworthy sentiment analysis system must be built upon an accurate natural language understanding software to obtain precise POS-tagging results, which are crucial to identifying diverse phrase combinations.

For the Curious…

Let’s take a closer look at the following phenomenon in a hotel review. Here, we can see again how negative constructions and intensifiers have a noteworthy impact on sentiment analysis:

A sentence-based sentiment scoring system based on rules can draw false conclusions since it will see that ‘not so good’ describes the hotel, assign a negative sentiment score, and move on to the next review. However, human readers will clearly see that this review actually tells a different story.

Even though the customer didn’t like the hotel itself, the view from the room contributes to better customer experience. Therefore, an automated sentiment analysis system must take into account every single word and structure to assign proper sentiment scores.

As seen before, an efficient sentiment analysis system must contain rules for every word combination in its sentiment library. Creating and maintaining these rules is not a piece of cake. And, after all, strict rules can’t always keep up with the evolution of natural language.

Instant messaging is all the rage right now, and it’s impossible for a text pattern analysis system to have so many rules at disposal as for every abbreviation, misspelling, acronym or pun that may appear in every text document. When something new comes up in a text document to which there is no rule to be applied, the system can’t assign a score.

In the end, if you need to do an analysis detecting subtle differences in meaning, you must look for a tool that employs both machine learning and natural language processing techniques.

A combination of ML and NLP for Sentiment Analysis

The main role of machine learning techniques in sentiment analysis is to automate the text analytics functions that sentiment analysis relies on (segmentation, POS-tagging, entity extraction…). For example, when data scientists train a machine learning model by feeding it with a great number of text documents containing pre-tagged examples, it will automatically detect sentiment analysis in future documents.

This is possible thanks to supervised and unsupervised machine learning techniques, such as neural networks and deep learning.

Machine learning also helps data analysts solve context-dependent problems caused by the evolution of natural language. For example, the adjective ‘burned-out’ may bear different meanings.

However, considering training methods as feeding machine learning models with thousand pre-tagged examples, the ML system can learn to understand what ‘burned-out’ means in the context of fire, versus in the context of work life.

Some hybrid sentiment analysis systems combine machine learning with natural language processing techniques to reach higher accuracy. At this point, it is important to make a difference between natural language processing and machine learning.

On the one hand, an NLP-based sentiment analysis becomes an effective tool to build a foundation for POS-tagging and sentiment analysis. On the other hand, machine learning techniques can help solve complex natural language processing tasks, such as understanding double-meanings through automated training.

A combination of ML and NLP techniques will, therefore, cover the entire text analytics procedure for sentiment analysis, from low-level segmentation and syntax analysis up to semantic differentiation depending on the context in which a word appears.

How Is Sentiment Analysis Used?

Generally speaking, sentiment analysis is mostly used as a tool for Voice of Customer and Voice of Employee with different purposes. Companies use sentiment analysis to understand how customers and employees feel about some topics, and to learn the main reason for those opinions.

These insights are then used to enhance customer/employee experience which contributes to higher incomes and stronger productivity for the company:

1. Sentiment Analysis for Customer Experience

In today’s social media world, most of your customers are using social networks, particularly Twitter, to talk and express their opinions about your brand, product, and services.

Analyzing tweets, online reviews and news articles is quite useful for social media managers or business analysts to get insights into how customers feel and act upon it afterward. For this task, automated sentiment analysis is an essential component as a language processing tool.

2. Sentiment Analysis for Employee Experience

A significant percentage of workers leave their jobs each year, while another portion is fired or let go. In this sense, HR teams are starting to take actions, with the help of data analytics, to understand such tendencies and reduce turnover while yielding a performance improvement.

Getting what employees are talking about and how they feel about your company is possible thanks to sentiment analysis systems, and this helps workforce analysts reduce employee churn.

Summing up, sentiment analysis tools serve to extract insights which are crucial to getting what users think and react on time to improve their user experience.

To this end, several machine learning and natural language processing techniques must be combined allowing enterprises to get more accurate results. Do you want to see it in a practical context? Go to our API and try it live now.

Further reading:

Where can I learn more about sentiment analysis?

• Sentiment Analysis for bot conversations.
• Standford NLP or Natural Language Toolkit (NLTK) open source distributions.
• A Practitioner´s Guide to NLP.
• Course on text mining.