NLP platform for Indian languages

India has the second largest population in the world after China with a fast growing economy. It is no surprise that many software and Internet companies are focusing on this fast growing market. Even though English is one of the official languages, not even 1% of Indian population speaks it.

The clear majority, 99.5%, speak languages such as Bengali, Gujarati, Hindi, Kannada, Marathi, Malayalam, Odia, Punjabi, Telugu, Tamil or Urdu, to name just a few of the 29 languages spoken in India at least by more than one million people.

This fact creates many challenges in developing applications for markets that rely on the understanding of text to function such as call center, social listening, search, virtual agents and market research. 

Text based applications need to understand language to achieve high quality support, so linguistic support must be developed for these languages. 

Linguistics includes functionalities such as part of speech tagging, lemmatization, phrase extraction, text categorization, entity extraction, topic extraction and parsing. There are many challenges to developing these types of NLP processing pipelines for Indic languages:

• Language complexity
• Differences in scripts
• Lack of language documented standards
• Difficulty in obtaining data

One of the toughest challenges to solve is the lack of literature about grammar, spellers or literature. Even if they are languages with thousands or millions native speakers there are not many resources available. 

Another difficulty our linguists faced is the variety of alphabets. Not only many Indic languages do not use Latin alphabet but also, they do use different scripts themselves. All of them are Brahmic derived alphabets, however there is a lot of difference among the languages spoken in North and South India.

To illustrate the differences among alphabets, take a look at the following example:

Bitext linguists team works jointly with native speakers to create complete dictionaries of all Indic languages that can be used for different purposes. For example, if combining these dictionaries with a lemmatizer algorithm will provide better results while searching for a specific term reducing the amount of noise.

Below you can see an example in few Indic languages of how our dictionaries work with lemmatization.

If you are interested in getting more information about our dictionaries and how you can use them for different applications like parsing, topic extraction or text categorization contact us!