NLP Labeling for AI

Training AI models requires large volumes of data, and this in turn requires increasing levels of precision and efficiency in data labeling. At Bitext, we offer advanced linguistic tools designed for automated pre-labeling of datasets to help scale Data Annotation and Labeling (DAL) projects. We offer a full range of solutions:

• Multilingual: up to 77 languages (English, Spanish, French, German, Italian, Portuguese, Arabic, Chinese, Japanese, Korean…)
• Multiple NLP functions: NER, Sentiment, POS Tagging, Anonymization / PII Detection, Intent Detection…
• As Software:
  • Extremely efficient performance: multiplatform C libraries, 500,000 words per second w/ 8 CPUs
  • Flexible deployment: as SDK or as API, both in cloud and on-premise
• As Data for GenAI Model Training:
  • Rich data dictionaries: 80 Million tagged words in 77 languages
  • Rich tagged corpora: 50 Billion tagged & categorized words in 77 languages

• Bitext NAMER (Named Entity Recognition)
• Topic-Based Sentiment Analysis
• POS Tagging
• Anonymization / PII Detection

Bitext NAMER (Named Entity Recognition)

Bitext NAMER (Named Entity Recognition) is a fundamental component in the realm of natural language processing (NLP) that identifies and categorizes key information within text, such as names of people, organizations, locations, dates, and other entities. At Bitext, our NER services are specifically designed to enhance the capabilities of labeling platforms by providing precise and automated pre-labeling. This automation allows for the rapid and accurate identification of entities within large datasets, significantly reducing the manual effort required and increasing the overall efficiency of the annotation process. By integrating our Bitext NAMER tools, platforms can ensure that their AI models are trained with high-quality, contextually relevant data, leading to better performance and more reliable outcomes.

Example:

John lives in New York → “John” – person name, “New York” – place

Available As:

• Software
  • SDK: 100KB/s of raw text per CPU core
  • On-premise or SaaS API
• Data (Dictionaries)
  • 100,000+ entities per language
• Annotated Corpora (for model training)
  • From public domain sources
  • Annotation of custom corpora available on demand

Features:

• 20 Entity types: person, place, company/brand, organization, phone, account number…
• Software and annotated corpora available in English, Spanish, French, German, Italian, Portuguese, Dutch…
• Dictionaries available in 77 languages

Example:

I hate my old phone → opinion: “hate” (negative), topic: “my old phone”

Available As:

• Software
  • SDK: 50KB/s of raw text per CPU core
  • On-premise or SaaS API
• Data (Dictionaries)
  • 10,000 words and expressions labeled for sentiment (per language)
• Annotated Corpora (for model training)
  • From public domain sources
  • Annotation of custom corpora available on demand

Features:

• Returns the polarity, magnitude and corresponding topic of opinions in text
• Software and dictionaries available in Catalan, Dutch, English, French, German, Italian, Portuguese, Spanish
• Annotated corpora available in 22 languages and 10 variants

POS Tagging

Example:

John runs back home → “John” – proper noun (“John”), “runs” – verb (“run”), “back” – preposition (“back”), “home” – noun (“home”)

Available As:

• Software
  • SDK: 400KB/s of raw text per CPU core
  • On-premise or SaaS API
• Data (Dictionaries)
  • 30,000+ root words and up to 20M inflected word forms per language
• Annotated Corpora (for model training)
  • From public domain sources
  • Annotation of custom corpora available on demand

Features:

• Returns the part of speech and lemma for each word in a sentence
• Software and corpora available in 21 languages
• Dictionaries available in 77 languages

Example:

My name is John and my account number is 1234567 → My name is XXXX and my account number is XXXX

Available As:

• Software
  • SDK: 400KB/s of raw text per CPU core
  • On-premise or SaaS API
• Annotated Corpora (for model training)
  • From public domain sources
  • Annotation of custom corpora available on demand

Features:

• Remove sensitive or personal information (PII) from text
• Software and corpora available in 25 languages

Deployment

Our NLP labeling tools are available as software (either an on-premise SDK or through our SaaS API), dictionaries or annotated corpora.

SDK –  for maximum performance and scalability

• Performance:
  • Throughput: 50KB/s to 400KB/s of raw text per CPU core (depending on the label type)
  • Memory usage: 400MB
• Storage Footprint: 30MB (no additional dependencies)
• Portable: C/C++ library that can be called from any programming language (C/C++, Python, Java…) and any platform (Linux, macOS, Windows, iOS…)

API – for ease of integration

• On-premise API
• SaaS API

Additional Resources

We also offer additional NLP Resources including computational lexicons, synonym dictionaries, custom text corpora…