Today I’m gonna tell you a short story, about how natural language processing helps the life of two individuals in a food & beverage company.
Alice and Bob work for NextGen which is a large F&B industry in the Asian market. Alice works in the product R&D department and Bob works for the strategic management department. Alice is tasked to look for new ingredients that the company can use to produce next-generation superfoods. Bob, on the other hand, keeps track of changing regulations and economic developments in the region so that the company can better prepare for what next to come and even stay ahead.
Nightmare of “Am I missing out?”
Though Alice and Bob work in different departments and their focus areas are different, they both do one thing in common. They both go through piles of news articles, magazines, journals etc to find information. For Alice, it is always a tedious job to keep an eye on what are the new ingredients out there which help make or enhance the company’s superfood portfolio. For Bob, not only did he go through all these news articles and magazines, he even got people on the ground to report to him what’s changing in their respective markets. But for both of them, it is simply an impossible task to go through the text and find relevant information. They merely go through about 5% of the data available for the day and miss out on the rest of the 95%. Add to that, they both cannot read and understand an article if that is in languages like Chinese, Vietnamese or Korean. To ease their burden, they both have subscribed to “Google News Alerts”, which sends an email with the most recent news for a topic (or ingredient) you key in. Problem is Alice doesn’t know what is gonna be the name of the next best superfood ingredient and Bob doesn’t know what is gonna be the next “Food Tax” or “Financial incentive”. That makes it challenging to even use google alerts.
NER — From a nightmare to a dream
As with most other companies, NextGen’s digital transformation takes place during the pandemic and they subscribed to an AI platform which is allegedly capable of processing tens of thousands of news, magazines, social media posts and even company’s own documents and extract necessary information just as a human analyst would. More importantly, it is capable of identifying new information even if the human user is not key-in the terms to look for. Sound of this is music to Alice and Bob’s ears but they were too sceptical.
Alice set up a project on the platform, she wanted to track what are the new ingredients that are there as healthier alternatives. In just a matter of minutes, the system throws out a bunch of superfood ingredients such as;
- Watermelon seeds
- Maqui berry
- Nut oils — Almond, Cashew, Hazelnut
Bob was fascinated by what he saw on the platform. He set up a project to look for food taxes around the Asian market. This is what the system throws out for him.
- Junk food tax in the Philippines
- Ban of junk food ads in Malaysia
- An alcohol tax hike in Japan
How this information is presented
What Alice and Bob really liked is how intuitive to use the system to find the right information and remove the noisier ones by simply dropping them to “Dustbin”. If they really liked a particular term, they dropped it into their infobox and set alerts to be notified when new information made available anywhere in the world related to that theme.
Behind the screen of NER
What made this magic possible is Natural Language Processing (NLP) and particularly Named Entity Recognition (NER). Identifying named things such as persons, locations, countries, products, companies from an unstructured text is known as named entity recognition.
The AI system built into this platform is capable of processing a text document and identifying Products (Eg: Watermelon Seeds, Maqui berry) and Concepts (Eg: Sugar rax, Junk food tax) without explicitly telling the system.
Unprocessed output directly from NER engine:
An example of NER engine is picking up concepts:
How is the AI system capable of finding these topics which were not inputted as terms to search for? Thanks to semantic data processing which, in any given text the NER model is able to process it not as a just a set of characters, but with the meaning of the words and phrases.
A named entity recognition model will help us identify first occurrences of “Apple” as a company, “Tim Cook” as a person, second occurrences of “apple” as a fruit, “Marina Bay Sands” as a location and “Singapore” as a country. An NER model can only identify predefined classes of entities and it is possible to train a model to identify business requirement specific classes such as Food Items, Brands, Needs/Wants etc.
We at AI palette works on multiple natural language processing tasks and particularly on named entity recognition. Going beyond language boundaries, we process data from English, Chinese, Thai, German and Vietnamese etc.
Do reach out to us on email@example.com if you are keen on different use cases of NER for your work.
If you are interested in joining us as an Intern, Data Scientist or a Consultant reach out to me firstname.lastname@example.org