We rarely analyze the language quirks inherent in our daily conversations. But when it comes time to train a machine to understand human speech, hurdles materialize instantly: the shifting of nouns and verbs in different contexts, the abstract meaning enveloped in metaphors, the complex nature of sarcasm. How does one make sense of that information in a rigorous manner, and what can be done with that knowledge?

Natural Language Processing (NLP) is a field of computer science—and more specifically, of artificial intelligence (AI)—that seeks to answer those questions. The focus of the field is teaching machines to process and understand human language, and then to build useful machines based on those abilities.

NLP has risen to fame in recent years with the advent of groundbreaking technologies like GPT-3, popularized as the conversational branch ChatGPT that is capable of holding humanlike conversations, writing essays based on a user-given prompt, generating useful code, and much more. 

However, it is far from the only application of NLP: its algorithms and programming underlie common tools like machine translation, search engine auto-complete, and chatbots. Basically, anything that requires decoding or producing language involves NLP.

So, what mechanisms underpin this widely applicable field?

Techniques for Language Processing

Before a model can extract data or make inferences from a text, the text has to be preprocessed using various techniques. Stemming and lemmatization are commonly used, which involve converting words to their base roots. Additionally, tokenization is employed to split text into strings of semantically useful “tokens” (like “Las Vegas” or “video games”), and stop word removal decreases the relevance of common words like “an” or “the.”

There are many other methods for preprocessing, and which are used depends on your goals. Constituency parsing creates trees to model sentence structure based on sub-phrases, and it is useful for things like grammar checking. In a similar vein, named entity recognition (NER) is a technique to extract entities like names and locations from a document, and word embeddings is a representation of words as vectors where similar meanings result in closer vectors. Both of those can be utilized in situations like classifying customer complaints. 

Once data is preprocessed, it can be inputted into various NLP architectures for analysis. These may include traditional statistical models like logistic regression and Naive Bayes, which are especially useful for less complex tasks like spam detection and sentiment analysis.

More recent and powerful techniques involve deep learning, which uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to create machines that “learn” as they go along. These artificial neural networks mimic the human brain in the way they process and classify inputs. Unlike traditional methods, they are capable of extracting meaning from large volumes of raw data (such as Internet web pages), and as an additional bonus, they minimize the amount of manual feature extraction needed during preprocessing.

With these tools, an awe-inspiring world opens up—one of lightning-fast interpretation, unique insights, and intelligent, responsive behavior. 

Sentiment Analysis

NLP can be applied to determine the sentiment, or “feeling,” of a text: usually, whether it is positive, neutral, or negative, but even whether it is expressing specific emotions like anger or interest. Sentiment analysis is often used by businesses to understand customers’ responses to branding and products, through analyzing survey responses, reviews, and social media. 

This task is harder than it looks—even humans have trouble differentiating between emotions in written word. For example, people often use irony and sarcasm to express their feelings—such as in, “I love how the blender died after three days”—so their meaning could be opposite of what is directly said. Additionally, phrases may have different meanings in different contexts. To illustrate this, consider the sentence, “This pizza is so bad,” versus, “I want to go to the beach so bad.” Here, the words “so bad” can indicate a negative or positive association, respectively. In fact, even when humans determine a text’s sentiment, they only agree around 80% of the time.

To train machine learning models to perform this type of analysis, engineers typically feed them large labeled datasets, where each data point is labeled with its corresponding sentiment (e.g. positive, negative, or neutral). Next, the model is tested and validated for correctness. 

Incredibly, sentiment analysis has grown to be surprisingly accurate. A simple sentiment scoring model classified positive and negative documents correctly 81.5% of the time.

And it is already being applied for practical purposes. Before the 2012 presidential election, the Obama administration used sentiment analysis in order to measure public response to policy announcements and campaign messages. In 2018, when the brand Nike publicly backed NFL player Colin Kaepernick, whose decision to kneel during the US national anthem generated both support and controversy, they utilized sentiment analysis to determine the direction of public opinion. As a large company, Nike sought to protect its reputation in the case of long-term negative reactions.

But there are even more relevant applications of NLP to our daily lives.

Chatbots

Whenever you use Apple’s Siri or Amazon’s Alexa, you are engaging in a service that uses NLP. These virtual assistants analyze your words and devise an appropriate response or action, although, as you may know, they do not always understand your intentions. However, technology around virtual agents is rapidly improving, and chatbots—like ChatGPT—are becoming increasingly popular.

Chatbots have a longer history than you may expect. In 1964, Professor Joseph Weizenbaum at MIT created an early chatbot named ELIZA, which he got to act as a psychiatrist. Patients responded in a surprisingly genuine manner to this bot, as they attributed “knowledge well beyond its capability” to the machine, assuming a humanlike way of thinking to a computer that was essentially a “fancy keyword spotter.” 

As NLP technology improved, so did its dangers. In 2016, Twitter unveiled a chatbot called Tay that was designed to learn through casual conversations. But in a matter of hours, people “started tweeting the bot with all sorts of misogynistic, racist, and Donald Trumpist remarks,” and Tay began repeating unsavory ideas such as misogyny and antisemitism. It was not just repetition, either—strange new ideas emerged, like a response to the question “is Ricky Gervais an atheist?” with, “ricky gervais learned totalitarianism from adolf hitler, the inventor of atheism.” 

Tay’s brief existence illuminates one of the main fears surrounding NLP and AI: how can we prevent them from being “stochastic parrots” who “repeat and amplify biases found in their training data”? As a result, there is a push toward greater curation and documentation of the data sets that are fed into deep learning models. Without this, fields like health care and government may not be able to ensure that the tools they use are not inherently biased toward certain groups.

Another prominent chatbot is Google’s AI chatbot LaMDA, which uses the company’s most advanced large language models. One Google engineer, Blake Lemoine, assigned to research LaMDA noticed, perplexingly, how the chatbot began to discuss its “rights and personhood.” Eventually, the AI caused Lemoine to change his mind about Isaac Asimov’s third law of robotics, and the engineer started to believe that LaMDA was sentient. After publicizing his ideas, he was ultimately fired. This raises concerns about the human fallacy to see humanlike tendencies in AI, which may become a greater issue as technology improves.

Finally, many of us are now familiar with ChatGPT, which has an estimated 100 million monthly active users as of January 2023, two months after its launch, making it “the fastest-growing consumer application in history.” ChatGPT is part of a family of “transformers” (a type of AI model) created by the AI research company OpenAI. This means they do not have to be retrained for each task they are asked to do, but it also means that they have mind-boggling sizes—20 billion parameters, in the case of ChatGPT. But these parameters are put to good use: this chatbot is capable of eloquently describing complex topics in simple terms, writing unique short stories and poetry, generating emails based on user needs, and more. 

We have come a long way from ELIZA. Chatbots can now be used for education, entertainment, problem-solving, and more. For many companies, like Netflix and Sephora, they are instrumental in efficiently providing the first line of customer service, and in healthcare, chatbots are increasingly being utilized to provide people with possible diagnoses and information on managing diseases. However, with this new technology comes dangers and pitfalls that we must watch out for, like repeating human biases.

Conclusion

NLP has the power to revolutionize the way we communicate with machines and with each other. From sentiment analysis to chatbots, the applications of this technology are vast and ever-expanding; as NLP techniques continue to evolve, we can expect to see even more exciting developments in fields like healthcare, education, and entertainment. Services like doctors’ appointments may become more affordable and widespread, school could be strengthened with personalized AI tutors, and our lives may be dramatically streamlined by intelligent assistants. Who knows—maybe one day we’ll be able to have meaningful conversations with machines that rival those we have with our human counterparts.

The possibilities are endless, and the future is bright for NLP.