Question Answering in NLTK in Python 3: A Comprehensive Guide to Question Answering

Ashish saini Leave a comment
Question Answering in NLTK | Innovate Yourself
49
0

Introduction to Question Answering in nltk:

Welcome, aspiring Python enthusiasts! If you’re on a quest to master the Python language and elevate your skills, you’re in the right place. In today’s digital era, data is abundant, and extracting meaningful insights is key. In this blog post, we’re going to delve into the fascinating realm of Natural Language Processing (NLP) with Python, specifically focusing on question answering using the Natural Language Toolkit (NLTK). Grab your coding gear, fire up PyCharm, and let’s embark on this knowledge-packed journey together!

Chapter 1: Unraveling Question Answering the NLTK Magic

Before we dive into the code, let’s take a moment to understand what NLTK is and why it’s a game-changer for NLP in Python. NLTK, or Natural Language Toolkit, is a powerful library that provides tools for working with human language data. It’s your Swiss Army knife for tasks like tokenization, stemming, tagging, parsing, and more.

Imagine you have a dataset containing paragraphs of text, and you want your Python script to answer questions about it. NLTK makes this seemingly complex task a breeze. It’s like having a linguistic superhero at your coding fingertips.

Chapter 2: Setting Up Your PyCharm Environment | Question Answering

Before we get our hands dirty with NLTK, let’s ensure your PyCharm environment is set up for success. Open PyCharm, create a new Python project, and install the NLTK library using the following command:

pip install nltk

Now, let’s download some essential Question Answering in NLTK resources:

import nltk

nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')

With the stage set, it’s time to move on to the real action.

Chapter 3: The Anatomy of Question Answering with NLTK

To demonstrate the power of NLTK in question answering, let’s work with a real-world dataset. We’ll use the famous “20 Newsgroups” dataset, a collection of approximately 20,000 newsgroup documents, spanning 20 different categories.

For the sake of brevity, let’s focus on one category – let’s say ‘comp.graphics.’

from sklearn.datasets import fetch_20newsgroups

# Load the dataset
newsgroups = fetch_20newsgroups(subset='train', categories=['comp.graphics'])

# Display a sample document
print(newsgroups.data[0])
From: [email protected] (Jerry Lee)
Subject: Cobra 2.0 1-b-1 Video card HELP ME!!!!
Organization: The TSoft BBS and Public Access Unix, +1 415 969 8238
Lines: 22

Does ANYONE out there in Net-land have any information on the Cobra 2.20
card?  The sticker on the end of the card reads
        Model: Cobra 1-B-1
        Bios:  Cobra v2.20

I Havn't been able to find anything about it from anyone!  If you have
any information on how to get a hold of the company which produces the
card or know where any drivers are for it, PLEASE let me know!

As far as I can tell, it's a CGA card that is taking up 2 of my 16-bit
ISA slots but when I enable the test patterns, it displays much more than
the usualy 4 CGA colors... At least 16 from what I can count.. Thanks!

              .------------------------------------------.
              : Internet: [email protected]          :
              :           [email protected]  :
              :           [email protected]    :
              :           [email protected]             :
              : UUCP    : apple.com!tsoft!bbs.mirage     :
              `------------------------------------------'

                    Computer and Video Imaging Major

Chapter 4: Tokenization – The First Step in Understanding Language | Question Answering

Tokenization is the process of breaking down a text into individual words or phrases, commonly referred to as tokens. NLTK’s word_tokenize function is your trusty sidekick for this task.

from nltk.tokenize import word_tokenize

# Tokenize the sample document
tokens = word_tokenize(newsgroups.data[0])

# Display the tokens
print(tokens)
# tokens for Question Answering in nltk
['From', ':', 'bbs.mirage', '@', 'tsoft.net', '(', 'Jerry', 'Lee', ')', 'Subject', ':', 'Cobra', '2.0', '1-b-1', 'Video', 'card', 'HELP', 'ME', '!', '!', '!', '!', 'Organization', ':', 'The', 'TSoft', 'BBS', 'and', 'Public', 'Access', 'Unix', ',', '+1', '415', '969', '8238', 'Lines', ':', '22', 'Does', 'ANYONE', 'out', 'there', 'in', 'Net-land', 'have', 'any', 'information', 'on', 'the', 'Cobra', '2.20', 'card', '?', 'The', 'sticker', 'on', 'the', 'end', 'of', 'the', 'card', 'reads', 'Model', ':', 'Cobra', '1-B-1', 'Bios', ':', 'Cobra', 'v2.20', 'I', 'Hav', "n't", 'been', 'able', 'to', 'find', 'anything', 'about', 'it', 'from', 'anyone', '!', 'If', 'you', 'have', 'any', 'information', 'on', 'how', 'to', 'get', 'a', 'hold', 'of', 'the', 'company', 'which', 'produces', 'the', 'card', 'or', 'know', 'where', 'any', 'drivers', 'are', 'for', 'it', ',', 'PLEASE', 'let', 'me', 'know', '!', 'As', 'far', 'as', 'I', 'can', 'tell', ',', 'it', "'s", 'a', 'CGA', 'card', 'that', 'is', 'taking', 'up', '2', 'of', 'my', '16-bit', 'ISA', 'slots', 'but', 'when', 'I', 'enable', 'the', 'test', 'patterns', ',', 'it', 'displays', 'much', 'more', 'than', 'the', 'usualy', '4', 'CGA', 'colors', '...', 'At', 'least', '16', 'from', 'what', 'I', 'can', 'count', '..', 'Thanks', '!', '.', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '.', ':', 'Internet', ':', 'jele', '@', 'eis.calstate.edu', ':', ':', 'bbs.mirage', '@', 'gilligan.tsoft.net', ':', ':', 'bbs.mirage', '@', 'tsoft.sf-bay.org', ':', ':', 'mirage', '@', 'thetech.com', ':', ':', 'UUCP', ':', 'apple.com', '!', 'tsoft', '!', 'bbs.mirage', ':', '`', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', '--', "'", 'Computer', 'and', 'Video', 'Imaging', 'Major']

Chapter 5: Part-of-Speech Tagging – Understanding the Role of Words | Question Answering

Now that we have our tokens, let’s unravel the grammatical mysteries by assigning parts of speech to each word using NLTK’s pos_tag function.

from nltk import pos_tag

# Perform part-of-speech tagging
pos_tags = pos_tag(tokens)

# Display the part-of-speech tags
print(pos_tags)
[('From', 'IN'), (':', ':'), ('bbs.mirage', 'NN'), ('@', 'JJ'), ('tsoft.net', 'NN'), ('(', '('), ('Jerry', 'NNP'), ('Lee', 'NNP'), (')', ')'), ('Subject', 'NN'), (':', ':'), ('Cobra', 'JJ'), ('2.0', 'CD'), ('1-b-1', 'JJ'), ('Video', 'NNP'), ('card', 'NN'), ('HELP', 'NNP'), ('ME', 'NNP'), ('!', '.'), ('!', '.'), ('!', '.'), ('!', '.'), ('Organization', 'NN'), (':', ':'), ('The', 'DT'), ('TSoft', 'NNP'), ('BBS', 'NNP'), ('and', 'CC'), ('Public', 'NNP'), ('Access', 'NNP'), ('Unix', 'NNP'), (',', ','), ('+1', 'VBD'), ('415', 'CD'), ('969', 'CD'), ('8238', 'CD'), ('Lines', 'NNS'), (':', ':'), ('22', 'CD'), ('Does', 'NNP'), ('ANYONE', 'NNP'), ('out', 'IN'), ('there', 'RB'), ('in', 'IN'), ('Net-land', 'NNP'), ('have', 'VBP'), ('any', 'DT'), ('information', 'NN'), ('on', 'IN'), ('the', 'DT'), ('Cobra', 'NNP'), ('2.20', 'CD'), ('card', 'NN'), ('?', '.'), ('The', 'DT'), ('sticker', 'NN'), ('on', 'IN'), ('the', 'DT'), ('end', 'NN'), ('of', 'IN'), ('the', 'DT'), ('card', 'NN'), ('reads', 'VBZ'), ('Model', 'NNP'), (':', ':'), ('Cobra', 'NNP'), ('1-B-1', 'JJ'), ('Bios', 'NNS'), (':', ':'), ('Cobra', 'NNP'), ('v2.20', 'NN'), ('I', 'PRP'), ('Hav', 'VBP'), ("n't", 'RB'), ('been', 'VBN'), ('able', 'JJ'), ('to', 'TO'), ('find', 'VB'), ('anything', 'NN'), ('about', 'IN'), ('it', 'PRP'), ('from', 'IN'), ('anyone', 'NN'), ('!', '.'), ('If', 'IN'), ('you', 'PRP'), ('have', 'VBP'), ('any', 'DT'), ('information', 'NN'), ('on', 'IN'), ('how', 'WRB'), ('to', 'TO'), ('get', 'VB'), ('a', 'DT'), ('hold', 'NN'), ('of', 'IN'), ('the', 'DT'), ('company', 'NN'), ('which', 'WDT'), ('produces', 'VBZ'), ('the', 'DT'), ('card', 'NN'), ('or', 'CC'), ('know', 'VB'), ('where', 'WRB'), ('any', 'DT'), ('drivers', 'NNS'), ('are', 'VBP'), ('for', 'IN'), ('it', 'PRP'), (',', ','), ('PLEASE', 'NNP'), ('let', 'VB'), ('me', 'PRP'), ('know', 'VB'), ('!', '.'), ('As', 'RB'), ('far', 'RB'), ('as', 'IN'), ('I', 'PRP'), ('can', 'MD'), ('tell', 'VB'), (',', ','), ('it', 'PRP'), ("'s", 'VBZ'), ('a', 'DT'), ('CGA', 'NNP'), ('card', 'NN'), ('that', 'WDT'), ('is', 'VBZ'), ('taking', 'VBG'), ('up', 'RP'), ('2', 'CD'), ('of', 'IN'), ('my', 'PRP$'), ('16-bit', 'JJ'), ('ISA', 'NNP'), ('slots', 'NNS'), ('but', 'CC'), ('when', 'WRB'), ('I', 'PRP'), ('enable', 'VBP'), ('the', 'DT'), ('test', 'NN'), ('patterns', 'NNS'), (',', ','), ('it', 'PRP'), ('displays', 'VBZ'), ('much', 'RB'), ('more', 'JJR'), ('than', 'IN'), ('the', 'DT'), ('usualy', 'JJ'), ('4', 'CD'), ('CGA', 'NNP'), ('colors', 'NNS'), ('...', ':'), ('At', 'IN'), ('least', 'JJS'), ('16', 'CD'), ('from', 'IN'), ('what', 'WP'), ('I', 'PRP'), ('can', 'MD'), ('count', 'VB'), ('..', 'JJ'), ('Thanks', 'NNS'), ('!', '.'), ('.', '.'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('.', '.'), (':', ':'), ('Internet', 'NN'), (':', ':'), ('jele', 'NN'), ('@', 'VBZ'), ('eis.calstate.edu', 'NN'), (':', ':'), (':', ':'), ('bbs.mirage', 'NN'), ('@', 'JJ'), ('gilligan.tsoft.net', 'NN'), (':', ':'), (':', ':'), ('bbs.mirage', 'NN'), ('@', 'JJ'), ('tsoft.sf-bay.org', 'JJ'), (':', ':'), (':', ':'), ('mirage', 'NN'), ('@', 'JJ'), ('thetech.com', 'NN'), (':', ':'), (':', ':'), ('UUCP', 'NN'), (':', ':'), ('apple.com', 'NN'), ('!', '.'), ('tsoft', 'NN'), ('!', '.'), ('bbs.mirage', 'NN'), (':', ':'), ('`', '``'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ('--', ':'), ("'", 'POS'), ('Computer', 'NNP'), ('and', 'CC'), ('Video', 'NNP'), ('Imaging', 'NNP'), ('Major', 'NNP')]

Chapter 6: Named Entity Recognition – Identifying Key Entities | Question Answering

In question answering, recognizing named entities is crucial. NLTK’s ne_chunk function helps identify entities like people, organizations, and locations.

from nltk import ne_chunk

# Perform named entity recognition
ner_result = ne_chunk(pos_tags)

# Display the named entities
print(ner_result)
(S
  From/IN
  :/:
  bbs.mirage/NN
  @/JJ
  tsoft.net/NN
  (/(
  (PERSON Jerry/NNP Lee/NNP)
  )/)
  Subject/NN
  :/:
  Cobra/JJ
  2.0/CD
  1-b-1/JJ
  Video/NNP
  card/NN
  (ORGANIZATION HELP/NNP)
  ME/NNP
  !/.
  !/.
  !/.
  !/.
  Organization/NN
  :/:
  The/DT
  (ORGANIZATION TSoft/NNP)
  BBS/NNP
  and/CC
  (PERSON Public/NNP Access/NNP Unix/NNP)
  ,/,
  +1/VBD
  415/CD
  969/CD
  8238/CD
  Lines/NNS
  :/:
  22/CD
  Does/NNP
  ANYONE/NNP
  out/IN
  there/RB
  in/IN
  (GPE Net-land/NNP)
  have/VBP
  any/DT
  information/NN
  on/IN
  the/DT
  Cobra/NNP
  2.20/CD
  card/NN
  ?/.
  The/DT
  sticker/NN
  on/IN
  the/DT
  end/NN
  of/IN
  the/DT
  card/NN
  reads/VBZ
  (PERSON Model/NNP)
  :/:
  Cobra/NNP
  1-B-1/JJ
  Bios/NNS
  :/:
  Cobra/NNP
  v2.20/NN
  I/PRP
  Hav/VBP
  n't/RB
  been/VBN
  able/JJ
  to/TO
  find/VB
  anything/NN
  about/IN
  it/PRP
  from/IN
  anyone/NN
  !/.
  If/IN
  you/PRP
  have/VBP
  any/DT
  information/NN
  on/IN
  how/WRB
  to/TO
  get/VB
  a/DT
  hold/NN
  of/IN
  the/DT
  company/NN
  which/WDT
  produces/VBZ
  the/DT
  card/NN
  or/CC
  know/VB
  where/WRB
  any/DT
  drivers/NNS
  are/VBP
  for/IN
  it/PRP
  ,/,
  (ORGANIZATION PLEASE/NNP)
  let/VB
  me/PRP
  know/VB
  !/.
  As/RB
  far/RB
  as/IN
  I/PRP
  can/MD
  tell/VB
  ,/,
  it/PRP
  's/VBZ
  a/DT
  (ORGANIZATION CGA/NNP)
  card/NN
  that/WDT
  is/VBZ
  taking/VBG
  up/RP
  2/CD
  of/IN
  my/PRP$
  16-bit/JJ
  (ORGANIZATION ISA/NNP)
  slots/NNS
  but/CC
  when/WRB
  I/PRP
  enable/VBP
  the/DT
  test/NN
  patterns/NNS
  ,/,
  it/PRP
  displays/VBZ
  much/RB
  more/JJR
  than/IN
  the/DT
  usualy/JJ
  4/CD
  CGA/NNP
  colors/NNS
  .../:
  At/IN
  least/JJS
  16/CD
  from/IN
  what/WP
  I/PRP
  can/MD
  count/VB
  ../JJ
  Thanks/NNS
  !/.
  ./.
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  ./.
  :/:
  Internet/NN
  :/:
  jele/NN
  @/VBZ
  eis.calstate.edu/NN
  :/:
  :/:
  bbs.mirage/NN
  @/JJ
  gilligan.tsoft.net/NN
  :/:
  :/:
  bbs.mirage/NN
  @/JJ
  tsoft.sf-bay.org/JJ
  :/:
  :/:
  mirage/NN
  @/JJ
  thetech.com/NN
  :/:
  :/:
  UUCP/NN
  :/:
  apple.com/NN
  !/.
  tsoft/NN
  !/.
  bbs.mirage/NN
  :/:
  `/``
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  --/:
  '/POS
  (ORGANIZATION Computer/NNP)
  and/CC
  (PERSON Video/NNP Imaging/NNP Major/NNP))

Chapter 7: Crafting Your Question Answering Algorithm

Now that we’ve laid the groundwork, let’s put it all together to create a simple question answering algorithm. For instance, let’s ask our model about graphic design software.

import nltk
from sklearn.datasets import fetch_20newsgroups

# Download NLTK resources
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')

# Load the "20 Newsgroups" dataset
newsgroups = fetch_20newsgroups(subset='train', categories=['comp.graphics'])

# Sample document
sample_document = newsgroups.data[0]

# Tokenization
tokens = nltk.word_tokenize(sample_document)

# Part-of-speech tagging
pos_tags = nltk.pos_tag(tokens)

# Named Entity Recognition
ner_result = nltk.ne_chunk(pos_tags)

def extract_entities(named_entities, entity_type):
    """
    Extracts entities of a specified type from the named entities result.
    """
    entities = [entity[0] for entity in named_entities if isinstance(entity, tuple) and entity[1] == entity_type]
    return entities

def question_answering_algorithm(question, named_entities):
    """
    Simple question-answering algorithm based on named entities.
    """
    if 'software' in question.lower():
        # Extract software-related entities
        software_entities = extract_entities(named_entities, 'ORGANIZATION')
        if software_entities:
            return f"The document mentions the following graphic design software: {', '.join(software_entities)}"
        else:
            return "No graphic design software is mentioned in the document."
    else:
        return "I'm sorry, I can't answer that question."

# Define the question
question = "What graphic design software is mentioned in the document?"

# Apply the question-answering algorithm
answer = question_answering_algorithm(question, ner_result)

# Display the answer
print(answer)

Chapter 8: Visualizing Results of Question Answering in nltk with Matplotlib

To enhance your understanding and impress your peers, let’s visualize the results using Matplotlib. For example, a bar chart showcasing the frequency of different named entities can be incredibly insightful.

import matplotlib.pyplot as plt
from collections import Counter

# Extract named entities
named_entities = [entity for entity in ner_result if isinstance(entity, tuple)]

# Count the frequency of each entity type
entity_counts = Counter(entity[1] for entity in named_entities)

# Plot the results
plt.bar(entity_counts.keys(), entity_counts.values())
plt.xlabel('Named Entity Types')
plt.ylabel('Frequency')
plt.title('Named Entity Frequency in the Document')
plt.show()
Question Answering in NLTK | Innovate Yourself

Conclusion of Question Answering in NLTK:

Congratulations, you’ve just scratched the surface of NLTK-powered question answering! As you continue your Python journey, remember that NLTK is a versatile tool that opens up endless possibilities in the world of natural language processing.

In this blog post, we explored tokenization, part-of-speech tagging, named entity recognition, and even crafted a simple question answering algorithm using NLTK. The “20 Newsgroups” dataset served as our real-world playground, and Matplotlib added a visual dimension to our insights.

Stay curious, keep coding, and before you know it, you’ll be wielding NLTK like a pro in your Python adventures.

Also, check out our other playlist Rasa Chatbot, Internet of things, Docker, Python Programming, Machine Learning, Natural Language Processing, MQTT, Tech News, ESP-IDF etc.
Become a member of our social family on youtube here.
Stay tuned and Happy Learning. ✌🏻😃
Happy coding, and may your NLP endeavors be both enlightening and rewarding! ❤️🔥🚀🛠️🏡💡

About Ashish saini

View all posts by Ashish saini →

You may like these posts

Leave a Reply