Skip to content

About

Predicting Next word from novel. (LSTM)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LSTM Next Word Predictor

A Deep Learning Natural Language Processing (NLP) application designed to generate text sequences. This project implements a Long Short-Term Memory (LSTM) neural network to predict the most probable next sequence of words based on user input, trained on literary works extracted from PDF documents.

The project features a modern, interactive web interface built with Streamlit for real-time inference.

📋 Table of Contents

  • Overview
  • Key Features
  • Technical Architecture
  • Project Structure
  • Installation & Setup
  • Usage
  • The Science Behind It

Overview

Predictive text models are the foundation of modern Large Language Models (LLMs). This project demonstrates the fundamental concepts of generative AI by building a word-level language model. It ingests raw text (specifically "Three Short Novels" by John Galt), processes it into numerical sequences, and trains a Recurrent Neural Network (RNN) variant known as LSTM to learn the probability distribution of the next word given a sequence of previous words.

Key Features

  • Deep Learning Core: Utilizes a robust LSTM network capable of capturing long-term dependencies in text data.
  • PDF Data Ingestion: Includes a pipeline to extract and clean text directly from PDF files (data.pdf).
  • Interactive Web UI: A clean, dark-themed Streamlit application that allows users to input seed text and visualize the model's generation in real-time.
  • Modern Tooling: Built using Python 3.12 and managed by uv for extremely fast dependency resolution and environment management.

Technical Architecture

The model is built using TensorFlow/Keras with the following architecture:

  1. Input Layer: Accepts tokenized sequences (N-grams).
  2. Embedding Layer (256 dimensions): Transforms sparse word indices into dense vectors, capturing semantic relationships between words (e.g., understanding that "King" and "Queen" are related).
  3. LSTM Layer (256 units): The core memory component. Unlike standard RNNs, LSTMs have internal mechanisms (gates) that regulate the flow of information, allowing the model to "remember" context from earlier in the sentence and "forget" irrelevant information.
  4. Dropout Layer (0.2): Randomly deactivates neurons during training to prevent overfitting.
  5. Dense Output Layer: A fully connected layer with a Softmax activation function that outputs a probability score for every word in the vocabulary. The word with the highest probability is selected as the prediction.

Project Structure

lstm-python/
├── app.py               # Streamlit web application for inference
├── data.pdf             # Source training corpus (John Galt novels)
├── lstm.ipynb           # Main Jupyter Notebook for data prep and model training
├── lstm.py              # Python script version of the training logic
├── rnn.ipynb            # Experimental notebook for SimpleRNN comparison
├── pyproject.toml       # Project metadata and dependencies
├── uv.lock              # Locked dependency versions for reproducible builds
├── lstm_model.keras     # (Artifact) The trained model file
└── tokenizer_config.pkl # (Artifact) The saved tokenizer configuration

Installation & Setup

This project uses uv for lightning-fast Python package management.

Prerequisites

  • Python 3.12 or higher
  • uv installed (if not, run pip install uv)

Steps

  1. Clone the repository

    git clone https://github.com/OmPatil44/LSTM-Python.git
    cd LSTM-Python
  2. Install Dependencies Initialize the virtual environment and sync dependencies using uv:

    uv sync

    Alternatively, if you prefer standard pip:

    pip install -r pyproject.toml

Usage

1. Training the Model

Before you can use the application, you must train the model on the provided dataset.

  1. Open lstm.ipynb in Jupyter or VS Code.
  2. Run all cells. This process will:
    • Extract text from data.pdf.
    • Tokenize the text and create sequences.
    • Train the LSTM model (default: 30-40 epochs).
    • Save lstm_model.keras and tokenizer_config.pkl.

2. Running the Application

Once the model files are generated, launch the web interface:

uv run streamlit run app.py

(Or streamlit run app.py if using standard pip)

Navigate to http://localhost:8501 in your browser. Enter a phrase (e.g., "The adventure of") and click Predict Next Words to see the model generate text.

The Science Behind It

Why LSTM?

While basic Recurrent Neural Networks (RNNs) can handle sequences, they suffer from the Vanishing Gradient Problem. As the sequence gets longer, the RNN struggles to propagate error signals back to the beginning of the sequence, effectively having a "short-term memory."

LSTM (Long Short-Term Memory) networks solve this by introducing a "Cell State" that runs through the entire chain with only minor linear interactions. It uses three gates to regulate this highway:

  1. Forget Gate: Decides what information to throw away from the cell state.
  2. Input Gate: Decides which new information to store in the cell state.
  3. Output Gate: Decides what to output based on the cell state.

This allows the model to maintain context over long paragraphs, making it ideal for text generation trained on novels.


Created by ompatil44

About

Predicting Next word from novel. (LSTM)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages