A powerful, modular web scraping application built with Streamlit, LangChain, and Crawl4AI. This tool leverages Large Language Models (LLMs) to intelligently extract structured data from websites based on natural language user prompts.
- AI-Powered Extraction: Describe what you want to extract in plain English (e.g., "Extract all product names and prices"), and the AI handles the rest.
- Multi-Provider Support: Seamlessly switch between:
- Ollama (Local LLMs like Llama 3, Mistral)
- Gemini API (Google's multimodal models)
- OpenRouter API (Access to various top-tier models)
- Dynamic Schema Generation: Automatically creates Pydantic models on the fly to ensure structured, consistent data output.
- Robust Backend: Uses
Crawl4AIfor efficient web crawling andLangChainfor orchestration. - User-Friendly Interface: Clean and responsive UI built with Streamlit.
- Data Export: Download extracted data directly as CSV.
web_scraping_project/
├── .streamlit/ # Streamlit configuration
├── app/
│ ├── ui/ # Frontend components (Sidebar, Main Page)
│ └── backend/ # Core logic (Scraper, LLM Service, Data Pipeline)
├── data/
│ └── output/ # Extracted CSV files
├── testing/ # Unit examinations for responses
├── pyproject.toml # Project dependencies
└── README.md # You are here
- Frontend: Streamlit
- Orchestration: LangChain (LCEL), LangGraph
- Crawling: Crawl4AI
- LLMs: Ollama, Google Gemini, OpenRouter
- Dependency Management: UV
- Python 3.12+
- Ollama (if using local models)
This project uses pyproject.toml for dependency management.
-
Clone the repository:
git clone https://github.com/OmPatil44/web_scraping.git cd web_scraping -
Install dependencies:
uv sync
-
Run the application:
uv run streamlit run app/ui/main.py
- Launch the App: Run the streamlit command mentioned above.
- Configure Sidebar:
- Select your LLM Provider (Ollama, Gemini, or OpenRouter).
- Enter your API Key (if required).
- Select your Model.
- Wait for the "Validated API Key" or check status.
- Scrape Data:
- Enter the Target URL (e.g.,
https://books.toscrape.com/). - Enter your Extraction Prompt (e.g., "Get the title, price, and rating of all books on this page").
- Click Start Scraping.
- Enter the Target URL (e.g.,
- Download: Once finished, preview the data and click Download CSV.
Contributions are welcome! Please check the templates for bug or feature reporting.



