A lightning-fast, zero-cloud CLI utility that cleans your Windows directories by parsing file contents and visual data β automatically destroying clones and generating intelligently named folder structures.
- Overview
- Features
- Architecture
- Project Structure
- Tech Stack
- Supported File Formats
- Prerequisites
- Installation
- Usage
- How It Works
- Configuration
- Roadmap
- Contributing
- License
Vortex uses state-of-the-art embedding models to intelligently group your files. Whether you have text documents, code, PDFs, or images β it semantically analyzes their content entirely locally. Your data never leaves your machine.
The Problem: Over time, directories accumulate duplicate files and an unstructured mess of documents, images, and code. Manually organizing them is tedious and error-prone.
The Solution: Vortex automates the entire process in a single command β deduplicating exact clones via SHA-256, then semantically clustering the remaining files using vector embeddings and DBSCAN, and finally reorganizing them into auto-named folders derived from their content.
- Recursive Directory Traversal β Walks the entire target directory tree, resolving all nested files.
- SHA-256 Chunked Hashing β Identifies exact binary duplicates efficiently, even for very large files, using 4 MB chunked reads.
- Storage Statistics β Gathers file size metadata before acting.
- Multi-Format Content Extraction β Pulls text from plain text files, code, PDFs (via PyMuPDF), and images (via Tesseract OCR).
- Local Embedding Generation β Generates text embeddings with
BAAI/bge-small-en-v1.5and image embeddings withQdrant/clip-ViT-B-32-vision, all viafastembed. - Vector Database β Manages semantic indexes in a local Qdrant instance (on-disk, no server needed).
- DBSCAN Clustering β Groups files by semantic proximity using cosine-distance DBSCAN with tuned
epsandmin_samplesper modality. - TF-IDF Directory Naming β Automatically generates human-readable folder names by extracting the most relevant terms from each cluster's combined content.
Implementation in progress.
An interactive review phase that will let you verify the proposed folder structure and directory names before Vortex commits changes to your filesystem.
flowchart TD
A["ποΈ Target Directory"] --> B["Phase 1: Deduplication"]
subgraph DEDUP ["src/dedup"]
B --> B1["dirTraversal β Recursive file discovery"]
B1 --> B2["fileHash β SHA-256 chunked hashing"]
B2 --> B3["fileStats β File size metadata"]
B3 --> B4{"Duplicates found?"}
B4 -- Yes --> B5["Remove exact clones"]
B4 -- No --> C
B5 --> C
end
C["Unique Files"] --> D["Phase 2: Clustering"]
subgraph CLUSTER ["src/clustering"]
D --> D1["fileContent β Multi-format text extraction"]
D1 --> D2["embeddings β Vector generation"]
D2 --> D2a["Text: BAAI/bge-small-en-v1.5 β 384d"]
D2 --> D2b["Image: Qdrant/clip-ViT-B-32-vision β 512d"]
D2a --> D3["Qdrant Vector DB β Local on-disk storage"]
D2b --> D3
D3 --> D4["dbscanModel β Cosine DBSCAN clustering"]
D4 --> D5["dirNaming β TF-IDF folder name generation"]
end
D5 --> E["Phase 3: HITL Review"]
E --> F["π Organized Directory"]
Vortex/
βββ main.py # CLI entry point (Typer + Rich)
βββ pyproject.toml # Project metadata & dependencies
βββ uv.lock # Locked dependency versions
βββ .python-version # Python 3.12
β
βββ src/
β βββ dedup/ # Phase 1 β Deduplication
β β βββ dirTraversal.py # Recursive directory walker
β β βββ fileHash.py # SHA-256 chunked file hashing
β β βββ fileStats.py # File size statistics
β β
β βββ clustering/ # Phase 2 β Semantic Clustering
β β βββ fileContent.py # Multi-format content extraction
β β βββ embeddings.py # Text & image embedding generation + Qdrant storage
β β βββ dbscanModel.py # DBSCAN clustering with data retrieval from Qdrant
β β βββ dirNaming.py # TF-IDF based directory name generation
β β
β βββ hitl/ # Phase 3 β Human-in-the-Loop (WIP)
β
βββ docs/ # Documentation (WIP)
| Category | Technology | Purpose |
|---|---|---|
| Language | Python β₯ 3.12 | Core runtime |
| CLI Framework | Typer + Rich | Command-line interface with styled output |
| Text Embeddings | FastEmbed (BAAI/bge-small-en-v1.5) |
384-dim text vectors |
| Image Embeddings | FastEmbed (Qdrant/clip-ViT-B-32-vision) |
512-dim image vectors |
| Vector Database | Qdrant (local on-disk mode) | Semantic index storage |
| Clustering | scikit-learn (DBSCAN) | Density-based grouping |
| PDF Parsing | PyMuPDF | Text extraction from PDFs |
| OCR | Tesseract + pytesseract | Image-to-text extraction |
| Image Processing | Pillow | Image loading for OCR |
| TF-IDF | scikit-learn (TfidfVectorizer) | Directory name generation |
| Package Manager | uv | Fast dependency management |
| Category | Extensions | Extraction Method |
|---|---|---|
| Plain Text / Code | .txt, .md, .csv, .json, .py, .js, .html |
Direct UTF-8 read |
| Documents | .pdf |
PyMuPDF text extraction |
| Images | .png, .jpg, .jpeg |
CLIP embeddings (visual) + Tesseract OCR (textual) |
Note: Images are embedded using CLIP for visual similarity clustering. OCR is used separately when text content is needed (e.g., for directory naming).
- Python 3.12+ β Download
- uv (recommended) β Install
- Tesseract OCR β Required for image text extraction.
- Download the Windows installer from UB Mannheim.
- Ensure
tesseract.exeis accessible via your systemPATH.
# Clone the repository
git clone https://github.com/44ompatil/Vortex.git
cd Vortex
# Install dependencies (uv recommended)
uv sync
# Or, using pip
pip install -e .Vortex exposes a sort command that orchestrates the full pipeline:
# Sort a directory
uv run python main.py sort <target-directory>
# Example
uv run python main.py sort "C:\Users\you\Downloads"| Command | Description |
|---|---|
sort <directory> |
Run the full dedup β cluster β organize pipeline on the target directory |
help |
Display available commands and usage information |
- Scan β Recursively discovers all files in the target directory.
- Dedup β Identifies and removes exact binary duplicates (SHA-256).
- Extract β Pulls text/visual content from each unique file.
- Embed β Generates vector embeddings (text: 384d, image: 512d).
- Cluster β Groups semantically similar files using DBSCAN.
- Name β Generates descriptive folder names via TF-IDF.
- Organize β Moves files into their newly created, named folders.
Files are recursively discovered via os.walk. Each file is hashed using SHA-256 with 4 MB chunked reads to handle large files efficiently. Files sharing the same hash are identified as exact duplicates β only one copy is kept, the rest are deleted.
Remaining unique files have their content extracted based on file type. Text content is embedded into 384-dimensional vectors using BAAI/bge-small-en-v1.5, while images are embedded into 512-dimensional vectors using Qdrant/clip-ViT-B-32-vision. All vectors are stored in a local Qdrant database.
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) then groups files by semantic similarity using cosine distance. The algorithm parameters are tuned separately for each modality:
| Modality | eps |
min_samples |
|---|---|---|
| Text | 0.25 | 2 |
| Image | 0.50 | 2 |
Note : These are the hyperparameters. They are subject to change. You can update this if the model's not working properly.
Each cluster is assigned a human-readable name generated by running TF-IDF on the combined text content of the cluster's files, extracting the top-2 most distinctive terms.
Coming soon. This phase will present the proposed folder structure for interactive review before files are moved.
Currently, model parameters are configured in-code:
| Parameter | Location | Default | Description |
|---|---|---|---|
chunkSize |
fileHash.py |
4194304 (4 MB) |
Byte chunk size for SHA-256 hashing |
txtEps |
dbscanModel.py |
0.25 |
DBSCAN epsilon for text clusters |
txtMinPts |
dbscanModel.py |
2 |
DBSCAN minimum points for text clusters |
imgEps |
dbscanModel.py |
0.50 |
DBSCAN epsilon for image clusters |
imgMinPts |
dbscanModel.py |
2 |
DBSCAN minimum points for image clusters |
top_n_words |
dirNaming.py |
2 |
Number of TF-IDF terms used in folder names |
PageSize |
dbscanModel.py |
30 |
Qdrant scroll page size |
- Recursive directory traversal
- SHA-256 chunked deduplication
- Multi-format content extraction (text, PDF, images)
- Local text & image embedding generation
- Qdrant vector storage
- DBSCAN semantic clustering
- TF-IDF auto-naming for directories
- Typer CLI with Rich output
- Human-in-the-Loop interactive review
- Config file support (YAML/TOML)
- Cross-modal clustering (text + image in unified space)
- Undo / dry-run mode
- Progress bars and summary statistics
- Additional file format support (
.docx,.xlsx,.pptx)
Contributions are welcome! Here's how to get started:
- Fork the repository.
- Create a feature branch β
git checkout -b feature/your-feature - Commit your changes β
git commit -m "Add your feature" - Push to the branch β
git push origin feature/your-feature - Open a Pull Request.
Please ensure your code follows the existing style and includes appropriate documentation.
Built with β€οΈ for anyone drowning in unorganized files.