-
Notifications
You must be signed in to change notification settings - Fork 1.4k
feat: Enhance filecache #823
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Yundi339
wants to merge
14
commits into
ithewei:master
Choose a base branch
from
Yundi339:feature/enhanced-filecache
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 6 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
a90085a
feat: add FileCacheEx - enhanced file cache with configurable header …
Yundi339 bc30edc
fix: address race conditions and safety issues in FileCacheEx
Yundi339 f14db65
refactor: align Windows compat with original libhv patterns
Yundi339 f9bae97
docs: add FileCacheEx documentation (en/cn) and fix comment style
Yundi339 f5e9a67
fix: add bounds check for negative reserved in resize_buf
Yundi339 7c7d63e
refactor: replace FileCache with FileCacheEx in HttpHandler and HttpS…
Yundi339 1f83fce
Merge branch 'ithewei:master' into feature/enhanced-filecache
Yundi339 a9790f6
Refactor FileCache and remove FileCacheEx
Yundi339 16c24e7
refactor: enhance FileCache with safe fallback in prepend_header and …
Yundi339 4fd86b4
refactor: improve FileCache Open method and enhance HttpHandler heade…
Yundi339 f1850b9
refactor: update FileCache to reset header_used on safe fallback and …
Yundi339 8cd32ab
refactor: update FileCache to change nread type from ssize_t to int i…
Yundi339 4c1d30c
rm deprecated comment
ithewei 731a0a8
Header too large for reserved space: send header first, then continue…
ithewei File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| # FileCacheEx — Enhanced File Cache | ||
|
|
||
| `FileCacheEx` is a drop-in replacement for `FileCache` that fixes thread-safety issues and provides runtime-configurable parameters. | ||
|
|
||
| ## Header | ||
|
|
||
| ```c++ | ||
| #include "FileCacheEx.h" | ||
| ``` | ||
|
|
||
| ## Improvements over FileCache | ||
|
|
||
| | Feature | FileCache | FileCacheEx | | ||
| |---------|-----------|-------------| | ||
| | HTTP header reserve | Compile-time 1024 bytes | Runtime configurable, default 4096 | | ||
| | `prepend_header()` | Returns `void`, silently drops overflow | Returns `bool`, `false` on overflow | | ||
| | Per-entry thread safety | No locking | Per-entry `std::mutex` | | ||
| | `resize_buf()` httpbuf | May leave dangling reference | Explicitly invalidated (UAF prevention) | | ||
| | File read | Single `read()`, may short-read | Loop with `EINTR` handling | | ||
| | Cache insertion | Before initialization completes | Deferred until fully initialized | | ||
| | Max file size | Compile-time `FILE_CACHE_MAX_SIZE` | Runtime `SetMaxFileSize()` | | ||
| | DLL export | None | `HV_EXPORT` | | ||
|
|
||
| ## Data Structures | ||
|
|
||
| ```c++ | ||
| typedef struct file_cache_ex_s { | ||
| mutable std::mutex mutex; // per-entry lock | ||
| std::string filepath; | ||
| struct stat st; | ||
| time_t open_time; | ||
| time_t stat_time; | ||
| uint32_t stat_cnt; | ||
| HBuf buf; // header_reserve + file_content | ||
| hbuf_t filebuf; // points into buf: file content | ||
| hbuf_t httpbuf; // points into buf: header + content | ||
| char last_modified[64]; | ||
| char etag[64]; | ||
| std::string content_type; | ||
| int header_reserve; // bytes reserved before content | ||
| int header_used; // bytes actually used by header | ||
|
|
||
| bool is_modified(); // caller must hold mutex | ||
| bool is_complete(); // caller must hold mutex | ||
| void resize_buf(size_t filesize, int reserved); // caller must hold mutex | ||
| void resize_buf(size_t filesize); | ||
| bool prepend_header(const char* header, int len); // thread-safe | ||
|
|
||
| // Thread-safe accessors | ||
| int get_header_reserve() const; | ||
| int get_header_used() const; | ||
| int get_header_remaining() const; | ||
| bool header_fits(int len) const; | ||
| } file_cache_ex_t; | ||
|
|
||
| typedef std::shared_ptr<file_cache_ex_t> file_cache_ex_ptr; | ||
| ``` | ||
|
|
||
| ## FileCacheEx Class | ||
|
|
||
| ```c++ | ||
| class HV_EXPORT FileCacheEx : public hv::LRUCache<std::string, file_cache_ex_ptr> { | ||
| public: | ||
| int stat_interval; // seconds between stat() checks, default 10 | ||
| int expired_time; // seconds before expiry, default 60 | ||
| int max_header_length; // header reserve per entry, default 4096 | ||
| int max_file_size; // max cached file size, default 4MB | ||
|
|
||
| explicit FileCacheEx(size_t capacity = 100); | ||
|
|
||
| file_cache_ex_ptr Open(const char* filepath, OpenParam* param); | ||
| bool Exists(const char* filepath) const; | ||
| bool Close(const char* filepath); | ||
| void RemoveExpiredFileCache(); | ||
|
|
||
| int GetMaxHeaderLength() const; | ||
| int GetMaxFileSize() const; | ||
| int GetStatInterval() const; | ||
| int GetExpiredTime() const; | ||
|
|
||
| void SetMaxHeaderLength(int len); | ||
| void SetMaxFileSize(int size); | ||
| }; | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ```c++ | ||
| FileCacheEx filecache(200); // LRU capacity = 200 | ||
| filecache.SetMaxHeaderLength(8192); // 8K header reserve | ||
| filecache.SetMaxFileSize(1 << 24); // 16MB max file | ||
| filecache.stat_interval = 5; | ||
| filecache.expired_time = 120; | ||
|
|
||
| FileCacheEx::OpenParam param; | ||
| file_cache_ex_ptr fc = filecache.Open("/var/www/index.html", ¶m); | ||
| if (fc) { | ||
| std::string header = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n"; | ||
| if (fc->prepend_header(header.c_str(), header.size())) { | ||
| // send fc->httpbuf (header + content) | ||
| } else { | ||
| // header exceeds reserve, fall back to separate send | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Migration from FileCache | ||
|
|
||
| 1. Replace `#include "FileCache.h"` with `#include "FileCacheEx.h"` | ||
| 2. Change types: `FileCache` → `FileCacheEx`, `file_cache_ptr` → `file_cache_ex_ptr` | ||
| 3. Handle `bool` return from `prepend_header()` | ||
| 4. Optional: configure via `SetMaxHeaderLength()` / `SetMaxFileSize()` | ||
|
|
||
| ## Thread Safety | ||
|
|
||
| - `FileCacheEx` inherits `hv::LRUCache` global mutex for LRU operations | ||
| - Each `file_cache_ex_s` entry has its own `std::mutex` for entry-level protection | ||
| - `prepend_header()` locks automatically; callers need no external synchronization | ||
| - `is_modified()`, `is_complete()`, `resize_buf()` require caller to hold the mutex |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| # FileCacheEx — 增强版文件缓存 | ||
|
|
||
| `FileCacheEx` 是 `FileCache` 的增强替代方案,解决了原始实现中的线程安全问题,并提供运行时可配置的参数。 | ||
|
|
||
| ## 头文件 | ||
|
|
||
| ```c++ | ||
| #include "FileCacheEx.h" | ||
| ``` | ||
|
|
||
| ## 与 FileCache 的区别 | ||
|
|
||
| | 特性 | FileCache | FileCacheEx | | ||
| |------|-----------|-------------| | ||
| | HTTP 头部预留空间 | 编译期固定 1024 字节 | 运行时可配置,默认 4096 字节 | | ||
| | `prepend_header()` | 返回 `void`,溢出静默丢弃 | 返回 `bool`,失败返回 `false` | | ||
| | 缓存条目线程安全 | 无锁保护 | 每条目 `std::mutex` | | ||
| | `resize_buf()` 后 httpbuf | 可能悬空引用 | 主动置空,防止 UAF | | ||
| | 文件读取 | 单次 `read()`,可能短读 | 循环读取,处理 `EINTR` | | ||
| | `put()` 时机 | 初始化前放入缓存 | 完全初始化后放入缓存 | | ||
| | 最大文件大小 | 编译期宏 `FILE_CACHE_MAX_SIZE` | 运行时 `SetMaxFileSize()` | | ||
| | DLL 导出 | 无 | `HV_EXPORT` | | ||
|
|
||
| ## 数据结构 | ||
|
|
||
| ```c++ | ||
| // 缓存条目 | ||
| typedef struct file_cache_ex_s { | ||
| mutable std::mutex mutex; // 条目级互斥锁 | ||
| std::string filepath; | ||
| struct stat st; | ||
| time_t open_time; | ||
| time_t stat_time; | ||
| uint32_t stat_cnt; | ||
| HBuf buf; // header_reserve + file_content | ||
| hbuf_t filebuf; // 指向 buf 中文件内容区域 | ||
| hbuf_t httpbuf; // 指向 buf 中 header + 文件内容区域 | ||
| char last_modified[64]; | ||
| char etag[64]; | ||
| std::string content_type; | ||
| int header_reserve; // 头部预留字节数 | ||
| int header_used; // 实际使用的头部字节数 | ||
|
|
||
| // 方法 | ||
| bool is_modified(); // 检查文件是否被修改(需持锁) | ||
| bool is_complete(); // 检查缓存是否完整(需持锁) | ||
| void resize_buf(size_t filesize, int reserved); // 重新分配缓冲区(需持锁) | ||
| void resize_buf(size_t filesize); // 使用当前 header_reserve | ||
| bool prepend_header(const char* header, int len); // 线程安全:写入 HTTP 头 | ||
|
|
||
| // 线程安全访问器 | ||
| int get_header_reserve() const; | ||
| int get_header_used() const; | ||
| int get_header_remaining() const; | ||
| bool header_fits(int len) const; | ||
| } file_cache_ex_t; | ||
|
|
||
| typedef std::shared_ptr<file_cache_ex_t> file_cache_ex_ptr; | ||
| ``` | ||
|
|
||
| ## FileCacheEx 类 | ||
|
|
||
| ```c++ | ||
| class HV_EXPORT FileCacheEx : public hv::LRUCache<std::string, file_cache_ex_ptr> { | ||
| public: | ||
| // 可配置参数 | ||
| int stat_interval; // stat() 检查间隔(秒),默认 10 | ||
| int expired_time; // 缓存过期时间(秒),默认 60 | ||
| int max_header_length; // 每条目头部预留字节,默认 4096 | ||
| int max_file_size; // 最大缓存文件大小,默认 4MB | ||
|
|
||
| explicit FileCacheEx(size_t capacity = 100); | ||
|
|
||
| // 打开文件并缓存 | ||
| file_cache_ex_ptr Open(const char* filepath, OpenParam* param); | ||
|
|
||
| // 检查文件是否在缓存中 | ||
| bool Exists(const char* filepath) const; | ||
|
|
||
| // 从缓存中移除文件 | ||
| bool Close(const char* filepath); | ||
|
|
||
| // 移除过期缓存条目 | ||
| void RemoveExpiredFileCache(); | ||
|
|
||
| // Getter | ||
| int GetMaxHeaderLength() const; | ||
| int GetMaxFileSize() const; | ||
| int GetStatInterval() const; | ||
| int GetExpiredTime() const; | ||
|
|
||
| // Setter | ||
| void SetMaxHeaderLength(int len); | ||
| void SetMaxFileSize(int size); | ||
| }; | ||
| ``` | ||
|
|
||
| ## 使用示例 | ||
|
|
||
| ```c++ | ||
| FileCacheEx filecache(200); // 最大 200 个缓存条目 | ||
| filecache.SetMaxHeaderLength(8192); // 8K 头部预留 | ||
| filecache.SetMaxFileSize(1 << 24); // 16MB 最大缓存文件 | ||
| filecache.stat_interval = 5; // 每 5 秒检查文件变更 | ||
| filecache.expired_time = 120; // 2 分钟过期 | ||
|
|
||
| FileCacheEx::OpenParam param; | ||
| file_cache_ex_ptr fc = filecache.Open("/var/www/index.html", ¶m); | ||
| if (fc) { | ||
| // 写入 HTTP 响应头 | ||
| std::string header = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n"; | ||
| if (fc->prepend_header(header.c_str(), header.size())) { | ||
| // 发送 fc->httpbuf (header + content) | ||
| } else { | ||
| // 头部超出预留空间,回退到普通发送 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## 迁移指南 | ||
|
|
||
| 从 `FileCache` 迁移到 `FileCacheEx`: | ||
|
|
||
| 1. 替换头文件引用:`#include "FileCache.h"` → `#include "FileCacheEx.h"` | ||
| 2. 替换类型:`FileCache` → `FileCacheEx`,`file_cache_ptr` → `file_cache_ex_ptr` | ||
| 3. 处理 `prepend_header()` 的 `bool` 返回值 | ||
| 4. 可选:利用 `SetMaxHeaderLength()` / `SetMaxFileSize()` 配置参数 | ||
|
|
||
| ## 线程安全说明 | ||
|
|
||
| - `FileCacheEx` 继承 `hv::LRUCache` 的全局互斥锁保护 LRU 操作 | ||
| - 每个 `file_cache_ex_s` 条目内置 `std::mutex` 保护条目级修改 | ||
| - `prepend_header()` 自动加锁,调用方无需额外同步 | ||
| - `is_modified()` / `is_complete()` / `resize_buf()` 需要调用方持锁 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FileCacheEx.his added to the CMake installed header list, but the repo also has a Makefile-based install flow (Makefile.vars) with its ownHTTP_SERVER_HEADERSlist. To keep both build systems consistent, please addhttp/server/FileCacheEx.htoMakefile.varsas well (otherwisemake installwon’t install the new public header).