Add hellishly slow level 13 compression#452
Conversation
|
Full benchamrk on the Silesia corpus:
and here a bit more explanation how it works: https://kirill.korins.ky/articles/hellishly-slow-level-13-deflate-compression/ |
|
@catap - Interesting! It seems tuned for textual information... does it benefit png images? How do your results compare to zopfli? I'm surprised at the cost of your approach, for <0.3% gain. |
|
@ace-dent base on table in turtledeflate, it loses :) |
|
but it can't have infinity loop like Efficient-Compression-Tool and turtledeflate |
|
Addressed MSVC warning which fails as error, and I bit puzzeled by rest of clang related issues which seems not related at all for changed code. |
|
The main branch had CI failures due to non-backwards compatible updates to clang, glibc, and vcpkg. I've fixed them, so they should be gone if you rebase. As for this pull request itself, it's a pretty big change and I don't have time to review it right now, sorry. I'll try to find time sometime. Historically libdeflate hasn't targeted the very end of the time/space tradeoff curve and the answer for that has just been "use zopfli", though. So I would wonder how this compares and if it's worth adding. |
Add the devil's compression level: an extremely slow, ratio first mode above level 12 that spends much more CPU time to squeeze DEFLATE output a little further. On the Silesia corpus, level 13 reduces total output by 86'990 bytes or 0.134% compared with level 12, while total runtime grows from 29.1 seconds to 1'670.5 seconds, making it about 57 times slower. Level 13 extends the near optimal parser with costlier parameters, larger text like blocks, broader parse scoring, delayed split commitment, and saved measured parses for final flushing. Expose the new level through the public API, command line tools, and compression level scripts.
|
@ebiggers thanks for suggestion and making it clear. This idea was to be a bit, just a bit, worse than zopfli but faster than zopfli, and have no infinity loop by design, like ECT and turtledeflate. I'll prepare some benchmarks in following days to make it clear how it behaves. |
3 participants
Add the devil's compression level: an extremely slow, ratio first mode above level 12 that spends much more CPU time to squeeze DEFLATE output a little further.
On the Silesia corpus, level 13 reduces total output by 86'990 bytes or 0.134% compared with level 12, while total runtime grows from 29.1 seconds to 1'670.5 seconds, making it about 57 times slower.
Level 13 extends the near optimal parser with costlier parameters, larger text like blocks, broader parse scoring, delayed split commitment, and saved measured parses for final flushing.
Expose the new level through the public API, command line tools, and compression level scripts.