Add graphics in terminal support: - Sixel and iTerm2 protocols#2973
Add graphics in terminal support: - Sixel and iTerm2 protocols#2973MatanZ wants to merge 11 commits into
Conversation
|
You are a hero my brother |
|
It works! but gifs are being duplicated on the terminal when expanding and collapsing the keyboard Regular images are being displayed fine
|
This is not a bug. img2sixel displays animations by sending each frame as a different sixel with a command to send the cursor to position (1,1) between them (CSI H). This causes each frame to overwrite the previous one. When you remove the keyboard, the screen gets taller, and termux handles this by scrolling down, so the previous frame which was at position (1,1), is now lower, so it is not covered by the following frames. |
|
oh, so everything's working fine then |
Here's a small guide for anyone else who wants to test this PR
|
|
Has issues with big images. For example when using Test image (7779x4191): https://user-images.githubusercontent.com/107305601/188685258-87eb0704-12c5-4b43-a47a-a1deae99e208.jpg Device config: Pixel 5, Android 13, 8 GiB RAM. |
Image displays fine on termux-monet with Device Config: Xiaomi POCO F1, Android 12L, 6GB RAM
|
|
I believe if MatanZ gets this imported and issue 142 gets solved, then MatanZ gets my $100 bounty on Bountysource for that issue. It will be well worth it!! :-D |
|
Thanks, but anyway there is some proper workaround needed to avoid OOM. Larger heap indeed can solve it for a particular case. Not all devices have lots of memory and part is always used by other apps including system, max heap size also vary between device models. Unfortunately I can't propose the best way to implement that, but I think refusing to display supersized bitmaps is better than crashing all sessions. For example if dimensions are too big, user can get a toast message describing the issue. Command line programs should not be able to crash terminal or cause resource overload by just sending some control sequence (no matter whether it is sixel or something else). |
I don't believe that making the terminal refuse to display oversized images is necessary, since most of command line programs always will be able to crash the application when overloaded or abused. That would only limit how the user should use his terminal, creating a "fake error" that doesn't even exist for some users, like the people who has enough RAM for displaying those images. Those people who can display the images wouldn't be able to. But if you guys decide that's the best thing to do, i propose placing a limit only for devices with less than 4GB RAM. |
Just so I can get more testing from the nice people who test my code:
|
|
Tested with imagemagick, and it's displaying fine on my side |
I hope this solves the problem. I put a try{}catch() block around each bitmap operation that allocates memory. This should be similar to what you suggest - ignoring large images, without hard coding a definition of large. I believe it is better without a toast notification. Usually when a terminal cannot (or does not want to) perform a certain operation it just ignores it, without notifying the user. Maybe a line in the log. |
|
@MatanZ Please use dedicated class for long sixel logic in Trying to catch Checking current memory before even trying to display large image could be done too. https://developer.android.com/topic/performance/graphics/load-bitmap#read-bitmap
This is the way. No toasts. Being done in other places too. |
The way graphics display is implemented, it is not possible to implement kitty protocol. It is possible to implement very simplified ("put image here"), but image and placement management, as well as combining text and images is impossible. |
Refactored some code to two new classes. I don't see anything in TerminalEmulator that belongs in another class, or which may become clearer.
I am not sure this is a good idea. I would have to add assumptions about how much memory is used by the android for various bimap manipulations (decode, scale, resize). I'll settle for trying anything the user asks for, and catching the errors. I did not crash termux with this, with all the above, and various other large images. |
|
Thanks. Will take another look later myself during merging. You can also let |
|
Seems like there is a problem with some GIFs. Notice the artifacts at the left part. Same as #142 (comment)? Gif file doesn't seem to be corrupted. screen-20220908-005128.2.mp4This doesn't happen with another gif: screen-20220908-005944.2.mp4 |
|
I cannot see those videos. And can you please include the actual files used, and the commands that cause the problem? |
|
Perhaps the issue with Termux I can reproduce it when connected over SSH to Termux from Konsole on laptop. But I can't reproduce the issue when animation is played via Screenshot with the problem (notice the block with lines on the left side): You can get the file from https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/Rotating_earth_%28large%29.gif/274px-Rotating_earth_%28large%29.gif libsixel package version differences:
The difference in a build time configuration. Though maybe OpenSUSE also applies some patches to the package. |
Fixes issue with some GIFs. See my comment at termux/termux-app#2973 (comment) Also enable libcurl support.
|
Commit termux/termux-packages@b72be3c resolves |
Fixes issue with some GIFs. See my comment at termux/termux-app#2973 (comment) Also enable libcurl support.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
If some package doesn't work - open a new issue in https://github.com/termux/termux-packages/issues. |
Sixel coverage:
Conclusion: All applications that uses and/or supports Sixel works properly iTerm2 coverage:
Conclusion: Not all applications that uses and/or supports iTerm2 works properly |
Report InfoUser Action: Crash DetailsCrash Thread: Crash Message: Stacktrace |
…age (`OSC 1337`) Support for displaying bitmaps inside the terminal has been added via `TerminalBitmap` which can be created from image `byte[]` or sixel bitmap. The bitmaps are sliced to character cell sized slices. The `TerminalBuffer` stores a map for bitmap number to the `TerminalBitmap` loaded in the terminal. The bitmap number and coordinates are encoded in the `long` `TerminalRow.mStyle` for the `TerminalRow` character of a column by `TerminalBitmap#buildOrThrow()` by getting encoded value from `TextStyle.encodeTerminalBitmap()`. The `TerminalRenderer.render()` then checks during rendering terminal output whether a character at a row/coloumn index is a bitmap instead of text by calling `TextStyle.isTerminalBitmap()`, then draws it using `Canvas.drawBitmap()` instead of `Canvas.drawText()`. Sixel images can be created with Sixel Device Control String command sent via `DCS q s..s ST` or `DCS P1; P2; P3; q s..s ST`. - The `TerminalEmulator` interprets sixel sequences, and sends them to `TerminalBuffer` for constructing a `TerminalSixel`. Once the sixel command has been completely processed, a `TerminalBitmap` is created from the `TerminalSixel`. If an error occurred during processing (like OOM), then remaining sixel command is completely read, but is ignored and no sixel is drawn (done by setting `mTerminalSixel` to `null` so that `TerminalBuffer.sixelReadData()` ignores further commands). - Since a sixel sequence can be very long to render a full image and can have length greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`), the entire sequence is not stored in the `mTerminalControlArgs` buffer before being processed as it will result in an overflow error, instead as soon as length crosses `TERMINAL_CONTROL_ARGS__MAX_LENGTH / 2` and a complete sixel sub command (`#`, `!`, or `"`) has been received, it is immediately processed, and then further commands are read after emptying buffer. - If "rough" horizontal and vertical size of image is received at start of sixel data string with a `Raster Attributes` command, like done by `img2sixel` command, then sixel commands args buffer capacity (`mTerminalControlArgs`) is increased and sixel bitmap in `TerminalSixel` is resized at start, instead of having to keep resizing buffer/bitmap as more sixel data is received, which has a performance hit due to memory reallocations and copying. - The `4` (sixel) value has been added to `CSI` `Primary Device Attributes` terminal response. The `img2sixel` command can be used to display sixel images after installing with `libsixel` package with `pkg install libsixel`, like with `img2sixel --width=1000px image.jpg`. To manually send an escape sequence, check the `digiater.nl` link below, but it is too cumbersome to create images large enough to be easy viewable in the terminal. See Also: - https://vt100.net/docs/vt3xx-gp/chapter14.html - https://en.wikipedia.org/wiki/Sixel - https://www.digiater.nl/openvms/decus/vax90b1/krypton-nasa/all-about-sixels.text iTerm images can be created with `1337` Operating System Control command. - Both `File=` and `MultipartFile=` (chunk based) protocols are supported. The `inline` parameter should be `1` to display inline images in the terminal. Downloading images to Downloads folder with the value `0` will be ignored as that is not supported. - The escape sequences/image data cannot be greater than `TERMINAL_CONTROL_ARGS__MAX_LENGTH` (`16384`) bytes if sent via (`File=`) protocol, otherwise it will be ignored with an overflow error. For larger images, send images via `MultipartFile=` protocol in chunks with `FilePart=`, the `imgcat` utility uses that with 200-byte chunks if `--legacy` flag is not passed. - The `TerminalEmulator` interprets iTerm images sequences and creates an `ITermImage` to process parameters and store the base64 encoded image sent. Once all the data has been received, which can be over multiple `OSC` commands for `MultipartFile=` protocol, the encoded image is decoded to a `byte[]`, which is then passed to `TerminalBuffer`, which creates a `TerminalBitmap` for the image. The `imgcat` utility can be used for sending images, like with `imgcat --width 1000px image.jpg` (`MultipartFile=`) or `imgcat --width 1000px --legacy image.jpg` (`File=`). To manually send an escape sequence, run `echo -en '\e]1337;File=inline=1;keepAspectRatio=0;width=1000px;:' ; base64 -w 0 ./image.jpg ; echo -e '\e\\'` (`File=`). See Also: - https://iterm2.com/documentation-images.html - https://iterm2.com/utilities/imgcat - https://github.com/gnachman/iTerm2-shell-integration/blob/d1d4012068c3c6761d5676c28ed73e0e2df2b715/utilities/imgcat
The `TerminalBitmap.MAX_BITMAP_SIZE` defines the max size of a Terminal bitmap for its pixels. Each pixel is stored on 4 bytes for a `Bitmap.Config.ARGB_8888` bitmap color config. The value should normally be between `100-200MB` depending on device and Android version. Check the variable docs for more info. The sixel image size cannot be greater than `TerminalBitmap.MAX_BITMAP_SIZE`. The repeat value for sixel Graphics Repeat Introducer command cannot be greater than `TerminalSixel.SIXEL__MAX_REPEAT` (`8192`). The iTerm image data sent for `File=` and `MultipartFile=` protocols cannot be greater than `TerminalBitmap.MAX_BITMAP_SIZE` bytes.
|
Thanks for the report @Kabouik. For https://vt100.net/docs/vt3xx-gp/chapter14.html#S1S14.2.1 For
You can grab APK build for current push from https://github.com/termux/termux-app/actions/runs/25403833943?pr=2973 |
@agnostic-apollo The example in the VT330/VT340 manual, while technically valid, is not an indication of the required format for Sixel sequences. And I suspect it's probably a typo, because the VT340 itself doesn't produce Sixel output like that when generating a hardcopy. According to the DEC STD 070 reference (§ 3.5.4.1 and 3.5.3.1), DCS parameters should be parsed in the same way as CSI parameters. A semicolon is a parameter separator, not a parameter terminator; zero-length parameter value represents a default value; and default parameter values can be omitted from the end of the parameter list. So something like |
…olon and allow not passing all optional parameters
|
@j4james Well, not sure if its a typo, even wikipedia says "Sixel mode is entered by sending the sequence https://en.wikipedia.org/wiki/Sixel
I wasn't allowing missing parameters before, but have fixed it with 2151d08 You can grab APK build for current push from https://github.com/termux/termux-app/actions/runs/25445369130?pr=2973 |
|
Awesome, thanks @agnostic-apollo, I can confirm 
|
|
Welcome. An android device or at least termux is likely going to be slower than desktop, but check if resolution of video being passed is not higher than display size, otherwise share the commands and video you are trying to view and I can try to test. Is video streaming app using sixel or iterm? |
It's just music album covers which are all 640×640. However, I don't have time to dive into it right now, but I believe there are many redraws occurring when the TUI is navigated, and that may exacerbate any slowness compared to desktop. It should be using |
|
Okay, let me know if you figure it out. iterm is 20x faster than sixel though. I'll wait a couple of days more for any more tests by users, before I merge and make a release. |
An official Termux release with sixel support? Awesome. However I just tested |
|
I can confirm that, for me,
curl https://iterm2.com/utilities/imgcat > $PREFIX/bin/imgcat
chmod +x $PREFIX/bin/imgcat
|
|
As I already said chafa will not work with iterm as it sends the image in tiff format regardless of source image format, and tiff is not supported by android platform libraries for decoding. To support tiff in termux end would require adding libtiff or some other external native/java library in the app, adding native libraries is complex for an Android app and I avoid adding dependencies unless necessary. And it may not be needed anyways as chafa maintainer says he plans to shift to PNG images, which will automatically work in termux when chafa shifts, you can track that in hpjansson/chafa#333. |
|
Instructions are also available at end of #2973 (comment) for how to use iterm protocol with just simple echo/base64 commands without any other external tool, like imgcat/chafa. |
|
chafa uses uncompressed tiff. It would be quite easy to decode without any external library. |
|
I haven't looked at how tiff format works, and how chafa is using it. And not sure whether a simpler decoding will work for other clients. I also don't have time to work or even research into it currently, I'm busy with termux rewrite, if you want work on it yourself right now, let me know and I'll hold off merging the pull. Otherwise can wait for chafa to switch to png or add basic tiff support in future and merge for now. |
|
I believe it is better to merge (and release) as soon as possible. chafa works in sixel mode, and other tools work also in iterm2 mode. It is always possible to add this later if desired. |
|
The link and explanation above suggests that a future version of chafa plans to switch to png for iterm2, which would solve the problem without termux intervention, I guess? in my opinion that makes it not a high priority unless many other iterm2 apps also require tiff support. |
|
Ok, merge it is then. |
|
Wifizone je suis très intéressé |
|
On April 17th the status was that approval of the current version was required from MatanZ, and on May 7th MatanZ gave approval of this version. In between those dates, on May 6th, an ETA of 2 days was given, and if the merging preparation work is still ongoing and more time is needed then that is fine, I just thought that I should mention that time passed since an ETA was given. |
|
Sorry, pull is ready to be merged, other than squashing commits and updating commit messages. I have been busy with finishing some of the termux-app rewrite stuff and couldn't shift attention from it to do a release as I have a limited context window size. Will get to it soon. |
|
Thanks for the update, based on the context I had just assumed that you were prepared to and wanted to merge this PR before finishing the major rewrite project, but if it would be easier and less stressful for you to postpone it until afterward I wouldn't mind that. |
|
I didn't mean the entire rewrite, I meant a part of it, have had to think on some complex issues and couldn't be distracted. |
18 participants
This commit implements graphics in the terminal, including two different protocols for placing images:
I ran this for a few days, and it seems to me that some people tried it from #142, and saw no crashes, so I hope it is reasonable safe.
TerminalBuffer for constructing a bitmap.
natural places ($,-,#), rather than collecting all in the buffer.
the style attribute is used to store which bitmap slice is displayed
in place of this character.
drawBitmap, instead of drawText.
Support iTerm2 inline image protocol (OSC 1337):
Small emulator changes:
CSI 14 tto give actual sizeCSI 16 t4(sixel) to device attributesFor those who tried this branch already, I force pushed to it in order to remove changes that may not be acceptable.
The main known issue is that if graphics sequence is interrupted (in specific places), the emulator remains stuck waiting for the sequence to end, thus ignoring all input. This requires terminal reset from the menu.