#127 [feature][ts] Add image component

[Anonymous]

Originally created by: endernoke

Continuation of [#126] (because I accidentally merged it)
Related to [#104]
This PR introduces a new modular and extensible image rendering system for terminal UIs, supporting multiple protocols for displaying images using different techniques.

Changes (high level:

• Added a unified Image component interface
• implemented some new image rendering protocols (halfBlock, braille, and sixel)
• Introduced a protocol factory for easy extensibility,

Image component

<Image> handles image fetching, resizing, and conversion to terminal-friendly output.
Props:

• src: file path or URL to image
• Accepts png/jpg, webp, possibly svg and tif but haven't tested
• width, height (optional): display width and height of the image (in terminal cells).
• If not set, width and height will be inferred from available space in parent component and the aspect ratio of the original image.
• Note that on most terminals the cell height is roughly twice the cell width, so for example a square image should have its height set to half its width to maintain aspect ratio
• protocol (optional): the rendering backend to use, currently either halfBlock or braille or sixel.

New image rendering protocols

ascii:

• Uses the existing ascii-art library to render images
• Output may either be colored or monochrome, and is dynamically selected based on whether the terminal supports colors
• Resolution: 1 pixel per terminal cell

halfBlock:

• Uses the unicode lower half block character ("▄",, U+2584) with ANSI escape sequences for text color and background color
• Resolution: 1x2 pixels per terminal cell

Braille:

• Uses eight dot braille characters (U+2800 to U+2899) to render a monochrome image
• Resolution: 2x4 pixels per terminal cell

sixel:

• Uses the Sixel protocol to directly render the image onto the terminal window
• Full resolution 🎉
• VERY EXPERIMENTAL (because the implementation requires fighting with Ink's text rendering)
  Some of the issues (wontfix):
• Flickering whenever anything in the DOM gets updated (re-render by Ink) and sometimes there's a race condition where the image won't be rendered at all after a re-render)
• Doesn't work if the Ink app size is longer than terminal height (but should work properly inside <FullScreen>)
• If the terminal doesn't support sixel (e.g. kitty), the raw image data will be printed all over the screen. I believe there's a supports-sixel package which can easily check sixel support and implement fallback. fixed with proper fallback

To-do / technical debt

• Add docstrings done
• Convert image protocol factory from class-based to functional for better compatibility with React done
• Implement ASCII rendering based on original feed display done
• Keep experimenting with iterm2, kitty, sixel, and other native terminal image protocols which allows full resolution.
  Since Ink doesn't support custom terminal rendering natively we need to manually figure out the correct coordinates in the terminal where the image should be rendered, then render the image via a side-effect, then clean up / redraw the image when component moves or unmounts (At this point I'm not very optimistic about this 💀)

[Anonymous]

Originally posted by: endernoke

@supreme-gg-gg could u try if sixel works on iTerm2

npm run start test-image sixel

[Anonymous]

Originally posted by: supreme-gg-gg

@supreme-gg-gg could u try if sixel works on iTerm2

npm run start test-image sixel

interesting bug:

╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│         Test Image 1         ││         Test Image 2         ││         Test Image 3         │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│         Test Image 4         ││         Test Image 5         ││         X                    │
│                              ││                              ││         Load failed          │
│                              ││                              ││         Test Image 6         │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
│                              ││                              ││                              │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
Ctrl+C to exit/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135
  return new Error(
         ^

Error: Expected positive integer for width but received 304.0909090909091 of type number
    at Object.invalidParameterError (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135:10)
    at Sharp.resize (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/resize.js:271:16)
    at generateImageOutput (file:///Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/dist/ui/components/image/Sixel.js:71:18)

Node.js v24.0.2

I was able to fix this by just rounding off the width and height when resizing the image, here in Sixel.tsx

const resizedImage = await image
                .resize(Math.round(width), Math.round(height))
                .ensureAlpha() // node-sixel requires alpha channel to be present
                .raw()
                .toBuffer({resolveWithObject: true});

not sure if this is the proper way since i haven't reviewed the code, alternatively we can just add this rounding to the calculate width function because it seems sharp requires integers

[Anonymous]

Originally posted by: supreme-gg-gg

The result looks like this

Not sure why the last two images failed to load but full resolution seems to work fine 😄 🥳

[Anonymous]

Originally posted by: endernoke

@supreme-gg-gg could u try if sixel works on iTerm2
npm run start test-image sixel

interesting bug:

```
╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│ Test Image 1 ││ Test Image 2 ││ Test Image 3 │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│ Test Image 4 ││ Test Image 5 ││ X │
│ ││ ││ Load failed │
│ ││ ││ Test Image 6 │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
Ctrl+C to exit/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135
return new Error(
^

Error: Expected positive integer for width but received 304.0909090909091 of type number
at Object.invalidParameterError (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135:10)
at Sharp.resize (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/resize.js:271:16)
at generateImageOutput (file:///Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/dist/ui/components/image/Sixel.js:71:18)

Node.js v24.0.2
```

I was able to fix this by just rounding off the width and height when resizing the image, here in Sixel.tsx

const resizedImage = await image .resize(Math.round(width), Math.round(height)) .ensureAlpha() // node-sixel requires alpha channel to be present .raw() .toBuffer({resolveWithObject: true});

not sure if this is the proper way since i haven't reviewed the code, alternatively we can just add this rounding to the calculate width function because it seems sharp requires integers

thanks for the catch, i'll just update calCulateImageSize

The result looks like this

Not sure why the last two images failed to load but full resolution seems to work fine 😄 🥳

This is weird since the image is supposed to fit available space in parent container, did you set an explicit width and height for the <Image>s?
If not can you give me the output of the following on ITerm2:

echo -ne "\033[14t"

as well as process.stdout.rows and process.stdout.columns

[Anonymous]

Originally posted by: supreme-gg-gg

@supreme-gg-gg could u try if sixel works on iTerm2
npm run start test-image sixel

interesting bug:
```
╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│ Test Image 1 ││ Test Image 2 ││ Test Image 3 │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
╭──────────────────────────────╮╭──────────────────────────────╮╭──────────────────────────────╮
│ Test Image 4 ││ Test Image 5 ││ X │
│ ││ ││ Load failed │
│ ││ ││ Test Image 6 │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
│ ││ ││ │
╰──────────────────────────────╯╰──────────────────────────────╯╰──────────────────────────────╯
Ctrl+C to exit/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135
return new Error(
^

Error: Expected positive integer for width but received 304.0909090909091 of type number
at Object.invalidParameterError (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/is.js:135:10)
at Sharp.resize (/Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/node_modules/sharp/lib/resize.js:271:16)
at generateImageOutput (file:///Users/supremegg/Documents/GitHub/instagram-cli/instagram-ts/dist/ui/components/image/Sixel.js:71:18)

Node.js v24.0.2
```

I was able to fix this by just rounding off the width and height when resizing the image, here in Sixel.tsx
const resizedImage = await image .resize(Math.round(width), Math.round(height)) .ensureAlpha() // node-sixel requires alpha channel to be present .raw() .toBuffer({resolveWithObject: true});

not sure if this is the proper way since i haven't reviewed the code, alternatively we can just add this rounding to the calculate width function because it seems sharp requires integers

thanks for the catch, i'll just update calCulateImageSize

The result looks like this

Not sure why the last two images failed to load but full resolution seems to work fine 😄 🥳

This is weird since the image is supposed to fit available space in parent container, did you set an explicit width and height for the <Image>s? If not can you give me the output of the following on ITerm2:

echo -ne "\033[14t"

as well as process.stdout.rows and process.stdout.columns

I did not change anything just ran the test command

The command output (looks very weird):

❯ echo -ne "\033[14t"
^[[4;707;1075t
 ~/Documents/GitHub                                                               base   21:30:48 
❯ ;707;1075t

> process.stdout.columns
105
> process.stdout.rows
35

[Anonymous]

Originally posted by: endernoke

Btw the last two images not loading is expected lol i intentionally added an invalid URL and a local file path for testing which I forgot to remove lmao

[Anonymous]

Originally posted by: endernoke

I did not change anything just ran the test command

The command output (looks very weird):

❯ echo -ne "\033[14t" ^[[4;707;1075t  ~/Documents/GitHub   base   21:30:48  ❯ ;70

The echo -ne "\033[14t" command should show your terminal dimensions, do you have a rough estimate of your terminal's screen size in pixels
Looking at the output the screen size calculated by the renderer is like half of your actual screen size for some reason.

[Anonymous]

Originally posted by: supreme-gg-gg

I did not change anything just ran the test command
The command output (looks very weird):
❯ echo -ne "\033[14t" ^[[4;707;1075t  ~/Documents/GitHub   base   21:30:48  ❯ ;70

The echo -ne "\033[14t" command should show your terminal dimensions, do you have a rough estimate of your terminal's screen size in pixels Looking at the output the screen size calculated by the renderer is like half of your actual screen size for some reason.

not sure how do you estimate the pixel size? I've only found info on getting cols and lines. Also you might need to patch the integer rounding it's still failing currently

[Anonymous]

Originally posted by: endernoke

not sure how do you estimate the pixel size? I've only found info on getting cols and lines. Also you might need to patch the integer rounding it's still failing currently

let's take this offline 💀

and yea i forgot to patch the rounding bug lmao will do it later

[Anonymous]

Originally posted by: endernoke

@supreme-gg-gg could you review and merge this thx, also pls remove the test-image command after you finish testing lol

I have decided not to fix the iterm2 incorrect image size since it only affects iterm2 I believe and we could not pinpoint the cause of the issue, I tested across kitty, konsole, wezterm, and windows terminal but couldn't reproduce the issue.
If anyone is experiencing it in other terminals feel free to open an issue and I'll investigate.

[Anonymous]

Originally posted by: supreme-gg-gg

Are we planning to release a library soon, this seems to be a great candidate and almost ready to go already. If not we can start integrating with the typescript client.

[Anonymous]

Originally posted by: endernoke

Are we planning to release a library soon, this seems to be a great candidate and almost ready to go already. If not we can start integrating with the typescript client.

Yes but still need a bit of refactoring done I will try to publish it tmr in flight 💀
I think we can start integrating now, when the package is available we can simply install and git rm in-house image components, the API should stay the same

[Anonymous]

Originally posted by: supreme-gg-gg

All tested and reviewed, I will open an issue for Sixel on iTerm2 in case we are chased by creditors. @endernoke this is one of the most impressive feature at Instagram CLI, you're the real 10x developer!!! Good luck with publishing the library

[Anonymous]

Ticket changed by: supreme-gg-gg

• status: open --> closed