#118 [feature] chat history summarization and command result display

[Anonymous]

Originally created by: endernoke

New Command

• :summarize (:s): Get message summary with local LLM
• Uses a OpenAI-compatible http endpoint for inferencing. This is supported by major local LLM runners like Ollama, LM Studio, llama.cpp, etc. and also works for remote endpoints for flexibility.
• New configuration values are added for configuring inference endpoints, model name temperature, etc.

UI Changes

• Command results, e.g. those from :help, :summarize will be displayed and wait for key press before being dismissed. This is to prevent the issue of not having enough time to read long informational results before they time out (usually after a few seconds).

[Anonymous]

Originally posted by: HaysonC

Thanks for the PR! This looks like a great addition. I have a few questions to help me understand it better:

1. Could you explain the configuration and model download process a bit more?
2. Is the system prompt user-configurable?
3. I'd like to contribute some insights here. What are the AI conventions and endpoints being used?

Also, this seems like a great candidate for a plugin architecture rather than a core component, which is semi-related to my issue [#114]. What are your thoughts on that?

[Anonymous]

Originally posted by: endernoke

Also, this seems like a great candidate for a plugin architecture rather than a core component, which is semi-related to my issue https://github.com/supreme-gg-gg/instagram-cli/issues/114. What are your thoughts on that?

Yes exactly this would be a perfect example of a plugin. I suppose since it's a stand-alone feature we can refactor when the plugin architecture is implemented. Right now we're trying to get some AI features shipped mainly for next round of marketing/promotion lol.

Users would need to download and set up their own local LLM for inference e.g. via ollama serve, then provide their inference endpoint to the InstagramCLI configuration. I've added docs for setting this up to README.

Currently available endpoints are:

• llm.endpoint: openai-compatible inference endpoint
• llm.model: optional model used for summarization
• llm.api_key: optional api key for remote inference (might be desirable to switch to an environment variable for this config instead)

I forgot to add system prompt to the configs lol, will do it later.

Currently the chat completions API is used i.e. .../v1/chat/completions. You can find the implementation in instagram/chat_ui/utils/chat_commands.py. I'd greatly appreciate insights into this since I'm not very familiar with local inferencing 😄

[Anonymous]

Originally posted by: supreme-gg-gg

Regarding images it is very feasible because we can just fetch the images like we do with the download command and convert to base64, then put them in the right ChatML format and send to the model in the exact same way (or might need to use the OpenAI type instead?):

{
  "messages": [
    {
      "role": "system",
      "content": [{ "type": "text", "text": "you are a helpful agent given text and images..." }]
    },
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "here is the complete text conversation" },
        { "type": "image", "image": "<base64 or image object>" },
        { "type": "image", "image": "<base64 or image object>" }
      ]
    }
 ]
}

I'm not entirely sure as to how much more compute would be required for using vision though.

We also would need to update config where the user needs to specify if their model supports vision, and add a flag in the :summarize command like --vision to indicate if we should include images.

however, this is not urgent at all we can let it rot 😄

[Anonymous]

Originally posted by: supreme-gg-gg

I realised the inline config is never implemented and added that, so users can configure stuff like llm endpoints without having to exit the chat interface with :config llm.endpoint=something

[Anonymous]

Originally posted by: supreme-gg-gg

Also added a support for streaming (llm.streaming) by allowing commands to return either str | Generator, helpful if some model takes very long to generator and especially in the future if we want more complex AI workflow, would be great if @endernoke you can review everything and merge

[Anonymous]

Originally posted by: endernoke

Regarding images it is very feasible because we can just fetch the images like we do with the download command and convert to base64, then put them in the right ChatML format and send to the model in the exact same way (or might need to use the OpenAI type instead?):

json { "messages": [ { "role": "system", "content": [{ "type": "text", "text": "you are a helpful agent given text and images..." }] }, { "role": "user", "content": [ { "type": "text", "text": "here is the complete text conversation" }, { "type": "image", "image": "<base64 or image object>" }, { "type": "image", "image": "<base64 or image object>" } ] } ] }
Yes I believe we would need OpenAI type for maximum compatibility

I'm not entirely sure as to how much more compute would be required for using vision though.

Not sure too lol haven't tried vision in local inferencing

We also would need to update config where the user needs to specify if their model supports vision, and add a flag in the :summarize command like --vision to indicate if we should include images.

however, this is not urgent at all we can let it rot 😄

Yea I guess we're entitled to borrowing some technical debt credits for now 😂 I think to support chat command flags we also need to revamp the existing command registry a bit.

[Anonymous]

Originally posted by: endernoke

Also added a support for streaming (llm.streaming) by allowing commands to return either str | Generator, helpful if some model takes very long to generator and especially in the future if we want more complex AI workflow, would be great if @endernoke you can review everything and merge

All changes lgtm 😄

[Anonymous]

Ticket changed by: endernoke

• status: open --> closed