Image inputs to OpenAI, Anthropic, Google, and Ollama providers

You can now upload images to a new Media Node and query with image inputs:

This is currently only limited to OpenAI, Anthropic, Google, and Ollama. If you want more provider support, consider making a PR. The relevant calls to LLMs are in utils.ts.

Image outputs to image inputs

Also enables generating images with one model and then passing them through prompt chaining to another, e.g.,

...Lots of other internal changes

This update required making lots of changes to how ChainForge stores media.

Images take up a lot of space in the browser. To handle this, ChainForge does not store the images in the browser (anymore). Instead we keep unique IDs to each image, derive from their SHA-256 hash, which we use to query the backend and pull the image only when it's necessary (e.g., viewing images in Table Views).

When running locally, ChainForge now copies media your upload into a media directory stored in the same place it puts Saved Flows.

As a consequence, ChainForge can now export and import cfzip bundles which are compressed .cforge (JSON) files alongside the media file(s) used in the flow.

(When running on the browser on chainforge.ai/play, ChainForge will continue to cache images in the front-end.)

These changes will make it easier to add data-heavy analysis pipelines, i.e. loading many documents and chunks into a RAG pipeline, in the near future.

If you encounter any problems with this release, please let us know.

Python 3.10+ are now the only versions supported by ChainForge.

We have deprecated support for 3.8 and 3.9 since the markitdown package required 3.10. This is a forward-thinking change since many other RAG-related packages now require ~3.10.

Developers

This update was initiated by the work of @loloMD and @RoyHEyono , who added the Media Node and image inputs to ChainForge's querying infrastructure. Thank you Roy and Loic! 🎉🙌📺