# PR Checklist
- [ ] Did you check if it works normally in all models? *ignore this
when it dosen't uses models*
- [x] Did you check if it works normally in all of web, local and node
hosted versions? if it dosen't, did you blocked it in those versions?
- [x] Did you added a type def?
# Description
This PR fixes the image input functionality for the Gemini model.
### Issue with Existing Code
The previous implementation attempted to process images in the following
way:
1. In the `OpenAIChat` type, if the `memo` field started with
`inlayImage`, the `content` field's value was copied into a variable
called `pendingImage`.
2. Later, if the `chat` role was `'user'` and `pendingImage` was not an
empty string, the code processed the image.
However, it does not seem to work as expected in the current state.
### Update
I updated the image input handling to align with the method used in
RisuAI for handling GPT’s image input. Specifically, the changes
include:
1. In `image.ts`, I explicitly specified the `gemini-exp` model.
2. If the `chat` object has a `multimodals` field and the `role` is
`user`:
- I created an array called `geminiParts` to store `GeminiPart` objects.
- The `chat.content` value is set as the `text` field of the
`GeminiPart` object in the array.
- I then iterated over `chat.multimodals` and created an object for each
`image` type, formatting it to match the Gemini structure, and added it
to the `geminiParts` array.
- After the iteration, the `geminiParts` array is assigned to the
`parts` field of `reformatedChat`.
### Notes
- I removed the previous non-functional code entirely. If this causes
any inconvenience or violates any conventions, I sincerely apologize.
- As the final name of the next-generation Gemini model is currently
unknown, I restricted the functionality to the **gemini-exp** model in
the `image.ts` file for now. This can be updated later when the official
name is confirmed.
The Gemini model is currently very widely used, so I kindly request you
to review the updated code. If you have any feedback or if the changes
are not acceptable, I completely understand if this PR is rejected.
Thank you for your time and consideration! Let me know if there's
anything I can improve or clarify.
Resolved bug when entire chat context is deleted and hypaV2Data becomes empty, throwing a undefined index error. Also changed it to typescript index.
Hopefully resolved issue where same chat is summarized over and over again by adding another field to mainChunks.
# Changelist:
## 1. Types
### MainChunks
Added id(int), and chatMemos(Set<string>)
Id: incremental int starting from 0
chatMemos: A set of UUID, containing which chat has been summarized in it.
### Chunks
mainChunkID: A connection of which mainChunk it has been split from
text: the split text data
## 2. Features
### CleanInvalidChunks
Called every time when chat is updated and Hypamemory is used.
Gets all the memo(UUID)s of current chats, and creates a set.
Then checks if each mainChunk's chatMemos set is subset of the enitre memo set. If not, the summarized part's chat is deleted/edited. The mainChunk is filtered out.
Concurrently, the chunks that are split from that mainChunk is also deleted.
Also revoked potentially problematic feature(add hypav2data chunk)
TODO:
1. On mid-context editing, currently that is not considered as deletion. Do have optional editedChatIndex to latter dive in more.
2. re-roll mainChunks(re-summarization) functionalities added, but not able to access it.
