kwaroran
cba3ff802c
Add selectable tokenizer supports on Ooba ( #281 )
...
# PR Checklist
- [ ] Did you check if it works normally in all models? *ignore this
when it dosen't uses models*
- [ ] Did you check if it works normally in all of web, local and node
hosted versions? if it dosen't, did you blocked it in those versions?
- [ ] Did you added a type def?
# Description
I write simple changes on code, which allow user to choose tokenizers.
As I write on https://github.com/kwaroran/RisuAI/issues/280 , differences
in tokenizers makes error when use mistral based models.
As I'm not good at javascript, I simply implement this work by write
name of tokenizer model, and select one on tokenizer.ts file.
I test it on my node RisuAI and I send long context to my own server.
As result, ooba returned 15858 as prompt tokens.
And as I test on official tokenizer implementations, it shows 1k
differences between llama tokenizer and mistral tokenizer.
So I think adding this option will help users use oobabooga with less
error.
2024-01-06 19:16:58 +09:00
kwaroran
8ba0d10d56
Update version to 1.71.0
2024-01-06 19:13:57 +09:00
kwaroran
70f25968b5
add transformers remoteHost again
2024-01-06 19:12:18 +09:00
kwaroran
497e0c6dfb
Update service worker and remote host URLs
2024-01-06 19:10:21 +09:00
kwaroran
f4d6fcb38c
Update characterCards.ts with VITS export support
2024-01-06 07:05:20 +09:00
kwaroran
7344e566f4
Add model selection for VitsModel
2024-01-06 06:45:18 +09:00
kwaroran
0ca4ec3695
Add VITS support
2024-01-06 00:28:06 +09:00
kwaroran
cfdd60bead
Remove TransformersBodyType
2024-01-05 23:58:23 +09:00
justpain02
ff4c67b993
Add selectable tokenizer supports on Ooba
2024-01-05 23:57:59 +09:00
kwaroran
56edb14e44
Update transformers import and add config parameter to runTransformers function
2024-01-05 23:54:01 +09:00
kwaroran
5ae3f35df6
Refactor Transformers
2024-01-05 23:51:37 +09:00
kwaroran
b6787b93a7
Fix extractor initialization in runEmbedding function
2024-01-05 23:33:27 +09:00
kwaroran
dcebac658f
[fix] embedding over 512 tokens
2024-01-05 23:29:57 +09:00
kwaroran
8ebaaca59b
Update version to 1.70.3
2024-01-05 19:58:37 +09:00
kwaroran
846441f04a
fix translator customscript
2024-01-05 19:58:19 +09:00
kwaroran
f373363038
Update version to 1.70.2
2024-01-04 17:48:53 +09:00
kwaroran
0b3c702f62
[fix] writer
2024-01-04 17:47:23 +09:00
kwaroran
495c4df698
[ver] 1.70.1
2024-01-04 17:09:43 +09:00
kwaroran
e5c65d7b09
[fix] welcome freezing
2024-01-04 17:09:07 +09:00
kwaroran
7c77b55e39
[fix] localwriter
2024-01-04 05:30:06 +09:00
kwaroran
a890a2acb7
[ver] 1.70.0
2024-01-04 03:58:07 +09:00
kwaroran
20bc35d194
[feat] change default italic for high contrast
2024-01-04 03:57:04 +09:00
kwaroran
5093fd9d0b
[fix] high constrast by default
2024-01-04 03:54:13 +09:00
kwaroran
7428fed704
[feat] new loading screen
2024-01-04 03:52:44 +09:00
kwaroran
34a4422af0
[feat] add horde default key
2024-01-04 03:49:21 +09:00
kwaroran
36c12de39f
[patchnote] 1.70.0
2024-01-04 03:48:06 +09:00
kwaroran
7aae1aa139
[feat] translations
2024-01-04 03:45:08 +09:00
kwaroran
4f77504d3f
[feat] add translate replace regex
2024-01-04 03:37:05 +09:00
kwaroran
5f2275eb22
[feat] better image saving
2024-01-04 03:18:11 +09:00
kwaroran
a13fdced2e
[fix] openai max context
2024-01-04 02:36:45 +09:00
kwaroran
036b23dbdc
[fix] block native
2024-01-04 02:35:08 +09:00
kwaroran
499e91ce6f
[fix] mobile saving
2024-01-04 02:34:22 +09:00
kwaroran
32e3060b19
[feat] better openrouter support
2024-01-04 02:32:53 +09:00
kwaroran
1d5d69b71b
[feat] finish new welcome
2024-01-04 02:14:05 +09:00
kwaroran
eed172f37e
[feat] recommend new template
2024-01-03 20:17:37 +09:00
kwaroran
d36399e409
[feat] new template
2024-01-03 20:17:03 +09:00
kwaroran
81b75947cd
[ref] apng typing
2024-01-03 19:43:13 +09:00
kwaroran
b10c973efb
[fix] apng support
2024-01-03 19:42:26 +09:00
kwaroran
23ef56cb33
[feat] fix horde support
2024-01-03 19:36:47 +09:00
kwaroran
99743f581a
[feat] add top_k for openrouter
2024-01-03 19:03:24 +09:00
kwaroran
a8e50dcd77
[feat] new pngchunk encode/decoderr
2024-01-03 04:13:19 +09:00
kwaroran
239d819383
[feat] add mobile writer support
2024-01-03 02:41:28 +09:00
kwaroran
8f496bc627
[fix] mobile save syste
2024-01-03 02:32:23 +09:00
kwaroran
f3cf1957f1
[feat] new welcome screen
2024-01-03 02:31:19 +09:00
kwaroran
2442b287ff
[feat] android support
2024-01-01 23:14:58 +09:00
kwaroran
0fb9beb5ea
[fix] newyear
2024-01-01 00:02:19 +09:00
kwaroran
151c7450d3
[fix] to localstring on chat
2023-12-31 22:28:38 +09:00
kwaroran
fa7da4c9e6
[fix] static
2023-12-31 22:25:03 +09:00
kwaroran
6d5e48b024
[fix] remove special day
2023-12-31 22:20:54 +09:00
kwaroran
0d1dd45cdf
[feat] new year
2023-12-31 22:19:57 +09:00
