5 Commits

Author	SHA1	Message	Date
kwaroran	f46df1af20	Add Llama3 tokenizer support	2024-04-23 23:02:27 +09:00
kwaroran	6f54c90187	[feat] add mistral support	2023-12-13 12:33:58 +09:00
kwaroran	560f6b56e3	[feat] llama tokenzier	2023-09-09 07:18:20 +09:00
kwaroran	401498b76d	[feat] nerdslash	2023-07-29 09:48:59 +09:00
kwaroran	e0b04784da	[feat] trin & claude tokenizer + novellist support	2023-06-25 21:43:28 +09:00