Commit Graph

213 Commits

Author SHA1 Message Date
Sean Hatfield
7fb76cfef0
add support for exporting to json and csv in workspace chats (#622)
* add support for exporting to json and csv in workspace chats

* safety encode URL options

* remove message about openai fine tuning on export success

* all defaults to jsonl
2024-01-18 17:59:51 -08:00
Sean Hatfield
08d33cfd8f
Merge pull request from GHSA-xmj6-g32r-fc5q
* remove support for import export anythingllm data

* remove unused imports
remove unused dep
update lockfile

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-18 13:57:23 -08:00
Sean Hatfield
56fa17caf2
create configurable topN per workspace (#616)
* create configurable topN per workspace

* Update TopN UI text
Fix fallbacks for all providers
Add SQLite CHECK to TOPN value

* merge with master
Update zilliz provider for variable TopN

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-18 12:34:20 -08:00
Timothy Carambat
683cd69a97
Display readable error message in chat failure (#621) 2024-01-18 12:01:53 -08:00
Timothy Carambat
658e7fa390
chore: Better VectorDb and Embedder error messages (#620)
* chore: propogate embedder and vectordb errors during document mutations

* add default value for errors on addDocuments
2024-01-18 11:40:48 -08:00
Timothy Carambat
0df86699e7
feat: Add support for Zilliz Cloud by Milvus (#615)
* feat: Add support for Zilliz Cloud by Milvus

* update placeholder text
update data handling stmt

* update zilliz descriptor
2024-01-17 18:00:54 -08:00
Sean Hatfield
3fe7a25759
add token context limit for native llm settings (#614)
Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 16:25:30 -08:00
Sean Hatfield
56dc49966d
add copy feature to assistant chat message (#611)
* add copy feature to assistant chat message

* fix tooltip not hiding on mobile

* fix: add tooltips
chore: breakout actions to extendable component + memoize
add CopyText to hook we can reuse
fix: Copy on code snippets broken, moved to event listener
fix: highlightjs patch for new API support
feat: add copy response support

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 16:22:06 -08:00
Sean Hatfield
c2c8fe9756
add support for mistral api (#610)
* add support for mistral api

* update docs to show support for Mistral

* add default temp to all providers, suggest different results per provider

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 14:42:05 -08:00
Sean Hatfield
90df37582b
Per workspace model selection (#582)
* WIP model selection per workspace (migrations and openai saves properly

* revert OpenAiOption

* add support for models per workspace for anthropic, localAi, ollama, openAi, and togetherAi

* remove unneeded comments

* update logic for when LLMProvider is reset, reset Ai provider files with master

* remove frontend/api reset of workspace chat and move logic to updateENV
add postUpdate callbacks to envs

* set preferred model for chat on class instantiation

* remove extra param

* linting

* remove unused var

* refactor chat model selection on workspace

* linting

* add fallback for base path to localai models

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 12:59:25 -08:00
Timothy Carambat
c61cbd1502
Add support for fetching single document in documents folder (#607) 2024-01-16 14:58:49 -08:00
Timothy Carambat
e973c1edbf
Preload onboarding (#606)
* no lazy onboarding

* no lazy onboarding steps

* Do not lazy load onboarding to prevent lazy-load white flash
2024-01-16 14:15:46 -08:00
Sean Hatfield
8b11288764
truncate title to shorter length so x button is not hidden for longer… (#603)
* truncate title to shorter length so x button is not hidden for longer title names in the citation modal

* absolutely position x button on citation modal
2024-01-16 13:43:32 -08:00
Timothy Carambat
bd158ce7b1
[Feat] Query mode to return no-result when no context found (#601)
* Query mode to return no-result when no context found

* update default error for sync chat

* remove unnecessary type conversion
2024-01-16 09:32:51 -08:00
Shuyoou
6faa0efaa8
Issue #543 support milvus vector db (#579)
* issue #543 support milvus vector db

* migrate Milvus to use MilvusClient instead of ORM
normalize env setup for docs/implementation
feat: embedder model dimension added

* update comments

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-12 13:23:57 -08:00
Timothy Carambat
3c859ba303
Change pwd check to O(1) check to prevent timing attacks - single user mode (#575)
Change pwd check to O(1) check to prevent timing attacks
2024-01-11 10:54:55 -08:00
Timothy Carambat
a4ace56a40
Patch minor XSS opportunity where user can self-XSS themselves. (#574)
Patch minor XSS opportunity where user can self-XSS themselvess. There is not real vuln here as any instance is not public facing
2024-01-11 09:57:59 -08:00
timothycarambat
4af9b9d5cc fix: relative/absolute url patch for vector count
connect #516
2024-01-10 19:27:39 -08:00
Timothy Carambat
4e2c0f04b4
Dynamic vector count on workspace settings (#567)
* Dynamic vector count on workspace settings
Add count to be workspace specific, fallback to system count
Update layout of data in settings
Update OpenAI per-token embedding price

* linting
2024-01-10 13:18:48 -08:00
Sean Hatfield
1d39b8a2ce
add Together AI LLM support (#560)
* add Together AI LLM support

* update readme to support together ai

* Patch togetherAI implementation

* add model sorting/option labels by organization for model selection

* linting + add data handling for TogetherAI

* change truthy statement
patch validLLMSelection method

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-10 12:35:30 -08:00
Sean Hatfield
5c3bb4b8cc
532 uiux add slash command modal (#555)
* WIP slash commands

* add slash command image

* WIP slash commands

* slash command menu feature complete

* move icons to slash command local

* update how slash command component works

* relint with new linter

* Finalize slash command input
Change empty workspace text layout
Patch dev unmount issues on Chatworkspace/index.jsx

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-09 13:07:09 -08:00
Timothy Carambat
cce48c2163
[Automated Builds] Patch lockfiles with new devdeps (#553)
patch lockfiles with new devdeps
2024-01-08 15:48:03 -08:00
Francisco Bischoff
990a2e85bf
devcontainer v1 (#297)
Implement support for GitHub codespaces and VSCode devcontainers
---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2024-01-08 15:31:06 -08:00
Sagar Jaglan
5172bceec3
#520 fixed unauthorized token error (#535) 2024-01-08 13:16:00 -08:00
Sean Hatfield
d95d1a9dfd
529 UI update llm embedder and vectordb selection pages (#533)
* move llm, embedder, vectordb items to components folder

* add backdrop blur to search in llm, embedder, vectordb preferences

* implement searchable llm preference in settings

* implement searchable embedder in settings

* remove unused useState from embedder preferences

* implement searchable vector database in settings

* fix save changes button not appearing on change for llm, embedder, and vectordb settings pages

* sort selected items in all settings and put selected item at top of list

* no auto-top for selection

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-04 18:20:58 -08:00
timothycarambat
a1b4ed43ba allow native when on embedder 2024-01-04 16:12:20 -08:00
Timothy Carambat
d8ca92df88
Onboarding V2 (#502)
* WIP onboarding v2

* Welcome screen for onboarding complete

* fix home page and WIP create skeleton for llm preference search/options

* render llms as options

* add search functionality to llm preference & add survey step

* fix openai settings undefined & create custom logo onboarding page

* add user setup UI

* add data handling & privacy onboarding screen

* add create workspace onboarding screen

* fix survey width in onboarding

* create vector database connection onboarding page

* add workspace image & all skeleton ui complete

* fix navigation buttons and ui tweaks to fit on screen

* WIP LLMPreference

* LLM Preference screen fully functional

* create components for vector db options and fix styling of azure options

* remove unneeded comment

* vector db connection onboarding screen complete

* minor ui tweak to searchbar

* user setup page fully working

* create workspace onboarding page fully working

* useNavigate for navigation between pages

* mobile layout, cleanup old files, survey functionality implemented

* fix default logo appearing when should be blank & password setup bug fix

* Modify flow of onboarding
todo: embedding set up

* Add embedder setup screen & insert into flow

* update embedding back button
auto-dismiss toasts on each step

* move page defs under imports
fix bg color on mobile styling

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2024-01-04 15:54:31 -08:00
timothycarambat
dc23961231 drop feeback path in paths 2024-01-03 15:44:26 -08:00
Timothy Carambat
e0a0a8976d
Add Ollama as LLM provider option (#494)
* Add support for Ollama as LLM provider
resolves #493
2023-12-27 17:21:47 -08:00
Timothy Carambat
24227e48a7
Add LLM support for Google Gemini-Pro (#492)
resolves #489
2023-12-27 17:08:03 -08:00
timothycarambat
7bee849c65 chore: Force VectorCache to always be on;
update file picker spacing for attributes
2023-12-20 10:45:03 -08:00
Timothy Carambat
ecf4295537
Add ability to grab youtube transcripts via doc processor (#470)
* Add ability to grab youtube transcripts via doc processor

* dynamic imports
swap out Github for Youtube in placeholder text
2023-12-18 17:17:26 -08:00
Timothy Carambat
452582489e
GitHub loader extension + extension support v1 (#469)
* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files

* noshow delete on custom-documents

* Add API key support because of rate limits

* WIP for frontend of data connectors

* wip

* Add frontend form for GitHub repo data connector

* remove console.logs
block custom-documents from being deleted

* remove _meta unused arg

* Add support for ignore pathing in request
Ignore path input via tagging

* Update hint
2023-12-18 15:48:02 -08:00
Timothy Carambat
2d700b13f6
fix: patch middle truncation func (#465)
resolves #456
2023-12-17 16:04:50 -08:00
Timothy Carambat
1c549e2844
fix: clarify custom truncate from npm truncate (#464)
chore: add @ where we can for less long imports
2023-12-17 15:58:49 -08:00
Timothy Carambat
65c7c0a518
fix: patch api key not persisting when setting LLM/Embedder (#458) 2023-12-16 10:21:36 -08:00
timothycarambat
d2e3506bb9 fix: transition on LLM and embedding screen
linting
2023-12-15 12:40:11 -08:00
Timothy Carambat
61db981017
feat: Embed on-instance Whisper model for audio/mp4 transcribing (#449)
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves #329

* additional logging

* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency

* update README

* update model size

* update supported filetypes
2023-12-15 11:20:13 -08:00
timothycarambat
b444171ef3 fix: Update copy code button on markdown to work with new highlightjs lib implementation 2023-12-12 16:25:22 -08:00
timothycarambat
7154f4f82b non-blocking send responses 2023-12-12 13:41:17 -08:00
Timothy Carambat
a84333901a
feat: implement questionnaire during onboarding (optional) (#429)
fix: PFP url check
2023-12-12 13:11:32 -08:00
Timothy Carambat
cba66150d7
patch: API key to localai service calls (#421)
connect #417
2023-12-11 14:18:28 -08:00
Timothy Carambat
8cc1455b72
feat: add support for variable chunk length (#415)
fix: cleanup code for embedding length clarify
resolves #388
2023-12-07 16:27:36 -08:00
Timothy Carambat
655ebd9479
[Feature] AnythingLLM use locally hosted Llama.cpp and GGUF files for inferencing (#413)
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev

* Add native embedder as an available embedder selection

* wrap model loader in try/catch

* print progress on download

* add built-in LLM support (expiermental)

* Update to progress output for embedder

* move embedder selection options to component

* saftey checks for modelfile

* update ref

* Hide selection when on hosted subdomain

* update documentation
hide localLlama when on hosted

* saftey checks for storage of models

* update dockerfile to pre-build Llama.cpp bindings

* update lockfile

* add langchain doc comment

* remove extraneous --no-metal option

* Show data handling for private LLM

* persist model in memory for N+1 chats

* update import
update dev comment on token model size

* update primary README

* chore: more readme updates and remove screenshots - too much to maintain, just use the app!

* remove screeshot link
2023-12-07 14:48:27 -08:00
Sean Hatfield
fcb591d364
Add user PFP support and context to logo (#408)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change

* fix for empty chat

* change mobile alert copy

* WIP adding PFP upload support

* WIP pfp for users

* edit account menu complete with change username/password and upload profile picture

* add pfp context to update all instances of usePfp hook on update

* linting

* add context for logo change to immediately update logo

* fix div with bullet points to use list-disc instead

* fix: small changes

* update multer file storage locations

* fix: use STORAGE_DIR for filepathing

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-07 14:11:51 -08:00
Timothy Carambat
f48e6b1a3e
chore: add @ as alias for frontend root (#414)
* chore: add @ as alias for frontend root

* fix: remove bad tag
2023-12-07 09:09:01 -08:00
timothycarambat
ef79649891 improve ms-sse-fetch to not re-fetch automatially 2023-12-06 12:04:10 -08:00
Timothy Carambat
88cdd8c872
Add built-in embedding engine into AnythingLLM (#411)
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev

* Add native embedder as an available embedder selection

* wrap model loader in try/catch

* print progress on download

* Update to progress output for embedder

* move embedder selection options to component

* forgot import

* add Data privacy alert updates for local embedder
2023-12-06 10:36:22 -08:00
Sean Hatfield
48764d6370
Gear icon appear on hover for workspace (#410)
* gear icon appear on hover for workspace

* put back user role check for default

* wrap in callback

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-05 15:58:57 -08:00
pritchey
732d07829f
401-Password Complexity Check Capability (#402)
* Added improved password complexity checking capability.

* Move password complexity checker as User.util
dynamically import required libraries depending on code execution flow
lint

* Ensure persistence of password requirements on restarts via env-dump
Copy example schema to docker env as well

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-05 09:13:06 -08:00
Sean Hatfield
7b30dd041b
Add down arrow to scroll user to bottom of page (#400)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change

* fix for empty chat

* change mobile alert copy

* fix div with bullet points to use list-disc instead

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-04 13:27:07 -08:00
Sean Hatfield
064e0c3408
Fix modal sizes to fit smaller screens & bug fixes (#399)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* change mobile alert copy

* fix div with bullet points to use list-disc instead

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-04 13:23:45 -08:00
Timothy Carambat
6fa8b0ce93
Add API key option to LocalAI (#407)
* Add API key option to LocalAI

* add api key for model dropdown selector
2023-12-04 08:38:15 -08:00
timothycarambat
203f5964d5 linting 2023-12-04 08:36:17 -08:00
Sean Hatfield
ca8bf52cce
Bug fixes (#394)
* fix black text for custom messages text

* fix file upload icon being stretched

* center github, docs, and discord icons in sidebar

* fix chat container being cut off on right side and tighter spacing between message

* fix default chat container being cut off on right side

* on create new workspace, take user to the workspace they just created instead of the home page

* add border to chat container and click outside user menu to close

* fix borders around all chat and settings containers to be consistent

* fix padding for default messages

* fix spacing between workspace items in sidebar

* fix margin around right side of chat, default, and settings containers to be the same as the left sidebar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-27 15:09:55 -06:00
Sean Hatfield
7edfccaf9a
Adding url uploads to document picker (#375)
* WIP adding url uploads to document picker

* fix manual script for uploading url to custom-documents

* fix metadata for url scraping

* wip url parsing

* update how async link scraping works

* docker-compose defaults added
no autocomplete on URLs

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 17:15:01 -08:00
Timothy Carambat
d766d128a2
move embedding options to their own component (#387) 2023-11-16 15:40:39 -08:00
Sean Hatfield
5ad8a5f2d0
Allow use of any embedder for any llm/update data handling modal (#386)
* allow use of any embedder for any llm/update data handling modal

* Apply embedder override and fallback to OpenAI and Azure models

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 15:19:49 -08:00
Sean Hatfield
2c2543b4d7
fix settings button path on single user mode (#383)
Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 14:37:42 -08:00
Sean Hatfield
73f342eb19
Warning about switching embedder or vectordb (#385)
* added warning modal to LLM preference

* added warning modal for changing embedder

* remove warning from LLM preference & add warning to vector database selection

* linting

* remove comments and move warning modal to component

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 14:35:14 -08:00
Sean Hatfield
dd33767059
365 update invite UI to match new UI (#370)
* update NewUserModal to match ui styles

* fix bg color of invite screen/auto login after accept invitation

* fix error text color

* cleanup

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:58:00 -08:00
Tobias Landenberger
a96a9d41a3
LocalAI for embeddings (#361)
* feature: add localAi as embedding provider

* chore: add LocalAI image

* chore: add localai embedding examples to docker .env.example

* update setting env
pull models from localai API

* update comments on embedder
Dont show cost estimation on UI

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:49:31 -08:00
timothycarambat
f269124760 fix type error for filetypes 2023-11-14 13:49:18 -08:00
Timothy Carambat
4bb99ab4bf
Support LocalAi as LLM provider by @tlandenberger (#373)
* feature: add LocalAI as llm provider

* update Onboarding/mgmt settings
Grab models from models endpoint for localai
merge with master

* update streaming for complete chunk streaming
update localAI LLM to be able to stream

* force schema on URL

---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: tlandenberger <tobiaslandenberger@gmail.com>
2023-11-14 12:31:44 -08:00
Timothy Carambat
6957bc3ec0
Robots.txt (#369)
* assume default model where appropriate

* merge with master and fix other model refs

* disallow robots

* add public file
2023-11-13 15:22:24 -08:00
Timothy Carambat
c22c50cca8
Enable chat streaming for LLMs (#354)
* [Draft] Enable chat streaming for LLMs

* stream only, move sendChat to deprecated

* Update TODO deprecation comments
update console output color for streaming disabled
2023-11-13 15:07:30 -08:00
Sean Hatfield
fa29003a46
Create manager role and limit default role (#351)
* added manager role to options

* block default role from editing workspace settings on workspace and text input box

* block default user from accessing settings at all

* create manager route

* let pass through if in single user mode

* fix permissions for manager and admin roles in settings

* fix settings button for single user and remove unneeded console.logs

* rename routes and paths for clarity

* admin, manager, default roles complete

* remove unneeded comments

* consistency changes

* manage permissions for mum modes

* update sidebar for single-user mode

* update comment on middleware
Modify permission setting for admins

* update render conditional

* Add role usage hint to each role

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-13 14:51:16 -08:00
Timothy Carambat
7fcf29d769
Add better code-block hightlighting and styles (#357)
* Add better code-block hightlighting and styles

* add fallback class
2023-11-10 16:31:21 -08:00
timothycarambat
6e72519caf patch onboarding 2023-11-09 15:19:52 -08:00
timothycarambat
195d3cfcc3 patch filename 2023-11-09 13:42:47 -08:00
Francisco Bischoff
f499f1ba59
Using OpenAI API locally (#335)
* Using OpenAI API locally

* Infinite prompt input and compression implementation (#332)

* WIP on continuous prompt window summary

* wip

* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff

* Implement compression for Anythropic
Fix lancedb sources

* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources

* Resolve Weaviate citation sources not working with schema

* comment cleanup

* disable import on hosted instances (#339)

* disable import on hosted instances

* Update UI on disabled import/export

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Add support for gpt-4-turbo 128K model (#340)

resolves #336
Add support for gpt-4-turbo 128K model

* 315 show citations based on relevancy score (#316)

* settings for similarity score threshold and prisma schema updated

* prisma schema migration for adding similarityScore setting

* WIP

* Min score default change

* added similarityThreshold checking for all vectordb providers

* linting

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

* rename localai to lmstudio

* forgot files that were renamed

* normalize model interface

* add model and context window limits

* update LMStudio tagline

* Fully working LMStudio integration

---------
Co-authored-by: Francisco Bischoff <984592+franzbischoff@users.noreply.github.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2023-11-09 12:33:21 -08:00
Timothy Carambat
1ec774ab2e
[Chore] replace all React-feather icons with phosphor icons fully (#349)
replace all React-feather icons with phosphor icons fully
remove package-lock.json files - yarn only
2023-11-09 08:55:20 -08:00
Sean Hatfield
458ffed0c7
added onboarding data handling modal (#342)
* added onboarding data handling modal

* adding data handling modal component

* update element to list
Update copy

* remove useEffect dep

* refactor onboarding navigation using history

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-08 19:00:12 -08:00
Sean Hatfield
997482ef8f
added JSONL export to workspace chats (#345)
* added JSONL export to workspace chats

* change permissions for workspace chat settings

* change permissions for workspace chat settings

* Show error for correct limit on fine-tune
Change sidebar position and permission
Remove check for MUM

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-08 17:36:54 -08:00
Timothy Carambat
88d4808c52
315 show citations based on relevancy score (#316)
* settings for similarity score threshold and prisma schema updated

* prisma schema migration for adding similarityScore setting

* WIP

* Min score default change

* added similarityThreshold checking for all vectordb providers

* linting

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2023-11-06 16:49:29 -08:00
Timothy Carambat
d34ec68702
Add support for gpt-4-turbo 128K model (#340)
resolves #336
Add support for gpt-4-turbo 128K model
2023-11-06 14:22:19 -08:00
Sean Hatfield
5673dc522b
disable import on hosted instances (#339)
* disable import on hosted instances

* Update UI on disabled import/export

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-06 13:23:12 -08:00
Timothy Carambat
be9d8b0397
Infinite prompt input and compression implementation (#332)
* WIP on continuous prompt window summary

* wip

* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff

* Implement compression for Anythropic
Fix lancedb sources

* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources

* Resolve Weaviate citation sources not working with schema

* comment cleanup
2023-11-06 13:13:53 -08:00
Sean Hatfield
eebafd7327
Replace old modals to match UI (#328)
* workspace not found modal updated to match ui

* update text preview modal to match new ui

* Mobile styles

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-02 16:17:02 -07:00
timothycarambat
c3abbfbf27 Fix admin chat pagination 2023-11-02 16:12:29 -07:00
Sean Hatfield
ab1e9d66b5
Use toast in workspace settings tab (#322)
* fix white border around user avatar

* add toast into settings menu on change/error
2023-11-01 18:21:30 -07:00
Sean Hatfield
1c5d55c425
Fix white border around user avatar glitch (#320)
fix white border around user avatar
2023-11-01 14:17:27 -07:00
Timothy Carambat
67c85f1550
Implement retrieval and use of fine-tune models (#314)
* Implement retrieval and use of fine-tune models
Cleanup LLM selection code
resolves #311

* Cleanup from PR bot
2023-10-31 11:38:28 -07:00
Timothy Carambat
5d56ab623b
Anthropic claude 2 support (#305)
* WIP Anythropic support for chat, chat and query w/context

* Add onboarding support for Anthropic

* cleanup

* fix Anthropic answer parsing
move embedding selector to general util
2023-10-30 15:44:03 -07:00
Sean Hatfield
27809b34b5
Added telemetry for onboarding completion (#295)
* added telemetry for onboarding completion

* minor changes

* linting and remove empty object

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-10-26 13:49:01 -07:00
Sean Hatfield
c8c0328346
Citations UI bug fix (#294)
citations spacing fix
2023-10-26 13:27:20 -07:00
Sean Hatfield
5f4d42edb8
Sidebar bug fix (#291)
truncate workspace name to prevent gear icon from being shown
2023-10-26 10:58:55 -07:00
Timothy Carambat
a8ec0d9584
Compensate for upper OpenAI emedding limit chunk size (#292)
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
2023-10-26 10:57:37 -07:00
Timothy Carambat
9f1a5d9dbc
Add back in selector on prompt input (#289)
resolves #286
2023-10-25 16:01:45 -07:00
Timothy Carambat
9f7b0c837a
Fix prompt overlaying on chats (#288)
resolves #287
2023-10-25 15:23:55 -07:00
timothycarambat
f6ae992c32 onboarding username password bug preventing multi user signup 2023-10-24 17:23:12 -07:00
timothycarambat
9b9644e931 remove typo 2023-10-23 14:07:04 -07:00
timothycarambat
81bb7bfc69 w-fit on workspace modal 2023-10-23 13:58:51 -07:00
Timothy Carambat
708068a09e
AnythingLLM UI overhaul (#278)
* v2 Login screen (#254)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* created reusable gradient for login screen

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 sidebar (#262)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* WIP sidebar redesign

* created reusable gradient for login screen

* remove dark mode items

* update new workspace button

* completed sidebar for desktop view

* add interactivity states

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* remove duplicated pkg

* v2 settings (#264)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* WIP sidebar redesign

* created reusable gradient for login screen

* remove dark mode items

* update new workspace button

* completed sidebar for desktop view

* WIP added colors/gradients to admin settings

* WIP fix discord logo import

* WIP settings redesign - added routes for general settings and restyled components

* WIP settings for LLM Preference, VectorDB, ExportImport

* settings menu UI complete WIP functionality

* settings fully functional/removed dark mode logo

* linting

* removing unneeded dependency

* Fix admin sidebar visibility
Fix API Keys location and work with single/mum
Fix Appearance location - WIP on funcitonality

* update api key page

* fix permissions for appearance

* Single user mode fixes

* fix multi user mode enabled

* fix import export

* Rename AdminSidebar to SettingsSidebar

* Fix mobile sidebar links

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* V2 user logout (#265)

* Add user logout button

* hide other 3 dot button

* wrap admin routes

* V2 workspace modal (#267)

Update new workspace modal
remove duplicate tailwind colors

* v2 Settings modal styles (#266)

* EditUserModal styles complete

* workspaces modals styles complete

* create invite link modal styles complete

* create new api key modal styles complete

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 Chats Redesign (#270)

* fix default message for new workspace

* prompt input box ui redesign complete

* ui tweak to prompt input

* WIP chat msg redesign

* chat container and historical chat messages redesign

* manage workspace modal appears when clicking upload a document on empty workspace

* fixed loading skeleton styles

* citations redesign complete

* restyle pending chat and prompt reply components

* default chat messages styles updated

* linting

* update how chats are returned

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Onboarding modal flow for first time setup (#274)

* WIP onboarding modal flow

* onboarding flow complete and private route redirection for onboarding setep

* redirect to home on onboarding complete

* add onboarding redirect using paths.onboarding()

* Apply changes to auth flow, onboarding determination, and flows

* remove formref

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 document picker (#275)

* remove unneeded comments

* WIP document picker UI

* WIP basic UI complete for document picker tab and settings tab

* linting

* settings menu complete, document row WIP

* WIP document picker loading from localFiles

* WIP file picker logic

* refactoring document picker to work with backend

* WIP refactoring document picker

* WIP refactor document picker to work with backend

* file uploading with dropzone working

* WIP deleting file when not embedded

* WIP embeddings

* WIP embedding with temp button and hardcoded paths

* WIP placeholder for WorkspaceDirectory component

* WIP WorkspaceDirectory

* WIP

* sort workspaceDocs and availibleDocs complete

* added directories util

* add and remove document from ws working

* v2 document picker complete

* reference modal ui bug fixes

* truncate function bug fix

* ManageWorkspace modal bug fixes

* blocking mobile users modal for workspace settings

* mobile ui fixes

* linting

* ui padding fixes

* citation bug fixes

* code review changes

* debounce handlers

* change tempFile object to array

* selection count fix

* Convert workspace modal to div
Memo workspace settings
update conditional rendering of workspace settings

* Show no documents

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* mobile sidebar styles

* padding on Mobile view
mobile sidebar items

* UI touchup

* suggestion implementations

* CSS fixes and animation perfomance change to GPU accelerated and 60fps

* change will-change

* remove transitions from onboarding modals, simplify on-change handlers

* Swap onboarding to memoized components and debounce onchange handlers

* remove console log

* remove Avenir font

---------

Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2023-10-23 13:10:34 -07:00
timothycarambat
ac2a0b0a06 reduce available types of models 2023-10-10 11:15:11 -07:00
timothycarambat
827a29d074 improve AUTH_TIMESTAMP clearing 2023-10-05 15:16:14 -07:00
timothycarambat
f9eec86672 Fix login modal popping up on protected routes 2023-10-05 14:34:30 -07:00
Timothy Carambat
62d39eb4fb
resolves #259 (#260)
Support API client for chroma
2023-09-29 13:20:06 -07:00
timothycarambat
a8933e56fd change member to default 2023-09-29 12:56:42 -07:00
Sean Hatfield
a126b5f5aa
Replace custom sqlite dbms with prisma (#239)
* WIP converted all sqlite models into prisma calls

* modify db setup and fix ApiKey model calls in admin.js

* renaming function params to be consistent

* converted adminEndpoints to utilize prisma orm

* converted chatEndpoints to utilize prisma orm

* converted inviteEndpoints to utilize prisma orm

* converted systemEndpoints to utilize prisma orm

* converted workspaceEndpoints to utilize prisma orm

* converting sql queries to prisma calls

* fixed default param bug for orderBy and limit

* fixed typo for workspace chats

* fixed order of deletion to account for sql relations

* fix invite CRUD and workspace management CRUD

* fixed CRUD for api keys

* created prisma setup scripts/docs for understanding how to use prisma

* prisma dependency change

* removing unneeded console.logs

* removing unneeded sql escape function

* linting and creating migration script

* migration from depreciated sqlite script update

* removing unneeded migrations in prisma folder

* create backup of old sqlite db and use transactions to ensure all operations complete successfully

* adding migrations to gitignore

* updated PRISMA.md docs for info on how to use sqlite migration script

* comment changes

* adding back migrations folder to repo

* Reviewing SQL and prisma integraiton on fresh repo

* update inline key replacement

* ensure migration script executes and maps foreign_keys regardless of db ordering

* run migration endpoint

* support new prisma backend

* bump version

* change migration call

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-09-28 14:00:03 -07:00