Commit Graph

90 Commits

Author SHA1 Message Date
Sean Hatfield
004b1f8db5
[FEAT] Implement new data connectors UI (#1034)
* WIP data connector redesign

* new UI for data connectors complete

* remove old data connector page/cleanup imports

* cleanup of UI and imports

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-04-05 14:25:41 -07:00
Sean Hatfield
6d80fe9602
[FIX] Add max height and scrolling to FolderSelectionPopup (#982)
* add max height and scrolling to FolderSelectionPopup

* hide scrollbar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-03-27 14:42:37 -07:00
Timothy Carambat
efe9dfa5e3
handle expired token being null and prevent constant rerender (#956)
* handle expired token being null and prevent constant rerender

* reset defaults
2024-03-26 16:47:25 -07:00
Sean Hatfield
129c456f78
[REFACTOR] Refactor & comment document picker modal (#929)
* implement alternating color rows for file picker

* implement alternating colors for workspace directory

* remove unneeded props/variables

* remove unused border classes

* WIP new folder UI

* remove unneeded expanded prop from filerow component

* folder creation UI and files object manipulation WIP

* folder creation & moving files complete

* add developer API support for creating folders and moving files

* update alternating row css for file picker

* remove unneeded props from merge

* add comments to ManageWorkspace modal and switch flex direction on smaller screen sizes

* drop Legacy and remove transitions post-merge

* switch flex direction on document picker on smaller screen sizes + css fixes for my documents

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-03-20 17:40:57 -07:00
Sean Hatfield
4f268dfeb6
[FEAT] Document picker folders for organization (#902)
* implement alternating color rows for file picker

* implement alternating colors for workspace directory

* remove unneeded props/variables

* remove unused border classes

* WIP new folder UI

* remove unneeded expanded prop from filerow component

* folder creation UI and files object manipulation WIP

* folder creation & moving files complete

* add developer API support for creating folders and moving files

* update alternating row css for file picker

* remove unneeded props from merge

* normalize paths for folders
priority to custom docs folder
silently fail on duplicate folders

* update folder icon to custom svg

* linting and move FolderIcon to JSX

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-03-20 16:10:30 -07:00
Sean Hatfield
1352b18b5f
[FEAT] Implement correct highlight colors on document picker (#897)
* implement alternating color rows for file picker

* implement alternating colors for workspace directory

* remove unneeded props/variables

* remove unused border classes

* remove unneeded expanded prop from filerow component
2024-03-14 10:52:35 -07:00
Sean Hatfield
fd2b13017e
[FEAT] On file upload, clear dropzone box of previously uploaded files (#894)
on file upload, clear uploader box from previously uploaded files
2024-03-12 11:40:16 -07:00
Timothy Carambat
147426704c
adjust max upload window height (#844) 2024-02-29 10:09:49 -08:00
Sean Hatfield
2974a969b5
[FEAT]: Pin to workspace UI update (#828)
* pin to workspace UI update

* tweak css

* change tooltip to white to match designs
2024-02-27 11:55:26 -08:00
Timothy Carambat
bbe3c0a5d6
Bulk remove files from file picker (#830)
Do bulk deletion on backend
2024-02-27 11:53:42 -08:00
Sean Hatfield
e87cba3468
[CHORE] Normalize styles of all input and select elements (#804)
* normalize styles of all input and select elements

* missed placeholder text input

* missed input fields on onboarding flow
2024-02-27 11:47:01 -08:00
Timothy Carambat
6d18d79bb7
Generic upload fallback as text file. (#808)
* Do not block any file upload
fallback unknown/unsupported types to text if possible

* reduce call for frontend

* patch
2024-02-26 13:43:54 -08:00
Timothy Carambat
791c0ee9dc
Enable ability to do full-text query on documents (#758)
* Enable ability to do full-text query on documents
Show alert modal on first pin for client
Add ability to use pins in stream/chat/embed

* typo and copy update

* simplify spread of context and sources
2024-02-21 13:15:45 -08:00
Sean Hatfield
858b2fcedb
[FEAT] New Workspace Settings Layout (#718)
* WIP new settings layout

* add suggested messages to general & appearance and clean up/make more user friendly

* lazy load workspace settings pages

* css fix on X button for document picker where button is barely clickable

* remove additional workspace settings page

* fix thread selection action when on thread

* refactor inputs into sub-components
remove unused paths

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-02-14 15:29:49 -08:00
timothycarambat
a01d4b847e drop temp loading state on workspace modal 2024-02-07 12:48:59 -08:00
Sean Hatfield
b94da205a5
fix truncate function to handle Chinese characters (#677)
* fix truncate function to handle Chinese characters

* revert changes to truncate function and cut off title using css

* add CSS ellipse on long file name

* fix uploaded middleTruncation as well as directory

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-02-06 11:30:49 -08:00
Sean Hatfield
608f28d745
[FEAT] create custom prompt suggestions per workspace (#664)
* create custom suggested chat messages per workspace

* update how suggestedChats are passed to chat window

* update mobile styles

* update edit change handler

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-02-06 11:24:33 -08:00
Sean Hatfield
9d41ff58e2
[FEAT] add support for new openai embedding models (#653)
* add support for new openai models

* QOL changes/improve logic for adding new openai embedding models

* add example file inputs for Openai embedding ENV selection;

* Fix if stmt conditional

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-29 08:48:27 -08:00
Hakeem Abbas
5614e2ed30
feature: Integrate Astra as vectorDBProvider (#648)
* feature: Integrate Astra as vectorDBProvider

feature: Integrate Astra as vectorDBProvider

* Update .env.example

* Add env.example to docker example file
Update spellcheck fo Astra
Update Astra key for vector selection
Update order of AstraDB options
Resize Astra logo image to 330x330
Update methods of Astra to take in latest vectorDB params like TopN and more
Update Astra interface to support default methods and avoid crash errors from 404 collections
Update Astra interface to comply to max chunk insertion limitations
Update Astra interface to dynamically set dimensionality from chunk 0 size on creation

* reset workspaces

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-26 13:07:53 -08:00
Sean Hatfield
21653b09fc
[FEAT] add gpt-4-turbo-preview (#651)
* add gpt-4-turbo-preview

* add gpt-4-turbo-preview to valid models
2024-01-26 13:03:50 -08:00
Sean Hatfield
62cea07599
add gpt-3.5-turbo-1106 model for openai LLM (#636)
* add gpt-3.5-turbo-1106 model for openai LLM

* add gpt-3.5-turbo-1106 as valid model for backend and per workspace model selection
2024-01-22 13:19:47 -08:00
Sean Hatfield
288ff0d18c
fix vector cache not deleting cache after unembedding items with folders (#630) 2024-01-22 13:03:05 -08:00
Sean Hatfield
56fa17caf2
create configurable topN per workspace (#616)
* create configurable topN per workspace

* Update TopN UI text
Fix fallbacks for all providers
Add SQLite CHECK to TOPN value

* merge with master
Update zilliz provider for variable TopN

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-18 12:34:20 -08:00
Sean Hatfield
c2c8fe9756
add support for mistral api (#610)
* add support for mistral api

* update docs to show support for Mistral

* add default temp to all providers, suggest different results per provider

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 14:42:05 -08:00
Sean Hatfield
90df37582b
Per workspace model selection (#582)
* WIP model selection per workspace (migrations and openai saves properly

* revert OpenAiOption

* add support for models per workspace for anthropic, localAi, ollama, openAi, and togetherAi

* remove unneeded comments

* update logic for when LLMProvider is reset, reset Ai provider files with master

* remove frontend/api reset of workspace chat and move logic to updateENV
add postUpdate callbacks to envs

* set preferred model for chat on class instantiation

* remove extra param

* linting

* remove unused var

* refactor chat model selection on workspace

* linting

* add fallback for base path to localai models

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-01-17 12:59:25 -08:00
Timothy Carambat
3c859ba303
Change pwd check to O(1) check to prevent timing attacks - single user mode (#575)
Change pwd check to O(1) check to prevent timing attacks
2024-01-11 10:54:55 -08:00
Timothy Carambat
4e2c0f04b4
Dynamic vector count on workspace settings (#567)
* Dynamic vector count on workspace settings
Add count to be workspace specific, fallback to system count
Update layout of data in settings
Update OpenAI per-token embedding price

* linting
2024-01-10 13:18:48 -08:00
timothycarambat
7bee849c65 chore: Force VectorCache to always be on;
update file picker spacing for attributes
2023-12-20 10:45:03 -08:00
Timothy Carambat
ecf4295537
Add ability to grab youtube transcripts via doc processor (#470)
* Add ability to grab youtube transcripts via doc processor

* dynamic imports
swap out Github for Youtube in placeholder text
2023-12-18 17:17:26 -08:00
Timothy Carambat
452582489e
GitHub loader extension + extension support v1 (#469)
* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files

* noshow delete on custom-documents

* Add API key support because of rate limits

* WIP for frontend of data connectors

* wip

* Add frontend form for GitHub repo data connector

* remove console.logs
block custom-documents from being deleted

* remove _meta unused arg

* Add support for ignore pathing in request
Ignore path input via tagging

* Update hint
2023-12-18 15:48:02 -08:00
Timothy Carambat
1c549e2844
fix: clarify custom truncate from npm truncate (#464)
chore: add @ where we can for less long imports
2023-12-17 15:58:49 -08:00
Timothy Carambat
61db981017
feat: Embed on-instance Whisper model for audio/mp4 transcribing (#449)
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves #329

* additional logging

* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency

* update README

* update model size

* update supported filetypes
2023-12-15 11:20:13 -08:00
Timothy Carambat
f48e6b1a3e
chore: add @ as alias for frontend root (#414)
* chore: add @ as alias for frontend root

* fix: remove bad tag
2023-12-07 09:09:01 -08:00
timothycarambat
203f5964d5 linting 2023-12-04 08:36:17 -08:00
Sean Hatfield
ca8bf52cce
Bug fixes (#394)
* fix black text for custom messages text

* fix file upload icon being stretched

* center github, docs, and discord icons in sidebar

* fix chat container being cut off on right side and tighter spacing between message

* fix default chat container being cut off on right side

* on create new workspace, take user to the workspace they just created instead of the home page

* add border to chat container and click outside user menu to close

* fix borders around all chat and settings containers to be consistent

* fix padding for default messages

* fix spacing between workspace items in sidebar

* fix margin around right side of chat, default, and settings containers to be the same as the left sidebar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-27 15:09:55 -06:00
Sean Hatfield
7edfccaf9a
Adding url uploads to document picker (#375)
* WIP adding url uploads to document picker

* fix manual script for uploading url to custom-documents

* fix metadata for url scraping

* wip url parsing

* update how async link scraping works

* docker-compose defaults added
no autocomplete on URLs

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 17:15:01 -08:00
Tobias Landenberger
a96a9d41a3
LocalAI for embeddings (#361)
* feature: add localAi as embedding provider

* chore: add LocalAI image

* chore: add localai embedding examples to docker .env.example

* update setting env
pull models from localai API

* update comments on embedder
Dont show cost estimation on UI

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:49:31 -08:00
timothycarambat
f269124760 fix type error for filetypes 2023-11-14 13:49:18 -08:00
Sean Hatfield
fa29003a46
Create manager role and limit default role (#351)
* added manager role to options

* block default role from editing workspace settings on workspace and text input box

* block default user from accessing settings at all

* create manager route

* let pass through if in single user mode

* fix permissions for manager and admin roles in settings

* fix settings button for single user and remove unneeded console.logs

* rename routes and paths for clarity

* admin, manager, default roles complete

* remove unneeded comments

* consistency changes

* manage permissions for mum modes

* update sidebar for single-user mode

* update comment on middleware
Modify permission setting for admins

* update render conditional

* Add role usage hint to each role

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-13 14:51:16 -08:00
Timothy Carambat
1ec774ab2e
[Chore] replace all React-feather icons with phosphor icons fully (#349)
replace all React-feather icons with phosphor icons fully
remove package-lock.json files - yarn only
2023-11-09 08:55:20 -08:00
Timothy Carambat
88d4808c52
315 show citations based on relevancy score (#316)
* settings for similarity score threshold and prisma schema updated

* prisma schema migration for adding similarityScore setting

* WIP

* Min score default change

* added similarityThreshold checking for all vectordb providers

* linting

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2023-11-06 16:49:29 -08:00
Timothy Carambat
be9d8b0397
Infinite prompt input and compression implementation (#332)
* WIP on continuous prompt window summary

* wip

* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff

* Implement compression for Anythropic
Fix lancedb sources

* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources

* Resolve Weaviate citation sources not working with schema

* comment cleanup
2023-11-06 13:13:53 -08:00
timothycarambat
c3abbfbf27 Fix admin chat pagination 2023-11-02 16:12:29 -07:00
Sean Hatfield
ab1e9d66b5
Use toast in workspace settings tab (#322)
* fix white border around user avatar

* add toast into settings menu on change/error
2023-11-01 18:21:30 -07:00
Timothy Carambat
5d56ab623b
Anthropic claude 2 support (#305)
* WIP Anythropic support for chat, chat and query w/context

* Add onboarding support for Anthropic

* cleanup

* fix Anthropic answer parsing
move embedding selector to general util
2023-10-30 15:44:03 -07:00
Timothy Carambat
a8ec0d9584
Compensate for upper OpenAI emedding limit chunk size (#292)
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
2023-10-26 10:57:37 -07:00
timothycarambat
81bb7bfc69 w-fit on workspace modal 2023-10-23 13:58:51 -07:00
Timothy Carambat
708068a09e
AnythingLLM UI overhaul (#278)
* v2 Login screen (#254)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* created reusable gradient for login screen

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 sidebar (#262)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* WIP sidebar redesign

* created reusable gradient for login screen

* remove dark mode items

* update new workspace button

* completed sidebar for desktop view

* add interactivity states

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* remove duplicated pkg

* v2 settings (#264)

* adding gradients for modal and sidebar

* adding font setup

* redesigned login screen for MultiUserAuth

* completed multi user mode login screen

* linting

* login screen for single user auth redesign complete

* WIP sidebar redesign

* created reusable gradient for login screen

* remove dark mode items

* update new workspace button

* completed sidebar for desktop view

* WIP added colors/gradients to admin settings

* WIP fix discord logo import

* WIP settings redesign - added routes for general settings and restyled components

* WIP settings for LLM Preference, VectorDB, ExportImport

* settings menu UI complete WIP functionality

* settings fully functional/removed dark mode logo

* linting

* removing unneeded dependency

* Fix admin sidebar visibility
Fix API Keys location and work with single/mum
Fix Appearance location - WIP on funcitonality

* update api key page

* fix permissions for appearance

* Single user mode fixes

* fix multi user mode enabled

* fix import export

* Rename AdminSidebar to SettingsSidebar

* Fix mobile sidebar links

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* V2 user logout (#265)

* Add user logout button

* hide other 3 dot button

* wrap admin routes

* V2 workspace modal (#267)

Update new workspace modal
remove duplicate tailwind colors

* v2 Settings modal styles (#266)

* EditUserModal styles complete

* workspaces modals styles complete

* create invite link modal styles complete

* create new api key modal styles complete

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 Chats Redesign (#270)

* fix default message for new workspace

* prompt input box ui redesign complete

* ui tweak to prompt input

* WIP chat msg redesign

* chat container and historical chat messages redesign

* manage workspace modal appears when clicking upload a document on empty workspace

* fixed loading skeleton styles

* citations redesign complete

* restyle pending chat and prompt reply components

* default chat messages styles updated

* linting

* update how chats are returned

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Onboarding modal flow for first time setup (#274)

* WIP onboarding modal flow

* onboarding flow complete and private route redirection for onboarding setep

* redirect to home on onboarding complete

* add onboarding redirect using paths.onboarding()

* Apply changes to auth flow, onboarding determination, and flows

* remove formref

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* v2 document picker (#275)

* remove unneeded comments

* WIP document picker UI

* WIP basic UI complete for document picker tab and settings tab

* linting

* settings menu complete, document row WIP

* WIP document picker loading from localFiles

* WIP file picker logic

* refactoring document picker to work with backend

* WIP refactoring document picker

* WIP refactor document picker to work with backend

* file uploading with dropzone working

* WIP deleting file when not embedded

* WIP embeddings

* WIP embedding with temp button and hardcoded paths

* WIP placeholder for WorkspaceDirectory component

* WIP WorkspaceDirectory

* WIP

* sort workspaceDocs and availibleDocs complete

* added directories util

* add and remove document from ws working

* v2 document picker complete

* reference modal ui bug fixes

* truncate function bug fix

* ManageWorkspace modal bug fixes

* blocking mobile users modal for workspace settings

* mobile ui fixes

* linting

* ui padding fixes

* citation bug fixes

* code review changes

* debounce handlers

* change tempFile object to array

* selection count fix

* Convert workspace modal to div
Memo workspace settings
update conditional rendering of workspace settings

* Show no documents

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* mobile sidebar styles

* padding on Mobile view
mobile sidebar items

* UI touchup

* suggestion implementations

* CSS fixes and animation perfomance change to GPU accelerated and 60fps

* change will-change

* remove transitions from onboarding modals, simplify on-change handlers

* Swap onboarding to memoized components and debounce onchange handlers

* remove console log

* remove Avenir font

---------

Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2023-10-23 13:10:34 -07:00
timothycarambat
ac2a0b0a06 reduce available types of models 2023-10-10 11:15:11 -07:00
timothycarambat
827a29d074 improve AUTH_TIMESTAMP clearing 2023-10-05 15:16:14 -07:00