* WIP openrouter integration
* add OpenRouter options to onboarding flow and data handling
* add todo to fix headers for rankings
* OpenRouter LLM support complete
* Fix hanging response stream with OpenRouter
update tagline
update comment
* update timeout comment
* wait for first chunk to start timer
* sort OpenRouter models by organization
* uppercase first letter of organization
* sort grouped models by org
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* show similarity score in citation chunks
* refactor combine like sources to handle separate similarity scores and improve UI for displaying chunks
* fix parseChunkSource function to work with links
* destructure properties in chunks mapping
* check chunk length in parseChunkSource
* change UI on how score is shown
* remove duplicate import
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add LLM support for perplexity
* update README & example env
* fix ENV keys in example env files
* slight changes for QA of perplexity support
* Update Perplexity AI name
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enable ability to do full-text query on documents
Show alert modal on first pin for client
Add ability to use pins in stream/chat/embed
* typo and copy update
* simplify spread of context and sources
* split CustomMessages and CustomLogo into separate components from appearance settings
* lint
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Remove useless icons in prompt bar
Add chatMode column to workspaces that defaults to chat
Add UI for toggle of chat mode with hint
Update UI for workspace settings to match designs
* WIP new settings layout
* add suggested messages to general & appearance and clean up/make more user friendly
* lazy load workspace settings pages
* css fix on X button for document picker where button is barely clickable
* remove additional workspace settings page
* fix thread selection action when on thread
* refactor inputs into sub-components
remove unused paths
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Include links in citations
force ChunkSource key to retain this information
old links will be unsupported
* show special icons depending on source
* remove console log
* reset server documents writeTo
* WIP RLHF works on historical messages
* refactor Actions component
* completed RLHF up and down votes for chats
* add defaults for HistoricalMessage params
* refactor RLHF implmenation
remove forwardRef on history items to prevent rerenders
* remove dup id
* Add rating to CSV output
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add more fields to csv export to make more useful
* refactor from review comments
* fix escapeCsv function
* catch export errors properly
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP custom footer icons
* UI for updating footer icons complete and backend to save/modify
* add backend for unprotected footer fetch
* break out footer into separate component and render footer items using a cache for 1 hour
* wip review
* refactor & cleanup
* Optimize footer form component
Optimize caching for footer icons
Add validation on SystemSetting upserts
Normalize fallback items for footer_data
* Adjust max icons to 3
* fix success message on remove
* fix success message on remove
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* hide security setting in settings sidebar when in multiusermode
* refactor sidebar to handle mobile view inside SettingsSidebar component
* forgot eventlogs on sidebar
move footer on settingSidebar to component
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP event logging - new table for events and new settings view for viewing
* WIP add logging
* UI for log rows
* rename files to Logging to prevent getting gitignore
* add metadata for all logging events and colored badges in logs page
* remove unneeded comment
* cleanup namespace for logging
* clean up backend calls
* update logging to show to => from settings changes
* add logging for invitations, created, deleted, and accepted
* add logging for user created, updated, suspended, or removed
* add logging for workspace deleted
* add logging for chat logs exported
* add logging for API keys, LLM, embedder, vector db, embed chat, and reset button
* modify event logs
* update to event log types
* simplify rendering of event badges
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* use filter instead of ref to delete row from workspace chat component on delete/fix backend invalid json error
* remove ref from pagination on embed chats and fix white screen bug
* remove unneeded import
* normalize response object
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* fix truncate function to handle Chinese characters
* revert changes to truncate function and cut off title using css
* add CSS ellipse on long file name
* fix uploaded middleTruncation as well as directory
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP embedded app
* WIP got response from backend in embedded app
* WIP streaming prints to embedded app
* implemented streaming and tailwind min for styling into embedded app
* WIP embedded app history functional
* load params from script tag into embedded app
* rough in modularization of embed chat
cleanup dev process for easier dev support
move all chat to components
todo: build process
todo: backend support
* remove eslint config
* Implement models and cleanup embed chat endpoints
Improve build process for embed
prod minification and bundle size awareness
WIP
* forgot files
* rename to embed folder
* introduce chat modal styles
* add middleware validations on embed chat
* auto open param and default greeting
* reset chat history
* Admin embed config page
* Admin Embed Chats mgmt page
* update embed
* nonpriv
* more style support
reopen if chat was last opened
* update comments
* remove unused imports
* allow change of workspace for embedconfig
* update failure to lookup message
* update reset script
* update instructions
* Add more styling options
Add sponsor text at bottom
Support dynamic container height
Loading animations
* publish new embed script
* Add back syntax highlighting and keep bundle small via dynamic script build
* add hint
* update readme
* update copy model for snippet with link to styles
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add support for new openai models
* QOL changes/improve logic for adding new openai embedding models
* add example file inputs for Openai embedding ENV selection;
* Fix if stmt conditional
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feature: Integrate Astra as vectorDBProvider
feature: Integrate Astra as vectorDBProvider
* Update .env.example
* Add env.example to docker example file
Update spellcheck fo Astra
Update Astra key for vector selection
Update order of AstraDB options
Resize Astra logo image to 330x330
Update methods of Astra to take in latest vectorDB params like TopN and more
Update Astra interface to support default methods and avoid crash errors from 404 collections
Update Astra interface to comply to max chunk insertion limitations
Update Astra interface to dynamically set dimensionality from chunk 0 size on creation
* reset workspaces
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add useModal hook and ModalWrapper component that will be used to replace all <dialog> modals for better browser support
* implement useModal hook and ModalWrapper component to replace all exisiting <dialog>
* migrate pinecone package to latest version and migrate pinecone vectordb provider class
* remove pinecone environment name env variable and update docs to reflect removal & serverless support complete
* migrate query for pinecone db
* typo in log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Implement total permission overhaul
Add explicit permissions on each flex and strict route
Patch issues with role escalation and CRUD of users
Patch permissions on all routes for coverage
Improve middleware to accept role array for clarity
* update comments
* remove permissions to API-keys for manager. Manager could generate API-key and using high-privelege api-key give themselves admin
* update sidebar permissions for multi-user and single user
* update options for mobile sidebar
* add support for exporting to json and csv in workspace chats
* safety encode URL options
* remove message about openai fine tuning on export success
* all defaults to jsonl
* create configurable topN per workspace
* Update TopN UI text
Fix fallbacks for all providers
Add SQLite CHECK to TOPN value
* merge with master
Update zilliz provider for variable TopN
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add copy feature to assistant chat message
* fix tooltip not hiding on mobile
* fix: add tooltips
chore: breakout actions to extendable component + memoize
add CopyText to hook we can reuse
fix: Copy on code snippets broken, moved to event listener
fix: highlightjs patch for new API support
feat: add copy response support
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add support for mistral api
* update docs to show support for Mistral
* add default temp to all providers, suggest different results per provider
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP model selection per workspace (migrations and openai saves properly
* revert OpenAiOption
* add support for models per workspace for anthropic, localAi, ollama, openAi, and togetherAi
* remove unneeded comments
* update logic for when LLMProvider is reset, reset Ai provider files with master
* remove frontend/api reset of workspace chat and move logic to updateENV
add postUpdate callbacks to envs
* set preferred model for chat on class instantiation
* remove extra param
* linting
* remove unused var
* refactor chat model selection on workspace
* linting
* add fallback for base path to localai models
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* truncate title to shorter length so x button is not hidden for longer title names in the citation modal
* absolutely position x button on citation modal
* issue #543 support milvus vector db
* migrate Milvus to use MilvusClient instead of ORM
normalize env setup for docs/implementation
feat: embedder model dimension added
* update comments
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Dynamic vector count on workspace settings
Add count to be workspace specific, fallback to system count
Update layout of data in settings
Update OpenAI per-token embedding price
* linting
* add Together AI LLM support
* update readme to support together ai
* Patch togetherAI implementation
* add model sorting/option labels by organization for model selection
* linting + add data handling for TogetherAI
* change truthy statement
patch validLLMSelection method
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Implement support for GitHub codespaces and VSCode devcontainers
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* move llm, embedder, vectordb items to components folder
* add backdrop blur to search in llm, embedder, vectordb preferences
* implement searchable llm preference in settings
* implement searchable embedder in settings
* remove unused useState from embedder preferences
* implement searchable vector database in settings
* fix save changes button not appearing on change for llm, embedder, and vectordb settings pages
* sort selected items in all settings and put selected item at top of list
* no auto-top for selection
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files
* noshow delete on custom-documents
* Add API key support because of rate limits
* WIP for frontend of data connectors
* wip
* Add frontend form for GitHub repo data connector
* remove console.logs
block custom-documents from being deleted
* remove _meta unused arg
* Add support for ignore pathing in request
Ignore path input via tagging
* Update hint
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves#329
* additional logging
* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency
* update README
* update model size
* update supported filetypes
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev
* Add native embedder as an available embedder selection
* wrap model loader in try/catch
* print progress on download
* add built-in LLM support (expiermental)
* Update to progress output for embedder
* move embedder selection options to component
* saftey checks for modelfile
* update ref
* Hide selection when on hosted subdomain
* update documentation
hide localLlama when on hosted
* saftey checks for storage of models
* update dockerfile to pre-build Llama.cpp bindings
* update lockfile
* add langchain doc comment
* remove extraneous --no-metal option
* Show data handling for private LLM
* persist model in memory for N+1 chats
* update import
update dev comment on token model size
* update primary README
* chore: more readme updates and remove screenshots - too much to maintain, just use the app!
* remove screeshot link
* fix sizing of onboarding modals & lint
* fix extra scrolling on mobile onboarding flow
* added message to use desktop for onboarding
* linting
* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change
* fix for empty chat
* change mobile alert copy
* WIP adding PFP upload support
* WIP pfp for users
* edit account menu complete with change username/password and upload profile picture
* add pfp context to update all instances of usePfp hook on update
* linting
* add context for logo change to immediately update logo
* fix div with bullet points to use list-disc instead
* fix: small changes
* update multer file storage locations
* fix: use STORAGE_DIR for filepathing
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev
* Add native embedder as an available embedder selection
* wrap model loader in try/catch
* print progress on download
* Update to progress output for embedder
* move embedder selection options to component
* forgot import
* add Data privacy alert updates for local embedder
