* add support for voyageai embedder
* remove unneeded import
* linting
* Add ENV examples
Update how chunks are processed for Voyage
use correct langchain import
Add data handling
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Support SQL Agent skill
* add MSSQL agent connector
* Add frontend to agent skills
remove FAKE_DB mock
reset skills to pickup child-skill dynamically
* add prompt examples for tools on untooled
* add better logging on SQL agents
* Wipe toolruns on each chat relay so tools can be used within the same session
* update comments
* implement regenerate response button
* wip on rerenders
* remove function that was duplicate
* update delete-chats function
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP slash presets
* WIP slash command customization CRUD + validations complete
* backend slash command support
* fix permission setting on new slash commands
rework form submit and pattern on frontend
* Add field updates for hooks,
required=true to field
add user<>command constraint to keep them unique
enforce uniquness via teritary uid field on table for multi and non-multi user
* reset migration
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs
* add comments
* enable few-shot prompting per function for OSS models
* WIP new login screen UI
* update prisma schema/create new models for pw recovery
* WIP password recovery backend
* WIP reset password flow
* WIP pw reset flow
* password reset logic complete & functional UI
* WIP login screen redesign for single and multi user
* create placeholder modal to display recovery codes
* implement UI for recovery code modals/download recovery codes
* multiuser desktop password reset UI/functionality complete
* support single user mode for pw reset
* mobile styles for all password reset/login flows complete
* lint
* remove single user password recovery
* create PasswordRecovery util file to make more readable
* do not drop-replace users table in migration
* review pr
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enable per-workspace provider/model combination
* cleanup
* remove resetWorkspaceChatModels and wipeWorkspaceModelPreference to prevent workspace from resetting model
* add space
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* members workspace settings menu and admin users UI updates
* change copy/fix admin and managers in workspace when not showing in UI
* remove existing workspace user mgmt modal
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Enable ability to do full-text query on documents
Show alert modal on first pin for client
Add ability to use pins in stream/chat/embed
* typo and copy update
* simplify spread of context and sources
Remove useless icons in prompt bar
Add chatMode column to workspaces that defaults to chat
Add UI for toggle of chat mode with hint
Update UI for workspace settings to match designs
* refactor stream/chat/embed-stram to be a single execution logic path so that it is easier to maintain and build upon
* no thread in sync chat since only api uses it
adjust import locations
* WIP RLHF works on historical messages
* refactor Actions component
* completed RLHF up and down votes for chats
* add defaults for HistoricalMessage params
* refactor RLHF implmenation
remove forwardRef on history items to prevent rerenders
* remove dup id
* Add rating to CSV output
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add more fields to csv export to make more useful
* refactor from review comments
* fix escapeCsv function
* catch export errors properly
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP custom footer icons
* UI for updating footer icons complete and backend to save/modify
* add backend for unprotected footer fetch
* break out footer into separate component and render footer items using a cache for 1 hour
* wip review
* refactor & cleanup
* Optimize footer form component
Optimize caching for footer icons
Add validation on SystemSetting upserts
Normalize fallback items for footer_data
* Adjust max icons to 3
* fix success message on remove
* fix success message on remove
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP event logging - new table for events and new settings view for viewing
* WIP add logging
* UI for log rows
* rename files to Logging to prevent getting gitignore
* add metadata for all logging events and colored badges in logs page
* remove unneeded comment
* cleanup namespace for logging
* clean up backend calls
* update logging to show to => from settings changes
* add logging for invitations, created, deleted, and accepted
* add logging for user created, updated, suspended, or removed
* add logging for workspace deleted
* add logging for chat logs exported
* add logging for API keys, LLM, embedder, vector db, embed chat, and reset button
* modify event logs
* update to event log types
* simplify rendering of event badges
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* use filter instead of ref to delete row from workspace chat component on delete/fix backend invalid json error
* remove ref from pagination on embed chats and fix white screen bug
* remove unneeded import
* normalize response object
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP embedded app
* WIP got response from backend in embedded app
* WIP streaming prints to embedded app
* implemented streaming and tailwind min for styling into embedded app
* WIP embedded app history functional
* load params from script tag into embedded app
* rough in modularization of embed chat
cleanup dev process for easier dev support
move all chat to components
todo: build process
todo: backend support
* remove eslint config
* Implement models and cleanup embed chat endpoints
Improve build process for embed
prod minification and bundle size awareness
WIP
* forgot files
* rename to embed folder
* introduce chat modal styles
* add middleware validations on embed chat
* auto open param and default greeting
* reset chat history
* Admin embed config page
* Admin Embed Chats mgmt page
* update embed
* nonpriv
* more style support
reopen if chat was last opened
* update comments
* remove unused imports
* allow change of workspace for embedconfig
* update failure to lookup message
* update reset script
* update instructions
* Add more styling options
Add sponsor text at bottom
Support dynamic container height
Loading animations
* publish new embed script
* Add back syntax highlighting and keep bundle small via dynamic script build
* add hint
* update readme
* update copy model for snippet with link to styles
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* migrate pinecone package to latest version and migrate pinecone vectordb provider class
* remove pinecone environment name env variable and update docs to reflect removal & serverless support complete
* migrate query for pinecone db
* typo in log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Implement total permission overhaul
Add explicit permissions on each flex and strict route
Patch issues with role escalation and CRUD of users
Patch permissions on all routes for coverage
Improve middleware to accept role array for clarity
* update comments
* remove permissions to API-keys for manager. Manager could generate API-key and using high-privelege api-key give themselves admin
* update sidebar permissions for multi-user and single user
* update options for mobile sidebar
* add support for exporting to json and csv in workspace chats
* safety encode URL options
* remove message about openai fine tuning on export success
* all defaults to jsonl
* WIP model selection per workspace (migrations and openai saves properly
* revert OpenAiOption
* add support for models per workspace for anthropic, localAi, ollama, openAi, and togetherAi
* remove unneeded comments
* update logic for when LLMProvider is reset, reset Ai provider files with master
* remove frontend/api reset of workspace chat and move logic to updateENV
add postUpdate callbacks to envs
* set preferred model for chat on class instantiation
* remove extra param
* linting
* remove unused var
* refactor chat model selection on workspace
* linting
* add fallback for base path to localai models
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add support for fetching single document in documents folder
* Add document object to upload + support link scraping via API
* hotfixes for documentation
* update api docs
* Dynamic vector count on workspace settings
Add count to be workspace specific, fallback to system count
Update layout of data in settings
Update OpenAI per-token embedding price
* linting
* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files
* noshow delete on custom-documents
* Add API key support because of rate limits
* WIP for frontend of data connectors
* wip
* Add frontend form for GitHub repo data connector
* remove console.logs
block custom-documents from being deleted
* remove _meta unused arg
* Add support for ignore pathing in request
Ignore path input via tagging
* Update hint
* wip: init refactor of document processor to JS
* add NodeJs PDF support
* wip: partity with python processor
feat: add pptx support
* fix: forgot files
* Remove python scripts totally
* wip:update docker to boot new collector
* add package.json support
* update dockerfile for new build
* update gitignore and linting
* add more protections on file lookup
* update package.json
* test build
* update docker commands to use cap-add=SYS_ADMIN so web scraper can run
update all scripts to reflect this
remove docker build for branch
* fix sizing of onboarding modals & lint
* fix extra scrolling on mobile onboarding flow
* added message to use desktop for onboarding
* linting
* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change
* fix for empty chat
* change mobile alert copy
* WIP adding PFP upload support
* WIP pfp for users
* edit account menu complete with change username/password and upload profile picture
* add pfp context to update all instances of usePfp hook on update
* linting
* add context for logo change to immediately update logo
* fix div with bullet points to use list-disc instead
* fix: small changes
* update multer file storage locations
* fix: use STORAGE_DIR for filepathing
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feature: add LocalAI as llm provider
* update Onboarding/mgmt settings
Grab models from models endpoint for localai
merge with master
* update streaming for complete chunk streaming
update localAI LLM to be able to stream
* force schema on URL
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: tlandenberger <tobiaslandenberger@gmail.com>
* added manager role to options
* block default role from editing workspace settings on workspace and text input box
* block default user from accessing settings at all
* create manager route
* let pass through if in single user mode
* fix permissions for manager and admin roles in settings
* fix settings button for single user and remove unneeded console.logs
* rename routes and paths for clarity
* admin, manager, default roles complete
* remove unneeded comments
* consistency changes
* manage permissions for mum modes
* update sidebar for single-user mode
* update comment on middleware
Modify permission setting for admins
* update render conditional
* Add role usage hint to each role
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* added JSONL export to workspace chats
* change permissions for workspace chat settings
* change permissions for workspace chat settings
* Show error for correct limit on fine-tune
Change sidebar position and permission
Remove check for MUM
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP on continuous prompt window summary
* wip
* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff
* Implement compression for Anythropic
Fix lancedb sources
* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources
* Resolve Weaviate citation sources not working with schema
* comment cleanup
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
* WIP converted all sqlite models into prisma calls
* modify db setup and fix ApiKey model calls in admin.js
* renaming function params to be consistent
* converted adminEndpoints to utilize prisma orm
* converted chatEndpoints to utilize prisma orm
* converted inviteEndpoints to utilize prisma orm
* converted systemEndpoints to utilize prisma orm
* converted workspaceEndpoints to utilize prisma orm
* converting sql queries to prisma calls
* fixed default param bug for orderBy and limit
* fixed typo for workspace chats
* fixed order of deletion to account for sql relations
* fix invite CRUD and workspace management CRUD
* fixed CRUD for api keys
* created prisma setup scripts/docs for understanding how to use prisma
* prisma dependency change
* removing unneeded console.logs
* removing unneeded sql escape function
* linting and creating migration script
* migration from depreciated sqlite script update
* removing unneeded migrations in prisma folder
* create backup of old sqlite db and use transactions to ensure all operations complete successfully
* adding migrations to gitignore
* updated PRISMA.md docs for info on how to use sqlite migration script
* comment changes
* adding back migrations folder to repo
* Reviewing SQL and prisma integraiton on fresh repo
* update inline key replacement
* ensure migration script executes and maps foreign_keys regardless of db ordering
* run migration endpoint
* support new prisma backend
* bump version
* change migration call
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
