* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files
* noshow delete on custom-documents
* Add API key support because of rate limits
* WIP for frontend of data connectors
* wip
* Add frontend form for GitHub repo data connector
* remove console.logs
block custom-documents from being deleted
* remove _meta unused arg
* Add support for ignore pathing in request
Ignore path input via tagging
* Update hint
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves#329
* additional logging
* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency
* update README
* update model size
* update supported filetypes
* wip: init refactor of document processor to JS
* add NodeJs PDF support
* wip: partity with python processor
feat: add pptx support
* fix: forgot files
* Remove python scripts totally
* wip:update docker to boot new collector
* add package.json support
* update dockerfile for new build
* update gitignore and linting
* add more protections on file lookup
* update package.json
* test build
* update docker commands to use cap-add=SYS_ADMIN so web scraper can run
update all scripts to reflect this
remove docker build for branch
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev
* Add native embedder as an available embedder selection
* wrap model loader in try/catch
* print progress on download
* add built-in LLM support (expiermental)
* Update to progress output for embedder
* move embedder selection options to component
* saftey checks for modelfile
* update ref
* Hide selection when on hosted subdomain
* update documentation
hide localLlama when on hosted
* saftey checks for storage of models
* update dockerfile to pre-build Llama.cpp bindings
* update lockfile
* add langchain doc comment
* remove extraneous --no-metal option
* Show data handling for private LLM
* persist model in memory for N+1 chats
* update import
update dev comment on token model size
* update primary README
* chore: more readme updates and remove screenshots - too much to maintain, just use the app!
* remove screeshot link
* fix sizing of onboarding modals & lint
* fix extra scrolling on mobile onboarding flow
* added message to use desktop for onboarding
* linting
* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change
* fix for empty chat
* change mobile alert copy
* WIP adding PFP upload support
* WIP pfp for users
* edit account menu complete with change username/password and upload profile picture
* add pfp context to update all instances of usePfp hook on update
* linting
* add context for logo change to immediately update logo
* fix div with bullet points to use list-disc instead
* fix: small changes
* update multer file storage locations
* fix: use STORAGE_DIR for filepathing
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev
* Add native embedder as an available embedder selection
* wrap model loader in try/catch
* print progress on download
* Update to progress output for embedder
* move embedder selection options to component
* forgot import
* add Data privacy alert updates for local embedder
* allow use of any embedder for any llm/update data handling modal
* Apply embedder override and fallback to OpenAI and Azure models
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feature: add LocalAI as llm provider
* update Onboarding/mgmt settings
Grab models from models endpoint for localai
merge with master
* update streaming for complete chunk streaming
update localAI LLM to be able to stream
* force schema on URL
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: tlandenberger <tobiaslandenberger@gmail.com>
* added manager role to options
* block default role from editing workspace settings on workspace and text input box
* block default user from accessing settings at all
* create manager route
* let pass through if in single user mode
* fix permissions for manager and admin roles in settings
* fix settings button for single user and remove unneeded console.logs
* rename routes and paths for clarity
* admin, manager, default roles complete
* remove unneeded comments
* consistency changes
* manage permissions for mum modes
* update sidebar for single-user mode
* update comment on middleware
Modify permission setting for admins
* update render conditional
* Add role usage hint to each role
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Using OpenAI API locally
* Infinite prompt input and compression implementation (#332)
* WIP on continuous prompt window summary
* wip
* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff
* Implement compression for Anythropic
Fix lancedb sources
* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources
* Resolve Weaviate citation sources not working with schema
* comment cleanup
* disable import on hosted instances (#339)
* disable import on hosted instances
* Update UI on disabled import/export
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add support for gpt-4-turbo 128K model (#340)
resolves#336
Add support for gpt-4-turbo 128K model
* 315 show citations based on relevancy score (#316)
* settings for similarity score threshold and prisma schema updated
* prisma schema migration for adding similarityScore setting
* WIP
* Min score default change
* added similarityThreshold checking for all vectordb providers
* linting
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* rename localai to lmstudio
* forgot files that were renamed
* normalize model interface
* add model and context window limits
* update LMStudio tagline
* Fully working LMStudio integration
---------
Co-authored-by: Francisco Bischoff <984592+franzbischoff@users.noreply.github.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* added JSONL export to workspace chats
* change permissions for workspace chat settings
* change permissions for workspace chat settings
* Show error for correct limit on fine-tune
Change sidebar position and permission
Remove check for MUM
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP on continuous prompt window summary
* wip
* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff
* Implement compression for Anythropic
Fix lancedb sources
* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources
* Resolve Weaviate citation sources not working with schema
* comment cleanup
* WIP Anythropic support for chat, chat and query w/context
* Add onboarding support for Anthropic
* cleanup
* fix Anthropic answer parsing
move embedding selector to general util
* include score value in similarityResponse for weaviate
* include score value in si
milarityResponse for qdrant
* include score value in si
milarityResponse for pinecone
* include score value in similarityResponse for chroma
* include score value in similarityResponse for lancedb
* distance to similarity
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Limit is due to POST body max size. Sufficiently large requests will abort automatically
We should report that error back on the frontend during embedding
Update vectordb providers to return on failed
* WIP converted all sqlite models into prisma calls
* modify db setup and fix ApiKey model calls in admin.js
* renaming function params to be consistent
* converted adminEndpoints to utilize prisma orm
* converted chatEndpoints to utilize prisma orm
* converted inviteEndpoints to utilize prisma orm
* converted systemEndpoints to utilize prisma orm
* converted workspaceEndpoints to utilize prisma orm
* converting sql queries to prisma calls
* fixed default param bug for orderBy and limit
* fixed typo for workspace chats
* fixed order of deletion to account for sql relations
* fix invite CRUD and workspace management CRUD
* fixed CRUD for api keys
* created prisma setup scripts/docs for understanding how to use prisma
* prisma dependency change
* removing unneeded console.logs
* removing unneeded sql escape function
* linting and creating migration script
* migration from depreciated sqlite script update
* removing unneeded migrations in prisma folder
* create backup of old sqlite db and use transactions to ensure all operations complete successfully
* adding migrations to gitignore
* updated PRISMA.md docs for info on how to use sqlite migration script
* comment changes
* adding back migrations folder to repo
* Reviewing SQL and prisma integraiton on fresh repo
* update inline key replacement
* ensure migration script executes and maps foreign_keys regardless of db ordering
* run migration endpoint
* support new prisma backend
* bump version
* change migration call
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Autodocument Swagger API with JSDocs on /v1/ endpoints for API access
implement single-player API keys
WIP Admin API Keys
* Create new api keys as both single and multi-user
* Add boot and telem
* Complete Admin API
* Complete endpoints
dark mode swagger
* update docs
* undo debug
* update docs and readme
* added ui for custom welcome messages and added label for custom logo in admin settings
* linting
* fixing img to use light/dark modes
* converted ChatBubble into component
* implemented backend for welcome messages and admin appearance page
* completed custom welcome messages for admin
* finished custom messages for single user mode
* merged with master and linted
* improved UI for appearance settings pages
* linted and merged with master
* small updates
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* implemented logo customization for single-user mode
* removing unneeded comments
* added dark and light mode support for default logo
* implemented dark and light mode switching in frontend
* fixed dark and light mode switching for failed to load logo from backend
* removed unneeded comment
* custom logos for admin implemented
* refactor logo mgmt functions
abstract logo management utils into their own file for simplicity
* added settings tab for appearance on single-user mode
* unchecking files with unneeded changes
* fixed appearance settings tab to be hidden on multiuser mode
* allow readall for logo
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Remove LangchainJS for chat support chaining
Implement runtime LLM selection
Implement AzureOpenAI Support for LLM + Emebedding
WIP on frontend
Update env to reflect the new fields
* Remove LangchainJS for chat support chaining
Implement runtime LLM selection
Implement AzureOpenAI Support for LLM + Emebedding
WIP on frontend
Update env to reflect the new fields
* Replace keys with LLM Selection in settings modal
Enforce checks for new ENVs depending on LLM selection
* improve citations to show all text chunks referred and expand the citation to view full referenced text
chunk text of same document together
* remove debug
* multi user wip
* WIP MUM features
* invitation mgmt
* suspend or unsuspend users
* workspace mangement
* manage chats
* manage chats
* add Support for admin system settings for users to delete workspaces and limit chats per user
* fix issue ith system var
update app to lazy load invite page
* cleanup and bug fixes
* wrong method
* update readme
* update readme
* update readme
* bump version to 0.1.0
* refactor: convert chunk embedding to one API call
* chore: lint
* fix chroma for batch and single vectorization of text
* Fix LanceDB multi and single vectorization
* Fix pinecone for single and multiple embeddings
---------
Co-authored-by: Jonathan Waltz <volcanicislander@gmail.com>
* Related to Issue #122, Implemented custom prompt in workspace settings.
* run linter
* Remove code duplication for chat prompt injection
---------
Co-authored-by: Francisco Bischoff <franzbischoff@gmail.com>
* Add chat/converstaion mode as the default chat mode
Show menu for toggling options for chat/query/reset command
Show chat status below input
resolves#61
* remove console logs
* implement dnd uploader
show file upload progress
write files to hotdirector
build simple flaskAPI to process files one off
* move document processor calls to util
build out dockerfile to run both procs at the same time
update UI to check for document processor before upload
* disable pragma update on boot
* dockerfile changes
* add filetype restrictions based on python app support response and show rejected files in the UI
* cleanup
* stub migrations on boot to prevent exit condition
* update CF template for AWS deploy
* 1. Define LLM Temperature as a workspace setting
2. Implement rudimentry table migration code for both new and existing repos to bring tables up to date
3. Trigger for workspace on update to update timestamp
4. Always fallback temp to 0.7
5. Extract WorkspaceModal into Tabbed content
6. Remove workspace name UNIQUE constraint (cannot be migrated :()
7. Add slug +seed when existing slug is already take
8. Seperate name from slug so display names can be changed
* remove blocking test return
* Updates for Linux for frontend/server
* frontend/server docker
* updated Dockerfile for deps related to node vectordb
* updates for collector in docker
* docker deps for ODT processing
* ignore another collector dir
* storage mount improvements; run as UID
* fix pypandoc version typo
* permissions fixes
* add start of lanceDB support
* lancedb initial support
* add null method for deletion of documents from namespace since LanceDB does not support
show warning modal on frontend for this
* update .env.example and lancedb methods for sourcing
* change export method
* update readme
