* wip bg workers for live document sync
* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it
* create frontend views and paths
Move elements to correct experimental scope
* update migration to delete runs on removal of watched document
* Add watch support to YouTube transcripts (#1716)
* Add watch support to YouTube transcripts
refactor how sync is done for supported types
* Watch specific files in Confluence space (#1718)
Add failure-prune check for runs
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* create tmp workflow modifications for beta image
* dual build
update copy of alert modals
* update job interval
* Add support for live-sync of Github files
* update copy for document sync feature
* hide Experimental features from UI
* update docs links
* [FEAT] Implement new settings menu for experimental features (#1735)
* implement new settings menu for experimental features
* remove unused context save bar
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* dont run job on boot
* unset workflow changes
* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs
* update jsDOC
* Linting and organization
* update modal copy for feature
---------
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
* add support for voyageai embedder
* remove unneeded import
* linting
* Add ENV examples
Update how chunks are processed for Voyage
use correct langchain import
Add data handling
---------
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
* Support SQL Agent skill
* add MSSQL agent connector
* Add frontend to agent skills
remove FAKE_DB mock
reset skills to pickup child-skill dynamically
* add prompt examples for tools on untooled
* add better logging on SQL agents
* Wipe toolruns on each chat relay so tools can be used within the same session
* update comments
* WIP slash presets
* WIP slash command customization CRUD + validations complete
* backend slash command support
* fix permission setting on new slash commands
rework form submit and pattern on frontend
* Add field updates for hooks,
required=true to field
add user<>command constraint to keep them unique
enforce uniquness via teritary uid field on table for multi and non-multi user
* reset migration
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs
* add comments
* enable few-shot prompting per function for OSS models
* Add Agent support for Ollama models
* azure, groq, koboldcpp agent support complete + WIP togetherai
* WIP gemini agent support
* WIP gemini blocked and will not fix for now
* azure fix
* merge fix
* add localai agent support
* azure untooled agent support
* merge fix
* refactor implementation of several agent provideers
* update bad merge comment
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add text gen web ui LLM provider support
* update README
* README typo
* update TextWebUI display name
patch workspace<>model support for provider
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs
* add comments
* enable few-shot prompting per function for OSS models
* Add Agent support for Ollama models
* improve json parsing for ollama text responses
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs
* add comments
* enable few-shot prompting per function for OSS models
* getChatCompletion working WIP streaming
* WIP
* working streaming WIP abort stream
* implement cohere embedder support
* remove inputType option from cohere embedder
* fix cohere LLM from not aborting stream when canceled by user
* Patch Cohere implemention
* add cohere to onboarding
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* if document is pinned, do not give queryRefusalResponse message
* forgot embed.js patch
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP agent support
* move agent folder
* wip frontend socket
* checkpoint
* fix schema
* Checkpoint for plugins and AgentHandler
* refactor plugins and agent arch
* agent error reporting and handling
* add frontend elements for agents in prompt input
* WIP integrations for agents
* enable web-search agent config from frontend
* persist chat history
* update alert
* update migration
remove console logs
update close state for agent invocations
* add examples to dockerignore
Extract statusResponse to its own component
* update close method
* wrap scraping rejections
* add RAG search as funciton
* Add telem and link highlight
* chat support
* patch memory
* Add rechart as a plugin option
* Toggles for abilites of default agent (system wide)
Validate values for agent skills
Enable dynamic loading of skills
UI for toggle of skills
* add UI for toggle of configs for agent
* toggle WS or WSS protocol
* update NGNIX proxy pass
* move components around and capture failed websocket creation
* fix name
* tmp docker image
* reset workflow
* safety mark functions
* telem on tool calls
* remove hardcode short circuit
* separate web-browser from scrape
* extract summarizer to util
add abort handlers and controller for langchain stuff so socket close kills process
* langchain summarize verbose when in dev
* chart styling improvements + add title to chart
* fix legend from being cutoff in chart downloads
* remove cursor blink
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* WIP new login screen UI
* update prisma schema/create new models for pw recovery
* WIP password recovery backend
* WIP reset password flow
* WIP pw reset flow
* password reset logic complete & functional UI
* WIP login screen redesign for single and multi user
* create placeholder modal to display recovery codes
* implement UI for recovery code modals/download recovery codes
* multiuser desktop password reset UI/functionality complete
* support single user mode for pw reset
* mobile styles for all password reset/login flows complete
* lint
* remove single user password recovery
* create PasswordRecovery util file to make more readable
* do not drop-replace users table in migration
* review pr
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Add support for Gemini-1.5 Pro
bump @google/generative-ai pkg
Toggle apiVersion if beta model selected
resolves#1109
* update API messages due to package change
* Enable per-workspace provider/model combination
* cleanup
* remove resetWorkspaceChatModels and wipeWorkspaceModelPreference to prevent workspace from resetting model
* add space
---------
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
* [FEAT] Added port checks in updateENV.validDockerizedUrl to prevent docker from assuming it cannot access localhost URLs
* [CHORE] Updated error message to include Linux URL
* Patch port checking for general loopbacks
* typo
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* implement new version of anthropic sdk and support new models
* remove handleAnthropicStream and move to handleStream inside anthropic provider
* update useGetProvidersModels for new anthropic models
* WIP openrouter integration
* add OpenRouter options to onboarding flow and data handling
* add todo to fix headers for rankings
* OpenRouter LLM support complete
* Fix hanging response stream with OpenRouter
update tagline
update comment
* update timeout comment
* wait for first chunk to start timer
* sort OpenRouter models by organization
* uppercase first letter of organization
* sort grouped models by org
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add LLM support for perplexity
* update README & example env
* fix ENV keys in example env files
* slight changes for QA of perplexity support
* Update Perplexity AI name
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* update HOW_TO_USE_DOCKER to help with Ollama setup using docker
* update HOW_TO_USE_DOCKER
* styles update
* create separate README for ollama and link to it in HOW_TO_USE_DOCKER
* styling update
* Enable ability to do full-text query on documents
Show alert modal on first pin for client
Add ability to use pins in stream/chat/embed
* typo and copy update
* simplify spread of context and sources
* refactor stream/chat/embed-stram to be a single execution logic path so that it is easier to maintain and build upon
* no thread in sync chat since only api uses it
adjust import locations
* WIP RLHF works on historical messages
* refactor Actions component
* completed RLHF up and down votes for chats
* add defaults for HistoricalMessage params
* refactor RLHF implmenation
remove forwardRef on history items to prevent rerenders
* remove dup id
* Add rating to CSV output
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add more fields to csv export to make more useful
* refactor from review comments
* fix escapeCsv function
* catch export errors properly
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP event logging - new table for events and new settings view for viewing
* WIP add logging
* UI for log rows
* rename files to Logging to prevent getting gitignore
* add metadata for all logging events and colored badges in logs page
* remove unneeded comment
* cleanup namespace for logging
* clean up backend calls
* update logging to show to => from settings changes
* add logging for invitations, created, deleted, and accepted
* add logging for user created, updated, suspended, or removed
* add logging for workspace deleted
* add logging for chat logs exported
* add logging for API keys, LLM, embedder, vector db, embed chat, and reset button
* modify event logs
* update to event log types
* simplify rendering of event badges
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* WIP embedded app
* WIP got response from backend in embedded app
* WIP streaming prints to embedded app
* implemented streaming and tailwind min for styling into embedded app
* WIP embedded app history functional
* load params from script tag into embedded app
* rough in modularization of embed chat
cleanup dev process for easier dev support
move all chat to components
todo: build process
todo: backend support
* remove eslint config
* Implement models and cleanup embed chat endpoints
Improve build process for embed
prod minification and bundle size awareness
WIP
* forgot files
* rename to embed folder
* introduce chat modal styles
* add middleware validations on embed chat
* auto open param and default greeting
* reset chat history
* Admin embed config page
* Admin Embed Chats mgmt page
* update embed
* nonpriv
* more style support
reopen if chat was last opened
* update comments
* remove unused imports
* allow change of workspace for embedconfig
* update failure to lookup message
* update reset script
* update instructions
* Add more styling options
Add sponsor text at bottom
Support dynamic container height
Loading animations
* publish new embed script
* Add back syntax highlighting and keep bundle small via dynamic script build
* add hint
* update readme
* update copy model for snippet with link to styles
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* add support for new openai models
* QOL changes/improve logic for adding new openai embedding models
* add example file inputs for Openai embedding ENV selection;
* Fix if stmt conditional
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* feature: Integrate Astra as vectorDBProvider
feature: Integrate Astra as vectorDBProvider
* Update .env.example
* Add env.example to docker example file
Update spellcheck fo Astra
Update Astra key for vector selection
Update order of AstraDB options
Resize Astra logo image to 330x330
Update methods of Astra to take in latest vectorDB params like TopN and more
Update Astra interface to support default methods and avoid crash errors from 404 collections
Update Astra interface to comply to max chunk insertion limitations
Update Astra interface to dynamically set dimensionality from chunk 0 size on creation
* reset workspaces
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* migrate pinecone package to latest version and migrate pinecone vectordb provider class
* remove pinecone environment name env variable and update docs to reflect removal & serverless support complete
* migrate query for pinecone db
* typo in log
---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
* Implement total permission overhaul
Add explicit permissions on each flex and strict route
Patch issues with role escalation and CRUD of users
Patch permissions on all routes for coverage
Improve middleware to accept role array for clarity
* update comments
* remove permissions to API-keys for manager. Manager could generate API-key and using high-privelege api-key give themselves admin
* update sidebar permissions for multi-user and single user
* update options for mobile sidebar
