anything-llm/server/utils
Timothy Carambat dc4ad6b5a9
[BETA] Live document sync (#1719)
* wip bg workers for live document sync

* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it

* create frontend views and paths
Move elements to correct experimental scope

* update migration to delete runs on removal of watched document

* Add watch support to YouTube transcripts (#1716)

* Add watch support to YouTube transcripts
refactor how sync is done for supported types

* Watch specific files in Confluence space (#1718)

Add failure-prune check for runs

* create tmp workflow modifications for beta image

* create tmp workflow modifications for beta image

* create tmp workflow modifications for beta image

* dual build
update copy of alert modals

* update job interval

* Add support for live-sync of Github files

* update copy for document sync feature

* hide Experimental features from UI

* update docs links

* [FEAT] Implement new settings menu for experimental features (#1735)

* implement new settings menu for experimental features

* remove unused context save bar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* dont run job on boot

* unset workflow changes

* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs

* update jsDOC

* Linting and organization

* update modal copy for feature

---------

Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2024-06-21 13:38:50 -07:00
..
agents Add support for searXNG search for agents (#1733) 2024-06-20 14:08:00 -07:00
AiProviders [FEAT] Add support for Claude Sonnet 3.5 model (#1731) 2024-06-20 10:13:53 -07:00
BackgroundWorkers [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
boot [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
chats [FEAT] Persist query mode refusal responses as chat history (#1727) 2024-06-20 15:44:19 -07:00
collectorApi [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
comKey [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
database Full developer api (#221) 2023-08-23 19:15:07 -07:00
DocumentManager Enable ability to do full-text query on documents (#758) 2024-02-21 13:15:45 -08:00
EmbeddingEngines [FEAT] Add LiteLLM embedding provider support (#1579) 2024-06-06 12:43:34 -07:00
EncryptionManager [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
files [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
helpers [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
http Validate max_tokens is number (#1445) 2024-05-17 21:44:55 -07:00
middleware [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00
PasswordRecovery Strengthen field validations on user Updates (#1201) 2024-04-26 16:46:04 -07:00
prisma Add built-in embedding engine into AnythingLLM (#411) 2023-12-06 10:36:22 -08:00
telemetry Replace custom sqlite dbms with prisma (#239) 2023-09-28 14:00:03 -07:00
TextSplitter Improve RAG results via chunkHeader append (#1473) 2024-05-21 14:43:39 -05:00
TextToSpeech Add Speech-to-text and Text-to-speech providers (#1394) 2024-05-14 11:57:21 -07:00
vectorDbProviders [BETA] Live document sync (#1719) 2024-06-21 13:38:50 -07:00