Commit Graph

443 Commits

Author SHA1 Message Date
Sean Hatfield
3f78ef413b
[FEAT] Support for gemini-1.0-pro model and fixes to prompt window limit (#1557)
support for gemini-1.0-pro model and fixes to prompt window limit
2024-05-29 08:17:35 +08:00
Sean Hatfield
c24b79c9d1
[FEAT] Bing Search API web search provider (#1519)
implement bing search engine for agents
2024-05-23 16:49:30 -07:00
Sean Hatfield
6a2d7aca28
[FEAT] Custom login screen icon + custom app name (#1500)
* implement custom icon on login screen for single & multi user + custom app name feature

* hide field when not relevant

* set customApp name

* show original anythingllm login logo unless custom logo is set

* nit-picks

* remove console log

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-23 14:14:53 -07:00
timothycarambat
6e8a327d98 merge with master 2024-05-23 12:58:36 -07:00
Timothy Carambat
a89812703b
repatch path normalization (#1516) 2024-05-23 12:52:04 -07:00
Timothy Carambat
c7e9240801
Add chat embed HTML ids (#1514) 2024-05-23 10:54:07 -07:00
timothycarambat
318025baee Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2024-05-23 10:46:22 -07:00
timothycarambat
bfc9a96e87 patch: bug when disabling password protection being invalid even though it should be disabled 2024-05-23 10:46:12 -07:00
Timothy Carambat
2f9b785f42
Patch handling of end chunk stream events for OpenAI endpoints (#1487)
* Patch handling of end chunk stream events for OpenAI endpoints

* update LiteLLM to use generic handler

* update for empty choices
2024-05-23 10:20:40 -07:00
Timothy Carambat
13fb63930b
Improve RAG responses via source backfilling (#1477)
* Improve RAG responses via source backfilling

* Hide irrelevant citations from UI
2024-05-23 09:56:57 -07:00
Sean Hatfield
cc7e7fb3ac
[FEAT] Add support for gemini-1.5-flash-latest model (#1502)
* add support for gemini-1.5-flash-latest

* update comment in gemini LLM provider
2024-05-23 09:42:30 -07:00
timothycarambat
961630f7d5 fix example call param 2024-05-23 08:29:25 -07:00
Sean Hatfield
acbad2f2cf
[FEAT] Clear all chats button + export button styles updates (#1495)
* implement clear all chats button + export button styles updates

* Hide clear button if no chats are visible
update clear function

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-22 16:27:56 -05:00
timothycarambat
c6ad94d81a Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-22 13:43:09 -05:00
timothycarambat
7bace20769 Improve VoyageAI error responses and textChunk handler
resolves #1491
2024-05-22 13:42:48 -05:00
timothycarambat
3ef009de73 enfore min and max username lengths to prevent DOS via spam-length names 2024-05-22 13:21:26 -05:00
Timothy Carambat
c2d37ccce5
Limit return object of user when returned in some endpoints (#1492) 2024-05-22 12:32:39 -05:00
timothycarambat
e2439c6d4c patch endpoint without exception handler 2024-05-22 11:34:15 -05:00
timothycarambat
1cfba76bb8 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-22 10:54:06 -05:00
Timothy Carambat
8c5a30db9d
Support dynamic context length - VoyageAI (#1489) 2024-05-22 10:53:49 -05:00
Shixian Sheng
a256db132d
Fixed links (#1485)
* Update CHROMA_SETUP.md

* Update ASTRA_SETUP.md
2024-05-22 10:06:39 -05:00
timothycarambat
9f327d015a update error handling for OpenAI providers 2024-05-22 09:58:10 -05:00
Timothy Carambat
196c4c13c4
Fix Swagger docs (#1479) 2024-05-21 21:30:37 -05:00
timothycarambat
c65ab6d863 merge with master 2024-05-21 14:48:16 -05:00
Timothy Carambat
b23cb1a90f
Improve RAG results via chunkHeader append (#1473) 2024-05-21 14:43:39 -05:00
Timothy Carambat
28eba636e9
Allow setting of safety thresholds for Gemini (#1466)
* Allow setting of safety thresholds for Gemini

* linting
2024-05-20 13:17:00 -05:00
Sean Hatfield
5bf4b4db58
[FEAT] Add support for Voyage AI embedder (#1401)
* add support for voyageai embedder

* remove unneeded import

* linting

* Add ENV examples
Update how chunks are processed for Voyage
use correct langchain import
Add data handling

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-05-19 13:20:23 -05:00
yonathan suarez
ecd5d3cb8a
Add API endpoint for updating pin status (#1449) 2024-05-19 12:47:47 -05:00
Timothy Carambat
9ace0e67e6
Validate max_tokens is number (#1445) 2024-05-17 21:44:55 -07:00
Timothy Carambat
1a5aacb001
Support multi-model whispers (#1444) 2024-05-17 21:31:29 -07:00
timothycarambat
3794ef8dfd patch: use workspace LLM for embed chats and fallback to system
resolves #1439
2024-05-17 17:33:19 -07:00
Timothy Carambat
f140139534
Patch WSS upgrade for manual HTTPS certs (#1429)
* Patch WSS upgrade for manual HTTPS certs

* update comment

* refactor
2024-05-17 14:03:25 -07:00
Timothy Carambat
cae6cee1b5
Do not go through LLM to embed when embedding documents (#1428) 2024-05-16 17:51:04 -07:00
Timothy Carambat
01cf2fed17
Make native embedder the fallback for all LLMs (#1427) 2024-05-16 17:25:05 -07:00
Sean Hatfield
826ef00da3
[FEAT] LiteLLM provider support (#1424)
* litellm LLM provider support

* fix lint error

* change import orders
fix issue with model retrieval

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-05-16 13:56:28 -07:00
timothycarambat
7c6c272051 add logging on more api endpoints for exceptions 2024-05-16 12:56:19 -07:00
Timothy Carambat
15cf921616
Support SQL Agent skill (#1411)
* Support SQL Agent skill

* add MSSQL agent connector

* Add frontend to agent skills
remove FAKE_DB mock
reset skills to pickup child-skill dynamically

* add prompt examples for tools on untooled

* add better logging on SQL agents

* Wipe toolruns on each chat relay so tools can be used within the same session

* update comments
2024-05-16 10:38:21 -07:00
timothycarambat
72c9fda6c9 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-14 12:50:17 -07:00
Sean Hatfield
612a7e1662
[FEAT] Website depth scraping data connector (#1191)
* WIP website depth scraping, (sort of works)

* website depth data connector stable + add maxLinks option

* linting + loading small ui tweak

* refactor website depth data connector for stability, speed, & readability

* patch: remove console log
Guard clause on URL validitiy check
reasonable overrides

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-05-14 12:49:14 -07:00
Timothy Carambat
b6be43be95
Add Speech-to-text and Text-to-speech providers (#1394)
* Add Speech-to-text and Text-to-speech providers

* add files and update comment

* update comments

* patch: bad playerRef check
2024-05-14 11:57:21 -07:00
Timothy Carambat
64b62290d7
Set gpt-4o as default for OpenAI (#1391) 2024-05-13 14:31:49 -07:00
Sean Hatfield
9ed2309757
[FEAT] Add API key support for Oobabooga Web UI (#1354)
* add api key support for oobabooga web ui

* dont expose API Key for TextWebGenUi

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-13 12:58:16 -07:00
timothycarambat
a87978d1d9 Make LanceDB the vector database default provider in backend to prevent issues where somehow this key is not set by the user resulting in a Pinecone error even though they never said they wanted Pinecone to be their vector db 2024-05-13 12:22:53 -07:00
timothycarambat
790fd58b97 no AuthToken unset for enable multi-user 2024-05-13 08:42:02 -07:00
Sean Hatfield
948ac8a3dd
[FIX] Validate messages schema for gemini provider (#1351)
validate messages schema for gemini provider
2024-05-10 17:33:25 -07:00
Timothy Carambat
7b18a36288
prevent accidental lockout from restricted chars in single pass mode (#1352)
* prevent accidental lockout from restrict chars in single pass mode

* update error message
2024-05-10 17:29:49 -07:00
Timothy Carambat
98953d70b7
Fix typo leading to agent history in threads not being unique in same workspace (#1350) 2024-05-10 16:09:34 -07:00
timothycarambat
6c1b5477e0 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-10 15:03:10 -07:00
Sean Hatfield
0a6a9e40c1
[FIX] Add max tokens field to generic OpenAI LLM connector (#1345)
* add max tokens field to generic openai llm connector

* add max_tokens property to generic openai agent provider
2024-05-10 14:49:02 -07:00
Sean Hatfield
734c5a9e96
[FEAT] Implement regenerate response button (#1341)
* implement regenerate response button

* wip on rerenders

* remove function that was duplicate

* update delete-chats function

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-10 14:47:29 -07:00
Sean Hatfield
d36c3ff8b2
[FEAT] Slash templates (#1314)
* WIP slash presets

* WIP slash command customization CRUD + validations complete

* backend slash command support

* fix permission setting on new slash commands
rework form submit and pattern on frontend

* Add field updates for hooks,
required=true to field
add user<>command constraint to keep them unique
enforce uniquness via teritary uid field on table for multi and non-multi user

* reset migration

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-10 12:35:33 -07:00
Timothy Carambat
81bc16cc39
More agent providers (#1316)
* add OpenRouter support

* add mistral agents
add perplexity agents
add textwebgenui agents
2024-05-08 16:04:18 -07:00
Sean Hatfield
8422f92542
Agent support for LLMs with no function calling (#1295)
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models

* Add Agent support for Ollama models

* azure, groq, koboldcpp agent support complete + WIP togetherai

* WIP gemini agent support

* WIP gemini blocked and will not fix for now

* azure fix

* merge fix

* add localai agent support

* azure untooled agent support

* merge fix

* refactor implementation of several agent provideers

* update bad merge comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-08 15:17:54 -07:00
timothycarambat
8aba28b02e Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-08 14:12:33 -07:00
Sean Hatfield
977a07db86
[FEAT] Text Generation Web UI LLM provider support (#1279)
* add text gen web ui LLM provider support

* update README

* README typo

* update TextWebUI display name
patch workspace<>model support for provider

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-08 11:56:30 -07:00
Imtiaz Mehmood
ad778dd36d
Feat/quick delete chat (#1302)
* feat:quick delete chat thread

* update:pull request template

* refactor bulk-deletion implementation

* unset pull_request_changes

* add border none for desktop support

* unset marks when toggling bulk mode

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-08 11:10:00 -07:00
Timothy Carambat
331d3741c9
Ollama agents (#1270)
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models

* Add Agent support for Ollama models

* improve json parsing for ollama text responses
2024-05-07 18:06:31 -07:00
Timothy Carambat
1b4559f57f
Add LMStudio agent support (generic) support (#1246)
* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models
2024-05-07 16:35:47 -07:00
timothycarambat
c60077a078 merge with master 2024-05-03 10:02:53 -07:00
Sean Hatfield
fc77b46800
[FEAT] KoboldCPP LLM Support (#1268)
* koboldcpp LLM support

* update .env.examples for koboldcpp support

* update LLM preference order
update koboldcpp comments

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-02 12:12:44 -07:00
Sean Hatfield
3caebc47b4
[FEAT] Cohere LLM and embedder support (#1233)
* getChatCompletion working WIP streaming

* WIP

* working streaming WIP abort stream

* implement cohere embedder support

* remove inputType option from cohere embedder

* fix cohere LLM from not aborting stream when canceled by user

* Patch Cohere implemention

* add cohere to onboarding

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-02 10:35:50 -07:00
Sean Hatfield
d02013fd71
[FIX] Document pinning does not count in query mode (#1250)
* if document is pinned, do not give queryRefusalResponse message

* forgot embed.js patch

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-02 10:27:09 -07:00
Timothy Carambat
244ce2e307
Prevent concurrent downloads on first-doc upload (#1267) 2024-05-02 10:15:11 -07:00
Timothy Carambat
47b7df4fc3
Clear chat window on /reset (#1261)
clear chat window on /reset
2024-05-01 20:39:58 -07:00
Sean Hatfield
9feaad79cc
[CHORE] Remove sendChat and streamChat in all LLM providers (#1260)
* remove sendChat and streamChat functions/references in all LLM providers

* remove unused imports

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-05-01 16:52:28 -07:00
timothycarambat
b69bf7cc27 Clearing of events does not reload anymore
updating workspace name does not result in reload anymore
set event log page size to 10
fix css issues with charts
2024-05-01 16:13:20 -07:00
timothycarambat
6150ff41ea Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-05-01 13:33:07 -07:00
Timothy Carambat
894f727903
Remove restrictions on pinned documents to use more context (#1248)
* Remove restrictions on pinned documents to use more contet

* update comment
2024-05-01 13:32:52 -07:00
Timothy Carambat
e61dfd80a5
Prevent i-framing of frontend UI to prevent unsafe embedding and/or clickjacking (#1200)
Prevent iframing of frontend UI to prevent unsafe embedding and/or clickjacking
2024-05-01 13:02:08 -07:00
Timothy Carambat
42e1d8e8ce
Customize refusal response for query mode (#1243)
* Customize refusal response for `query` mode

* remove border for desktop
2024-04-30 16:14:30 -07:00
Timothy Carambat
547d4859ef
Bump openai package to latest (#1234)
* Bump `openai` package to latest
Tested all except localai

* bump LocalAI support with latest image

* add deprecation notice

* linting
2024-04-30 12:33:42 -07:00
Timothy Carambat
94017e2b51
bump langchain deps (#1231)
* bump langchain deps

* patch native and ollama providers remove deprecated deps

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2024-04-30 12:04:24 -07:00
Timothy Carambat
bf435b2861
Adjust how text is split depending on input type (#1238)
resolves #1230
2024-04-30 10:11:56 -07:00
Timothy Carambat
ca63012c0f
bump lancedb dep (#1229) 2024-04-29 09:52:22 -07:00
timothycarambat
5ae20d50ea patch agent invocation regression
resolves #1211
2024-04-27 12:39:45 -07:00
Timothy Carambat
1b35bcbeab
Strengthen field validations on user Updates (#1201)
* Strengthen field validations on user Updates

* update writables
2024-04-26 16:46:04 -07:00
timothycarambat
df2c01b176 patch OpenRouter model fetcher when key is not present 2024-04-26 15:58:30 -07:00
Timothy Carambat
2e813846dc
Agent skill: chart generation (#1103)
* WIP agent support

* move agent folder

* wip frontend socket

* checkpoint

* fix schema

* Checkpoint for plugins and AgentHandler

* refactor plugins and agent arch

* agent error reporting and handling

* add frontend elements for agents in prompt input

* WIP integrations for agents

* enable web-search agent config from frontend

* persist chat history

* update alert

* update migration
remove console logs
update close state for agent invocations

* add examples to dockerignore
Extract statusResponse to its own component

* update close method

* wrap scraping rejections

* add RAG search as funciton

* Add telem and link highlight

* chat support

* patch memory

* Add rechart as a plugin option

* Toggles for abilites of default agent (system wide)
Validate values for agent skills
Enable dynamic loading of skills
UI for toggle of skills

* add UI for toggle of configs for agent

* toggle WS or WSS protocol

* update NGNIX proxy pass

* move components around and capture failed websocket creation

* fix name

* tmp docker image

* reset workflow

* safety mark functions

* telem on tool calls

* remove hardcode short circuit

* separate web-browser from scrape

* extract summarizer to util
add abort handlers and controller for langchain stuff so socket close kills process

* langchain summarize verbose when in dev

* chart styling improvements + add title to chart

* fix legend from being cutoff in chart downloads

* remove cursor blink

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2024-04-26 11:18:55 -07:00
Sean Hatfield
348b36bf85
[FEAT] Confluence data connector (#1181)
* WIP Confluence data connector backend

* confluence data connector complete

* confluence citations

* fix citation for confluence

* Patch confulence integration

* fix Citation Icon for confluence

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-04-25 17:53:38 -07:00
Sean Hatfield
11f6419c3c
[FEAT] Implement new login screen UI & multi-user password reset (#1074)
* WIP new login screen UI

* update prisma schema/create new models for pw recovery

* WIP password recovery backend

* WIP reset password flow

* WIP pw reset flow

* password reset logic complete & functional UI

* WIP login screen redesign for single and multi user

* create placeholder modal to display recovery codes

* implement UI for recovery code modals/download recovery codes

* multiuser desktop password reset UI/functionality complete

* support single user mode for pw reset

* mobile styles for all password reset/login flows complete

* lint

* remove single user password recovery

* create PasswordRecovery util file to make more readable

* do not drop-replace users table in migration

* review pr

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-04-25 16:52:30 -07:00
timothycarambat
d25933930b merge with master 2024-04-25 07:35:55 -07:00
timothycarambat
dfaaf1680f update perplexity models
resolves #1188
2024-04-25 07:34:28 -07:00
Timothy Carambat
df17fbda36
Add generic OpenAI endpoint support (#1178)
* Add generic OpenAI endpoint support

* allow any input for model in case provider does not support models endpoint
2024-04-23 13:06:07 -07:00
Timothy Carambat
ac6ca13f60
1173 dynamic cache openrouter (#1176)
* patch agent invocation rule

* Add dynamic model cache from OpenRouter API for context length and available models
2024-04-23 11:10:54 -07:00
Timothy Carambat
9c00db7d3b
patch agent invocation rule (#1175) 2024-04-23 09:59:00 -07:00
Sean Hatfield
897e168fd1
[FEAT] Add support for more groq models (Llama 3 and Gemma) (#1143)
add support for more groq models
2024-04-22 13:14:27 -07:00
timothycarambat
37f4db36a6 resolves #1156
Fixed prisma command response on api workspace update-embeddings
2024-04-21 11:45:34 -07:00
timothycarambat
e1372a81d4 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-04-20 18:22:41 -07:00
Timothy Carambat
c65f890afc
Add LMStudio embedding endpoint support (#1141)
* Add LMStudio embedding endpoint support

* update alive path check for HEAD
remove commented JSX

* update comment
2024-04-19 15:36:07 -07:00
Timothy Carambat
b3c7c002db
Warn user when changing embedder with current progress already (#1135)
* Warn user when changing embedder with current progress already

* update comment
2024-04-19 09:51:58 -07:00
Timothy Carambat
58b744771f
Add support for Gemini-1.5 Pro (#1134)
* Add support for Gemini-1.5 Pro
bump @google/generative-ai pkg
Toggle apiVersion if beta model selected
resolves #1109

* update API messages due to package change
2024-04-19 08:59:46 -07:00
timothycarambat
e28c0469f4 bump togetherai models Apr 18, 2024
resolves #1126
2024-04-18 16:28:43 -07:00
timothycarambat
bd5b3f5d00 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-04-18 14:54:05 -07:00
Timothy Carambat
91ea37d17a
Patch model selection for openai (#1128)
prevent rerenders of web-browsing for agents
update openai latest pkg name
2024-04-18 13:32:50 -07:00
Timothy Carambat
9655880cf0
Update all vector dbs to filter duplicate source documents that may be pinned (#1122)
* Update all vector dbs to filter duplicate parents

* cleanup
2024-04-17 18:04:39 -07:00
Timothy Carambat
41978765bc
Improve document summarizer by being able to reference by direct name versus ID, which sometimes models fail to chain (#1121) 2024-04-17 14:34:58 -07:00
Timothy Carambat
81fd82e133
model specific summarization (#1119)
* model specific summarization

* update guard functions

* patch model picker and key inputs
2024-04-17 14:04:51 -07:00
timothycarambat
45505630a6 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into render 2024-04-17 11:55:57 -07:00
Timothy Carambat
9449fcd737
Add Anthropic agent support with new API and tool_calling (#1116)
* Add Anthropic agent support with new API and tool_calling

* patch useProviderHook to unset default models on provider change
2024-04-17 11:54:58 -07:00
Timothy Carambat
8ebe1a515b
Gracefully handle bad agent auth (#1115)
use provider that is set
2024-04-16 16:42:06 -07:00