Commit Graph

263 Commits

Author SHA1 Message Date
Timothy Carambat
ecf4295537
Add ability to grab youtube transcripts via doc processor (#470)
* Add ability to grab youtube transcripts via doc processor

* dynamic imports
swap out Github for Youtube in placeholder text
2023-12-18 17:17:26 -08:00
Timothy Carambat
452582489e
GitHub loader extension + extension support v1 (#469)
* feat: implement github repo loading
fix: purge of folders
fix: rendering of sub-files

* noshow delete on custom-documents

* Add API key support because of rate limits

* WIP for frontend of data connectors

* wip

* Add frontend form for GitHub repo data connector

* remove console.logs
block custom-documents from being deleted

* remove _meta unused arg

* Add support for ignore pathing in request
Ignore path input via tagging

* Update hint
2023-12-18 15:48:02 -08:00
Timothy Carambat
2d700b13f6
fix: patch middle truncation func (#465)
resolves #456
2023-12-17 16:04:50 -08:00
Timothy Carambat
1c549e2844
fix: clarify custom truncate from npm truncate (#464)
chore: add @ where we can for less long imports
2023-12-17 15:58:49 -08:00
timothycarambat
2def9a67a1 docs: update default storage path to just write to user home dir for docker instructions 2023-12-17 15:48:56 -08:00
Ikko Eltociear Ashimine
4fe25cebe3
Update README.md (#454)
proprotional -> proportional
2023-12-17 10:44:27 -08:00
Timothy Carambat
65c7c0a518
fix: patch api key not persisting when setting LLM/Embedder (#458) 2023-12-16 10:21:36 -08:00
lunamidori5
ecfd146891
Update Dockerfile (added ffmpeg) (#451) 2023-12-15 19:35:38 -08:00
timothycarambat
d2e3506bb9 fix: transition on LLM and embedding screen
linting
2023-12-15 12:40:11 -08:00
Timothy Carambat
61db981017
feat: Embed on-instance Whisper model for audio/mp4 transcribing (#449)
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves #329

* additional logging

* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency

* update README

* update model size

* update supported filetypes
2023-12-15 11:20:13 -08:00
Timothy Carambat
719521c307
Document Processor v2 (#442)
* wip: init refactor of document processor to JS

* add NodeJs PDF support

* wip: partity with python processor
feat: add pptx support

* fix: forgot files

* Remove python scripts totally

* wip:update docker to boot new collector

* add package.json support

* update dockerfile for new build

* update gitignore and linting

* add more protections on file lookup

* update package.json

* test build

* update docker commands to use cap-add=SYS_ADMIN so web scraper can run
update all scripts to reflect this
remove docker build for branch
2023-12-14 15:14:56 -08:00
timothycarambat
5f6a013139 Change server bootup log 2023-12-14 13:52:11 -08:00
Timothy Carambat
1e98da07bc
docs: placeholder for model downloads folder (#446) 2023-12-14 10:31:14 -08:00
Timothy Carambat
d094cc3c10
docs: update deployment templates to just pull the docker image (#445)
docs: update deployment templates to just pull the docker image and boot that instead of building
2023-12-13 19:55:23 -08:00
Timothy Carambat
cc3343ba79
add Docker internal URL to READMEs (#441)
add Docker internal URL
2023-12-13 11:59:14 -08:00
Timothy Carambat
da0cec7aa2
patch: remove unidecode as it was transliterating non-latin chars (#434)
resolves #298
2023-12-13 11:54:55 -08:00
timothycarambat
b444171ef3 fix: Update copy code button on markdown to work with new highlightjs lib implementation 2023-12-12 16:25:22 -08:00
Timothy Carambat
37cdb845a4
patch: implement @lunamidori hotfix for LocalAI streaming chunk overflows (#433)
* patch: implement @lunamidori hotfix for LocalAI streaming chunk overflows
resolves #416

* change log to error log

* log trace

* lint
2023-12-12 16:20:06 -08:00
Timothy Carambat
d4f4d85492
patch: fix non-latin filenames being encoded improperly during upload and chat (#432)
patch: fix non-latin filenames being messed up during upload and chat
connect #169
resolves #427
2023-12-12 16:07:23 -08:00
timothycarambat
7154f4f82b non-blocking send responses 2023-12-12 13:41:17 -08:00
Timothy Carambat
a84333901a
feat: implement questionnaire during onboarding (optional) (#429)
fix: PFP url check
2023-12-12 13:11:32 -08:00
Timothy Carambat
ce9233c258
feat: enable HTML uploads from UI (#422)
resolves #418
2023-12-11 14:40:33 -08:00
Timothy Carambat
cba66150d7
patch: API key to localai service calls (#421)
connect #417
2023-12-11 14:18:28 -08:00
Ikko Eltociear Ashimine
fef5169ca4
chore: update README.md (#420)
standlone -> standalone
2023-12-11 11:32:26 -08:00
Timothy Carambat
8cc1455b72
feat: add support for variable chunk length (#415)
fix: cleanup code for embedding length clarify
resolves #388
2023-12-07 16:27:36 -08:00
timothycarambat
48dd99b028 chore: update docker boot script for storage location export 2023-12-07 15:11:48 -08:00
Timothy Carambat
655ebd9479
[Feature] AnythingLLM use locally hosted Llama.cpp and GGUF files for inferencing (#413)
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev

* Add native embedder as an available embedder selection

* wrap model loader in try/catch

* print progress on download

* add built-in LLM support (expiermental)

* Update to progress output for embedder

* move embedder selection options to component

* saftey checks for modelfile

* update ref

* Hide selection when on hosted subdomain

* update documentation
hide localLlama when on hosted

* saftey checks for storage of models

* update dockerfile to pre-build Llama.cpp bindings

* update lockfile

* add langchain doc comment

* remove extraneous --no-metal option

* Show data handling for private LLM

* persist model in memory for N+1 chats

* update import
update dev comment on token model size

* update primary README

* chore: more readme updates and remove screenshots - too much to maintain, just use the app!

* remove screeshot link
2023-12-07 14:48:27 -08:00
timothycarambat
fecfb0fafc chore: remove unused NO_DEBUG env 2023-12-07 14:14:30 -08:00
Sean Hatfield
fcb591d364
Add user PFP support and context to logo (#408)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change

* fix for empty chat

* change mobile alert copy

* WIP adding PFP upload support

* WIP pfp for users

* edit account menu complete with change username/password and upload profile picture

* add pfp context to update all instances of usePfp hook on update

* linting

* add context for logo change to immediately update logo

* fix div with bullet points to use list-disc instead

* fix: small changes

* update multer file storage locations

* fix: use STORAGE_DIR for filepathing

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-07 14:11:51 -08:00
Timothy Carambat
f48e6b1a3e
chore: add @ as alias for frontend root (#414)
* chore: add @ as alias for frontend root

* fix: remove bad tag
2023-12-07 09:09:01 -08:00
timothycarambat
33de34f8dc add embedding engine to telem 2023-12-07 08:53:37 -08:00
timothycarambat
ef79649891 improve ms-sse-fetch to not re-fetch automatially 2023-12-06 12:04:10 -08:00
timothycarambat
79cdb8631a fix: fix logo fetching raising errors in server 2023-12-06 11:56:07 -08:00
timothycarambat
ce52f54cda Documentation update 2023-12-06 11:38:40 -08:00
Timothy Carambat
1a6f599971
update docker build instructions (#412)
* update docker build instructions

* cleanup
2023-12-06 11:27:38 -08:00
Timothy Carambat
88cdd8c872
Add built-in embedding engine into AnythingLLM (#411)
* Implement use of native embedder (all-Mini-L6-v2)
stop showing prisma queries during dev

* Add native embedder as an available embedder selection

* wrap model loader in try/catch

* print progress on download

* Update to progress output for embedder

* move embedder selection options to component

* forgot import

* add Data privacy alert updates for local embedder
2023-12-06 10:36:22 -08:00
Sean Hatfield
48764d6370
Gear icon appear on hover for workspace (#410)
* gear icon appear on hover for workspace

* put back user role check for default

* wrap in callback

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-05 15:58:57 -08:00
pritchey
732d07829f
401-Password Complexity Check Capability (#402)
* Added improved password complexity checking capability.

* Move password complexity checker as User.util
dynamically import required libraries depending on code execution flow
lint

* Ensure persistence of password requirements on restarts via env-dump
Copy example schema to docker env as well

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-05 09:13:06 -08:00
Sean Hatfield
7b30dd041b
Add down arrow to scroll user to bottom of page (#400)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* add arrow to scroll to bottom (debounced) and fix chat scrolling to always scroll to very bottom on message history change

* fix for empty chat

* change mobile alert copy

* fix div with bullet points to use list-disc instead

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-04 13:27:07 -08:00
Sean Hatfield
064e0c3408
Fix modal sizes to fit smaller screens & bug fixes (#399)
* fix sizing of onboarding modals & lint

* fix extra scrolling on mobile onboarding flow

* added message to use desktop for onboarding

* linting

* change mobile alert copy

* fix div with bullet points to use list-disc instead

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-12-04 13:23:45 -08:00
Timothy Carambat
6fa8b0ce93
Add API key option to LocalAI (#407)
* Add API key option to LocalAI

* add api key for model dropdown selector
2023-12-04 08:38:15 -08:00
timothycarambat
203f5964d5 linting 2023-12-04 08:36:17 -08:00
Sean Hatfield
ca8bf52cce
Bug fixes (#394)
* fix black text for custom messages text

* fix file upload icon being stretched

* center github, docs, and discord icons in sidebar

* fix chat container being cut off on right side and tighter spacing between message

* fix default chat container being cut off on right side

* on create new workspace, take user to the workspace they just created instead of the home page

* add border to chat container and click outside user menu to close

* fix borders around all chat and settings containers to be consistent

* fix padding for default messages

* fix spacing between workspace items in sidebar

* fix margin around right side of chat, default, and settings containers to be the same as the left sidebar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-27 15:09:55 -06:00
Timothy Carambat
55d319b527
Rehash password for admin-user pwd updates (#398)
resolved #397
2023-11-27 12:47:07 -06:00
timothycarambat
29d3c603f2 update AnythingLLM promo images 2023-11-20 15:14:38 -08:00
timothycarambat
c6f0e4cdc4 update docker instructions 2023-11-17 20:15:11 -08:00
timothycarambat
17dfeae0da dockerfile default UID/GUID 2023-11-17 19:21:47 -08:00
timothycarambat
14c56a9bfc bump build workflow 2023-11-17 19:18:26 -08:00
timothycarambat
8176381513 build docker image and push on master merge 2023-11-17 19:15:23 -08:00
timothycarambat
b583aa74fd remove prints 2023-11-16 17:17:52 -08:00