anything-llm/server/storage
Timothy Carambat 61db981017
feat: Embed on-instance Whisper model for audio/mp4 transcribing (#449)
* feat: Embed on-instance Whisper model for audio/mp4 transcribing
resolves #329

* additional logging

* add placeholder for tmp folder in collector storage
Add cleanup of hotdir and tmp on collector boot to prevent hanging files
split loading of model and file conversion into concurrency

* update README

* update model size

* update supported filetypes
2023-12-15 11:20:13 -08:00
..
assets AnythingLLM UI overhaul (#278) 2023-10-23 13:10:34 -07:00
documents Docker support (#34) 2023-06-13 11:26:11 -07:00
models feat: Embed on-instance Whisper model for audio/mp4 transcribing (#449) 2023-12-15 11:20:13 -08:00
vector-cache Docker support (#34) 2023-06-13 11:26:11 -07:00
README.md Enable debug mode of ENV at runtime (#111) 2023-06-26 11:38:38 -07:00

AnythingLLM Storage

This folder is for the local or disk storage of ready-to-embed documents, vector-cached embeddings, and the disk-storage of LanceDB and the local SQLite database.

This folder should contain the following folders. documents lancedb (if using lancedb) vector-cache and a file named exactly anythingllm.db

Common issues

SQLITE_FILE_CANNOT_BE_OPENED in the server log = The DB file does not exist probably because the node instance does not have the correct permissions to write a file to the disk. To solve this..

  • Local dev

    • Create a anythingllm.db empty file in this directory. Thats all. No need to reboot the server or anything. If your permissions are correct this should not ever occur since the server will create the file if it does not exist automatically.
  • Docker Instance

    • Get your AnythingLLM docker container id with docker ps -a. The container must be running to execute the next commands.

    • Run docker container exec -u 0 -t <ANYTHINGLLM DOCKER CONTAINER ID> mkdir -p /app/server/storage /app/server/storage/documents /app/server/storage/vector-cache /app/server/storage/lancedb

    • Run docker container exec -u 0 -t <ANYTHINGLLM DOCKER CONTAINER ID> touch /app/server/storage/anythingllm.db

    • Run docker container exec -u 0 -t <ANYTHINGLLM DOCKER CONTAINER ID> chown -R anythingllm:anythingllm /app/collector /app/server

    • The above commands will create the appropriate folders inside of the docker container and will persist as long as you do not destroy the container and volume. This will also fix any ownership issues of folder files in the collector and the server.