anything-llm/collector/scripts/watch/main.py
Timothy Carambat c4eb46ca19
Upload and process documents via UI + document processor in docker image (#65)
* implement dnd uploader
show file upload progress
write files to hotdirector
build simple flaskAPI to process files one off

* move document processor calls to util
build out dockerfile to run both procs at the same time
update UI to check for document processor before upload
* disable pragma update on boot
* dockerfile changes

* add filetype restrictions based on python app support response and show rejected files in the UI

* cleanup

* stub migrations on boot to prevent exit condition

* update CF template for AWS deploy
2023-06-16 16:01:27 -07:00

22 lines
688 B
Python

import os
from .filetypes import FILETYPES
from .utils import move_source
RESERVED = ['__HOTDIR__.md']
def watch_for_changes(directory):
for raw_doc in os.listdir(directory):
if os.path.isdir(f"{directory}/{raw_doc}") or raw_doc in RESERVED: continue
filename, fileext = os.path.splitext(raw_doc)
if filename in ['.DS_Store'] or fileext == '': continue
if fileext not in FILETYPES.keys():
print(f"{fileext} not a supported file type for conversion. Removing from hot directory.")
move_source(new_destination_filename=raw_doc, failed=True)
continue
FILETYPES[fileext](
directory=directory,
filename=filename,
ext=fileext,
)