2022-07-22 17:20:12 +02:00
< h1 align = "center" > Lama Cleaner< / h1 >
< p align = "center" > A free and open-source inpainting tool powered by SOTA AI model.< / p >
< p align = "center" >
< a href = "https://github.com/Sanster/lama-cleaner" >
< img alt = "total download" src = "https://pepy.tech/badge/lama-cleaner" / >
< / a >
< a href = "https://pypi.org/project/lama-cleaner/" >
< img alt = "version" src = "https://img.shields.io/pypi/v/lama-cleaner" / >
< / a >
< a href = "https://colab.research.google.com/drive/1e3ZkAJxvkK3uzaTGu91N9TvI_Mahs0Wb?usp=sharing" >
< img alt = "Open in Colab" src = "https://colab.research.google.com/assets/colab-badge.svg" / >
< / a >
2022-11-16 14:18:51 +01:00
< a href = "https://huggingface.co/spaces/Sanster/Lama-Cleaner-lama" >
< img alt = "Hugging Face Spaces" src = "https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue" / >
< / a >
2022-10-09 15:32:13 +02:00
< a href = "https://www.python.org/downloads/" >
< img alt = "python version" src = "https://img.shields.io/badge/python-3.7+-blue.svg" / >
< / a >
2022-10-08 06:55:51 +02:00
< a href = "https://hub.docker.com/r/cwq1913/lama-cleaner" >
< img alt = "version" src = "https://img.shields.io/docker/pulls/cwq1913/lama-cleaner" / >
< / a >
2022-07-22 17:20:12 +02:00
< / p >
2022-10-20 16:36:32 +02:00
https://user-images.githubusercontent.com/3998421/196976498-ba1ad3ab-fa18-4c55-965f-5c6683141375.mp4
2022-07-22 17:20:12 +02:00
## Features
- Completely free and open-source
- Fully self-hosted
2022-09-26 03:16:11 +02:00
- Classical image inpainting algorithm powered by [cv2 ](https://docs.opencv.org/3.4/df/d3d/tutorial_py_inpainting.html )
2022-07-22 17:20:12 +02:00
- Multiple SOTA AI models
2022-03-24 14:36:47 +01:00
1. [LaMa ](https://github.com/saic-mdal/lama )
1. [LDM ](https://github.com/CompVis/latent-diffusion )
2022-07-22 17:20:12 +02:00
1. [ZITS ](https://github.com/DQiaole/ZITS_inpainting )
2022-08-22 17:24:02 +02:00
1. [MAT ](https://github.com/fenglinglwb/MAT )
2022-09-04 15:23:58 +02:00
1. [FcF ](https://github.com/SHI-Labs/FcF-Inpainting )
2022-10-20 15:03:43 +02:00
1. [SD1.5 ](https://github.com/runwayml/stable-diffusion )
2022-07-22 17:20:12 +02:00
- Support CPU & GPU
2022-09-04 15:23:58 +02:00
- Various inpainting [strategy ](#inpainting-strategy )
2022-07-22 17:20:12 +02:00
- Run as a desktop APP
## Usage
2022-10-05 17:15:09 +02:00
< details >
< summary > 1. Remove any unwanted things on the image< / summary >
| Usage | Before | After |
| ---------------------- | --------------------------------------------- | --------------------------------------------------- |
| Remove unwanted things |  |  |
| Remove unwanted person |  |  |
| Remove Text |  |  |
| Remove watermark |  |  |
< / details >
< details >
< summary > 2. Fix old photo< / summary >
| Usage | Before | After |
| ------------- | ----------------------------------- | ----------------------------------------------- |
| Fix old photo |  |  |
< / details >
< details >
< summary > 3. Replace something on the image < / summary >
| Usage | Before | After |
| ---------------------- | ------------------------ | -------------------------------------------------------------- |
| Text Driven Inpainting |  | Prompt: a fox sitting on a bench< br />  |
< / details >
2021-12-12 07:54:37 +01:00
2022-07-22 17:20:12 +02:00
## Quick Start
2021-11-15 08:22:34 +01:00
2022-10-05 17:15:09 +02:00
The easiest way to use Lama Cleaner is to install it using `pip` :
2022-04-18 16:40:23 +02:00
```bash
pip install lama-cleaner
2022-03-04 06:44:53 +01:00
2022-10-05 17:15:09 +02:00
# Models will be downloaded at first time used
2022-07-22 17:20:12 +02:00
lama-cleaner --model=lama --device=cpu --port=8080
# Lama Cleaner is now running at http://localhost:8080
2022-04-18 16:40:23 +02:00
```
2022-10-21 03:05:58 +02:00
For stable-diffusion model, you need to [accepting the terms to access ](https://huggingface.co/runwayml/stable-diffusion-inpainting ), and
get an access token from here [huggingface access token ](https://huggingface.co/docs/hub/security-tokens )
2022-10-05 17:15:09 +02:00
If you prefer to use docker, you can check out [docker ](#docker )
2022-10-24 12:32:35 +02:00
If you hava no idea what is docker or pip, please check [One Click Installer ](./scripts/README.md )
2022-10-05 17:15:09 +02:00
Available command line arguments:
2022-03-04 06:44:53 +01:00
2022-10-24 12:32:35 +02:00
| Name | Description | Default |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- | -------- |
| --model | lama/ldm/zits/mat/fcf/sd1.5 See details in [Inpaint Model ](#inpainting-model ) | lama |
2022-10-21 03:05:58 +02:00
| --hf_access_token | stable-diffusion need [huggingface access token ](https://huggingface.co/docs/hub/security-tokens ) to download model | |
2022-10-24 12:32:35 +02:00
| --sd-run-local | Once the model as downloaded, you can pass this arg and remove `--hf_access_token` | |
| --sd-disable-nsfw | Disable stable-diffusion NSFW checker. | |
| --sd-cpu-textencoder | Always run stable-diffusion TextEncoder model on CPU. | |
| --device | cuda or cpu | cuda |
| --port | Port for backend flask web server | 8080 |
| --gui | Launch lama-cleaner as a desktop application | |
| --gui_size | Set the window size for the application | 1200 900 |
| --input | Path to image you want to load by default | None |
| --debug | Enable debug mode for flask web server | |
2022-03-04 06:44:53 +01:00
2022-07-22 17:20:12 +02:00
## Inpainting Model
2022-06-12 16:06:29 +02:00
2022-09-26 03:16:11 +02:00
| Model | Description | Config |
| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cv2 | :+1: No GPU is required, and for simple backgrounds, the results may even be better than AI models. | |
| LaMa | :+1: Generalizes well on high resolutions(~2k)< br / > | |
2022-09-19 10:06:37 +02:00
| LDM | :+1: Possible to get better and more detail result < br /> :+1: The balance of time and quality can be achieved by adjusting `steps` < br /> :neutral_face: Slower than GAN model< br /> :neutral_face: Need more GPU memory | `Steps` : You can get better result with large steps, but it will be more time-consuming < br /> `Sampler` : ddim or [plms ](https://arxiv.org/abs/2202.09778 ). In general plms can get [better results ](https://github.com/Sanster/lama-cleaner/releases/tag/0.13.0 ) with fewer steps |
2022-09-26 03:16:11 +02:00
| ZITS | :+1: Better holistic structures compared with previous methods < br /> :neutral_face: Wireframe module is **very** slow on CPU | `Wireframe` : Enable edge and line detect |
| MAT | TODO | |
| FcF | :+1: Better structure and texture generation < br / > :neutral_face: Only support fixed size (512x512) input | |
2022-10-20 15:03:43 +02:00
| SD1.5 | :+1: SOTA text-to-image diffusion model | |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
< details >
< summary > See model comparison detail< / summary >
**LaMa vs LDM**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | LaMa | LDM |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs ZITS**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | ZITS | LaMa |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
Image is from [ZITS ](https://github.com/DQiaole/ZITS_inpainting ) paper. I didn't find a good example to show the advantages of ZITS and let me know if you have a good example. There can also be possible problems with my code, if you find them, please let me know too!
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs FcF**
2022-06-12 16:06:29 +02:00
2022-09-04 15:23:58 +02:00
| Original Image | Lama | FcF |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
< / details >
2022-09-04 15:23:58 +02:00
## Inpainting Strategy
Lama Cleaner provides three ways to run inpainting model on images, you can change it in the settings dialog.
2022-10-08 16:15:21 +02:00
| Strategy | Description | VRAM | Speed |
| ------------ | ------------------------------------------------------------------------------------------------------ | ------ | ----------------- |
| **Original** | Use the resolution of the original image | High | :zap: |
| **Resize** | Resize the image to a smaller size before inpainting. The area outside the mask will not loss quality. | Midium | :zap: :zap: |
| **Crop** | Crop masking area from the original image to do inpainting | Low | :zap: :zap: :zap: |
2021-11-15 08:22:34 +01:00
2022-11-02 11:24:04 +01:00
## Download Model Manually
2022-07-25 04:16:01 +02:00
If you have problems downloading the model automatically when lama-cleaner start,
you can download it manually. By default lama-cleaner will load model from `TORCH_HOME=~/.cache/torch/hub/checkpoints/` ,
you can set `TORCH_HOME` to other folder and put the models there.
2022-11-16 03:07:20 +01:00
- GitHub:
2022-07-25 04:16:01 +02:00
- [LaMa ](https://github.com/Sanster/models/releases/tag/add_big_lama )
- [LDM ](https://github.com/Sanster/models/releases/tag/add_ldm )
- [ZITS ](https://github.com/Sanster/models/releases/tag/add_zits )
2022-09-04 15:23:58 +02:00
- [MAT ](https://github.com/Sanster/models/releases/tag/add_mat )
- [FcF ](https://github.com/Sanster/models/releases/tag/add_fcf )
2022-07-25 04:16:01 +02:00
- Baidu:
- https://pan.baidu.com/s/1vUd3BVqIpK6e8N_EA_ZJfw
- passward: flsu
2021-11-15 08:22:34 +01:00
## Development
2022-02-05 12:58:25 +01:00
Only needed if you plan to modify the frontend and recompile yourself.
2022-04-25 15:08:19 +02:00
### Frontend
2021-11-15 08:22:34 +01:00
2022-03-04 06:44:53 +01:00
Frontend code are modified from [cleanup.pictures ](https://github.com/initml/cleanup.pictures ), You can experience their
great online services [here ](https://cleanup.pictures/ ).
2021-11-15 08:22:34 +01:00
- Install dependencies:`cd lama_cleaner/app/ && yarn`
2022-04-04 15:51:33 +02:00
- Start development server: `yarn start`
2021-11-15 08:22:34 +01:00
- Build: `yarn build`
2021-11-15 20:11:46 +01:00
2021-11-15 23:41:59 +01:00
## Docker
2021-11-16 14:21:41 +01:00
2022-11-08 02:16:21 +01:00
You can use [pre-build docker image ](https://github.com/Sanster/lama-cleaner#run-docker-cpu ) to run Lama Cleaner. The model will be downloaded to the cache directory when first time used.
2022-10-05 17:15:09 +02:00
You can mount existing cache directory to start the container,
so you don't have to download the model every time you start the container.
2021-11-15 23:41:59 +01:00
2022-10-05 17:15:09 +02:00
The cache directories for different models correspond as follows:
2021-11-16 14:21:41 +01:00
2022-10-05 17:15:09 +02:00
- lama/ldm/zits/mat/fcf: /root/.cache/torch
2022-10-20 15:03:43 +02:00
- sd1.5: /root/.cache/huggingface
2021-11-15 23:51:27 +01:00
### Run Docker (cpu)
2021-11-16 14:21:41 +01:00
2021-11-15 23:51:27 +01:00
```
2022-10-05 17:15:09 +02:00
docker run -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-02 11:24:04 +01:00
--rm cwq1913/lama-cleaner:cpu-0.24.4 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cpu --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-15 23:51:27 +01:00
### Run Docker (gpu)
2021-11-16 14:21:41 +01:00
2022-10-08 06:55:51 +02:00
- cuda11.6
- pytorch1.12.1
- minimum nvidia driver 510.39.01+
2021-11-15 20:11:46 +01:00
```
2022-10-05 17:15:09 +02:00
docker run --gpus all -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-02 11:24:04 +01:00
--rm cwq1913/lama-cleaner:gpu-0.24.4 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cuda --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-16 14:21:41 +01:00
Then open [http://localhost:8080 ](http://localhost:8080 )
2022-10-05 17:15:09 +02:00
### Build Docker image
cpu only
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/CPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
gpu & cpu
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/GPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
