2022-07-22 17:20:12 +02:00
< h1 align = "center" > Lama Cleaner< / h1 >
< p align = "center" > A free and open-source inpainting tool powered by SOTA AI model.< / p >
< p align = "center" >
< a href = "https://github.com/Sanster/lama-cleaner" >
< img alt = "total download" src = "https://pepy.tech/badge/lama-cleaner" / >
< / a >
< a href = "https://pypi.org/project/lama-cleaner/" >
< img alt = "version" src = "https://img.shields.io/pypi/v/lama-cleaner" / >
< / a >
< a href = "https://colab.research.google.com/drive/1e3ZkAJxvkK3uzaTGu91N9TvI_Mahs0Wb?usp=sharing" >
< img alt = "Open in Colab" src = "https://colab.research.google.com/assets/colab-badge.svg" / >
< / a >
2022-10-09 15:32:13 +02:00
< a href = "https://www.python.org/downloads/" >
< img alt = "python version" src = "https://img.shields.io/badge/python-3.7+-blue.svg" / >
< / a >
2022-10-08 06:55:51 +02:00
< a href = "https://hub.docker.com/r/cwq1913/lama-cleaner" >
< img alt = "version" src = "https://img.shields.io/docker/pulls/cwq1913/lama-cleaner" / >
< / a >
2022-07-22 17:20:12 +02:00
< / p >
2022-10-20 16:36:32 +02:00
https://user-images.githubusercontent.com/3998421/196976498-ba1ad3ab-fa18-4c55-965f-5c6683141375.mp4
2022-07-22 17:20:12 +02:00
## Features
- Completely free and open-source
- Fully self-hosted
2022-09-26 03:16:11 +02:00
- Classical image inpainting algorithm powered by [cv2 ](https://docs.opencv.org/3.4/df/d3d/tutorial_py_inpainting.html )
2022-07-22 17:20:12 +02:00
- Multiple SOTA AI models
2022-03-24 14:36:47 +01:00
1. [LaMa ](https://github.com/saic-mdal/lama )
1. [LDM ](https://github.com/CompVis/latent-diffusion )
2022-07-22 17:20:12 +02:00
1. [ZITS ](https://github.com/DQiaole/ZITS_inpainting )
2022-08-22 17:24:02 +02:00
1. [MAT ](https://github.com/fenglinglwb/MAT )
2022-09-04 15:23:58 +02:00
1. [FcF ](https://github.com/SHI-Labs/FcF-Inpainting )
2022-10-20 15:03:43 +02:00
1. [SD1.5 ](https://github.com/runwayml/stable-diffusion )
2022-07-22 17:20:12 +02:00
- Support CPU & GPU
2022-09-04 15:23:58 +02:00
- Various inpainting [strategy ](#inpainting-strategy )
2022-07-22 17:20:12 +02:00
- Run as a desktop APP
## Usage
2022-10-05 17:15:09 +02:00
< details >
< summary > 1. Remove any unwanted things on the image< / summary >
| Usage | Before | After |
| ---------------------- | --------------------------------------------- | --------------------------------------------------- |
| Remove unwanted things |  |  |
| Remove unwanted person |  |  |
| Remove Text |  |  |
| Remove watermark |  |  |
< / details >
< details >
< summary > 2. Fix old photo< / summary >
| Usage | Before | After |
| ------------- | ----------------------------------- | ----------------------------------------------- |
| Fix old photo |  |  |
< / details >
< details >
< summary > 3. Replace something on the image < / summary >
| Usage | Before | After |
| ---------------------- | ------------------------ | -------------------------------------------------------------- |
| Text Driven Inpainting |  | Prompt: a fox sitting on a bench< br />  |
< / details >
2021-12-12 07:54:37 +01:00
2022-07-22 17:20:12 +02:00
## Quick Start
2021-11-15 08:22:34 +01:00
2022-10-05 17:15:09 +02:00
The easiest way to use Lama Cleaner is to install it using `pip` :
2022-04-18 16:40:23 +02:00
```bash
pip install lama-cleaner
2022-03-04 06:44:53 +01:00
2022-10-05 17:15:09 +02:00
# Models will be downloaded at first time used
2022-07-22 17:20:12 +02:00
lama-cleaner --model=lama --device=cpu --port=8080
# Lama Cleaner is now running at http://localhost:8080
2022-04-18 16:40:23 +02:00
```
2022-10-21 03:05:58 +02:00
For stable-diffusion model, you need to [accepting the terms to access ](https://huggingface.co/runwayml/stable-diffusion-inpainting ), and
get an access token from here [huggingface access token ](https://huggingface.co/docs/hub/security-tokens )
2022-10-05 17:15:09 +02:00
If you prefer to use docker, you can check out [docker ](#docker )
2022-10-24 12:32:35 +02:00
If you hava no idea what is docker or pip, please check [One Click Installer ](./scripts/README.md )
2022-10-05 17:15:09 +02:00
Available command line arguments:
2022-03-04 06:44:53 +01:00
2022-10-24 12:32:35 +02:00
| Name | Description | Default |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- | -------- |
| --model | lama/ldm/zits/mat/fcf/sd1.5 See details in [Inpaint Model ](#inpainting-model ) | lama |
2022-10-21 03:05:58 +02:00
| --hf_access_token | stable-diffusion need [huggingface access token ](https://huggingface.co/docs/hub/security-tokens ) to download model | |
2022-10-24 12:32:35 +02:00
| --sd-run-local | Once the model as downloaded, you can pass this arg and remove `--hf_access_token` | |
| --sd-disable-nsfw | Disable stable-diffusion NSFW checker. | |
| --sd-cpu-textencoder | Always run stable-diffusion TextEncoder model on CPU. | |
| --device | cuda or cpu | cuda |
| --port | Port for backend flask web server | 8080 |
| --gui | Launch lama-cleaner as a desktop application | |
| --gui_size | Set the window size for the application | 1200 900 |
| --input | Path to image you want to load by default | None |
| --debug | Enable debug mode for flask web server | |
2022-03-04 06:44:53 +01:00
2022-07-22 17:20:12 +02:00
## Inpainting Model
2022-06-12 16:06:29 +02:00
2022-09-26 03:16:11 +02:00
| Model | Description | Config |
| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cv2 | :+1: No GPU is required, and for simple backgrounds, the results may even be better than AI models. | |
| LaMa | :+1: Generalizes well on high resolutions(~2k)< br / > | |
2022-09-19 10:06:37 +02:00
| LDM | :+1: Possible to get better and more detail result < br /> :+1: The balance of time and quality can be achieved by adjusting `steps` < br /> :neutral_face: Slower than GAN model< br /> :neutral_face: Need more GPU memory | `Steps` : You can get better result with large steps, but it will be more time-consuming < br /> `Sampler` : ddim or [plms ](https://arxiv.org/abs/2202.09778 ). In general plms can get [better results ](https://github.com/Sanster/lama-cleaner/releases/tag/0.13.0 ) with fewer steps |
2022-09-26 03:16:11 +02:00
| ZITS | :+1: Better holistic structures compared with previous methods < br /> :neutral_face: Wireframe module is **very** slow on CPU | `Wireframe` : Enable edge and line detect |
| MAT | TODO | |
| FcF | :+1: Better structure and texture generation < br / > :neutral_face: Only support fixed size (512x512) input | |
2022-10-20 15:03:43 +02:00
| SD1.5 | :+1: SOTA text-to-image diffusion model | |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
< details >
< summary > See model comparison detail< / summary >
**LaMa vs LDM**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | LaMa | LDM |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs ZITS**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | ZITS | LaMa |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
Image is from [ZITS ](https://github.com/DQiaole/ZITS_inpainting ) paper. I didn't find a good example to show the advantages of ZITS and let me know if you have a good example. There can also be possible problems with my code, if you find them, please let me know too!
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs FcF**
2022-06-12 16:06:29 +02:00
2022-09-04 15:23:58 +02:00
| Original Image | Lama | FcF |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
< / details >
2022-09-04 15:23:58 +02:00
## Inpainting Strategy
Lama Cleaner provides three ways to run inpainting model on images, you can change it in the settings dialog.
2022-10-08 16:15:21 +02:00
| Strategy | Description | VRAM | Speed |
| ------------ | ------------------------------------------------------------------------------------------------------ | ------ | ----------------- |
| **Original** | Use the resolution of the original image | High | :zap: |
| **Resize** | Resize the image to a smaller size before inpainting. The area outside the mask will not loss quality. | Midium | :zap: :zap: |
| **Crop** | Crop masking area from the original image to do inpainting | Low | :zap: :zap: :zap: |
2021-11-15 08:22:34 +01:00
2022-07-25 04:16:01 +02:00
## Download Model Mannually
If you have problems downloading the model automatically when lama-cleaner start,
you can download it manually. By default lama-cleaner will load model from `TORCH_HOME=~/.cache/torch/hub/checkpoints/` ,
you can set `TORCH_HOME` to other folder and put the models there.
- Github:
- [LaMa ](https://github.com/Sanster/models/releases/tag/add_big_lama )
- [LDM ](https://github.com/Sanster/models/releases/tag/add_ldm )
- [ZITS ](https://github.com/Sanster/models/releases/tag/add_zits )
2022-09-04 15:23:58 +02:00
- [MAT ](https://github.com/Sanster/models/releases/tag/add_mat )
- [FcF ](https://github.com/Sanster/models/releases/tag/add_fcf )
2022-07-25 04:16:01 +02:00
- Baidu:
- https://pan.baidu.com/s/1vUd3BVqIpK6e8N_EA_ZJfw
- passward: flsu
2021-11-15 08:22:34 +01:00
## Development
2022-02-05 12:58:25 +01:00
Only needed if you plan to modify the frontend and recompile yourself.
2022-04-25 15:08:19 +02:00
### Frontend
2021-11-15 08:22:34 +01:00
2022-03-04 06:44:53 +01:00
Frontend code are modified from [cleanup.pictures ](https://github.com/initml/cleanup.pictures ), You can experience their
great online services [here ](https://cleanup.pictures/ ).
2021-11-15 08:22:34 +01:00
- Install dependencies:`cd lama_cleaner/app/ && yarn`
2022-04-04 15:51:33 +02:00
- Start development server: `yarn start`
2021-11-15 08:22:34 +01:00
- Build: `yarn build`
2021-11-15 20:11:46 +01:00
2021-11-15 23:41:59 +01:00
## Docker
2021-11-16 14:21:41 +01:00
2022-10-05 17:15:09 +02:00
You can use [pre-build docker image]() to run Lama Cleaner. The model will be downloaded to the cache directory when first time used.
You can mount existing cache directory to start the container,
so you don't have to download the model every time you start the container.
2021-11-15 23:41:59 +01:00
2022-10-05 17:15:09 +02:00
The cache directories for different models correspond as follows:
2021-11-16 14:21:41 +01:00
2022-10-05 17:15:09 +02:00
- lama/ldm/zits/mat/fcf: /root/.cache/torch
2022-10-20 15:03:43 +02:00
- sd1.5: /root/.cache/huggingface
2021-11-15 23:51:27 +01:00
### Run Docker (cpu)
2021-11-16 14:21:41 +01:00
2021-11-15 23:51:27 +01:00
```
2022-10-05 17:15:09 +02:00
docker run -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-01 02:09:43 +01:00
--rm lama-cleaner:cpu-0.24.4 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cpu --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-15 23:51:27 +01:00
### Run Docker (gpu)
2021-11-16 14:21:41 +01:00
2022-10-08 06:55:51 +02:00
- cuda11.6
- pytorch1.12.1
- minimum nvidia driver 510.39.01+
2021-11-15 20:11:46 +01:00
```
2022-10-05 17:15:09 +02:00
docker run --gpus all -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-01 02:09:43 +01:00
--rm lama-cleaner:gpu-0.24.4 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cuda --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-16 14:21:41 +01:00
Then open [http://localhost:8080 ](http://localhost:8080 )
2022-10-05 17:15:09 +02:00
### Build Docker image
cpu only
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/CPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
gpu & cpu
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/GPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
