2022-07-22 17:20:12 +02:00
< h1 align = "center" > Lama Cleaner< / h1 >
< p align = "center" > A free and open-source inpainting tool powered by SOTA AI model.< / p >
< p align = "center" >
< a href = "https://github.com/Sanster/lama-cleaner" >
< img alt = "total download" src = "https://pepy.tech/badge/lama-cleaner" / >
< / a >
< a href = "https://pypi.org/project/lama-cleaner/" >
< img alt = "version" src = "https://img.shields.io/pypi/v/lama-cleaner" / >
< / a >
< a href = "https://colab.research.google.com/drive/1e3ZkAJxvkK3uzaTGu91N9TvI_Mahs0Wb?usp=sharing" >
< img alt = "Open in Colab" src = "https://colab.research.google.com/assets/colab-badge.svg" / >
< / a >
2022-11-21 07:02:44 +01:00
2022-11-16 14:18:51 +01:00
< a href = "https://huggingface.co/spaces/Sanster/Lama-Cleaner-lama" >
< img alt = "Hugging Face Spaces" src = "https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue" / >
< / a >
2022-11-21 07:02:44 +01:00
2022-12-11 15:50:37 +01:00
< a href = "" >
< img alt = "python version" src = "https://img.shields.io/pypi/pyversions/lama-cleaner" / >
2022-10-09 15:32:13 +02:00
< / a >
2022-10-08 06:55:51 +02:00
< a href = "https://hub.docker.com/r/cwq1913/lama-cleaner" >
< img alt = "version" src = "https://img.shields.io/docker/pulls/cwq1913/lama-cleaner" / >
< / a >
2022-07-22 17:20:12 +02:00
< / p >
2022-10-20 16:36:32 +02:00
https://user-images.githubusercontent.com/3998421/196976498-ba1ad3ab-fa18-4c55-965f-5c6683141375.mp4
2022-07-22 17:20:12 +02:00
## Features
- Completely free and open-source
- Fully self-hosted
2022-12-05 01:48:54 +01:00
- [Windows 1-Click Installer ](./scripts/README.md )
2022-09-26 03:16:11 +02:00
- Classical image inpainting algorithm powered by [cv2 ](https://docs.opencv.org/3.4/df/d3d/tutorial_py_inpainting.html )
2022-07-22 17:20:12 +02:00
- Multiple SOTA AI models
2023-01-03 15:03:02 +01:00
1. [LaMa ](https://github.com/saic-mdal/lama )
2. [LDM ](https://github.com/CompVis/latent-diffusion )
3. [ZITS ](https://github.com/DQiaole/ZITS_inpainting )
4. [MAT ](https://github.com/fenglinglwb/MAT )
5. [FcF ](https://github.com/SHI-Labs/FcF-Inpainting )
6. [SD1.5/SD2 ](https://github.com/runwayml/stable-diffusion )
7. [Manga ](https://github.com/msxie92/MangaInpainting )
2023-01-08 14:13:36 +01:00
8. [Paint by Example ](https://github.com/Fantasy-Studio/Paint-by-Example ) [YouTube Demo ](https://www.youtube.com/watch?v=NSAN3TzfhaI&ab_channel=PanicByte )
2022-07-22 17:20:12 +02:00
- Support CPU & GPU
2022-09-04 15:23:58 +02:00
- Various inpainting [strategy ](#inpainting-strategy )
2022-07-22 17:20:12 +02:00
- Run as a desktop APP
2023-01-08 14:13:36 +01:00
- [Interactive Segmentation ](https://github.com/Sanster/lama-cleaner/releases/tag/0.28.0 ) on any object. [YouTube Demo ](https://www.youtube.com/watch?v=xHdo8a4Mn2g&ab_channel=PanicByte )
2022-07-22 17:20:12 +02:00
## Usage
2023-01-08 14:13:36 +01:00
A great introductory [youtube video ](https://www.youtube.com/watch?v=aYia7Jvbjno&ab_channel=Aitrepreneur ) made by **
Aitrepreneur**
2022-12-15 14:30:20 +01:00
2022-10-05 17:15:09 +02:00
< details >
< summary > 1. Remove any unwanted things on the image< / summary >
2022-11-21 07:02:44 +01:00
| Usage | Before | After |
2023-01-03 15:03:02 +01:00
| ----------------------------- | --------------------------------------------- | --------------------------------------------------- |
2022-11-21 07:02:44 +01:00
| Remove unwanted things |  |  |
| Remove unwanted person |  |  |
| Remove text |  |  |
| Remove watermark |  |  |
| Remove text balloons on manga |  |  |
2022-10-05 17:15:09 +02:00
< / details >
< details >
< summary > 2. Fix old photo< / summary >
| Usage | Before | After |
| ------------- | ----------------------------------- | ----------------------------------------------- |
| Fix old photo |  |  |
< / details >
< details >
< summary > 3. Replace something on the image < / summary >
2022-12-11 15:15:34 +01:00
SD1.5/SD2
2022-10-05 17:15:09 +02:00
| Usage | Before | After |
| ---------------------- | ------------------------ | -------------------------------------------------------------- |
| Text Driven Inpainting |  | Prompt: a fox sitting on a bench< br />  |
2022-12-11 15:15:34 +01:00
Paint by Example
2023-01-03 15:03:02 +01:00
| Original Image | Example Image | Result Image |
| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| < img src = "https://user-images.githubusercontent.com/3998421/206908542-c6465ca3-6414-4593-8318-0c8b569e7682.jpg" > | [<img src="https://user-images.githubusercontent.com/3998421/206908517-bd7f62d2-464a-43bc-892e-dbea45f7b104.jpeg"> ](https://youtu.be/NSAN3TzfhaI ) | [<img src="https://user-images.githubusercontent.com/3998421/206903752-0463a0cf-146d-4125-a969-8fe20127a09b.jpeg"> ](https://youtu.be/NSAN3TzfhaI ) |
2022-12-11 15:15:34 +01:00
2022-10-05 17:15:09 +02:00
< / details >
2021-12-12 07:54:37 +01:00
2022-07-22 17:20:12 +02:00
## Quick Start
2021-11-15 08:22:34 +01:00
2022-10-05 17:15:09 +02:00
The easiest way to use Lama Cleaner is to install it using `pip` :
2022-04-18 16:40:23 +02:00
```bash
pip install lama-cleaner
2022-03-04 06:44:53 +01:00
2022-10-05 17:15:09 +02:00
# Models will be downloaded at first time used
2022-07-22 17:20:12 +02:00
lama-cleaner --model=lama --device=cpu --port=8080
# Lama Cleaner is now running at http://localhost:8080
2022-04-18 16:40:23 +02:00
```
2022-10-05 17:15:09 +02:00
If you prefer to use docker, you can check out [docker ](#docker )
2022-10-24 12:32:35 +02:00
If you hava no idea what is docker or pip, please check [One Click Installer ](./scripts/README.md )
2022-10-05 17:15:09 +02:00
Available command line arguments:
2022-03-04 06:44:53 +01:00
2023-01-08 14:13:36 +01:00
| Name | Description | Default |
| -------------------- | ------------------------------------------------------------------------------------------------------------- | -------- |
| --model | lama/ldm/zits/mat/fcf/sd1.5/manga/sd2/paint_by_example See details in [Inpaint Model ](#inpainting-model ) | lama |
| --sd-disable-nsfw | Disable stable-diffusion NSFW checker. | |
| --sd-cpu-textencoder | Always run stable-diffusion TextEncoder model on CPU. | |
| --sd-enable-xformers | Enable xFormers optimizations. See: [facebookresearch/xformers ](https://github.com/facebookresearch/xformers ) | |
2023-01-12 13:07:49 +01:00
| --local-files-only | Once the model as downloaded, you can pass this arg to avoid diffusers connect to Hugging Face server | |
2023-01-08 14:13:36 +01:00
| --cpu-offload | sd/paint_by_example model, offloads all models to CPU, sacrifice speed for reducing vRAM usage. |
| --no-half | Using full precision for sd/paint_by_exmaple model | |
| --device | cuda / cpu / mps | cuda |
| --port | Port for backend flask web server | 8080 |
| --gui | Launch lama-cleaner as a desktop application | |
| --gui_size | Set the window size for the application | 1200 900 |
| --input | Path to image you want to load by default | None |
| --debug | Enable debug mode for flask web server | |
2022-03-04 06:44:53 +01:00
2022-07-22 17:20:12 +02:00
## Inpainting Model
2022-06-12 16:06:29 +02:00
2022-09-26 03:16:11 +02:00
| Model | Description | Config |
| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cv2 | :+1: No GPU is required, and for simple backgrounds, the results may even be better than AI models. | |
| LaMa | :+1: Generalizes well on high resolutions(~2k)< br / > | |
2022-09-19 10:06:37 +02:00
| LDM | :+1: Possible to get better and more detail result < br /> :+1: The balance of time and quality can be achieved by adjusting `steps` < br /> :neutral_face: Slower than GAN model< br /> :neutral_face: Need more GPU memory | `Steps` : You can get better result with large steps, but it will be more time-consuming < br /> `Sampler` : ddim or [plms ](https://arxiv.org/abs/2202.09778 ). In general plms can get [better results ](https://github.com/Sanster/lama-cleaner/releases/tag/0.13.0 ) with fewer steps |
2022-09-26 03:16:11 +02:00
| ZITS | :+1: Better holistic structures compared with previous methods < br /> :neutral_face: Wireframe module is **very** slow on CPU | `Wireframe` : Enable edge and line detect |
| MAT | TODO | |
| FcF | :+1: Better structure and texture generation < br / > :neutral_face: Only support fixed size (512x512) input | |
2022-10-20 15:03:43 +02:00
| SD1.5 | :+1: SOTA text-to-image diffusion model | |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
< details >
< summary > See model comparison detail< / summary >
**LaMa vs LDM**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | LaMa | LDM |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs ZITS**
2022-06-12 16:06:29 +02:00
2022-07-22 17:20:12 +02:00
| Original Image | ZITS | LaMa |
| ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-11-21 07:02:44 +01:00
Image is from [ZITS ](https://github.com/DQiaole/ZITS_inpainting ) paper. I didn't find a good example to show the
advantages of ZITS and let me know if you have a good example. There can also be possible problems with my code, if you
find them, please let me know too!
2022-06-12 16:06:29 +02:00
2022-10-08 16:15:21 +02:00
**LaMa vs FcF**
2022-06-12 16:06:29 +02:00
2022-11-21 07:02:44 +01:00
| Original Image | LaMa | FcF |
2022-09-04 15:23:58 +02:00
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|  |  |  |
2022-06-12 16:06:29 +02:00
2022-11-21 07:02:44 +01:00
**LaMa vs Manga**
Manga model works better on high-quality manga image then LaMa model.
Original Image
2023-01-03 15:03:02 +01:00
| Model | 1080x740 | 1470x1010 |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Manga |  |  |
| [LaMa ](https://github.com/saic-mdal/lama ) |  |  |
2022-11-21 07:02:44 +01:00
2022-10-08 16:15:21 +02:00
< / details >
2022-09-04 15:23:58 +02:00
## Inpainting Strategy
Lama Cleaner provides three ways to run inpainting model on images, you can change it in the settings dialog.
2022-10-08 16:15:21 +02:00
| Strategy | Description | VRAM | Speed |
| ------------ | ------------------------------------------------------------------------------------------------------ | ------ | ----------------- |
| **Original** | Use the resolution of the original image | High | :zap: |
| **Resize** | Resize the image to a smaller size before inpainting. The area outside the mask will not loss quality. | Midium | :zap: :zap: |
| **Crop** | Crop masking area from the original image to do inpainting | Low | :zap: :zap: :zap: |
2021-11-15 08:22:34 +01:00
2022-11-02 11:24:04 +01:00
## Download Model Manually
2022-07-25 04:16:01 +02:00
If you have problems downloading the model automatically when lama-cleaner start,
you can download it manually. By default lama-cleaner will load model from `TORCH_HOME=~/.cache/torch/hub/checkpoints/` ,
you can set `TORCH_HOME` to other folder and put the models there.
2022-11-16 03:07:20 +01:00
- GitHub:
2023-01-03 15:03:02 +01:00
- [LaMa ](https://github.com/Sanster/models/releases/tag/add_big_lama )
- [LDM ](https://github.com/Sanster/models/releases/tag/add_ldm )
- [ZITS ](https://github.com/Sanster/models/releases/tag/add_zits )
- [MAT ](https://github.com/Sanster/models/releases/tag/add_mat )
- [FcF ](https://github.com/Sanster/models/releases/tag/add_fcf )
2022-07-25 04:16:01 +02:00
- Baidu:
2023-01-03 15:03:02 +01:00
- https://pan.baidu.com/s/1vUd3BVqIpK6e8N_EA_ZJfw
- passward: flsu
2022-07-25 04:16:01 +02:00
2021-11-15 08:22:34 +01:00
## Development
2022-02-05 12:58:25 +01:00
Only needed if you plan to modify the frontend and recompile yourself.
2022-04-25 15:08:19 +02:00
### Frontend
2021-11-15 08:22:34 +01:00
2022-03-04 06:44:53 +01:00
Frontend code are modified from [cleanup.pictures ](https://github.com/initml/cleanup.pictures ), You can experience their
great online services [here ](https://cleanup.pictures/ ).
2021-11-15 08:22:34 +01:00
- Install dependencies:`cd lama_cleaner/app/ && yarn`
2022-04-04 15:51:33 +02:00
- Start development server: `yarn start`
2021-11-15 08:22:34 +01:00
- Build: `yarn build`
2021-11-15 20:11:46 +01:00
2021-11-15 23:41:59 +01:00
## Docker
2021-11-16 14:21:41 +01:00
2022-11-21 07:02:44 +01:00
You can use [pre-build docker image ](https://github.com/Sanster/lama-cleaner#run-docker-cpu ) to run Lama Cleaner. The
model will be downloaded to the cache directory when first time used.
2022-10-05 17:15:09 +02:00
You can mount existing cache directory to start the container,
so you don't have to download the model every time you start the container.
2021-11-15 23:41:59 +01:00
2022-10-05 17:15:09 +02:00
The cache directories for different models correspond as follows:
2021-11-16 14:21:41 +01:00
2022-10-05 17:15:09 +02:00
- lama/ldm/zits/mat/fcf: /root/.cache/torch
2022-10-20 15:03:43 +02:00
- sd1.5: /root/.cache/huggingface
2021-11-15 23:51:27 +01:00
### Run Docker (cpu)
2021-11-16 14:21:41 +01:00
2021-11-15 23:51:27 +01:00
```
2022-10-05 17:15:09 +02:00
docker run -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-17 15:20:42 +01:00
--rm cwq1913/lama-cleaner:cpu-0.26.1 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cpu --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-15 23:51:27 +01:00
### Run Docker (gpu)
2021-11-16 14:21:41 +01:00
2022-10-08 06:55:51 +02:00
- cuda11.6
- pytorch1.12.1
- minimum nvidia driver 510.39.01+
2021-11-15 20:11:46 +01:00
```
2022-10-05 17:15:09 +02:00
docker run --gpus all -p 8080:8080 \
-v /path/to/torch_cache:/root/.cache/torch \
-v /path/to/huggingface_cache:/root/.cache/huggingface \
2022-11-17 15:20:42 +01:00
--rm cwq1913/lama-cleaner:gpu-0.26.1 \
2022-10-05 17:15:09 +02:00
lama-cleaner --device=cuda --port=8080 --host=0.0.0.0
2021-11-15 20:11:46 +01:00
```
2021-11-16 14:21:41 +01:00
Then open [http://localhost:8080 ](http://localhost:8080 )
2022-10-05 17:15:09 +02:00
### Build Docker image
cpu only
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/CPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
gpu & cpu
```
2022-11-01 02:09:43 +01:00
docker build -f --build-arg version=0.x.0 ./docker/GPUDockerfile -t lamacleaner .
2022-10-05 17:15:09 +02:00
```
