How to Delete Background from Videos and Add a New Background via CLI

Sep 23, 20243 min read

Recently, I was looking for tools to delete the background from a video I needed for a presentation. Unfortunately, the quality of the free online tools I tried wasn't up to par. After some exploration, I found some open-source alternatives that delivered much better results. In this post, I’ll walk you through the process I followed and what I learned along the way.

I’ll introduce an open-source library called Rembg and demonstrate how to delete the background from a single image. Next, I'll explain how to extract frames from a video and apply Rembg for background removal. Finally, I’ll show you how to add a new background to those images and combine them back into a video.

Prerequisites

Before we start, let's make sure we have the following tools installed:

Python: For running AI models.
FFmpeg: It is a powerful Linux tool for video processing. Here, we will use it to extract video images and combine images to video.
ImageMagick: Linux tool for manipulating images. Here, we will use it to add background to the image.

You can install these tools using:

# Ubuntu procedure
sudo apt update

# Install Python
sudo apt install python3

# Install FFmpeg
sudo apt-get install ffmpeg

# Install rembg Python library
pip install rembg[gpu,cli]

# Install ImageMagick
sudo apt-get install imagemagick

Make sure you have homebrew for MacOS.

# Mac
brew update

# Install FFmpeg
brew install ffmpeg

# Install Python
brew install python

# Install rembg Python library
pip install rembg[gpu,cli]

# Install ImageMagick
brew install imagemagick

Removing background from an Image

Let's start with this image.

Run this command

rembg i anime.png output.png

Note: when you run the above command for the first time, you will have some models downloaded.

I got this output

In the above image, the AI tool has deleted some parts of the ears. That's because we use a generic model for background removal.

Let's run Rembg with anime specific model.

rembg i -m isnet-anime anime.png output.png

Here is the output we get

Now, the result is much better.

Adding background image

We take the following background image and add the above image to it.

Image Magic has a tool called convert that helps us to add the image. Here add the deleted background image to the center of the background image.

convert background.png output.png -gravity center -composite final.png

Background removal for Video

The above process can be repeated frame by frame to process in the video.

Step 1: Extract Frames from the Video

First, we need to extract all the frames from the video. For Illustration, I'll be saving each frame as an image for us to understand better. The extraction of images can be done easily using FFmpeg.

Before we proceed to execute the command, check the FPS of the video and pass that input below.

ffmpeg -i input_video.mp4 -vf "fps=25" frames/frame_%04d.png

Here’s a breakdown of the command:

`-i input_video.mp4`: Input video file.
`-vf "fps=25"`: This parameter will take the snapshot of the current. Extracts 25 frames per second.
`frames/frame_%04d.png`: Saves each frame as a PNG image in the `frames/` directory with sequential numbering.

Once the command is finished, we get this output.

Step 2: Delete Background from Each Frame

Now that we have the frames, we will use the `rembg` library to remove the background of each frame.

mkdir frames_no_bg
rembg p frames/ -m isnet-anime frames_no_bg/

Note: Depending on your CPU/GPU & video length, this process might take a while.

This loop goes through each image in the `frames/` directory, removes the background, and saves the result in the `frames_no_bg/` directory.

The above command will result in the following output:

Performance

	Time taken to generate 1s video
CPU	12.7 sec
GPU (Nvidia 4070)	3.6 sec

Step 3: Add a New Background to Each Frame

After removing the backgrounds, the next step is to overlay a new background onto each frame. You can composite the frames onto a new background image using ImageMagick's' convert' command.

for i in frames_no_bg/*.png; do file_name=$(basename ${i}); convert ../space.png $i -gravity center -composite frames_new_bg/$file_name; done

Explanation:

`background.png`: The new background image.
`-gravity center`: Centers the foreground image onto the background.
`-composite`: Combines the two images.

Step 4: Reassemble the Frames into a Video with Audio

Now that the frames have a new background, we can reassemble them into a video. We’ll also merge the original audio back into the final video.

ffmpeg -r 25 -i frames_new_bg/frame_0%3d.png -s 1280x720  final-video.mov

Explanation:

`-i frames_new_bg/frame_0%3d.png`: Loading all frames in sequential order.
`-r 25` & `-s 1280x720`: Setting default framerate and resolution.

Conclusion

With these simple steps, you can change the background of your video using only command-line tools. By breaking the video down into frames, modifying each frame, and then reassembling it, you gain full control over the look and feel of your video content.

Feel free to contact me with any questions regarding the above process.