Video Capture in Arcade

[Akascape]

I created a shader filter and used it with a video as an input channel, it is working properly but I want to save the rendered frames and make a video out of it.

Code:

import arcade
from arcade.experimental.shadertoy import Shadertoy
import cv2 

SCREEN_WIDTH = 400
SCREEN_HEIGHT = 300
SCREEN_TITLE = "ShaderToy Video"


class ShadertoyVideo(arcade.Window):

    def __init__(self, width, height, title):
        super().__init__(width, height, title, resizable=True)
        self.shadertoy = Shadertoy(
            self.get_framebuffer_size(),
            """
            void mainImage( out vec4 fragColor, in vec2 fragCoord )
            {
                // Normalized pixel coordinates (from 0 to 1)
                vec2 suv = fragCoord/iResolution.xy;

                fragColor = vec4(1.5 * sin(suv.y * iResolution.y/3. + iTime * 20.));
                fragColor = 1.- floor(abs(fragColor));
                fragColor *= vec4(sin(suv.y), 0, cos( 1. - suv.y * 2.) , 1);
                fragColor *= texture(iChannel0, suv);
            } 
            """,
        )
        # INSERT YOUR OWN VIDEO HERE
        self.video = cv2.VideoCapture("Example.mp4")
        width, height = (
            int(self.video.get(cv2.CAP_PROP_FRAME_WIDTH)),
            int(self.video.get(cv2.CAP_PROP_FRAME_HEIGHT)),
        )
        self.video_texture = self.ctx.texture((width, height), components=3)
        self.video_texture.wrap_x = self.ctx.CLAMP_TO_EDGE
        self.video_texture.wrap_y = self.ctx.CLAMP_TO_EDGE
        self.video_texture.swizzle = "BGR1"
        self.shadertoy.channel_0 = self.video_texture
        self.set_size(width, height)

    def on_draw(self):
        self.clear()
        self.shadertoy.render()

    def on_update(self, delta_time: float):
        self.shadertoy.time += delta_time
        self.next_frame()

    def on_resize(self, width: float, height: float):
        super().on_resize(width, height)
        self.shadertoy.resize(self.get_framebuffer_size())

    def next_frame(self):
        exists, frame = self.video.read()
        frame = cv2.flip(frame, 0)
        if exists:
            self.video_texture.write(frame)


if __name__ == "__main__":
    ShadertoyVideo(SCREEN_WIDTH, SCREEN_HEIGHT, SCREEN_TITLE)
    arcade.run()

Is there any way to save the new rendered frames and make a video? Please help :)

[einarf]

There is a simple capture module on moderngl-window we thought about porting into arcade. It simply pipes raw frames in stdin : https://github.com/moderngl/moderngl-window/tree/master/moderngl_window/capture

You can probably easily adapt that one to arcade.

[Akascape]

I am still confused on what to do :(, can't we just get the frames by any method and use opencv to convert it?

[einarf]

I guess you could check out the VideoWriter in opencv, but I haven't used that before.

To access the raw pixel data from the window framebuffer you can window.ctx.screen.read() to get the byte data.

[Akascape]

@einarf Ya, I know about the VideoWriter method of opencv, but we have to get the frame data of the window one by one to write the video.

[einarf]

You know how to get the frame data now. Be free to ask questions if you are stuck. We also have a discord server that might be a better place to have conversations about this. The plan was to add a simple capture module in arcade 2.7 at some point, so maybe this can help getting the work done.

[Akascape]

@einarf I have used this method cv2.imwrite("out.png", self.ctx.screen.read()) to write the image file but this is giving errors.

[einarf]

I think opencv wants a numpy array with a certain shape and data type. When asking for help you should always include the actual error.

Again: Discord is better for this if possible

[Akascape]

@einarf Discord link?

[einarf]

Front page in arcade docs.

[pushfoo]

Given this is now a unique ticket, I crossed it off of #1342.