<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Video Pipeline Details
This notebook goes into detail about the stages of the video pipeline in the base overlay and is written for people who want to create and integrate their own video IP. For most regular input and output use cases the high level wrappers of HDMIIn and HDMIOut should be used.
Both the input and output pipelines in the base overlay consist of four stages, an HDMI frontend, a colorspace converter, a pixel format converter, and the video DMA. For the input the stages are arranged Frontend -> Colorspace Converter -> Pixel Format -> VDMA with the order reversed for the output side. The aim of this notebook is to give you enough information to use each stage separately and be able to modify the pipeline for your own ends.
Before exploring the pipeline we'll import the entire pynq.lib.video module where all classes relating to the pipelines live. We'll also load the base overlay to serve as an example.
The following table shows the IP responsible for each stage in the base overlay which will be referenced throughout the rest of the notebook
|Stage             | Input IP                               | Output IP                          |
|------------------|
Step1:  HDMI Frontend
The HDMI frontend modules wrap all of the clock and timing logic. The HDMI input frontend can be used independently from the rest of the pipeline by accessing its driver from the base overlay.
Step2:  Creating the device will signal to the computer that a monitor is connected. Starting the frontend will wait attempt to detect the video mode, blocking until a lock can be achieved. Once the frontend is started the video mode will be available.
Step3:  The HDMI output frontend can be accessed in a similar way.
Step4:  and the mode must be set prior to starting the output. In this case we are just going to use the same mode as the input.
Step5:  Note that nothing will be displayed on the screen as no video data is currently being send.
Colorspace conversion
The colorspace converter operates on each pixel independently using a 3x4 matrix to transform the pixels. The converter is programmed with a list of twelve coefficients in the folling order
Step6:  Pixel format conversion
The pixel format converters convert between the 24-bit signal used by the HDMI frontends and the colorspace converters to either an 8, 24, or 32 bit signal. 24-bit mode passes the input straight through, 32-bit pads the additional pixel with 0 and 8-bit mode selects the first channel in the pixel. This is exposed by a single property to set or get the number of bits.
Step7:  Video DMA
The final element in the pipeline is the video DMA which transfers video frames to and from memory. The VDMA consists of two channels, one for each direction which operate completely independently. To use a channel its mode must be set prior to start being called. After the DMA is started readframe and writeframe transfer frames. Frames are only transferred once with the call blocking if necessary. asyncio coroutines are available as readframe_async and writeframe_async which yield instead of blocking. A frame of the size of the output can be retrieved from the VDMA by calling writechannel.newframe(). This frame is not guaranteed to be initialised to blank so should be completely written before being handed back.
Step8:  In this case, because we are only using 8 bits per pixel, only the red channel is read and displayed.
The two channels can be tied together which will ensure that the input is always mirrored to the output
Step9:  Frame Ownership
The VDMA driver has a strict method of frame ownership. Any frames returned by readframe or newframe are owned by the user and should be destroyed by the user when no longer needed by calling frame.freebuffer(). Frames handed back to the VDMA with writeframe are no longer owned by the user and should not be touched - the data may disappear at any time.
Cleaning up
It is vital to stop the VDMA before reprogramming the bitstream otherwise the memory system of the chip can be placed into an undefined state. If the monitor does not power on when starting the VDMA this is the likely cause.
<ASSISTANT_TASK:>
Python Code:
from pynq.overlays.base import BaseOverlay
from pynq.lib.video import *
base = BaseOverlay("base.bit")
Explanation: Video Pipeline Details
This notebook goes into detail about the stages of the video pipeline in the base overlay and is written for people who want to create and integrate their own video IP. For most regular input and output use cases the high level wrappers of HDMIIn and HDMIOut should be used.
Both the input and output pipelines in the base overlay consist of four stages, an HDMI frontend, a colorspace converter, a pixel format converter, and the video DMA. For the input the stages are arranged Frontend -> Colorspace Converter -> Pixel Format -> VDMA with the order reversed for the output side. The aim of this notebook is to give you enough information to use each stage separately and be able to modify the pipeline for your own ends.
Before exploring the pipeline we'll import the entire pynq.lib.video module where all classes relating to the pipelines live. We'll also load the base overlay to serve as an example.
The following table shows the IP responsible for each stage in the base overlay which will be referenced throughout the rest of the notebook
|Stage             | Input IP                               | Output IP                          |
|------------------|:---------------------------------------|:-----------------------------------|
|Frontend (Timing) |video/hdmi_in/frontend/vtc_in         |video/hdmi_out/frontend/vtc_out   |
|Frontend (Other)  |video/hdmi_in/frontend/axi_gpio_hdmiin|video/hdmi_out/frontend/axi_dynclk|
|Colour Space      |video/hdmi_in/color_convert           |video/hdmi_out/color_convert      |
|Pixel Format      |video/hdmi_in/pixel_pack              |video/hdmi_outpixel_unpack        |
|VDMA              |video/axi_vdma                        |video/axi_vdam                    |
End of explanation
hdmiin_frontend = base.video.hdmi_in.frontend
Explanation: HDMI Frontend
The HDMI frontend modules wrap all of the clock and timing logic. The HDMI input frontend can be used independently from the rest of the pipeline by accessing its driver from the base overlay.
End of explanation
hdmiin_frontend.start()
hdmiin_frontend.mode
Explanation: Creating the device will signal to the computer that a monitor is connected. Starting the frontend will wait attempt to detect the video mode, blocking until a lock can be achieved. Once the frontend is started the video mode will be available.
End of explanation
hdmiout_frontend = base.video.hdmi_out.frontend
Explanation: The HDMI output frontend can be accessed in a similar way.
End of explanation
hdmiout_frontend.mode = hdmiin_frontend.mode
hdmiout_frontend.start()
Explanation: and the mode must be set prior to starting the output. In this case we are just going to use the same mode as the input.
End of explanation
colorspace_in = base.video.hdmi_in.color_convert
colorspace_out = base.video.hdmi_out.color_convert
bgr2rgb = [0, 0, 1,
           0, 1, 0, 
           1, 0, 0,
           0, 0, 0]
colorspace_in.colorspace = bgr2rgb
colorspace_out.colorspace = bgr2rgb
colorspace_in.colorspace
Explanation: Note that nothing will be displayed on the screen as no video data is currently being send.
Colorspace conversion
The colorspace converter operates on each pixel independently using a 3x4 matrix to transform the pixels. The converter is programmed with a list of twelve coefficients in the folling order:
|     |in1 |in2 |in3 | 1  |
|-----|----|----|----|----|
|out1 |c1  |c2  |c3  |c10 |
|out2 |c4  |c5  |c6  |c11 |
|out3 |c7  |c8  |c9  |c12 |
Each coefficient should be a floating point number between -2 and +2.
The pixels to and from the HDMI frontends are in BGR order so a list of coefficients to convert from the input format to RGB would be:
[0, 0, 1,
 0, 1, 0,
 1, 0, 0,
 0, 0, 0]
reversing the order of the pixels and not adding any bias.
The driver for the colorspace converters has a single property that contains the list of coefficients.
End of explanation
pixel_in = base.video.hdmi_in.pixel_pack
pixel_out = base.video.hdmi_out.pixel_unpack
pixel_in.bits_per_pixel = 8
pixel_out.bits_per_pixel = 8
pixel_in.bits_per_pixel
Explanation: Pixel format conversion
The pixel format converters convert between the 24-bit signal used by the HDMI frontends and the colorspace converters to either an 8, 24, or 32 bit signal. 24-bit mode passes the input straight through, 32-bit pads the additional pixel with 0 and 8-bit mode selects the first channel in the pixel. This is exposed by a single property to set or get the number of bits.
End of explanation
inputmode = hdmiin_frontend.mode
framemode = VideoMode(inputmode.width, inputmode.height, 8)
vdma = base.video.axi_vdma
vdma.readchannel.mode = framemode
vdma.readchannel.start()
vdma.writechannel.mode = framemode
vdma.writechannel.start()
frame = vdma.readchannel.readframe()
vdma.writechannel.writeframe(frame)
Explanation: Video DMA
The final element in the pipeline is the video DMA which transfers video frames to and from memory. The VDMA consists of two channels, one for each direction which operate completely independently. To use a channel its mode must be set prior to start being called. After the DMA is started readframe and writeframe transfer frames. Frames are only transferred once with the call blocking if necessary. asyncio coroutines are available as readframe_async and writeframe_async which yield instead of blocking. A frame of the size of the output can be retrieved from the VDMA by calling writechannel.newframe(). This frame is not guaranteed to be initialised to blank so should be completely written before being handed back.
End of explanation
vdma.readchannel.tie(vdma.writechannel)
Explanation: In this case, because we are only using 8 bits per pixel, only the red channel is read and displayed.
The two channels can be tied together which will ensure that the input is always mirrored to the output
End of explanation
vdma.readchannel.stop()
vdma.writechannel.stop()
Explanation: Frame Ownership
The VDMA driver has a strict method of frame ownership. Any frames returned by readframe or newframe are owned by the user and should be destroyed by the user when no longer needed by calling frame.freebuffer(). Frames handed back to the VDMA with writeframe are no longer owned by the user and should not be touched - the data may disappear at any time.
Cleaning up
It is vital to stop the VDMA before reprogramming the bitstream otherwise the memory system of the chip can be placed into an undefined state. If the monitor does not power on when starting the VDMA this is the likely cause.
End of explanation
<END_TASK>