Video as texture for openGLES
<p>I've a local video that I want to pass as a texture to an openGL shader.
I'm aware of a number of posts covering related topics, some being old or weird, and some I could not get to work.</p>

<p>It sounds that the way to go is:</p>

<ul>
<li>load the video</li>
<li>get the video output as a <code>CVPixelBuffer</code></li>
<li>then approaches vary around yuv vs rgb, <code>CVOpenGLESTextureCacheCreateTextureFromImage</code> vs <code>glTexImage2D</code> etc. If there's no specific reason to use yuv, I'd rather stick to rgb.</li>
</ul>

<p>My code is able to render <code>UIImages</code> but I could not adapt it to video.</p>

<p>It seems that <code>CVOpenGLESTextureCacheCreateTextureFromImage</code> is now recommended over <code>glTexImage2D</code> to pass the video frame to the openGL program. <a href="https://stackoverflow.com/questions/27268663/video-as-texture-in-opengles2-0">Some</a> convert the video output buffer as an image and then pass it down the pipeline, but this sounds inefficient.</p>

<p>As a start, here is how I get the video pixel buffer that I pass to the view managing the GL program (you can probably skip this as I think it works ok):</p>

<pre><code>import UIKit
import AVFoundation

class ViewController: UIViewController {
    // video things
    var videoOutput: AVPlayerItemVideoOutput!
    var player: AVPlayer!
    var playerItem: AVPlayerItem!
    var isVideoReady = false

    override func viewDidLoad() {
        super.viewDidLoad()
        self.setupVideo()
    }

    func setupVideo() -&gt; Void {
        let url = Bundle.main.urlForResource("myVideoName", withExtension: "mp4")!

        let outputSettings: [String: AnyObject] = ["kCVPixelBufferPixelFormatTypeKey": Int(kCVPixelFormatType_32BGRA)]
        self.videoOutput = AVPlayerItemVideoOutput.init(pixelBufferAttributes: outputSettings)
        self.player = AVPlayer()
        let asset = AVURLAsset(url: url)


        asset.loadValuesAsynchronously(forKeys: ["playable"]) {
            var error: NSError? = nil
            let status = asset.statusOfValue(forKey: "playable", error: &amp;error)
            switch status {
            case .loaded:
                self.playerItem = AVPlayerItem(asset: asset)
                self.playerItem.add(self.videoOutput)
                self.player.replaceCurrentItem(with: self.playerItem)
                self.isVideoReady = true
            case .failed:
                print("failed")
            case .cancelled:
                print("cancelled")
            default:
                print("default")
            }
        }
    }

    // this function is called just before that the openGL program renders
    // and can be used to update the texture. (all the GL program is already initialized at this point)
    func onGlRefresh(glView: OpenGLView) -&gt; Void {
        if self.isVideoReady {
            let pixelBuffer = self.videoOutput.copyPixelBuffer(forItemTime: self.playerItem.currentTime(), itemTimeForDisplay: nil)
            glView.pixelBuffer = pixelBuffer
        }
    }
}
</code></pre>

<p>This seems to work fine, even though I'm not able to really test it :)</p>

<p>So now I've an <code>CVPixelBuffer</code> available (as soon as the video is loaded)
How can I pass it to a GL program?</p>

<p>This code works for an <code>CGImage?</code></p>

<pre><code>    // textureSource is an CGImage?
    guard let textureSource = textureSource else { return }
    let width: Int = textureSource.width
    let height: Int = textureSource.height

    let spriteData = UnsafeMutablePointer&lt;GLubyte&gt;(calloc(Int(UInt(CGFloat(width) * CGFloat(height) * 4)), sizeof(GLubyte.self)))

    let colorSpace = textureSource.colorSpace!

    let spriteContext: CGContext = CGContext(data: spriteData, width: width, height: height, bitsPerComponent: 8, bytesPerRow: width*4, space: colorSpace, bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)!
    spriteContext.draw(in: CGRect(x: 0, y: 0, width: CGFloat(width), height: CGFloat(height)), image: textureSource)

    glBindTexture(GLenum(GL_TEXTURE_2D), _textureId!)
    glTexImage2D(GLenum(GL_TEXTURE_2D), 0, GL_RGBA, GLsizei(width), GLsizei(height), 0, GLenum(GL_RGBA), UInt32(GL_UNSIGNED_BYTE), spriteData)

    free(spriteData)
</code></pre>

<p>but I could not get my head around how to adapt it efficiently to a <code>CVPixelBuffer</code></p>

<p>I'm happy to share more code if needed, but I thought this post was already long enough :)</p>

<p>========== EDIT ==========</p>

<p>I've looked at a bunch of repos (that all copy from Apple's <a href="https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Introduction/Intro.html" rel="nofollow noreferrer">CameraRipple</a> and Ray Wenderlich's <a href="https://www.raywenderlich.com/3664/opengl-tutorial-for-ios-opengl-es-2-0" rel="nofollow noreferrer">tutorial</a>) and <a href="https://github.com/gsabran/Swift-openGL-Video/tree/d7f7487bff2ca1d2a5d016bf0731d34ef5cb6f28" rel="nofollow noreferrer">here</a> is the github repo of what I've so far (I'll keep it alive to preserve the link) It's not ideal but I don't want to paste too much code here. I've been able to get some video texturing to work but:</p>

<ul>
<li>the color are wrong</li>
<li>the display in the simulator is different from the one on a device. In the simulator only the left half of the video is displayed (and covers the entire screen) and there are some vertical aberrations.</li>
</ul>

<p>The simulator issues looks like they might be related to XCode 8 being in beta, but I'm not sure about that...</p>