Android ImageReader get NV21 format?
<p>I do not have a background in imaging or graphics, so please bear with me :)</p>

<p>I am using <a href="https://github.com/bytedeco/javacv" rel="nofollow noreferrer">JavaCV</a> in one of my projects. In the <a href="https://github.com/bytedeco/javacv/blob/master/samples/RecordActivity.java" rel="nofollow noreferrer">examples</a>, a <code>Frame</code> is constructed which has a buffer of a <a href="https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/Frame.java#L108" rel="nofollow noreferrer">certain size</a>. </p>

<p>When using the <code>public void onPreviewFrame(byte[] data, Camera camera)</code> function in Android, copying this <code>data</code> byte array is no problem if you declare the <code>Frame</code> as <code>new Frame(frameWidth, frameHeight, Frame.DEPTH_UBYTE, 2);</code> where <code>frameWidth</code> and <code>frameHeight</code> are declared as </p>

<pre><code>Camera.Size previewSize = cameraParam.getPreviewSize();
int frameWidth = previewSize.width;
int frameHeight = previewSize.height;
</code></pre>

<p>Recently, Android added a method to capture your screen. Naturally, I wanted to grab those images and also covert them to <code>Frame</code>s. I modified <a href="https://github.com/googlesamples/android-ScreenCapture" rel="nofollow noreferrer">the example code from Google</a> to use the <a href="https://developer.android.com/reference/android/media/ImageReader.html" rel="nofollow noreferrer">ImageReader</a>. </p>

<p>This <code>ImageReader</code> is constructed as <code>ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, PixelFormat.RGBA_8888, 2);</code>. So currently it uses the RGBA_8888 pixel format. I use the following code to copy the bytes to the <code>Frame</code>, which is instantiated as <code>new Frame(DISPLAY_WIDTH, DISPLAY_HEIGHT, Frame.DEPTH_UBYTE, 2);</code>:</p>

<pre><code>ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
mImage.close();
((ByteBuffer) frame.image[0].position(0)).put(bytes);
</code></pre>

<p>But this gives me a <code>java.nio.BufferOverflowException</code>. I printed the sizes of both buffers and the Frame's buffer size is 691200 whereas the <code>bytes</code> array above is of size <code>1413056</code>. Figuring out how this latter number is constructed failed because <a href="https://android.googlesource.com/platform/frameworks/base.git/+/master/media/java/android/media/ImageReader.java#847" rel="nofollow noreferrer">I ran into this native call</a>. So clearly, this won't work out.</p>

<p>After quite a bit of digging I found out that the <a href="http://developer.android.com/reference/android/graphics/ImageFormat.html#NV21" rel="nofollow noreferrer">NV21 image format</a> is "the default format for Camera preview images, when not otherwise set with setPreviewFormat(int)", but the <a href="http://developer.android.com/reference/android/media/ImageReader.html#newInstance%28int,%20int,%20int,%20int%29" rel="nofollow noreferrer">ImageReader class does not support the NV21 format</a> (see the format parameter). So that's tough luck. In the documentation it also reads that 
"For the android.hardware.camera2 API, the YUV_420_888 format is recommended for YUV output instead."</p>

<p>So I tried creating an ImageReader like this <code>ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, ImageFormat.YUV_420_888, 2);</code>, but this gives me <code>java.lang.UnsupportedOperationException: The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23.</code> so that won't work either.</p>

<p>As a last resort, I tried to convert RGBA_8888 to YUV myself using e.g. <a href="https://stackoverflow.com/questions/5960247/convert-bitmap-array-to-yuv-ycbcr-nv21">this post</a>, but I fail to understand how I can obtain an <code>int[] rgba</code> as per the answer.</p>

<p>So, TL;DR <strong>how can I obtain  NV21 image data like you get in Android's <code>public void onPreviewFrame(byte[] data, Camera camera)</code> camera function to instantiate my <code>Frame</code> and work with it using Android's ImageReader (and Media Projection)?</strong> </p>

<h2>Edit (25-10-2016)</h2>

<p>I have created the following conversion runnable to go from RGBA to NV21 format:</p>

<pre><code>private class updateImage implements Runnable {

    private final Image mImage;

    public updateImage(Image image) {
        mImage = image;
    }

    @Override
    public void run() {

        int mWidth = mImage.getWidth();
        int mHeight = mImage.getHeight();

        // Four bytes per pixel: width * height * 4.
        byte[] rgbaBytes = new byte[mWidth * mHeight * 4];
        // put the data into the rgbaBytes array.
        mImage.getPlanes()[0].getBuffer().get(rgbaBytes);

        mImage.close(); // Access to the image is no longer needed, release it.

        // Create a yuv byte array: width * height * 1.5 ().
        byte[] yuv = new byte[mWidth * mHeight * 3 / 2];
        RGBtoNV21(yuv, rgbaBytes, mWidth, mHeight);
        ((ByteBuffer) yuvImage.image[0].position(0)).put(yuv);
    }

    void RGBtoNV21(byte[] yuv420sp, byte[] argb, int width, int height) {
        final int frameSize = width * height;

        int yIndex = 0;
        int uvIndex = frameSize;

        int A, R, G, B, Y, U, V;
        int index = 0;
        int rgbIndex = 0;

        for (int i = 0; i &lt; height; i++) {
            for (int j = 0; j &lt; width; j++) {

                R = argb[rgbIndex++];
                G = argb[rgbIndex++];
                B = argb[rgbIndex++];
                A = argb[rgbIndex++]; // Ignored right now.

                // RGB to YUV conversion according to
                // https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV444_to_RGB888_conversion
                Y = ((66 * R + 129 * G + 25 * B + 128) &gt;&gt; 8) + 16;
                U = ((-38 * R - 74 * G + 112 * B + 128) &gt;&gt; 8) + 128;
                V = ((112 * R - 94 * G - 18 * B + 128) &gt;&gt; 8) + 128;

                // NV21 has a plane of Y and interleaved planes of VU each sampled by a factor
                // of 2 meaning for every 4 Y pixels there are 1 V and 1 U.
                // Note the sampling is every other pixel AND every other scanline.
                yuv420sp[yIndex++] = (byte) ((Y &lt; 0) ? 0 : ((Y &gt; 255) ? 255 : Y));
                if (i % 2 == 0 &amp;&amp; index % 2 == 0) {
                    yuv420sp[uvIndex++] = (byte) ((V &lt; 0) ? 0 : ((V &gt; 255) ? 255 : V));
                    yuv420sp[uvIndex++] = (byte) ((U &lt; 0) ? 0 : ((U &gt; 255) ? 255 : U));
                }
                index++;
            }
        }
    }
}
</code></pre>

<p>The <code>yuvImage</code> object is initialized as <code>yuvImage = new Frame(DISPLAY_WIDTH, DISPLAY_HEIGHT, Frame.DEPTH_UBYTE, 2);</code>, the <code>DISPLAY_WIDTH</code> and <code>DISPLAY_HEIGHT</code> are just two integers specifying the display size. 
This is the code where a background handler handles the onImageReady:</p>

<pre><code>private final ImageReader.OnImageAvailableListener mOnImageAvailableListener
            = new ImageReader.OnImageAvailableListener() {

        @Override
        public void onImageAvailable(ImageReader reader) {
            mBackgroundHandler.post(new updateImage(reader.acquireNextImage()));
        }

    };

...

mImageReader = ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, PixelFormat.RGBA_8888, 2);
mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);
</code></pre>

<p>The methods work and I at least don't get any errors, but the output image is malformed. What is going wrong in my conversion? An example image that is being created:</p>

<p><img src="https://gyazo.com/5095490f2f461b2d6334da1c68c611e4.png" alt="example of malformed image"></p>

<h2>Edit (15-11-2016)</h2>

<p>I have modified the <code>RGBtoNV21</code> function to be the following:</p>

<pre><code>void RGBtoNV21(byte[] yuv420sp, int width, int height) {
    try {
        final int frameSize = width * height;

        int yIndex = 0;
        int uvIndex = frameSize;
        int pixelStride = mImage.getPlanes()[0].getPixelStride();
        int rowStride = mImage.getPlanes()[0].getRowStride();
        int rowPadding = rowStride - pixelStride * width;
        ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();

        Bitmap bitmap = Bitmap.createBitmap(getResources().getDisplayMetrics(), width, height, Bitmap.Config.ARGB_8888);

        int A, R, G, B, Y, U, V;
        int offset = 0;

        for (int i = 0; i &lt; height; i++) {
            for (int j = 0; j &lt; width; j++) {

                // Useful link: https://stackoverflow.com/questions/26673127/android-imagereader-acquirelatestimage-returns-invalid-jpg

                R = (buffer.get(offset) &amp; 0xff) &lt;&lt; 16;     // R
                G = (buffer.get(offset + 1) &amp; 0xff) &lt;&lt; 8;  // G
                B = (buffer.get(offset + 2) &amp; 0xff);       // B
                A = (buffer.get(offset + 3) &amp; 0xff) &lt;&lt; 24; // A
                offset += pixelStride;

                int pixel = 0;
                pixel |= R;     // R
                pixel |= G;  // G
                pixel |= B;       // B
                pixel |= A; // A
                bitmap.setPixel(j, i, pixel);

                // RGB to YUV conversion according to
                // https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV444_to_RGB888_conversion
//                        Y = ((66 * R + 129 * G + 25 * B + 128) &gt;&gt; 8) + 16;
//                        U = ((-38 * R - 74 * G + 112 * B + 128) &gt;&gt; 8) + 128;
//                        V = ((112 * R - 94 * G - 18 * B + 128) &gt;&gt; 8) + 128;

                Y = (int) Math.round(R *  .299000 + G *  .587000 + B *  .114000);
                U = (int) Math.round(R * -.168736 + G * -.331264 + B *  .500000 + 128);
                V = (int) Math.round(R *  .500000 + G * -.418688 + B * -.081312 + 128);

                // NV21 has a plane of Y and interleaved planes of VU each sampled by a factor
                // of 2 meaning for every 4 Y pixels there are 1 V and 1 U.
                // Note the sampling is every other pixel AND every other scanline.
                yuv420sp[yIndex++] = (byte) ((Y &lt; 0) ? 0 : ((Y &gt; 255) ? 255 : Y));
                if (i % 2 == 0 &amp;&amp; j % 2 == 0) {
                    yuv420sp[uvIndex++] = (byte) ((V &lt; 0) ? 0 : ((V &gt; 255) ? 255 : V));
                    yuv420sp[uvIndex++] = (byte) ((U &lt; 0) ? 0 : ((U &gt; 255) ? 255 : U));
                }
            }
            offset += rowPadding;
        }

        File file = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES).getAbsolutePath(), "/Awesomebitmap.png");
        FileOutputStream fos = new FileOutputStream(file);
        bitmap.compress(Bitmap.CompressFormat.PNG, 100, fos);
    } catch (Exception e) {
        Timber.e(e, "Converting image to NV21 went wrong.");
    }
}
</code></pre>

<p>Now the image is no longer malformed, but the chroma is off. </p>

<p><img src="https://gyazo.com/0d9947a036c0924f4e09d55ea730e4b8.png" alt="Wrong color chroma"></p>

<p>The right side is the bitmap that is being created in that loop, the left side is the NV21 saved to an image. So the RGB pixels are processed correctly. Clearly the chroma is off, but the RGB to YUV conversion should be the same one as depicted by wikipedia. What could be wrong here?</p>