How to get real time image processing using constrained high speed capture session in Android?
<p>I'm working on a project that requires real time image processing using
Android smartphone's camera (actually, the Samsung Galaxy S7).</p>

<p>Briefly, the main requirements/conditions are:</p>

<ul>
<li>I need to capture and process the images in real-time at high frame rates 
(ideally 120fps or 240fps, but 60fps would be a good start);</li>
<li>There is no need to preview the images on display;</li>
<li>I just need grayscale images (working with nv21 format, this is the first part of the image data).
So, for efficiency, I would prefer not to convert the images from a native format to jpg and then decode and compute the grayscale data;</li>
<li>I don't need high resolution images (640x480 would be fine) and the frame processing itself is relatively simple and can be done very fast
(I just need to scan the grayscale data and extract some basic information);</li>
</ul>

<p>I have tried a conventional capture using the Camera2 Api with an <code>ImageReader</code> surface, but the
best I could get was 37fps (just capturing, not processing), even when turning off the auto control mode, setting bigger ranges for the capture request, 
changing the exposure time, frame duration, etc.</p>

<p>Now, I'm trying to solve the problem using the <code>CameraConstrainedHighSpeedCaptureSession</code> class from the Camera2 Api.
However, the Android reference material says that it should be used for high speed VIDEO RECORDING use case.
Also, according to the documentation, the method <code>createConstrainedHighSpeedCaptureSession</code> requires a surface that
must be either a video encoder surface (that could be acquired from a <code>MediaRecorder</code>) or a preview surface (obtained from <code>SurfaceView</code>, <code>SurfaceTexture</code>).
I think that such capture mode uses an special kind of surface/buffer (probably a faster and more efficient one).</p>

<p>Using a preview surface (<code>SurfaceView</code>, <code>SurfaceTexture</code>), I think I could get stuck in 60fps (the refresh rate of the display). So,
I'm looking for a way to get access to the <code>BufferQueue</code> under the <code>MediaRecorder</code>'s surface, maybe using NDK/JNI.
The idea is to get access to the raw frame data passed from the camera device to <code>MediaRecorder</code>, before the frames be encoded for video purpose.</p>

<p>Please, is this possible? How it could be achieved? Is there a better way?</p>

<p>As an alternative, I have read a litte about <code>FileDescriptor</code> with the intention of redirecting the video frames generated by <code>MediaRecorder</code>
to a buffer in memory and then try to get access to those frames as they are generated, but it seems to be very inefficient and 
the delay may not be tolerated by the application.</p>