Heat haze/distortion effect in OpenGL (GLSL) and how it should be achieved
<p>you can skip to the TL;DR at the bottom for the conclusion. I preferred to provide as much information as I could, so as to help narrow down the question further.</p>

<p>I've been having an issue with a heat haze effect I've been working on.
This is the sort of effect that I was thinking of but since this is a rather generalized system it would apply to any so called screen space refraction:<a href="https://i.stack.imgur.com/NQxl9.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/NQxl9.jpg" alt="Overwatch ((C) 2016) heat haze on bike"></a>
The haze effect is not where my issue lies as it is just a distortion of sampling coordinates, rather it's with what is sampled. My first approach was to render the distortions to another render target. This method was fairly successful but has a major downfall that's easy to foresee if you've dealt with screen space textures before. the problem is that because of the offset to the sampling coordinate, if an object is in front of the refractor, its edges will be taken into the refraction calculation.
<a href="https://i.stack.imgur.com/SZ0Vt.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/SZ0Vt.jpg" alt="looks fine"></a>
as you can see it looks fine when all the geometry is either the environment (no depth test) or back geometry. <a href="https://i.stack.imgur.com/cfzuA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/cfzuA.png" alt="looks bad"></a> and here with a cube closer than the refractor. As you can see it, there is this effect I'll call bleeding of the closer geometry.</p>

<p>relevant shader code for reference:</p>

<pre><code>/* transparency.frag */
layout (location = 0) out vec4 out_color; // frag color
layout (location = 1) out vec4 bright; // used for bloom effect
layout (location = 2) out vec4 deform; // deform buffer
[...]
void main(void) {
    [...]
    vec2 n = __sample_noise_texture_with_time__{};
    deform = vec4(n * .1, 0, 1);
    out_color = vec4(0, 0, 0, .0);
    bright = vec4(0.0, 0.0, 0.0, .9);
}
</code></pre>

<hr>

<pre><code>/* post_process.frag */
in vec2 texel;

uniform sampler2D screen_t;
uniform sampler2D depth_t;
uniform sampler2D bright_t;
uniform sampler2D deform_t;
[...]
void main(void) {
    [...]
    vec3 noise_sample = texture(deform_t, texel).xyz;
    vec2 texel_c = texel + noise_sample.xy;
    [sample screen and bloom with texel_c, gama corect, output to color buffer]
}
</code></pre>

<p>To try to combat this, I tried a technique that involved comparing depth components. to do this, i made the transparent object write its frag_depth tp the z component of my deform buffer like so</p>

<pre><code>/* transparency.frag */
[...]
deform = vec4(n * .1, gl_FragCoord.z, 1);
[...]
</code></pre>

<p>and then to determine what is in front of what a quick check in the post processing shader.</p>

<pre><code>[...]
float dist = texture(depth_t, texel_c).x;
float dist1 = noise_sample.z; // what i wrote to the deform buffer z
if (dist + .01 &lt; dist1) { /* do something liek draw debug */ }
[...]
</code></pre>

<p>this worked somewhat but broke down as i moved away, even i i linearized the depth values and compared the distances.
<a href="https://i.stack.imgur.com/UoWRt.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UoWRt.png" alt="color keyed debug"></a>
EDIT 3: added better screenshots for the depth test phase
<a href="https://i.stack.imgur.com/ZRbFr.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ZRbFr.png" alt="better color keyed debug"></a>
(In yellow where it's sampling something that's in front, couldn't be bothered to make it render the polygons as well so i drew them in)
<a href="https://i.stack.imgur.com/Zh7MZ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Zh7MZ.png" alt="failing from further away"></a>
(and here demonstrating it partially failing the depth comparison test from further away)</p>

<p>I also had some 'fun' with another technique where i passed the color buffer directly to the transparency shader and had it output the sample to its color output. In theory if the scene is Z sorted, this should produce the desired result. i'll let you be the judge of that.
<a href="https://i.stack.imgur.com/LZFVC.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/LZFVC.jpg" alt="enter image description here"></a>
(I have a few guesses as to what the patterns that emerge are since they are similar to the rasterisation patterns of GPUs however that's not very relevant sine that 'solution' was more of a desperation effort than anything)</p>

<p>TL;DR and Formal Question: I've had a go at a few techniques based on my knowledge and haven't been able to find much literature on the subject. so my question is: How do you realize sch effects as heat haze/distortion (that do not cover the whole screen might i add) and is there literature on the subject. For reference to what sort of effect I would be looking at, see my Overwatch screenshot and all other similar effects in the game.</p>

<p>Thought I would also mention just for completeness sake I'm running OpenGL 4.5 (on windows) with most shaders being version 4.00, and am working with a custom engine.</p>

<p>EDIT: If you want information about the software part of the engine feel free to ask. I didn't include any because it I didn't deem it relevant however i'd be glad to provide specs and code snippets as well as more shaders on demand.</p>

<p>EDIT 2: I thought i'd also mention that this could be achieved by using a second render pass and a clipping plane however, that would be costly and feels unnecessary since the viewpoint is the same. It might be that's this is the only solution but i don't believe so.</p>

<p>Thanks for your answers in advance!</p>