How to unproject a vector in 3d
<p>I am developing an 3D engine from scratch and I am trying to unproject a vector. I use my own Math library called <code>ALMath.js</code>. As far as I know, to convert 2d screen coordinates into 3d world coordinates I need multiply the vector formed by x and y coordinates in the canvas by the inverse of ViewProjection Matrix. This is the code for unproject:</p>

<pre><code>unproject : function (vector){
    var viewMatrix = camera.viewMatrix;
    var projectionMatrix = camera.projectionMatrix;

    var viewProjection = viewMatrix.multiply(projectionMatrix);
    var inverseViewProjection = viewProjection.getInverse();

    var x = ((vector.x -0) / (AL3D.width)) *2 -1;
    var y = ((vector.y -0) / (AL3D.height)) * 2 -1;
    var z = 2*vector.z-1;
    var w = 1;
    var vec4 = new ALMath.Vector4(x,y,z,w);
    var transformedVector = inverseViewProjection.multiplyByVector4(vec4);
    var wordCoords = new ALMath.Vector3(transformedVector.x/transformedVector.w,transformedVector.y/transformedVector.w,transformedVector.z/transformedVector.w);
    return wordCoords;
  }
</code></pre>

<p>The <code>ALMath</code> library works fine. I use it around all engine (compute the model-view-projection, create projection matrix, do the inverse ...) and works well. In fact I check the result of the operations using <code>Octave</code> (alternative to <code>matlab</code>) and the result is the same with <code>ALMath</code>.</p>

<p>The problem is that if I click in the upper left corner:</p>

<pre><code>canvas.addEventListener('click', function(event) {
            var rect = canvas.getBoundingClientRect();
            var x = event.pageX - rect.left,
            y = event.pageY - rect.top;
            var vector = camera.unproject(new ALMath.Vector3(x,y,0));
         });
</code></pre>

<p>with <code>x = 0</code> and <code>y = 2</code> I get a <code>vector = (-0.12131, -0.25894, -0.79409)</code> and I know that it is wrong because If I set the cube mesh in that position I see that this is not the upper left corner.</p>

<p>I have programmed a lookAt function in the camera class</p>

<pre><code>lookAt : function (eye, target, up)
</code></pre>

<p>As a example I show the operations for <code>x = 0</code> and <code>y = 2</code> with <code>octave</code>.</p>

<p>Firs I set my camera as follow:</p>

<pre><code>camera = new AL3D.PerspectiveCamera(40, window.innerWidth/window.innerHeight);
        camera.lookAt(new ALMath.Vector3(), new ALMath.Vector3(0,-0.5,-2), new ALMath.Vector3(0,1,0));
</code></pre>

<p>And this is step by step calculations in <code>octave</code> that match with javascript code result</p>

<pre><code>viewMatrix =

   1.00000   0.00000   0.00000   0.00000
   0.00000   0.97014   0.24254   0.00000
   0.00000  -0.24254   0.97014   0.00000
   0.00000   0.00000   0.00000   1.00000

projectionMatrix =

   1.37374   0.00000   0.00000   0.00000
   0.00000   2.82610   0.00000   0.00000
   0.00000   0.00000  -1.00020  -0.20002
   0.00000   0.00000  -1.00000   0.00000

octave:7&gt; viewProjectionMatrix = viewMatrix * projectionMatrix 
viewProjectionMatrix =

   1.37374   0.00000   0.00000   0.00000
   0.00000   2.74171  -0.24258  -0.04851
   0.00000  -0.68543  -0.97034  -0.19405
   0.00000   0.00000  -1.00000   0.00000

octave:8&gt; inverseViewProjectionMatrix = inv(viewProjectionMatrix)
inverseViewProjectionMatrix =

   0.72794   0.00000   0.00000  -0.00000
   0.00000   0.34328  -0.08582   0.00000
   0.00000   0.00000   0.00000  -1.00000
   0.00000  -1.21256  -4.85023   5.00050

AL3D.width = 1366
AL3D.height = 664
x = -1
y = -0.9939759036144579
z = -1
w = 1

octave:9&gt; vector = [ -1 -0.9939759036144579 -1 1]
vector =

  -1.00000  -0.99398  -1.00000   1.00000

octave:10&gt; transformedVector = vector * inverseViewProjectionMatrix 
transformedVector =

  -0.72794  -1.55377  -4.76492   6.00050
// Perspective division
octave:12&gt; result = [ transformedVector(1)/transformedVector(4) transformedVector(2)/transformedVector(4) transformedVector(3)/transformedVector(4)]
result =

  -0.12131  -0.25894  -0.79409
</code></pre>

<p>Maybe I am forgeting something, but I don't know. What is wrong in my logic. Thanks.</p>

<p><strong>Edit</strong>: The problem seem to be the view matrix. This is the code for my view matrix:</p>

<pre><code>lookAt : function(eye, target, up){
        var eye = eye || new ALMath.Vector3();
        var up = up || new ALMath.Vector3();
        var target = target || new ALMath.Vector3();

        var c = this.components;

        var z = target.sub(eye);
        z = z.normalize();

        var x = z.cross(up);
        x = x.normalize();

        var y = x.cross(z);
        y = y.normalize();

        c[0] = x.x; c[1] = x.y; c[2] = x.z; 
        c[4] = y.x; c[5] = y.y; c[6] = y.z; 
        c[8] = -z.x; c[9] = -z.y; c[10] = -z.z; 
c[12] = -x.dot(eye); c[13] = -y.dot(eye); c[14] = z.dot(eye);

        return this;
    },
</code></pre>

<p>and this is for projection:</p>

<pre><code>perspectiveProjection : function ( fov, aspect, zNear, zFar ) {
        var a = aspect;

        var tan=Math.tan(ALMath.degToRad(0.5*fov)),
            A=-(zFar+zNear)/(zFar-zNear),
            B=(-2*zFar*zNear)/(zFar-zNear);

        var c = this.components;

        c[ 0 ] = 0.5/tan;       c[ 4 ] = 0;             c[ 8 ] = 0;         c[ 12 ] = 0;
        c[ 1 ] = 0;             c[ 5 ] = (0.5*a/tan);   c[ 9 ] = 0;         c[ 13 ] = 0;
        c[ 2 ] = 0;             c[ 6 ] = 0;             c[ 10 ] = A;        c[ 14 ] = B;
        c[ 3 ] = 0;             c[ 7 ] = 0;             c[ 11 ] =-1;        c[ 15 ] = 0;

        return this;
    },
</code></pre>

<p>My projection matrix is fine. But my view matrix is not the same that the view matrix compute in the gman library: <a href="https://github.com/greggman/twgl.js/blob/master/src/m4.js" rel="nofollow noreferrer">https://github.com/greggman/twgl.js/blob/master/src/m4.js</a> in m4.js the matrix is computed</p>

<pre><code>c[0] = x.x; c[1] = x.y; c[2] = x.z; c[3] = 0;
c[4] = y.x; c[5] = y.y; c[6] = y.z; c[7] = 0;
c[8] = -z.x; c[9] = -z.y; c[10] = -z.z; c[11] = 0;
c[12] = eye.x; c[13] = eye.y; c[14] = eye.z; c[15] = 1;
</code></pre>

<p>instead of</p>

<pre><code>c[0] = x.x; c[1] = x.y; c[2] = x.z; 
c[4] = y.x; c[5] = y.y; c[6] = y.z; 
c[8] = -z.x; c[9] = -z.y; c[10] = -z.z; 
c[12] = -x.dot(eye); c[13] = -y.dot(eye); c[14] = z.dot(eye);
</code></pre>

<p>Note that in my math library I computed it with dot between axis and eye as said here: <a href="https://stackoverflow.com/questions/349050/calculating-a-lookat-matrix">Calculating a LookAt matrix</a></p>

<p>So,that thread is wrong? I should use eye directly instead dot product between axis and eye?</p>

<p>If I run the script posted by gman I get the following ouput:</p>

<pre><code>frustum points
0 -0.414 -0.207 -0.500
1 0.414 -0.207 -0.500
2 -0.414 0.207 -0.500
3 0.414 0.207 -0.500
4 -82.843 -41.421 -100.000
5 82.843 -41.421 -100.000
6 -82.843 41.421 -100.000
7 82.843 41.421 -100.000

camerafrustum points
0 1.666 2.120 3.080
1 1.080 2.120 3.666
2 1.497 2.458 2.911
3 0.911 2.458 3.497
4 134.224 25.915 19.067
5 17.067 25.915 136.224
6 100.403 93.555 -14.754
7 -16.754 93.555 102.403

screen points (should match width, height)
0 148.858 -47.653 4.029
1 111.806 -38.903 3.734
2 147.454 -72.303 4.217
3 108.845 -59.000 3.876
4 951.911 101.710 9.651
5 61.823 20.354 3.229
6 -833.522 732.104 -10.661
7 25.094 -97.340 4.035

unprojected (should match cameraFrustum points)
0 1.666 2.120 3.080
1 1.080 2.120 3.666
2 1.497 2.458 2.911
3 0.911 2.458 3.497
4 134.224 25.915 19.067
5 17.067 25.915 136.226
6 100.404 93.556 -14.754
7 -16.754 93.557 102.405
</code></pre>

<p>Is the same that result posted by gman but it differs in the <code>screen points (should match width, height)</code> section.</p>

<p>If I run the gman script in my pc with this directive: <code>&lt;script src="https://twgljs.org/dist/2.x/twgl-full.min.js"&gt;&lt;/script&gt;</code> the output is the same that gman posted</p>

<pre><code>screen points (should match width, height)
0 -0.000 -0.000 -1.000
1 300.000 -0.000 -1.000
2 -0.000 150.000 -1.000
3 300.000 150.000 -1.000
4 0.000 0.000 1.000
5 300.000 0.000 1.000
6 -0.000 150.000 1.000
7 300.000 150.000 1.000
</code></pre>

<p>But If I download <code>https://twgljs.org/dist/2.x/twgl-full.min.js</code> and store in the same directory that html file and use the directive <code>&lt;script src="twgl.js"&gt;&lt;/script&gt;</code> in the html file, the output is like my Math library, this is:</p>

<pre><code>screen points (should match width, height)
    0 148.858 -47.653 4.029
    1 111.806 -38.903 3.734
    2 147.454 -72.303 4.217
    3 108.845 -59.000 3.876
    4 951.911 101.710 9.651
    5 61.823 20.354 3.229
    6 -833.522 732.104 -10.661
    7 25.094 -97.340 4.035
</code></pre>

<p>The script adapted to my library is the following:</p>

<pre><code>function main()
{

var width = 300;
var height = 150;
var aspect = width / height
var fieldOfView = Math.PI * 0.25; // 45 degrees
var zNear = 0.5;
var zFar  = 100;
var projection = new ALMath.Matrix4();
projection = projection.perspectiveProjection(45, aspect, zNear, zFar);

var eye = new ALMath.Vector3(1, 2, 3);
var target = new ALMath.Vector3(4, 5, 6);
var up = new ALMath.Vector3(0, 1, 0);

var camera = new ALMath.Matrix4();
camera = camera.lookAt(eye, target, up);
var view = camera.getInverse();
var viewProjection = view.multiply(projection);
var inverseViewProjection = viewProjection.getInverse();

function getFrustumPoints(fieldOfView, aspect, zNear, zFar) {

  var f = 1 / Math.tan(fieldOfView / 2);
  var nearY = zNear / f;
  var nearX = nearY * aspect;
  var farY = zFar / f;
  var farX = farY * aspect;

  return [
    new ALMath.Vector3(-nearX, -nearY, -zNear),
    new ALMath.Vector3( nearX, -nearY, -zNear),
    new ALMath.Vector3(-nearX,  nearY, -zNear),
    new ALMath.Vector3( nearX,  nearY, -zNear),
    new ALMath.Vector3(-farX, -farY, -zFar),
    new ALMath.Vector3( farX, -farY, -zFar),
    new ALMath.Vector3(-farX,  farY, -zFar),
    new ALMath.Vector3( farX,  farY, -zFar),
  ];
}

function projectScreenPoint(width, height, projection, point) {
  var c = projection.transformPoint(point);
  return new ALMath.Vector3((c.x * 0.5 + 0.5) * width,(c.y * 0.5 + 0.5) * height, c.z);

}

function unproject(width, height, inverseViewProjection, p) {
  return inverseViewProjection.transformPoint(new ALMath.Vector3(p.x / width * 2 - 1, p.y / height * 2 - 1, p.z));    
}

function showPoints(label, points) {
  log(label);
  points.forEach((p, ndx) =&gt; log(ndx, p.x.toFixed(3), p.y.toFixed(3), p.z.toFixed(3)));
}

var frustumPoints = getFrustumPoints(fieldOfView, aspect, zNear, zFar);
showPoints("frustum points", frustumPoints);

var cameraFrustumPoints = frustumPoints.map(
  p =&gt; camera.transformPoint(p));
showPoints("camerafrustum points", cameraFrustumPoints);

var screenPoints = cameraFrustumPoints.map(
  p =&gt; projectScreenPoint(width, height, viewProjection, p));
showPoints("screen points (should match width, height)", screenPoints);

var unprojectedPoints = screenPoints.map(
  p =&gt; unproject(width, height, inverseViewProjection, p));
showPoints("unprojected (should match cameraFrustum points)",
           unprojectedPoints);


function log(...args) {
  var elem = document.createElement("pre");
  elem.textContent = [...args].join(' ');
  document.body.appendChild(elem);
}

}
</code></pre>

<p>So the questions now are:</p>

<ul>
<li>I should use eye directly instead dot product between axis and eye?</li>
<li>Why the screen points differs, from my library and local stored gman's library and from the</li>
</ul>