Mouse picking with ray casting in DirectX
<p>I'm trying to implement raycasting with DirectX in C++ but have run into problems. I've tried two approaches, using <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/microsoft.directx_sdk.transformation.xmvector3unproject(v=vs.85).aspx" rel="nofollow noreferrer">XMVector3Unproject</a> and following the instructions provided <a href="https://stackoverflow.com/a/19153090/6805787">here on stackoverflow</a> a while ago.</p>

<p>Unproject Approach:</p>

<pre><code>//p is mouse location vector

//Don't know if I need to normalize mouse input
//p.x = (2.0f * p.x) / g_nScreenWidth - 1.0f;
//p.y = 1.0f - (2.0f * p.y) / g_nScreenHeight;

Vector3 orig = XMVector3Unproject(Vector3(p.x, p.y, 0),
                 0,
                 0,
                 g_nScreenWidth,
                 g_nScreenHeight,
                 0,
                 1,
                 GameRenderer.m_matProj,
                 GameRenderer.m_matView,
                 GameRenderer.m_matWorld);

Vector3 dest = XMVector3Unproject(Vector3(p.x, p.y, 1),
                 0,
                 0,
                 g_nScreenWidth,
                 g_nScreenHeight,
                 0,
                 1,
                 GameRenderer.m_matProj,
                 GameRenderer.m_matView,
                 GameRenderer.m_matWorld);

Vector3 direction = dest - orig;
direction.Normalize();

//shoot a bullet to visualize the ray for testing
g_cObjectManager.createObject(BUL_OBJ, "bullet", orig, direction*100);
</code></pre>

<p>Matrix Inversion Approach:</p>

<pre><code>//Normalized device coordinates
p.x = (2.0f * p.x) / g_nScreenWidth - 1.0f;
p.y = 1.0f - (2.0f * p.y) / g_nScreenHeight;

XMVECTOR det; //Determinant, needed for matrix inverse function call
Vector3 origin = Vector3(p.x, p.y, 0);
Vector3 faraway = Vector3(p.x, p.y, 1);

XMMATRIX invViewProj = XMMatrixInverse(&amp;det, GameRenderer.m_matView * GameRenderer.m_matProj);
Vector3 rayorigin = XMVector3Transform(origin, invViewProj);
Vector3 rayend = XMVector3Transform(faraway, invViewProj);
Vector3 raydirection = rayend - rayorigin;
raydirection.Normalize();

g_cObjectManager.createObject(BUL_OBJ, "bullet", rayorigin, raydirection * 5);
</code></pre>

<p>I assume at least the matrix inversion approach from stackoverflow should work, but for some reason my attempt doesn't. Do I need the world matrix as well, or is there some step I'm missing?</p>

<p>This is my first stackoverflow post, so if anything is unclear or more information is needed please let me know.</p>