How to get (X,Y) vectors from position and landmark
<p>I currently have an agent in a map, whose position is known as <code>myPos=(myX,myY)</code>, but whose orientation <code>myOri=(oriX,oriY)</code> is unknown. I also see a landmark at position <code>lm=(lmX,lmY)</code> and I have both Cartesian and polar coordinates from my point of view to the landmark, as <code>relLM=(relX,relY)</code> and <code>polLM=(r,theta)</code>, respectively.</p>

<p>My goal is to find how my orientation vector is related with the X and Y axis, as <code>XX=(xX, xY)</code> and <code>YY=(yX, yY)</code>. Assume for the following examples that X grows to the right and Y grows upwards, and that an agent with 0 rotation follows the X axis (so an agent looking right has <code>XX=(1,0)</code> and <code>YY=(0,1)</code>. <em>This follows from the <a href="https://en.wikipedia.org/wiki/Radian#/media/File:Degree-Radian_Conversion.svg" rel="nofollow noreferrer">intuition</a> where 0 angle rotation is on the X axis, PI/2 rotation is on the Y, PI is on -X, 3PI/2 is on -Y and 2PI is X.</em></p>

<p><strong>Example)</strong> If <code>myOri=(1,1)</code> (agent is facing top right), then <code>XX=(1, -1)</code> (since the X axis is top right to him) and <code>YY=(1, 1)</code> (the Y axis is top left). In the picture below, X and Y are shown in red and green. My agent is in blue and the landmark in pink. Hence, our initial data are <code>myPos=(0,-2)</code>, <code>lm=(0,-1)</code>, <code>relLM=(~0.7,~0.7)</code>.</p>

<p><a href="https://i.stack.imgur.com/U6LUR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/U6LUR.png" alt="Example image"></a></p>

<p>By knowing <code>myPos</code> and <code>lmPos</code>, as well as <code>relLM</code>, this should be possible. However, I'm having trouble finding the appropriate vectors. What is the correct algorithm?</p>

<pre><code>bool someFunction(Landmark *lm, Vector2f myPos, Vector2f *xx, Vector2f *yy){
    // Vector from agent to landmark
    Vector2f agentToLandmark(lm-&gt;position.x - myPos.x,
            lm-&gt;position.y - myPos.y);

    // Vector from agent to landmark, regarding agent's orientation
    Vector2f agentFacingLandmark = lm-&gt;relPosition.toCartesian();        

    // Set the XX and YY values
    // how?
}
</code></pre>

<p><em>My problem is actually in 3D, but using 2D makes the problem easier to explain.</em></p>