{"action":"created","issue":{"url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/15","labels_url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/15/comments","events_url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/15/events","html_url":"https://github.com/DiffSharp/DiffSharp/issues/15","id":123779064,"number":15,"title":"Forward on forward mode question","user":{"login":"mrakgr","id":6266635,"avatar_url":"https://avatars.githubusercontent.com/u/6266635?v=3","gravatar_id":"","url":"https://api.github.com/users/mrakgr","html_url":"https://github.com/mrakgr","followers_url":"https://api.github.com/users/mrakgr/followers","following_url":"https://api.github.com/users/mrakgr/following{/other_user}","gists_url":"https://api.github.com/users/mrakgr/gists{/gist_id}","starred_url":"https://api.github.com/users/mrakgr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrakgr/subscriptions","organizations_url":"https://api.github.com/users/mrakgr/orgs","repos_url":"https://api.github.com/users/mrakgr/repos","events_url":"https://api.github.com/users/mrakgr/events{/privacy}","received_events_url":"https://api.github.com/users/mrakgr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/DiffSharp/DiffSharp/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-24T09:11:29Z","updated_at":"2015-12-25T08:43:45Z","closed_at":null,"body":"    let inline r x y = (x*x)*(y*y)\r\n\r\n    let a = D 3.0f |> makeForward 0u (D 1.0f)\r\n    let a' = a |> makeForward 1u (D 1.0f)\r\n\r\n    let b = D 2.0f |> makeForward 0u (D 0.0f)\r\n    let b' = b |> makeForward 1u (D 0.0f)\r\n\r\n    let q = r a' b'\r\n\r\nI am wondering whether it would be possible to take second derivatives using forward on forward. I already know how to do it using reverse over forward, but I've read that forward on forward should be possible as well."},"comment":{"url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/comments/167210534","html_url":"https://github.com/DiffSharp/DiffSharp/issues/15#issuecomment-167210534","issue_url":"https://api.github.com/repos/DiffSharp/DiffSharp/issues/15","id":167210534,"user":{"login":"mrakgr","id":6266635,"avatar_url":"https://avatars.githubusercontent.com/u/6266635?v=3","gravatar_id":"","url":"https://api.github.com/users/mrakgr","html_url":"https://github.com/mrakgr","followers_url":"https://api.github.com/users/mrakgr/followers","following_url":"https://api.github.com/users/mrakgr/following{/other_user}","gists_url":"https://api.github.com/users/mrakgr/gists{/gist_id}","starred_url":"https://api.github.com/users/mrakgr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrakgr/subscriptions","organizations_url":"https://api.github.com/users/mrakgr/orgs","repos_url":"https://api.github.com/users/mrakgr/repos","events_url":"https://api.github.com/users/mrakgr/events{/privacy}","received_events_url":"https://api.github.com/users/mrakgr/received_events","type":"User","site_admin":false},"created_at":"2015-12-25T08:43:45Z","updated_at":"2015-12-25T08:43:45Z","body":"Ah, right. I worded my question badly. What I really meant to ask was whether it was possible to calculate the Hessian using forward on forward. For the above example it comes to this:\r\n\r\n    val it : D * DM = (D 36.0f, DM [[18.0f; 24.0f]\r\n                                    [24.0f; 8.0f]])\r\n\r\nI can get a single column using forward and then the reverse like so:\r\n\r\n    let a = D 3.0f |> makeForward 0u (D 0.0f)\r\n    let a' = a |> makeReverse 1u\r\n\r\n    let b = D 2.0f |> makeForward 0u (D 1.0f)\r\n    let b' = b |> makeReverse 1u\r\n\r\n    let q = r a' b'\r\n\r\n    q |> reverseProp (D 1.0f)\r\n\r\n    a'\r\n    b'\r\n\r\nIt is also possible to take the diagonal elements using the example above. What I meant to ask is, purely out of curiosity, whether it was possible to take the non diagonal elements using forward on forward.\r\n\r\nOne way that I've discovered is like this:\r\n\r\n    let a = D 3.0f |> makeForward 0u (D 1.0f)\r\n    let a' = a |> makeForward 1u (D 1.0f)\r\n\r\n    let b = D 2.0f |> makeForward 2u (D 1.0f)\r\n    let b' = b |> makeForward 3u (D 1.0f)\r\n\r\n    let q = r a' b'\r\n\r\nIf one sets the perturbation tags like so, it will go deeper and calculate all sorts of derivatives:\r\n\r\n    DF\r\n        (DF\r\n           (DF (DF (D 36.0f,D 24.0f,0u),DF (D 24.0f,D 8.0f,0u),1u),\r\n            DF (DF (D 36.0f,D 24.0f,0u),DF (D 24.0f,D 8.0f,0u),1u),2u),\r\n         DF\r\n           (DF (DF (D 36.0f,D 24.0f,0u),DF (D 24.0f,D 8.0f,0u),1u),\r\n            DF (DF (D 18.0f,D 12.0f,0u),DF (D 12.0f,D 4.0f,0u),1u),2u),3u)\r\n\r\nI can see a 18 and 8 and 24 in there (a whole bunch of them,) so I can tell that it is calculating the Hessian among others, but I cannot see a way to do it on an even level. For example:\r\n\r\n    let a = D 3.0f |> makeForward 0u (D 0.0f)\r\n    let a' = a |> makeForward 1u (D 0.0f)\r\n\r\n    let b = D 2.0f |> makeForward 0u (D 0.0f)\r\n    let b' = b |> makeForward 1u (D 0.0f)\r\n\r\n    let q = r a' b'\r\n\r\nThe above does not give me the first row and column of the Hessian but simply zero.\r\n\r\n    let a = D 3.0f |> makeForward 0u (D 1.0f)\r\n    let a' = a |> makeForward 1u (D 1.0f)\r\n\r\n    let b = D 2.0f |> makeForward 0u (D 1.0f)\r\n    let b' = b |> makeForward 1u (D 0.0f)\r\n\r\n    let q = r a' b'\r\n\r\nThe above example gives me 32 as the second derivative which is obviously wrong.\r\n\r\nMaybe this is a misunderstanding on my part and forward on forward is really made to be used for Hessians.\r\n\r\n"}}