{"action":"created","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/143","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/143/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/143/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/143/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/143","id":64196014,"number":143,"title":"Statistically correct calculation of means, correlations and time-lagged correlations in TICA","user":{"login":"franknoe","id":6495810,"avatar_url":"https://avatars.githubusercontent.com/u/6495810?v=3","gravatar_id":"","url":"https://api.github.com/users/franknoe","html_url":"https://github.com/franknoe","followers_url":"https://api.github.com/users/franknoe/followers","following_url":"https://api.github.com/users/franknoe/following{/other_user}","gists_url":"https://api.github.com/users/franknoe/gists{/gist_id}","starred_url":"https://api.github.com/users/franknoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/franknoe/subscriptions","organizations_url":"https://api.github.com/users/franknoe/orgs","repos_url":"https://api.github.com/users/franknoe/repos","events_url":"https://api.github.com/users/franknoe/events{/privacy}","received_events_url":"https://api.github.com/users/franknoe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-25T06:59:58Z","updated_at":"2015-03-25T10:58:36Z","closed_at":null,"body":"Initial points:\r\n\r\n   * Perhaps it is correct to compute mean and instantaneous correlation matrix on t-tau data points only, in order to have a consistent normalization. In any case we are not computing the real mean (which would requiring reweighting out-of equilibrium data), so perhaps all is needed for the empirical mean is that it is consistently computed with the time-lagged quantities, such that all normalizations are OK and we can expect to get eigenvalues below 1.\r\n   * Fabian's suggestion: It should hold that $c_ii(tau) \\le c_ii(0)$. This is not guaranteed in the current implementation.\r\n\r\nI believe Fabian is right. The consequence is that for a time-lagged (TICA) problem with lag time $\\tau$, the quantities should **not** be computed on the largest possible set of data, but rather consistently:\r\n\r\n![screen shot 2015-03-25 at 08 12 02](https://cloud.githubusercontent.com/assets/6495810/6819754/a8cce78c-d2c6-11e4-9c3e-a9fa71d01035.png)\r\n\r\nOr, if we **define** $N(\\tau) = T-\\tau$ where $T$ is the total number of trajectory frames, and $N$ is the number of frames in our X chunks (X in the code), then simply:\r\n\r\n![screen shot 2015-03-25 at 08 12 43](https://cloud.githubusercontent.com/assets/6495810/6819764/beda1932-d2c6-11e4-8335-bb808fe3ae9e.png)\r\n\r\nI think this is the right solution. Another argument is that in MSMs (special case of the variation principle with characteristic basis functions) we do exactly this:\r\n\r\n![screen shot 2015-03-25 at 08 15 14](https://cloud.githubusercontent.com/assets/6495810/6819794/1d7c4ea6-d2c7-11e4-9105-4362c1501044.png)\r\n\r\nNote the $-1$ is missing because we do not remove an empirical mean.\r\n\r\nThis solution does not only seem to be correct, but will also lead to the most simple code. We will not have to do any dirty tricks, but we can rather compute $N(\\tau)$ by counting the number of frames in our X-variable, directly implement the second set of equations above and do *not* need to care about the trajectory ends for $\\mu_i$ and $c_ij(0)$.\r\n\r\nDo you agree or disagree?"},"comment":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/comments/85982139","html_url":"https://github.com/markovmodel/PyEMMA/issues/143#issuecomment-85982139","issue_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/143","id":85982139,"user":{"login":"gph82","id":7518004,"avatar_url":"https://avatars.githubusercontent.com/u/7518004?v=3","gravatar_id":"","url":"https://api.github.com/users/gph82","html_url":"https://github.com/gph82","followers_url":"https://api.github.com/users/gph82/followers","following_url":"https://api.github.com/users/gph82/following{/other_user}","gists_url":"https://api.github.com/users/gph82/gists{/gist_id}","starred_url":"https://api.github.com/users/gph82/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gph82/subscriptions","organizations_url":"https://api.github.com/users/gph82/orgs","repos_url":"https://api.github.com/users/gph82/repos","events_url":"https://api.github.com/users/gph82/events{/privacy}","received_events_url":"https://api.github.com/users/gph82/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T10:58:36Z","updated_at":"2015-03-25T10:58:36Z","body":"Hi, this indeed seems to make more sense.\r\n\r\nI have one question, though. Even if only slightly, doesn't this lead to a $\\tau$-dependent mean and a $\\tau$-dependent instantaneous correlation matrix? \r\n\r\nI know that for \"well behaved\", \"converged\" data, a leaving out $\\tau$ frames to estimate the inst. correlation and the mean should not matter. It shouldn't even matter much if it's the first $\\tau$ frames, or the last $\\tau$ frames, or some $\\tau$ random frames in between, because mean and correlation should be stationary for this type of data. \r\n\r\nWhat about for data that is *not* that well behaved? Are there any foreseeable consequences (at all). Perhaps there aren't any, I do not know. "}}