{"action":"created","issue":{"url":"https://api.github.com/repos/jeroenjanssens/sos/issues/4","labels_url":"https://api.github.com/repos/jeroenjanssens/sos/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/jeroenjanssens/sos/issues/4/comments","events_url":"https://api.github.com/repos/jeroenjanssens/sos/issues/4/events","html_url":"https://github.com/jeroenjanssens/sos/pull/4","id":90996328,"number":4,"title":"Added the .copy operator to ensure passing by value.","user":{"login":"Fokko","id":1134248,"avatar_url":"https://avatars.githubusercontent.com/u/1134248?v=3","gravatar_id":"","url":"https://api.github.com/users/Fokko","html_url":"https://github.com/Fokko","followers_url":"https://api.github.com/users/Fokko/followers","following_url":"https://api.github.com/users/Fokko/following{/other_user}","gists_url":"https://api.github.com/users/Fokko/gists{/gist_id}","starred_url":"https://api.github.com/users/Fokko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fokko/subscriptions","organizations_url":"https://api.github.com/users/Fokko/orgs","repos_url":"https://api.github.com/users/Fokko/repos","events_url":"https://api.github.com/users/Fokko/events{/privacy}","received_events_url":"https://api.github.com/users/Fokko/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-06-25T15:29:03Z","updated_at":"2015-07-16T13:46:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/jeroenjanssens/sos/pulls/4","html_url":"https://github.com/jeroenjanssens/sos/pull/4","diff_url":"https://github.com/jeroenjanssens/sos/pull/4.diff","patch_url":"https://github.com/jeroenjanssens/sos/pull/4.patch"},"body":"Hi Jeroen Janssens,\r\n\r\nI think I have found a bug in the Python implementation. Currently I am working on an implementation in Scala for the Apache Spark platform the scale the algorithm across a large number of clusters. I used the Python script to create unit-tests to verify the workings of the algorithm. I could not find any obvious mistakes, so I did a step by step comparison.\r\n\r\nWithin Scala the values are assigned as an immutable value which is different as in Python. I found out that this caused a difference in result. To mimic this behavior I added the `.copy()` method on the value which gives a copy. Otherwise, for example `beta[i] == 1`, `betamax == inf` and `Hdiff <= 0`. `betamax` gets assigned as `1`, which is the value of `beta at that moment`, after that the precision of `beta` gets adjusted to `.5`, and then betamax gets also adjusted to `.5`, which does not seem like a correct behavior to me.\r\n\r\n```python\r\n# If not, increase or decrease precision\r\nif Hdiff > 0:\r\n    betamin = beta[i].copy()\r\n    if betamax == np.inf or betamax == -np.inf:\r\n        beta[i] = beta[i] * 2.0\r\n    else:\r\n        beta[i] = (beta[i] + betamax) / 2.0\r\nelse:\r\n    betamax = beta[i].copy()\r\n    if betamin == np.inf or betamin == -np.inf:\r\n        beta[i] = beta[i] / 2.0\r\n    else:\r\n        beta[i] = (beta[i] + betamin) / 2.0\r\n```\r\n\r\nBefore:\r\n```python\r\n~/Desktop/sos-correct/bin$ < iris.csv ./sos -p 30 | sort -nr | head\r\n0.92552418\r\n0.91794955\r\n0.81657372\r\n0.79410068\r\n0.77251273\r\n0.76652991\r\n0.71135211\r\n0.69634175\r\n0.69305280\r\n0.68967627\r\n```\r\nAfter the application of the `.copy()`:\r\n\r\n```python\r\n~/Desktop/sos-correct/bin$ < iris.csv ./sos -p 30 | sort -nr | head\r\n0.88569072\r\n0.82241037\r\n0.74932441\r\n0.71168871\r\n0.69498432\r\n0.68975787\r\n0.67562369\r\n0.66990241\r\n0.65935384\r\n0.65740135\r\n```"},"comment":{"url":"https://api.github.com/repos/jeroenjanssens/sos/issues/comments/121961280","html_url":"https://github.com/jeroenjanssens/sos/pull/4#issuecomment-121961280","issue_url":"https://api.github.com/repos/jeroenjanssens/sos/issues/4","id":121961280,"user":{"login":"Fokko","id":1134248,"avatar_url":"https://avatars.githubusercontent.com/u/1134248?v=3","gravatar_id":"","url":"https://api.github.com/users/Fokko","html_url":"https://github.com/Fokko","followers_url":"https://api.github.com/users/Fokko/followers","following_url":"https://api.github.com/users/Fokko/following{/other_user}","gists_url":"https://api.github.com/users/Fokko/gists{/gist_id}","starred_url":"https://api.github.com/users/Fokko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fokko/subscriptions","organizations_url":"https://api.github.com/users/Fokko/orgs","repos_url":"https://api.github.com/users/Fokko/repos","events_url":"https://api.github.com/users/Fokko/events{/privacy}","received_events_url":"https://api.github.com/users/Fokko/received_events","type":"User","site_admin":false},"created_at":"2015-07-16T13:46:33Z","updated_at":"2015-07-16T13:46:33Z","body":"Sure, I am curious where the difference in results comes from. Please let me know.\r\n\r\nAs far as I know, outlier detection is not yet implemented in Apache Spark. Sparks' own [MLLib](https://spark.apache.org/mllib/) has some general machine learning tools, among other unsupervised algorithms like k-Means. Recently a [Spark third-party packages repository](http://spark-packages.org/) has been set up. I am planning to submit the SOS-algorithm to this repo when is it finished, still need to do some testing and cleanup.\r\n\r\nAlgorithms like LOF are quite hard to implement in Apache Spark as the results are dependent of the n-th nearest neighbors, this would require multiple shuffles across the data-set. As SOS works on rows of data which makes it easier to distribute the computations across multiple nodes. \r\n\r\nThe main problem however, as with LOF and SOS, is the computational cost of the distance matrix which is quadratic to the input size."}}