{"action":"created","issue":{"url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/107","labels_url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/107/labels{/name}","comments_url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/107/comments","events_url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/107/events","html_url":"https://github.com/apa512/clj-rethinkdb/issues/107","id":114288023,"number":107,"title":"net/read-response* bottleneck","user":{"login":"KeeganMyers","id":5679159,"avatar_url":"https://avatars.githubusercontent.com/u/5679159?v=3","gravatar_id":"","url":"https://api.github.com/users/KeeganMyers","html_url":"https://github.com/KeeganMyers","followers_url":"https://api.github.com/users/KeeganMyers/followers","following_url":"https://api.github.com/users/KeeganMyers/following{/other_user}","gists_url":"https://api.github.com/users/KeeganMyers/gists{/gist_id}","starred_url":"https://api.github.com/users/KeeganMyers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KeeganMyers/subscriptions","organizations_url":"https://api.github.com/users/KeeganMyers/orgs","repos_url":"https://api.github.com/users/KeeganMyers/repos","events_url":"https://api.github.com/users/KeeganMyers/events{/privacy}","received_events_url":"https://api.github.com/users/KeeganMyers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-30T15:08:02Z","updated_at":"2015-11-02T23:04:37Z","closed_at":null,"body":"I have been running performance benchmarks between the old application I have been refactoring (rails with Mariadb back-end), and the current Clojure rethink stack. The results for basic crud have been disappointing. This lead me to initially assume that Rethinkdb its self may be slower than Mariadb for basic crud. So I tested this by connecting to rethink in ruby. But the result is substantially faster than in Clojure\r\n\r\nBenchmark.ms {1000.times.each {r.db('arcDev').table('resource').get(\"a60401f0-7124-11e5-81af-b16d74de8809\").run()}\r\n\r\n559.1058265417814 ms\r\n\r\n(time (with-open [conn (r/connect :host \"172.17.0.10\" :port 28015 :db \"arcDev\")] (dotimes [n 1000] (-> (r/db \"arcDev\") (r/table \"resource\") (r/get \"c5144730-7124-11e5-81af-b16d74de8809\") (r/run conn)))))\r\n\"Elapsed time: 40018.837416 msecs\"\r\n\r\nAfter profiling the Clojure implementation I found that most time spent in this query occurs in read-response*. This seems to be directly related to two factors \r\n\r\n1 casting bytes to int seems to be somewhat slower than DataInputStream.readInt or .readLong\r\n2 reading from unbuffered inputStreams is known to be slower than making use of either bufferedInputStream or a buffered reader.\r\n\r\nMaking use of a buffered stream seems to give marginal improvements. But I think performance might be even better if we replace java.io with nio.2. The only limiting factor is that core.async + nio.2 may not account for back pressure well. \r\n\r\nI have forked the repo and will create a pull request if I can find a way to significantly improve performance."},"comment":{"url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/comments/153186527","html_url":"https://github.com/apa512/clj-rethinkdb/issues/107#issuecomment-153186527","issue_url":"https://api.github.com/repos/apa512/clj-rethinkdb/issues/107","id":153186527,"user":{"login":"KeeganMyers","id":5679159,"avatar_url":"https://avatars.githubusercontent.com/u/5679159?v=3","gravatar_id":"","url":"https://api.github.com/users/KeeganMyers","html_url":"https://github.com/KeeganMyers","followers_url":"https://api.github.com/users/KeeganMyers/followers","following_url":"https://api.github.com/users/KeeganMyers/following{/other_user}","gists_url":"https://api.github.com/users/KeeganMyers/gists{/gist_id}","starred_url":"https://api.github.com/users/KeeganMyers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KeeganMyers/subscriptions","organizations_url":"https://api.github.com/users/KeeganMyers/orgs","repos_url":"https://api.github.com/users/KeeganMyers/repos","events_url":"https://api.github.com/users/KeeganMyers/events{/privacy}","received_events_url":"https://api.github.com/users/KeeganMyers/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T23:04:37Z","updated_at":"2015-11-02T23:04:37Z","body":"I wanted to provide an update on this issue. At the moment I have decreased the query time to a fairly meaningful degree\r\n\r\n(time (dotimes [n 1000] (-> (query/db \"arcDev\") (query/table \"resource\") (query/get \"a60401f0-7124-11e5-81af-b16d74de8809\") (query/run conn))))\r\n\"Elapsed time: 666.877891 msecs\"\r\n\r\nI removed most of the raw java.io logic and replaced it with a netty based abstraction clj-tcp. I was not able to devote as much time as I would have liked to this project over the weekend. But at this point I should just need to fix a race condition in read-response* and conduct far more scientific bench-marking.\r\n\r\nI also added an optarg to changes so that we can make use of the squash parameter. \r\n"}}