{"action":"created","issue":{"url":"https://api.github.com/repos/crate/crate/issues/2506","labels_url":"https://api.github.com/repos/crate/crate/issues/2506/labels{/name}","comments_url":"https://api.github.com/repos/crate/crate/issues/2506/comments","events_url":"https://api.github.com/repos/crate/crate/issues/2506/events","html_url":"https://github.com/crate/crate/issues/2506","id":104576717,"number":2506,"title":"How to increase the speed of this benchmark?","user":{"login":"mojovski","id":90824,"avatar_url":"https://avatars.githubusercontent.com/u/90824?v=3","gravatar_id":"","url":"https://api.github.com/users/mojovski","html_url":"https://github.com/mojovski","followers_url":"https://api.github.com/users/mojovski/followers","following_url":"https://api.github.com/users/mojovski/following{/other_user}","gists_url":"https://api.github.com/users/mojovski/gists{/gist_id}","starred_url":"https://api.github.com/users/mojovski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mojovski/subscriptions","organizations_url":"https://api.github.com/users/mojovski/orgs","repos_url":"https://api.github.com/users/mojovski/repos","events_url":"https://api.github.com/users/mojovski/events{/privacy}","received_events_url":"https://api.github.com/users/mojovski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-02T21:38:01Z","updated_at":"2015-09-03T06:46:05Z","closed_at":null,"body":"On my local machine (localhost:4200) I created this table:\r\n```\r\n    create table my_table (\r\n        x integer primary key,\r\n        y integer primary key,\r\n        z integer primary key,\r\n        sha1_file string\r\n    );\r\n```\r\nand ran this python benchmark inserting 1Mio elements.\r\n```\r\nfrom crate import client\r\nimport time\r\nconnection = client.connect() #(crate_host)\r\ncursor = connection.cursor()\r\nn=100\r\nt0=time.clock()\r\nfor x in range(n):\r\n\tfor y in range(n):\r\n\t\tfor z in range(n):\t\t\r\n\t\t\tcursor.execute(\"\"\"INSERT INTO my_table\r\n\t\t\t\t(x, y, z) VALUES (?, ?, ?)\"\"\",\r\n\t\t\t\t(x,y,z))\r\nt1=time.clock()\r\nelapsed=t1-t0\r\nprint \"Inserting \"+str(n**3)+\" elements took: \"+str(elapsed)+\" seconds\"\r\n```\r\n\r\nAnd I receive:\r\n```\r\nInserting 1000000 elements took 622.201936 seconds\r\n```\r\n\r\nIs there a way to increase the speed?\r\n\r\nPS: I need a speed 100x faster than that. :/"},"comment":{"url":"https://api.github.com/repos/crate/crate/issues/comments/137352244","html_url":"https://github.com/crate/crate/issues/2506#issuecomment-137352244","issue_url":"https://api.github.com/repos/crate/crate/issues/2506","id":137352244,"user":{"login":"dobe","id":48020,"avatar_url":"https://avatars.githubusercontent.com/u/48020?v=3","gravatar_id":"","url":"https://api.github.com/users/dobe","html_url":"https://github.com/dobe","followers_url":"https://api.github.com/users/dobe/followers","following_url":"https://api.github.com/users/dobe/following{/other_user}","gists_url":"https://api.github.com/users/dobe/gists{/gist_id}","starred_url":"https://api.github.com/users/dobe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dobe/subscriptions","organizations_url":"https://api.github.com/users/dobe/orgs","repos_url":"https://api.github.com/users/dobe/repos","events_url":"https://api.github.com/users/dobe/events{/privacy}","received_events_url":"https://api.github.com/users/dobe/received_events","type":"User","site_admin":false},"created_at":"2015-09-03T06:46:05Z","updated_at":"2015-09-03T06:46:05Z","body":"you should either use execute_many if it is ok for your use-case to batch the inserts (see https://crate.io/docs/projects/crate-python/stable/client.html#inserting-data) \r\n\r\nalso a bottleneck is the sequential call you are doing here. try to insert in parallel with threads or prccesses, then it should be faster. \r\n\r\ni don't know the use case, but if you want to import lot of data you might want use copy from, see https://crate.io/docs/stable/best_practice/data_import.html \r\n\r\nin order to be able to insert 160000 rows per second (according to your requirements) you need more clients or use copy from. we achieved about 330k tweet inserts per second on an 8 node ssd cluster with copy from."}}