{"push_id":609745957,"size":3,"distinct_size":3,"ref":"refs/heads/improve-import-users-task","head":"1d27dbe8b827bac3af65dbff1999a918dff3c227","before":"e08fea00b8eff81f0b6da39c2ff11b09e443d662","commits":[{"sha":"aca4ebbdc1e056a7e18a9a1ea1f4b2c6df12bf47","author":{"email":"55107e193e648a27778fa98736b2e8e24b3cd6e1@karen-chan.com","name":"karen chan"},"message":"Improve time and memory usage of import users task\n\n - Use transactions to shorten the execution time.\n - Remove the result hash table that was storing all the user objects, instead\n   just write to the output csv directly.\n - Change \"Imported XXX users...\" to be printed only every 100 users.\n\nHere are some benchmarks I did on my local machine.  Based on my code changes,\nwith various bits commented out, I got these time and memory usage:\n\n|                  | Original   | No transaction | One transaction | Multiple transactions |\n| ---------------  | ----------:| --------------:| ---------------:| ---------------------:|\n| **Time (real)**  |            |                |                 |                       |\n| 4000 users       |  6m52.556s |       8m8.919s |       3m17.362s |             2m50.438s |\n| 10000 users      | 18m11.743s |     18m48.104s |      11m23.677s |              8m4.523s |\n| **Memory (rsz)** |            |                |                 |                       |\n| 4000 users       |   89.132MB |       10.128MB |       18.2220MB |              59.080MB |\n| 10000 users      |  207.560MB |       14.704MB |       451.980MB |              65.008MB |\n\nThe multiple transactions was configured to be one transaction per 1000 users.\n\nThe resident memory usage is calculated inside `ImportUsers#read` by getting\nthe memory usage at the beginning and at the end and calculating the\ndifference.  The memory usage was measured using ````ps -o rsz #{Process.pid} |\ntail -1`.to_i```\n\nThe time usage was measured by ```time rake accounts:import_users```.\n\nTowards #152","distinct":true,"url":"https://api.github.com/repos/karenc/accounts/commits/aca4ebbdc1e056a7e18a9a1ea1f4b2c6df12bf47"},{"sha":"0b5b2cf68c1599a1f01ca3bae537fc4d0c157d86","author":{"email":"3d07a85bcf8c861e99d9e9b5021708e397105dd4@gmail.com","name":"JP Slavinsky"},"message":"Merge pull request #161 from karenc/improve-import-users-task\n\nImprove import users task","distinct":true,"url":"https://api.github.com/repos/karenc/accounts/commits/0b5b2cf68c1599a1f01ca3bae537fc4d0c157d86"},{"sha":"1d27dbe8b827bac3af65dbff1999a918dff3c227","author":{"email":"55107e193e648a27778fa98736b2e8e24b3cd6e1@karen-chan.com","name":"karen chan"},"message":"Use smarter_csv in import users task\n\nUse smarter_csv instead of the built in ruby csv library because\nsmarter_csv allows reading the csv file in chunks.\n\nIt didn't improve the execution time or the memory usage of the import\nusers task, but it simplifies the code and makes it easier to read.\n\nTowards #152","distinct":true,"url":"https://api.github.com/repos/karenc/accounts/commits/1d27dbe8b827bac3af65dbff1999a918dff3c227"}]}