{"action":"created","issue":{"url":"https://api.github.com/repos/syncany/syncany/issues/195","labels_url":"https://api.github.com/repos/syncany/syncany/issues/195/labels{/name}","comments_url":"https://api.github.com/repos/syncany/syncany/issues/195/comments","events_url":"https://api.github.com/repos/syncany/syncany/issues/195/events","html_url":"https://github.com/syncany/syncany/issues/195","id":39916554,"number":195,"title":"Support for 200k+ files; find bottlenecks and test with many files","user":{"login":"binwiederhier","id":664597,"avatar_url":"https://avatars.githubusercontent.com/u/664597?v=3","gravatar_id":"","url":"https://api.github.com/users/binwiederhier","html_url":"https://github.com/binwiederhier","followers_url":"https://api.github.com/users/binwiederhier/followers","following_url":"https://api.github.com/users/binwiederhier/following{/other_user}","gists_url":"https://api.github.com/users/binwiederhier/gists{/gist_id}","starred_url":"https://api.github.com/users/binwiederhier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binwiederhier/subscriptions","organizations_url":"https://api.github.com/users/binwiederhier/orgs","repos_url":"https://api.github.com/users/binwiederhier/repos","events_url":"https://api.github.com/users/binwiederhier/events{/privacy}","received_events_url":"https://api.github.com/users/binwiederhier/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/syncany/syncany/labels/prio:high","name":"prio:high","color":"e11d21"},{"url":"https://api.github.com/repos/syncany/syncany/labels/type:bug","name":"type:bug","color":"eb6420"},{"url":"https://api.github.com/repos/syncany/syncany/labels/what:core","name":"what:core","color":"eb6420"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/syncany/syncany/milestones/2","html_url":"https://github.com/syncany/syncany/milestones/Syncany%20Beta","labels_url":"https://api.github.com/repos/syncany/syncany/milestones/2/labels","id":944077,"number":2,"title":"Syncany Beta","description":"","creator":{"login":"binwiederhier","id":664597,"avatar_url":"https://avatars.githubusercontent.com/u/664597?v=3","gravatar_id":"","url":"https://api.github.com/users/binwiederhier","html_url":"https://github.com/binwiederhier","followers_url":"https://api.github.com/users/binwiederhier/followers","following_url":"https://api.github.com/users/binwiederhier/following{/other_user}","gists_url":"https://api.github.com/users/binwiederhier/gists{/gist_id}","starred_url":"https://api.github.com/users/binwiederhier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binwiederhier/subscriptions","organizations_url":"https://api.github.com/users/binwiederhier/orgs","repos_url":"https://api.github.com/users/binwiederhier/repos","events_url":"https://api.github.com/users/binwiederhier/events{/privacy}","received_events_url":"https://api.github.com/users/binwiederhier/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":7,"state":"open","created_at":"2015-01-20T19:27:34Z","updated_at":"2015-11-08T07:25:54Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2014-08-10T20:59:06Z","updated_at":"2015-12-29T03:57:12Z","closed_at":"2015-11-08T07:25:54Z","body":"As we saw in #115, Syncany currently has a few issues with large amounts of files. It can handle 200k+ files, but apparently has issues with 400k+ files.\r\n\r\nThis issue's purpose is to \r\n- [ ] Test performance with many files\r\n- [ ] Find bottlenecks that make processing slow (read stuff to memory)\r\n- [ ] Find memory leaks to free up memory for important stuff (dedup, et al); *see #382*\r\n\r\nLet us know what your experiences are with many files, or ask on IRC (#syncany on Freenode)."},"comment":{"url":"https://api.github.com/repos/syncany/syncany/issues/comments/167714734","html_url":"https://github.com/syncany/syncany/issues/195#issuecomment-167714734","issue_url":"https://api.github.com/repos/syncany/syncany/issues/195","id":167714734,"user":{"login":"ruipgpinheiro","id":11169970,"avatar_url":"https://avatars.githubusercontent.com/u/11169970?v=3","gravatar_id":"","url":"https://api.github.com/users/ruipgpinheiro","html_url":"https://github.com/ruipgpinheiro","followers_url":"https://api.github.com/users/ruipgpinheiro/followers","following_url":"https://api.github.com/users/ruipgpinheiro/following{/other_user}","gists_url":"https://api.github.com/users/ruipgpinheiro/gists{/gist_id}","starred_url":"https://api.github.com/users/ruipgpinheiro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruipgpinheiro/subscriptions","organizations_url":"https://api.github.com/users/ruipgpinheiro/orgs","repos_url":"https://api.github.com/users/ruipgpinheiro/repos","events_url":"https://api.github.com/users/ruipgpinheiro/events{/privacy}","received_events_url":"https://api.github.com/users/ruipgpinheiro/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T03:57:12Z","updated_at":"2015-12-29T03:57:12Z","body":"I attempted to `sy down` about 6 times, and it never completes successfuly for this folder, always with \"GC Overhead limit exceeded\" or \"Heap space exceeded\".\r\n\r\nThe last time, I saved what seems to be the important part of the log:\r\n\r\n\t29-12-15 1:39:00.543 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:00.605 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1175768,3681 operation 14987,62 txts 329787\r\n\t29-12-15 1:39:15.222 | DownOperation   | main           | INFO :   + Applying database version (CMWvHvSNqMWlvFDdYMwK144)\r\n\t29-12-15 1:39:15.222 | DownOperation   | main           | FINE :   + Contents: DatabaseVersion [header=CMWvHvSNqMWlvFDdYMwK/(CMWvHvSNqMWlvFDdYMwK144)/T=14498985\r\n\t15311, chunks=1173, multiChunks=13, fileContents=1123, fileHistories=3099]\r\n\t29-12-15 1:39:24.473 | DatabaseVersion | main           | SEVE : SQL Error:\r\n\tjava.sql.BatchUpdateException: java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n\t\t\tat org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)\r\n\t\t\tat org.syncany.database.dao.ChunkSqlDao.writeChunks(ChunkSqlDao.java:69)\r\n\t\t\tat org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:146)\r\n\t\t\tat org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:108)\r\n\t\t\tat org.syncany.database.SqlDatabase.writeDatabaseVersion(SqlDatabase.java:201)\r\n\t\t\tat org.syncany.operations.down.DownOperation.persistDatabaseVersion(DownOperation.java:527)\r\n\t\t\tat org.syncany.operations.down.DownOperation.persistDatabaseVersions(DownOperation.java:514)\r\n\t\t\tat org.syncany.operations.down.DownOperation.applyChangesAndPersistDatabase(DownOperation.java:495)\r\n\t\t\tat org.syncany.operations.down.DownOperation.applyWinnersBranch(DownOperation.java:470)\r\n\t\t\tat org.syncany.operations.down.DownOperation.execute(DownOperation.java:190)\r\n\t\t\tat org.syncany.cli.DownCommand.execute(DownCommand.java:56)\r\n\t\t\tat org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:392)\r\n\t\t\tat org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:382)\r\n\t\t\tat org.syncany.cli.CommandLineClient.start(CommandLineClient.java:198)\r\n\t\t\tat org.syncany.Syncany.main(Syncany.java:52)\r\n\t29-12-15 1:39:26.345 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:26.345 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1191999,3681 operation 16231,0 txts 334271\r\n\t29-12-15 1:39:26.485 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:26.485 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1213316,3681 operation 21317,0 txts 334271\r\n\t29-12-15 1:39:26.626 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:26.626 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1238248,3681 operation 24932,0 txts 334271\r\n\t29-12-15 1:39:26.735 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:26.751 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1263071,3697 operation 24823,16 txts 334271\r\n\t29-12-15 1:39:26.844 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 15\r\n\t29-12-15 1:39:26.844 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1288005,3712 operation 24934,15 txts 334271\r\n\t29-12-15 1:39:27.000 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.000 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1312946,3712 operation 24941,0 txts 334271\r\n\t29-12-15 1:39:27.094 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.109 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1337893,3727 operation 24947,15 txts 334271\r\n\t29-12-15 1:39:27.219 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.234 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1362884,3742 operation 24991,15 txts 334271\r\n\t29-12-15 1:39:27.343 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.343 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1387872,3742 operation 24988,0 txts 334271\r\n\t29-12-15 1:39:27.453 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.468 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1412755,3757 operation 24883,15 txts 334271\r\n\t29-12-15 1:39:27.546 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.546 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1437732,3757 operation 24977,0 txts 334271\r\n\t29-12-15 1:39:27.640 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.655 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1462621,3772 operation 24889,15 txts 334271\r\n\t29-12-15 1:39:27.749 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.749 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1487594,3772 operation 24973,0 txts 334271\r\n\t29-12-15 1:39:27.843 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.858 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1512556,3787 operation 24962,15 txts 334271\r\n\t29-12-15 1:39:27.936 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:27.952 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1537418,3803 operation 24862,16 txts 334271\r\n\t29-12-15 1:39:28.030 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.045 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1562276,3818 operation 24858,15 txts 334271\r\n\t29-12-15 1:39:28.155 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.155 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1587178,3818 operation 24902,0 txts 334271\r\n\t29-12-15 1:39:28.233 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.248 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1612013,3833 operation 24835,15 txts 334271\r\n\t29-12-15 1:39:28.326 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.342 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1636849,3849 operation 24836,16 txts 334271\r\n\t29-12-15 1:39:28.451 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.451 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1661583,3849 operation 24734,0 txts 334271\r\n\t29-12-15 1:39:28.529 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.529 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1686558,3849 operation 24975,0 txts 334271\r\n\t29-12-15 1:39:28.654 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.654 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1711480,3849 operation 24922,0 txts 334271\r\n\t29-12-15 1:39:28.763 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.779 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1736230,3865 operation 24750,16 txts 334271\r\n\t29-12-15 1:39:28.903 | hsqldb.db.HSQLD | main           | FINE : copyShadow [size, time] 16396 0\r\n\t29-12-15 1:39:28.903 | hsqldb.db.HSQLD | main           | FINE : cache save rows [count,time] totals 1761129,3865 operation 24899,0 txts 334271\r\n\t29-12-15 1:39:29.200 | CommandLineClie | main           | SEVE : Command org.syncany.cli.DownCommand@7225790e FAILED.\r\n\tjava.lang.RuntimeException: Cannot persist database.\r\n\t\t\tat org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:118)\r\n\t\t\tat org.syncany.database.SqlDatabase.writeDatabaseVersion(SqlDatabase.java:201)\r\n\t\t\tat org.syncany.operations.down.DownOperation.persistDatabaseVersion(DownOperation.java:527)\r\n\t\t\tat org.syncany.operations.down.DownOperation.persistDatabaseVersions(DownOperation.java:514)\r\n\t\t\tat org.syncany.operations.down.DownOperation.applyChangesAndPersistDatabase(DownOperation.java:495)\r\n\t\t\tat org.syncany.operations.down.DownOperation.applyWinnersBranch(DownOperation.java:470)\r\n\t\t\tat org.syncany.operations.down.DownOperation.execute(DownOperation.java:190)\r\n\t\t\tat org.syncany.cli.DownCommand.execute(DownCommand.java:56)\r\n\t\t\tat org.syncany.cli.CommandLineClient.runLocally(CommandLineClient.java:392)\r\n\t\t\tat org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:382)\r\n\t\t\tat org.syncany.cli.CommandLineClient.start(CommandLineClient.java:198)\r\n\t\t\tat org.syncany.Syncany.main(Syncany.java:52)\r\n\tCaused by: java.sql.BatchUpdateException: java.lang.OutOfMemoryError: GC overhead limit exceeded\r\n\t\t\tat org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)\r\n\t\t\tat org.syncany.database.dao.ChunkSqlDao.writeChunks(ChunkSqlDao.java:69)\r\n\t\t\tat org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:146)\r\n\t\t\tat org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:108)\r\n\t\t\t... 11 more\r\n\tError: Cannot persist database.\r\n\t\t   Refer to help page using '--help'."}}