{"action":"created","issue":{"url":"https://api.github.com/repos/Mindwerks/worldengine/issues/138","labels_url":"https://api.github.com/repos/Mindwerks/worldengine/issues/138/labels{/name}","comments_url":"https://api.github.com/repos/Mindwerks/worldengine/issues/138/comments","events_url":"https://api.github.com/repos/Mindwerks/worldengine/issues/138/events","html_url":"https://github.com/Mindwerks/worldengine/pull/138","id":111890017,"number":138,"title":"Make use of Numpy-Arrays","user":{"login":"tcld","id":12257447,"avatar_url":"https://avatars.githubusercontent.com/u/12257447?v=3","gravatar_id":"","url":"https://api.github.com/users/tcld","html_url":"https://github.com/tcld","followers_url":"https://api.github.com/users/tcld/followers","following_url":"https://api.github.com/users/tcld/following{/other_user}","gists_url":"https://api.github.com/users/tcld/gists{/gist_id}","starred_url":"https://api.github.com/users/tcld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tcld/subscriptions","organizations_url":"https://api.github.com/users/tcld/orgs","repos_url":"https://api.github.com/users/tcld/repos","events_url":"https://api.github.com/users/tcld/events{/privacy}","received_events_url":"https://api.github.com/users/tcld/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-10-16T18:49:54Z","updated_at":"2015-10-18T20:00:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Mindwerks/worldengine/pulls/138","html_url":"https://github.com/Mindwerks/worldengine/pull/138","diff_url":"https://github.com/Mindwerks/worldengine/pull/138.diff","patch_url":"https://github.com/Mindwerks/worldengine/pull/138.patch"},"body":"Another branch, as promised (https://github.com/Mindwerks/worldengine/issues/136). I replaced almost all arrays with numpy-arrays. Biomes are not (yet) done and everything that is *not* mentioned in the serialization-tests is probably not touched, either. (I had to start somewhere.)\r\n\r\nHere some performance-test results:\r\n\r\nParameters: \r\n```\r\n  width        = 384\r\n  height       = 256\r\n  plates       =   5\r\n  oceansurface = 65% (only effective in the numpy_repl-branch, which is based on ocean_level_fix)\r\n  seed         = 13047\r\n```\r\n\r\nGeneration times for every branch:\r\n```\r\n  master     = 39.57 s\r\n  perf_opt   = 29.22 s\r\n  numpy_repl = 28.40 s\r\n  (numpy_repl + platec1.4.0 = 21.48 s)\r\n  (numpy_repl + platec1.4.0+ small improvements = 20.44 s)\r\n```\r\n\r\nImages (master - perf_opt - numpy_repl - platec1.4.0):\r\n![seed_13047_biome](https://cloud.githubusercontent.com/assets/12257447/10550427/29988688-7447-11e5-98b9-6ad1b2e3490a.png)\r\n![seed_13047_biome](https://cloud.githubusercontent.com/assets/12257447/10550451/45c0ce06-7447-11e5-88cb-5eb4eff38e14.png)\r\n![seed_13047_biome](https://cloud.githubusercontent.com/assets/12257447/10550456/49513da8-7447-11e5-87c1-b75cc60e6249.png)\r\n![seed_13047_biome](https://cloud.githubusercontent.com/assets/12257447/10550761/fdbcbfa0-7448-11e5-8b2b-740931c7c2fa.png)\r\n\r\nThe latest changes were not supposed to speed things up too much, only some of the excessively used functions were replaced. I did what I could while keeping an eye on the target of getting numpy ready. In the future it won't be a pain to get a bit of optimization done anymore. And maybe the memory footprint will have changed a bit, I still have no way of reliably checking that.\r\nI will do some tests with larger maps and see if anything changed there. (And I am really looking forward to the new version of platec. :) )\r\n\r\nAll tests except two are passed, although I did run one of the regeneration-scripts in the tests-folder to make it happen. At this point, especially due to the change to the seeding, worldengine-data should be updated a bit anyway, I think. (One of the failed tests, maybe even both, can probably be passed with fresh data - maybe it has to be Python 2/3-specific, though.)\r\n\r\nEDIT: I took a look at the new platec version and added the time above. In total we are down to roughly 50% of the generation time from a couple of days ago! :)\r\n\r\nEDIT2: As far as I see, the failed test comes from the fact that the old world-file was generated without numpy-arrays. I wouldn't know what to do about that - on my system regenerating the test-world was enough, though.\r\n\r\nEDIT3: I did a little more profiling this evening. The air gets thin now, there are only three methods left that slightly justify spending time on with optimization (world.is_land()~11% of total time, world.tiles_around()~20%, common.anti_alias_point()~20% - that's roughly 50% in total!). And after looking at them for a while and trying some things I really couldn't come up with any improvement.\r\nA funny sidenote is that the function calls for my testmap went done from more than 300 million to less than 100 million. Half of which is attributed to is_land(). Speed-wise this might be the end of the road, unless a lot more work is invested.\r\n\r\n```\r\n         95833445 function calls (95705636 primitive calls) in 92.104 seconds\r\n\r\n   Ordered by: internal time\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n  4014080   19.543    0.000   19.543    0.000 worldengine/../worldengine/common.py:97(anti_alias_point)\r\n  1262482   18.308    0.000   27.963    0.000 worldengine/../worldengine/world.py:375(tiles_around)\r\n     1150   14.349    0.012   14.349    0.012 {built-in method step} -- this refers to platecs step()-method\r\n 56451705   10.417    0.000   10.417    0.000 worldengine/../worldengine/world.py:341(is_land)\r\n        1    2.312    2.312    4.155    4.155 worldengine/../worldengine/simulations/erosion.py:136(river_sources)\r\n```"},"comment":{"url":"https://api.github.com/repos/Mindwerks/worldengine/issues/comments/149042318","html_url":"https://github.com/Mindwerks/worldengine/pull/138#issuecomment-149042318","issue_url":"https://api.github.com/repos/Mindwerks/worldengine/issues/138","id":149042318,"user":{"login":"psi29a","id":1122069,"avatar_url":"https://avatars.githubusercontent.com/u/1122069?v=3","gravatar_id":"","url":"https://api.github.com/users/psi29a","html_url":"https://github.com/psi29a","followers_url":"https://api.github.com/users/psi29a/followers","following_url":"https://api.github.com/users/psi29a/following{/other_user}","gists_url":"https://api.github.com/users/psi29a/gists{/gist_id}","starred_url":"https://api.github.com/users/psi29a/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/psi29a/subscriptions","organizations_url":"https://api.github.com/users/psi29a/orgs","repos_url":"https://api.github.com/users/psi29a/repos","events_url":"https://api.github.com/users/psi29a/events{/privacy}","received_events_url":"https://api.github.com/users/psi29a/received_events","type":"User","site_admin":false},"created_at":"2015-10-18T20:00:12Z","updated_at":"2015-10-18T20:00:12Z","body":"```\r\n  File \"/home/travis/build/Mindwerks/worldengine/worldengine/world.py\", line 483, in elevation_at\r\n    return self.elevation['data'].T[pos]\r\nAttributeError: 'list' object has no attribute 'T'\r\n```\r\n\r\nIs this a typo?"}}