{"action":"created","issue":{"url":"https://api.github.com/repos/micropython/micropython/issues/1577","labels_url":"https://api.github.com/repos/micropython/micropython/issues/1577/labels{/name}","comments_url":"https://api.github.com/repos/micropython/micropython/issues/1577/comments","events_url":"https://api.github.com/repos/micropython/micropython/issues/1577/events","html_url":"https://github.com/micropython/micropython/pull/1577","id":114699298,"number":1577,"title":"Persistent bytecode v2","user":{"login":"dpgeorge","id":6187689,"avatar_url":"https://avatars.githubusercontent.com/u/6187689?v=3","gravatar_id":"","url":"https://api.github.com/users/dpgeorge","html_url":"https://github.com/dpgeorge","followers_url":"https://api.github.com/users/dpgeorge/followers","following_url":"https://api.github.com/users/dpgeorge/following{/other_user}","gists_url":"https://api.github.com/users/dpgeorge/gists{/gist_id}","starred_url":"https://api.github.com/users/dpgeorge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpgeorge/subscriptions","organizations_url":"https://api.github.com/users/dpgeorge/orgs","repos_url":"https://api.github.com/users/dpgeorge/repos","events_url":"https://api.github.com/users/dpgeorge/events{/privacy}","received_events_url":"https://api.github.com/users/dpgeorge/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-11-02T22:33:11Z","updated_at":"2015-11-04T12:28:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/micropython/micropython/pulls/1577","html_url":"https://github.com/micropython/micropython/pull/1577","diff_url":"https://github.com/micropython/micropython/pull/1577.diff","patch_url":"https://github.com/micropython/micropython/pull/1577.patch"},"body":"This is an updated version of #1527 and provides support for portable/persistent bytecode.  It has the following features:\r\n\r\n- Config variables MICROPY_PORTABLE_CODE and MICROPY_PORTABLE_CODE_SAVE.\r\n\r\n- Without any of the configs enabled (the default) the code/ROM size and RAM usage of bytecode is almost the same as before.  There has been some code refactoring and shuffling, but conceptually everything is the same.\r\n\r\n- With MICROPY_PORTABLE_CODE enabled the bytecode changes: qstrs are stored within the bytecode but with a fixed 2-byte encoding (should be plenty for all use cases) and pointers are stored in a constant table outside the bytecode.  There is only a small increase in RAM usage with this option enabled (maybe 1-3% increase).\r\n\r\n- MICROPY_PORTABLE_CODE costs about 1060 bytes ROM for stmhal.\r\n\r\n- With MICROPY_PORTABLE_CODE_SAVE enabled, bytecode can be saved to a .mpc file.  This is portable code that can be loaded and linked (at compile or run time) into any uPy instance.\r\n\r\n- With MICROPY_PORTABLE_CODE enabled, one can load and import .mpc files and execute them.  No lexer/parser/compiler is needed.  But the bytecode must be loaded into RAM because the qstrs need to be linked (ie point to the global qstr table of the uPy instance).\r\n\r\n- There is a script called tools/mpcdump.py which can read a .mpc file and convert it into C source, which can then be compiled with your program to create a properly frozen Python script.  These frozen scripts can be executed directly from ROM and include all constant objects in ROM as well (eg bignum, float, tuple).  RAM is only needed when executing (eg to populate the globals dict of the module).\r\n\r\nSo we can now do the following:\r\n\r\n- Have proper frozen module support that requires no compiling and no RAM to store the compiled bytecode.  This means we could potentially start to rewrite parts of uPy in Python itself, if we wanted.\r\n\r\n- Provide an option to disable the parser/lexer/compiler to save a lot (roughly 20k) of code size and just embed frozen scripts and execute those (no REPL though!).  This would give complete Python applications in under 64k (Thumb2 arch).\r\n\r\n- Compile scripts offline, upload the .mpc files to the filesystem of pyboard/wipy/etc and execute them (import them) directly.  This would save time during bootup of the board (no compiling needed) and reduce RAM pressure of compilation stage.  Would be useful for drivers (eg onewire.mpc).\r\n\r\nThings that are missing:\r\n\r\n- The ability to add persistent bytecode to a uPy executable after it has been linked, and execute that bytecode from ROM.  Use case would be microbit: we have a given firmware and want to append the user's compiled code to the firmware so it can be executed without using much RAM.\r\n\r\n- Ability to make native/viper code persistent.  This could be done, and would be nice to do, and would share a lot of code with dynamic loadable native C modules.\r\n\r\nThe one issue with .mpc files is that they are not 100% portable: bytecode differs if MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE is enabled or not.  This is annoying because that feature is enabled on unix port to get a big speed boost, but not enabled on any other port (because it increases RAM usage of bytecode, and makes the VM non-deterministic in terms of execution speed).  It means you need a different build of unix binary to compile code for pyboard/wipy/etc.\r\n"},"comment":{"url":"https://api.github.com/repos/micropython/micropython/issues/comments/153705647","html_url":"https://github.com/micropython/micropython/pull/1577#issuecomment-153705647","issue_url":"https://api.github.com/repos/micropython/micropython/issues/1577","id":153705647,"user":{"login":"danicampora","id":7749335,"avatar_url":"https://avatars.githubusercontent.com/u/7749335?v=3","gravatar_id":"","url":"https://api.github.com/users/danicampora","html_url":"https://github.com/danicampora","followers_url":"https://api.github.com/users/danicampora/followers","following_url":"https://api.github.com/users/danicampora/following{/other_user}","gists_url":"https://api.github.com/users/danicampora/gists{/gist_id}","starred_url":"https://api.github.com/users/danicampora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danicampora/subscriptions","organizations_url":"https://api.github.com/users/danicampora/orgs","repos_url":"https://api.github.com/users/danicampora/repos","events_url":"https://api.github.com/users/danicampora/events{/privacy}","received_events_url":"https://api.github.com/users/danicampora/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T12:28:52Z","updated_at":"2015-11-04T12:28:52Z","body":"Wow, everything sounds super awesome to me :-)"}}