{"action":"created","comment":{"url":"https://api.github.com/repos/akka/akka/pulls/comments/36950509","id":36950509,"diff_hunk":"@@ -0,0 +1,386 @@\n+.. _persistence-schema-evolution-scala:\n+\n+##############################\n+Persistence - Schema Evolution\n+##############################\n+\n+When working on long running projects using :ref:`persistence-scala`, or any kind of `Event Sourcing`_ architectures,\n+schema evolution becomes one of the more important technical aspects of developing your application since both\n+requirements as well as our own understanding of the business domain may (and will) change in time.\n+\n+In fact, if a project matures to the point where you need to evolve its schema to adapt to changing business\n+requirements you can view this as first signs of its success – if you wouldn't need to adapt anything over an apps\n+lifecycle that could mean that no-one is really using it actively.\n+\n+In this chapter we will investigate various schema evolution strategies and techniques from which you can pick and\n+choose the ones that match your domain and challange at hand.\n+\n+.. note::\n+  This page proposes a number of possible solutions to the schema evolution problem and explains how some of the\n+  utilities Akka provides can be used to achieve this, it is by no means a complete (closed) set of solutions.\n+\n+  Sometimes based on the capabilities of your serialization formats you may be able to evolve your schema in\n+  different ways than outlined in the sections below. If you discover useful patterns or techniques for schema\n+  evolution feel free to submit Pull Requests to this page to extend it.\n+\n+\n+Schema evolution in event-sourced systems\n+=========================================\n+\n+In recent years we have observed a tremendous move towards immutable append-only datastores, with event-sourcing being\n+the prime technique successfully being used in these settings. For an excellent overview why and how immutable data makes scalability\n+and systems design much simpler you may want to read Pat Helland's excellent `Immutability Changes Everything`_ whitepaper.\n+\n+Since with `Event Sourcing`_ the **events are immutable** and usually never deleted – the way schema evolution is handled in such\n+systems differs from how one would go about it in a mutable database setting (e.g. in typical CRUD database applications).\n+The system needs to be able to continue to work in the presence of \"old\" events which were stored under the \"old\" schema,\n+and we want to limit the complexity to the data layer, exposing a consistent view over all of the events of a given type\n+to :class:`PersistentActor`` s and persistence queries, which allows these layers to focus on the business problems instead\n+handling the different schemas explicitly in the business logic layers.\n+\n+\n+In summary, schema evolution in event sourced systems exposes the following characteristics:\n+  - Allow the system to continue operating without large scale migrations to be applied,\n+  - Allow the system to read \"old\" events from the underlying storage, however present them in a \"new\" view to the application logic,\n+  - Transparently promote events to the latest versions during recovery (or queries) such that the business logic need not consider multiple versions of events\n+\n+.. _Immutability Changes Everything: http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf\n+.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html\n+\n+Picking the right serialization format\n+======================================\n+\n+Picking the serialization format is one of the most important decisions you will be making in setting up\n+your project. It affects which kind of evolutions are simple (or hard) to do, how much work is required to\n+add a new datatype and last but not least performance serialization.\n+\n+Provided default serializers\n+----------------------------\n+\n+Akka Persistence provides `Google Protocol Buffers`_ based serializers (using :ref:`Akka Serialization <serialization-scala>`)\n+for it's own message types such as ``PersistentRepr``, ``AtomicWrite`` and snapshots. Journal plugin implementations\n+*may* choose to use those provided serializers, or pick a serializer which suits the underlying database more.\n+\n+.. note::\n+  Serialization is **NOT** handled automatically by Akka Persistence itself. Instead, it only provides the above described\n+  serializers, and in case a ``AsyncWriteJournal`` plugin implementation chooses to use them directly, the above serialization\n+  scheme will be used.\n+\n+  Please refer to your write journal's documentation to learn more how it handles serialization!\n+\n+  For example, some journals may choose to not use Akka Serialization *at all* and instead store the data in a format\n+  that is more \"native\" for the underlying datastore, e.g. using JSON or some other kind of format that the target\n+  datastore understands natively.\n+\n+The below figure explains how the default serialization scheme works, and how it fits together with serializing the\n+user provided message itself, which we will from here on refer to as the ``payload`` (highlighted in yellow):\n+\n+.. figure:: ../images/persistent-message-envelope.png\n+   :align: center\n+\n+   Akka Persistence provided serializers wrap the user payload in an envelope containing all persistence-relevant information.\n+   **The payload will be serialized using the user configured serializer (if none is provided explicitly, Java serialization will be used).**\n+\n+The blue colored regions of the ``PersistentMessage`` indicate what is serialized using the generated protocol buffers\n+serializers, and the yellow payload indicates the user provided event (by calling ``persist(payload)(...)``).\n+As you can see, the ``PersistentMessage`` acts as an envelope around the payload, adding various fields related to the\n+origin of the event (``persistenceId``, ``sequenceNr`` and more).\n+\n+More advanced techniques (e.g. :ref:`remove-event-class-scala`) will dive into using the manifests for increasing the\n+flexibility of the persisted vs. exposed types even more, however for now we will focus on the simpler evolution techniques,\n+concerning simply configuring the payload serializers.\n+\n+By default the ``payload`` will be serialized using Java Serialization. This is fine for testing and initial phases\n+of your development (while you're still figuring out things and the data will not need to stay persisted forever),\n+however once you move to production you really should *pick a different serializer for your payloads*.\n+\n+.. warning::\n+  Do not rely on Java serialization (which will be picked by Akka by default if you don't specify any serializaers)\n+  for *serious* application development! It does not lean itself well to evolving schemas over long periods of time,\n+  and its performance is also not very high (it never was designed for high-throughput scenarios).\n+\n+.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/\n+.. _Apache Thrift: https://thrift.apache.org/\n+\n+Configuring payload serializers\n+-------------------------------\n+This section aims to highlight the complete basics on how to define custom serializers using :ref:`Akka Serialization <serialization-scala>`.\n+Many journal plugin implementations use Akka Serialization, thus it is tremendously important to understand how to configure\n+it to work with your event classes.\n+\n+.. note::\n+  Read the :ref:`Akka Serialization <serialization-scala>` docs to learn more about defining custom serializers,\n+  to improve performance and maintainability of your system. Do not depend on Java serialization for production deployments.\n+\n+The below snippet explains in the minimal amount of lines how a custom serializer can be registered.\n+For more in-depth explanations on how serialization picks the serializer to use etc, please refer to its documentation.\n+\n+First we start by defining our domain model class, here representing a person:\n+.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer-model\n+\n+Next we implement a serializer (or extend an existing one to be able to handle the new ``Person`` class):\n+\n+.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer\n+\n+And finally we register the serializer and bind it to handle the ``docs.persistence.Person`` class:\n+\n+.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer-config\n+\n+Deserialization will be performed by the same serializer which serialized the message initially\n+because of the ``identifier`` being stored together with the message.\n+\n+For more advanced techniques for using serializers refer to the :ref:`Akka Serialization <serialization-scala>` documentation,\n+giving the :ref:`string-manifest-serializer-scala` special attention, as it is tremendously useful for Persistence based applications\n+(as we will see in some of the below explained evolutions).\n+\n+\n+Serializing to human-readable documents\n+---------------------------------------\n+Some projects may prefer to store events in human-readable formats instead of formats optimised for write/read throughput.\n+The style of serialization used in Akka Persistence is completely left up to the concrete Journal plugin implementations -\n+this means that some community provided journals may offer human readable serialization, however it is not the default style\n+to persist events.\n+\n+Additionally some datastores are able to store \"documents\" as JSON (or BSON) natively, wich may have additional benefits","path":"akka-docs/rst/scala/persistence-schema-evolution.rst","position":144,"original_position":144,"commit_id":"6d471b5c8f544ce5e54d29ed5ca8f838c2d760ee","original_commit_id":"6d471b5c8f544ce5e54d29ed5ca8f838c2d760ee","user":{"login":"ktoso","id":120979,"avatar_url":"https://avatars.githubusercontent.com/u/120979?v=3","gravatar_id":"","url":"https://api.github.com/users/ktoso","html_url":"https://github.com/ktoso","followers_url":"https://api.github.com/users/ktoso/followers","following_url":"https://api.github.com/users/ktoso/following{/other_user}","gists_url":"https://api.github.com/users/ktoso/gists{/gist_id}","starred_url":"https://api.github.com/users/ktoso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktoso/subscriptions","organizations_url":"https://api.github.com/users/ktoso/orgs","repos_url":"https://api.github.com/users/ktoso/repos","events_url":"https://api.github.com/users/ktoso/events{/privacy}","received_events_url":"https://api.github.com/users/ktoso/received_events","type":"User","site_admin":false},"body":"thanks","created_at":"2015-08-13T08:24:10Z","updated_at":"2015-08-13T08:24:10Z","html_url":"https://github.com/akka/akka/pull/18084#discussion_r36950509","pull_request_url":"https://api.github.com/repos/akka/akka/pulls/18084","_links":{"self":{"href":"https://api.github.com/repos/akka/akka/pulls/comments/36950509"},"html":{"href":"https://github.com/akka/akka/pull/18084#discussion_r36950509"},"pull_request":{"href":"https://api.github.com/repos/akka/akka/pulls/18084"}}},"pull_request":{"url":"https://api.github.com/repos/akka/akka/pulls/18084","id":40998817,"html_url":"https://github.com/akka/akka/pull/18084","diff_url":"https://github.com/akka/akka/pull/18084.diff","patch_url":"https://github.com/akka/akka/pull/18084.patch","issue_url":"https://api.github.com/repos/akka/akka/issues/18084","number":18084,"state":"open","locked":false,"title":"+doc #17613 document persistence schema evolution","user":{"login":"ktoso","id":120979,"avatar_url":"https://avatars.githubusercontent.com/u/120979?v=3","gravatar_id":"","url":"https://api.github.com/users/ktoso","html_url":"https://github.com/ktoso","followers_url":"https://api.github.com/users/ktoso/followers","following_url":"https://api.github.com/users/ktoso/following{/other_user}","gists_url":"https://api.github.com/users/ktoso/gists{/gist_id}","starred_url":"https://api.github.com/users/ktoso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktoso/subscriptions","organizations_url":"https://api.github.com/users/ktoso/orgs","repos_url":"https://api.github.com/users/ktoso/repos","events_url":"https://api.github.com/users/ktoso/events{/privacy}","received_events_url":"https://api.github.com/users/ktoso/received_events","type":"User","site_admin":false},"body":"Will resolve https://github.com/akka/akka/issues/17613\r\n\r\nI had a section on \"rename event\" but it was \"either protobuf does it for you, or refer to deleting events\" will work a bit more on covering all cases, need a break from this document for a while though.","created_at":"2015-07-28T10:57:45Z","updated_at":"2015-08-13T08:24:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3cc2a8a8c3fd1eb239229ed91eecb9b7c16c3187","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/akka/akka/pulls/18084/commits","review_comments_url":"https://api.github.com/repos/akka/akka/pulls/18084/comments","review_comment_url":"https://api.github.com/repos/akka/akka/pulls/comments{/number}","comments_url":"https://api.github.com/repos/akka/akka/issues/18084/comments","statuses_url":"https://api.github.com/repos/akka/akka/statuses/6d471b5c8f544ce5e54d29ed5ca8f838c2d760ee","head":{"label":"ktoso:wip-schema-migration-omg-its-a-small-book-ktoso","ref":"wip-schema-migration-omg-its-a-small-book-ktoso","sha":"6d471b5c8f544ce5e54d29ed5ca8f838c2d760ee","user":{"login":"ktoso","id":120979,"avatar_url":"https://avatars.githubusercontent.com/u/120979?v=3","gravatar_id":"","url":"https://api.github.com/users/ktoso","html_url":"https://github.com/ktoso","followers_url":"https://api.github.com/users/ktoso/followers","following_url":"https://api.github.com/users/ktoso/following{/other_user}","gists_url":"https://api.github.com/users/ktoso/gists{/gist_id}","starred_url":"https://api.github.com/users/ktoso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktoso/subscriptions","organizations_url":"https://api.github.com/users/ktoso/orgs","repos_url":"https://api.github.com/users/ktoso/repos","events_url":"https://api.github.com/users/ktoso/events{/privacy}","received_events_url":"https://api.github.com/users/ktoso/received_events","type":"User","site_admin":false},"repo":{"id":4462204,"name":"akka","full_name":"ktoso/akka","owner":{"login":"ktoso","id":120979,"avatar_url":"https://avatars.githubusercontent.com/u/120979?v=3","gravatar_id":"","url":"https://api.github.com/users/ktoso","html_url":"https://github.com/ktoso","followers_url":"https://api.github.com/users/ktoso/followers","following_url":"https://api.github.com/users/ktoso/following{/other_user}","gists_url":"https://api.github.com/users/ktoso/gists{/gist_id}","starred_url":"https://api.github.com/users/ktoso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktoso/subscriptions","organizations_url":"https://api.github.com/users/ktoso/orgs","repos_url":"https://api.github.com/users/ktoso/repos","events_url":"https://api.github.com/users/ktoso/events{/privacy}","received_events_url":"https://api.github.com/users/ktoso/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ktoso/akka","description":"Akka Project","fork":true,"url":"https://api.github.com/repos/ktoso/akka","forks_url":"https://api.github.com/repos/ktoso/akka/forks","keys_url":"https://api.github.com/repos/ktoso/akka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ktoso/akka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ktoso/akka/teams","hooks_url":"https://api.github.com/repos/ktoso/akka/hooks","issue_events_url":"https://api.github.com/repos/ktoso/akka/issues/events{/number}","events_url":"https://api.github.com/repos/ktoso/akka/events","assignees_url":"https://api.github.com/repos/ktoso/akka/assignees{/user}","branches_url":"https://api.github.com/repos/ktoso/akka/branches{/branch}","tags_url":"https://api.github.com/repos/ktoso/akka/tags","blobs_url":"https://api.github.com/repos/ktoso/akka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ktoso/akka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ktoso/akka/git/refs{/sha}","trees_url":"https://api.github.com/repos/ktoso/akka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ktoso/akka/statuses/{sha}","languages_url":"https://api.github.com/repos/ktoso/akka/languages","stargazers_url":"https://api.github.com/repos/ktoso/akka/stargazers","contributors_url":"https://api.github.com/repos/ktoso/akka/contributors","subscribers_url":"https://api.github.com/repos/ktoso/akka/subscribers","subscription_url":"https://api.github.com/repos/ktoso/akka/subscription","commits_url":"https://api.github.com/repos/ktoso/akka/commits{/sha}","git_commits_url":"https://api.github.com/repos/ktoso/akka/git/commits{/sha}","comments_url":"https://api.github.com/repos/ktoso/akka/comments{/number}","issue_comment_url":"https://api.github.com/repos/ktoso/akka/issues/comments{/number}","contents_url":"https://api.github.com/repos/ktoso/akka/contents/{+path}","compare_url":"https://api.github.com/repos/ktoso/akka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ktoso/akka/merges","archive_url":"https://api.github.com/repos/ktoso/akka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ktoso/akka/downloads","issues_url":"https://api.github.com/repos/ktoso/akka/issues{/number}","pulls_url":"https://api.github.com/repos/ktoso/akka/pulls{/number}","milestones_url":"https://api.github.com/repos/ktoso/akka/milestones{/number}","notifications_url":"https://api.github.com/repos/ktoso/akka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ktoso/akka/labels{/name}","releases_url":"https://api.github.com/repos/ktoso/akka/releases{/id}","created_at":"2012-05-27T14:05:17Z","updated_at":"2015-04-03T11:58:56Z","pushed_at":"2015-08-12T16:18:47Z","git_url":"git://github.com/ktoso/akka.git","ssh_url":"git@github.com:ktoso/akka.git","clone_url":"https://github.com/ktoso/akka.git","svn_url":"https://github.com/ktoso/akka","homepage":"http://akka.io","size":169230,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"akka:master","ref":"master","sha":"629b0373120ee7302330d135222555b36056cc3e","user":{"login":"akka","id":1496237,"avatar_url":"https://avatars.githubusercontent.com/u/1496237?v=3","gravatar_id":"","url":"https://api.github.com/users/akka","html_url":"https://github.com/akka","followers_url":"https://api.github.com/users/akka/followers","following_url":"https://api.github.com/users/akka/following{/other_user}","gists_url":"https://api.github.com/users/akka/gists{/gist_id}","starred_url":"https://api.github.com/users/akka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akka/subscriptions","organizations_url":"https://api.github.com/users/akka/orgs","repos_url":"https://api.github.com/users/akka/repos","events_url":"https://api.github.com/users/akka/events{/privacy}","received_events_url":"https://api.github.com/users/akka/received_events","type":"Organization","site_admin":false},"repo":{"id":130013,"name":"akka","full_name":"akka/akka","owner":{"login":"akka","id":1496237,"avatar_url":"https://avatars.githubusercontent.com/u/1496237?v=3","gravatar_id":"","url":"https://api.github.com/users/akka","html_url":"https://github.com/akka","followers_url":"https://api.github.com/users/akka/followers","following_url":"https://api.github.com/users/akka/following{/other_user}","gists_url":"https://api.github.com/users/akka/gists{/gist_id}","starred_url":"https://api.github.com/users/akka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akka/subscriptions","organizations_url":"https://api.github.com/users/akka/orgs","repos_url":"https://api.github.com/users/akka/repos","events_url":"https://api.github.com/users/akka/events{/privacy}","received_events_url":"https://api.github.com/users/akka/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/akka/akka","description":"Akka Project","fork":false,"url":"https://api.github.com/repos/akka/akka","forks_url":"https://api.github.com/repos/akka/akka/forks","keys_url":"https://api.github.com/repos/akka/akka/keys{/key_id}","collaborators_url":"https://api.github.com/repos/akka/akka/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/akka/akka/teams","hooks_url":"https://api.github.com/repos/akka/akka/hooks","issue_events_url":"https://api.github.com/repos/akka/akka/issues/events{/number}","events_url":"https://api.github.com/repos/akka/akka/events","assignees_url":"https://api.github.com/repos/akka/akka/assignees{/user}","branches_url":"https://api.github.com/repos/akka/akka/branches{/branch}","tags_url":"https://api.github.com/repos/akka/akka/tags","blobs_url":"https://api.github.com/repos/akka/akka/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/akka/akka/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/akka/akka/git/refs{/sha}","trees_url":"https://api.github.com/repos/akka/akka/git/trees{/sha}","statuses_url":"https://api.github.com/repos/akka/akka/statuses/{sha}","languages_url":"https://api.github.com/repos/akka/akka/languages","stargazers_url":"https://api.github.com/repos/akka/akka/stargazers","contributors_url":"https://api.github.com/repos/akka/akka/contributors","subscribers_url":"https://api.github.com/repos/akka/akka/subscribers","subscription_url":"https://api.github.com/repos/akka/akka/subscription","commits_url":"https://api.github.com/repos/akka/akka/commits{/sha}","git_commits_url":"https://api.github.com/repos/akka/akka/git/commits{/sha}","comments_url":"https://api.github.com/repos/akka/akka/comments{/number}","issue_comment_url":"https://api.github.com/repos/akka/akka/issues/comments{/number}","contents_url":"https://api.github.com/repos/akka/akka/contents/{+path}","compare_url":"https://api.github.com/repos/akka/akka/compare/{base}...{head}","merges_url":"https://api.github.com/repos/akka/akka/merges","archive_url":"https://api.github.com/repos/akka/akka/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/akka/akka/downloads","issues_url":"https://api.github.com/repos/akka/akka/issues{/number}","pulls_url":"https://api.github.com/repos/akka/akka/pulls{/number}","milestones_url":"https://api.github.com/repos/akka/akka/milestones{/number}","notifications_url":"https://api.github.com/repos/akka/akka/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/akka/akka/labels{/name}","releases_url":"https://api.github.com/repos/akka/akka/releases{/id}","created_at":"2009-02-16T12:51:54Z","updated_at":"2015-08-13T01:29:15Z","pushed_at":"2015-08-13T08:15:27Z","git_url":"git://github.com/akka/akka.git","ssh_url":"git@github.com:akka/akka.git","clone_url":"https://github.com/akka/akka.git","svn_url":"https://github.com/akka/akka","homepage":"http://akka.io","size":266818,"stargazers_count":4313,"watchers_count":4313,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1265,"mirror_url":null,"open_issues_count":734,"forks":1265,"open_issues":734,"watchers":4313,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/akka/akka/pulls/18084"},"html":{"href":"https://github.com/akka/akka/pull/18084"},"issue":{"href":"https://api.github.com/repos/akka/akka/issues/18084"},"comments":{"href":"https://api.github.com/repos/akka/akka/issues/18084/comments"},"review_comments":{"href":"https://api.github.com/repos/akka/akka/pulls/18084/comments"},"review_comment":{"href":"https://api.github.com/repos/akka/akka/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/akka/akka/pulls/18084/commits"},"statuses":{"href":"https://api.github.com/repos/akka/akka/statuses/6d471b5c8f544ce5e54d29ed5ca8f838c2d760ee"}}}}