{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/815","labels_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/815/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/815/comments","events_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/815/events","html_url":"https://github.com/aws/aws-sdk-ruby/issues/815","id":76445902,"number":815,"title":"Handling DynamoDB \"unprocessed_items\" from batch_write_item is too difficult","user":{"login":"smeyfroi","id":203996,"avatar_url":"https://avatars.githubusercontent.com/u/203996?v=3","gravatar_id":"","url":"https://api.github.com/users/smeyfroi","html_url":"https://github.com/smeyfroi","followers_url":"https://api.github.com/users/smeyfroi/followers","following_url":"https://api.github.com/users/smeyfroi/following{/other_user}","gists_url":"https://api.github.com/users/smeyfroi/gists{/gist_id}","starred_url":"https://api.github.com/users/smeyfroi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smeyfroi/subscriptions","organizations_url":"https://api.github.com/users/smeyfroi/orgs","repos_url":"https://api.github.com/users/smeyfroi/repos","events_url":"https://api.github.com/users/smeyfroi/events{/privacy}","received_events_url":"https://api.github.com/users/smeyfroi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-05-14T18:24:20Z","updated_at":"2015-05-15T08:16:06Z","closed_at":null,"body":"DynamoDB batch_write_item API can fail and return \"unprocessed_items\". The docs say that the format of this returned data can be used in a subsequent batch_write_item, but that's not quite true. You can't simply do something like ```client.batch_write_item(request_items: response.unprocessed_items)``` because the response is not a hash as expected in the params.\r\n\r\nWhat I've ended up doing is something like\r\n```ruby\r\nclient.batch_write_item(request_items: unprocessed_items_to_request_items(response.unprocessed_items))\r\n```\r\n\r\nwhere that helper method munges the response structure like this (```deep_transform_keys``` comes from ActiveSupport):\r\n\r\n```ruby\r\n    def self.unprocessed_items_to_request_items(unprocessed_items)\r\n      unprocessed_items.as_json.deep_transform_keys do |key|\r\n        if %w{ item put_request n s b ss ns bs m l null bool delete_request }.include? key\r\n          key.to_sym\r\n        else\r\n          key\r\n        end\r\n      end\r\n    end\r\n```\r\n\r\n(This is required because the table_name key to the hash supplied to put_request needs to be a string. There may be other cases that I've not encountered yet.)\r\n\r\nIt would be quite good not to require dirty hacks like this, if possible. :-)\r\n\r\n(This is SDKv2 by the way)"},"comment":{"url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/comments/102313089","html_url":"https://github.com/aws/aws-sdk-ruby/issues/815#issuecomment-102313089","issue_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/815","id":102313089,"user":{"login":"smeyfroi","id":203996,"avatar_url":"https://avatars.githubusercontent.com/u/203996?v=3","gravatar_id":"","url":"https://api.github.com/users/smeyfroi","html_url":"https://github.com/smeyfroi","followers_url":"https://api.github.com/users/smeyfroi/followers","following_url":"https://api.github.com/users/smeyfroi/following{/other_user}","gists_url":"https://api.github.com/users/smeyfroi/gists{/gist_id}","starred_url":"https://api.github.com/users/smeyfroi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smeyfroi/subscriptions","organizations_url":"https://api.github.com/users/smeyfroi/orgs","repos_url":"https://api.github.com/users/smeyfroi/repos","events_url":"https://api.github.com/users/smeyfroi/events{/privacy}","received_events_url":"https://api.github.com/users/smeyfroi/received_events","type":"User","site_admin":false},"created_at":"2015-05-15T08:16:06Z","updated_at":"2015-05-15T08:16:06Z","body":"Hmmm that's odd. OK here's the error, which includes output from the code below (excuse the mess: you can see the working version in the stuff that's commented out):\r\n\r\n```ruby\r\n    # Note AWS API limit of 25 puts per request so we batch the put\r\n    # and add a delay between batches\r\n    # TODO: limit of 1MB per request in total not enforced here\r\n    def self.put_items(table_name, items)\r\n      __start_time = Time.now if DEBUG\r\n      in_batches_of_25(items) do |items_batch, last_iteration|\r\n        put_requests = items_batch.map do |item|\r\n          {\r\n            put_request: {\r\n              item: item\r\n            }\r\n          }\r\n        end\r\n        request_items = {\r\n          table_name => put_requests\r\n        }\r\n        loop do\r\n          response = client.batch_write_item(request_items: request_items)\r\n          sleep NICENESS_PAUSE unless last_iteration\r\n\r\n          break if response.unprocessed_items.size == 0\r\nrequest_items = response.unprocessed_items\r\n#          request_items = unprocessed_items_to_request_items(response.unprocessed_items)\r\nRails.logger.info(\"request_items\")\r\nRails.logger.info(request_items)\r\n\r\n          sleep NICENESS_PAUSE  * 2  # sleep again: we're not managing to store all items\r\n        end\r\n      end\r\n    ensure\r\n      ::Rails.logger.debug(\"put_items #{table_name} (#{items.length}): #{(Time.now() - __start_time)*1000}ms\") if DEBUG\r\n    end\r\n\r\n#    def self.unprocessed_items_to_request_items(unprocessed_items)\r\n#      unprocessed_items.as_json.deep_transform_keys do |key|\r\n#        if %w{ item put_request n s b ss ns bs m l null bool delete_request }.include? key\r\n#          key.to_sym\r\n#        else\r\n#          key\r\n#        end\r\n#      end\r\n#    end\r\n```\r\n\r\n```\r\n2015-05-15T08:03:48.914Z 23049 TID-ox45wrunc CreateHistoricGroupMetricsWorker JID-25a46a9675cc7fc03ec44445 INFO: request_items\r\n2015-05-15T08:03:48.914Z 23049 TID-ox45wrunc CreateHistoricGroupMetricsWorker JID-25a46a9675cc7fc03ec44445 INFO: {\"test_cc_group_metrics\"=>[#<struct put_request=#<struct item={\"timestamp\"=>#<struct s=nil, n=\"1431406800000.0\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"_class\"=>#<struct s=\"GroupMetric\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"interval\"=>#<struct s=nil, n=\"900\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id\"=>#<struct s=nil, n=\"73\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id_interval\"=>#<struct s=\"73:900\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"organisation_id\"=>#<struct s=nil, n=\"4\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"config_resource_statistics\"=>#<struct s=\"{\\\"1\\\":{\\\"probe\\\":{\\\"l\\\":{\\\"Connect\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":58},\\\"FirstByte\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":216},\\\"Transfer\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":1},\\\"Total\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":275}},\\\"h\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":99.979},\\\"u\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":100.0},\\\"s\\\":{\\\"2XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"3XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"4XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"5XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null}}}}}\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>}>, delete_request=nil>, #<struct put_request=#<struct item={\"timestamp\"=>#<struct s=nil, n=\"1431407700000.0\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"_class\"=>#<struct s=\"GroupMetric\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"interval\"=>#<struct s=nil, n=\"900\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id\"=>#<struct s=nil, n=\"73\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id_interval\"=>#<struct s=\"73:900\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"organisation_id\"=>#<struct s=nil, n=\"4\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"config_resource_statistics\"=>#<struct s=\"{\\\"1\\\":{\\\"probe\\\":{\\\"l\\\":{\\\"Connect\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":61},\\\"FirstByte\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":224},\\\"Transfer\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":0},\\\"Total\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":285}},\\\"h\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":99.983},\\\"u\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":100.0},\\\"s\\\":{\\\"2XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"3XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"4XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"5XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null}}}}}\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>}>, delete_request=nil>, #<struct put_request=#<struct item={\"timestamp\"=>#<struct s=nil, n=\"1431408600000.0\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"_class\"=>#<struct s=\"GroupMetric\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"interval\"=>#<struct s=nil, n=\"900\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id\"=>#<struct s=nil, n=\"73\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"group_id_interval\"=>#<struct s=\"73:900\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"organisation_id\"=>#<struct s=nil, n=\"4\", b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>, \"config_resource_statistics\"=>#<struct s=\"{\\\"1\\\":{\\\"probe\\\":{\\\"l\\\":{\\\"Connect\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":56},\\\"FirstByte\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":228},\\\"Transfer\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":0},\\\"Total\\\":{\\\"Unit\\\":\\\"ms\\\",\\\"Avg\\\":284}},\\\"h\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":99.98699999999999},\\\"u\\\":{\\\"Unit\\\":\\\"%\\\",\\\"Avg\\\":100.0},\\\"s\\\":{\\\"2XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"3XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"4XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null},\\\"5XX\\\":{\\\"Unit\\\":\\\"\\\",\\\"Avg\\\":null}}}}}\", n=nil, b=nil, ss=nil, ns=nil, bs=nil, m=nil, l=nil, null=nil, bool=nil>}>, delete_request=nil>]}\r\n2015-05-15T08:03:49.931Z 23049 TID-ox45wrunc CreateHistoricGroupMetricsWorker JID-25a46a9675cc7fc03ec44445 ERROR: parameter validator found 3 errors:\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][0] to be a hash\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][1] to be a hash\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][2] to be a hash\r\n2015-05-15T08:03:49.931Z 23049 TID-ox45wrunc CreateHistoricGroupMetricsWorker JID-25a46a9675cc7fc03ec44445 ERROR: parameter validator found 3 errors:\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][0] to be a hash\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][1] to be a hash\r\n  - expected params[:request_items][\"test_cc_group_metrics\"][2] to be a hash\r\n2015-05-15T08:03:49.931Z 23049 TID-ox45wrunc CreateHistoricGroupMetricsWorker JID-25a46a9675cc7fc03ec44445 ERROR: /geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/param_validator.rb:24:in `validate!'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/param_validator.rb:9:in `validate!'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/plugins/param_validation.rb:21:in `call'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/plugins/response_target.rb:18:in `call'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/request.rb:70:in `send_request'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.0.41/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/lib/dynamodb/driver.rb:73:in `block (2 levels) in put_items'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/lib/dynamodb/driver.rb:72:in `loop'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/lib/dynamodb/driver.rb:72:in `block in put_items'\r\n/geome/releases/93ebf9aea0a4de93b442d41170a42a23877ec91e/lib/dynamodb/driver.rb:52:in `block in in_batches_of_25'\r\n```"}}