{"push_id":738760196,"size":101,"distinct_size":101,"ref":"refs/heads/hummingbird_production","head":"585fa6239d0f0064f07f89bc7f032291e92a0b5b","before":"111316f8b89757ec6061741f9e9007c136104425","commits":[{"sha":"aa4866eb6014d7fbb8531eb5d927d5611ebfd06d","author":{"email":"f23d94b2a735b6193d3882ae31308d2e41e8282b@redhat.com","name":"Thiago da Silva"},"message":"move replication code to ReplicatedObjectController\n\nMoving _connect_put_node, send_file, _transfer_data and _store_object\nmethods to ReplicatedObjectController. Each one of these methods are\nspecific to replication policy.  EC policy implements their own\nversion of them.\n\nOf these four methods, only _store_object and _connect_put_node are\nrequired to be implemented by the policy specific Object Controllers.\n\nChange-Id: Ifc72461b77dbfdaae9d63417f1286e8b5da3ca4e\nSigned-off-by: Thiago da Silva <thiago@redhat.com>","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/aa4866eb6014d7fbb8531eb5d927d5611ebfd06d"},{"sha":"e54781a2aa29808aff654861d8a0aafd24b6620c","author":{"email":"f23d94b2a735b6193d3882ae31308d2e41e8282b@redhat.com","name":"Thiago da Silva"},"message":"add object post and delete methods to BaseObjectController\n\nAdding post and delete methods to BaseObjectController that can\nbe overriden by ObjectCOntroller subclasses. These methods are\nsimilar to the PUT and GET methods that were introduced as part\nof the EC work\n\nChange-Id: I197364bc3e2f2287c0afc8948863e3cdeab90383\nSigned-off-by: Thiago da Silva <thiago@redhat.com>","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/e54781a2aa29808aff654861d8a0aafd24b6620c"},{"sha":"4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10","author":{"email":"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@swiftstack.com","name":"Samuel Merritt"},"message":"EC: support multiple ranges for GET requests\n\nThis commit lets clients receive multipart/byteranges responses (see\nRFC 7233, Appendix A) for erasure-coded objects. Clients can already\ndo this for replicated objects, so this brings EC closer to feature\nparity (ha!).\n\nGetOrHeadHandler got a base class extracted from it that treats an\nHTTP response as a sequence of byte-range responses. This way, it can\ncontinue to yield whole fragments, not just N-byte pieces of the raw\nHTTP response, since an N-byte piece of a multipart/byteranges\nresponse is pretty much useless.\n\nThere are a couple of bonus fixes in here, too. For starters, download\nresuming now works on multipart/byteranges responses. Before, it only\nworked on 200 responses or 206 responses for a single byte\nrange. Also, BufferedHTTPResponse grew a readline() method.\n\nAlso, the MIME response for replicated objects got tightened up a\nlittle. Before, it had some leading and trailing CRLFs which, while\nallowed by RFC 7233, provide no benefit. Now, both replicated and EC\nmultipart/byteranges avoid extraneous bytes. This let me re-use the\nContent-Length calculation in swob instead of having to either hack\naround it or add extraneous whitespace to match.\n\nChange-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10"},{"sha":"af734b3fb6fdbc11b8a0b775256e7642eb485b8d","author":{"email":"c42914e1d5908882129e35da47936a272424299c@hp.com","name":"Mark Seger"},"message":"Change usage help and Attention messages to warnings\n\nChange-Id: I1396aaffe36e739606f15f7fef37b11bd83f1fc1","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/af734b3fb6fdbc11b8a0b775256e7642eb485b8d"},{"sha":"0cf95b055549fd76aba38ea682ad4cc0a74de7fa","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"EC: support multiple ranges for GET requests\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/0cf95b055549fd76aba38ea682ad4cc0a74de7fa"},{"sha":"93de6c73e587a672663fa0997f439b73b336769c","author":{"email":"e49e4e527f4cd5c299a75e1c8e0b181aeadfc549@jp.fujitsu.com","name":"Hisashi Osanai"},"message":"Add metadata size tests on the border\n\nThis patch adds a test for a 204 when a single metadata item in a POST causes\nthe backend aggregate constraints check to be on the border.\n\nBackground:\nOverall metadata size constraint is enforced in the container and\naccount backends as well as in the proxy controllers.\nWhereas the proxy controller can check that constraints are not exceeded\nby a single PUT or POST request, the backend checks that constraints are not\nexceeded by the aggregate of all PUTs and POSTs.\n\nThe change [1] added a test for a 400 when a single metadata item in\na POST causes the backend aggregate constraints check to go over limit.\n\n[1] I1489e29686013cbd3d70283d8756b548aea3c2e1\n\nChange-Id: Iac86ea71240ddde177e625c279c21aef67659d10","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/93de6c73e587a672663fa0997f439b73b336769c"},{"sha":"c953e8cc9049e94cd01239d5e4a30a56f26a97d3","author":{"email":"48a255bf75f0d65fa07d8f658bfa862c57b1124b@us.ibm.com","name":"Minwoo Bae"},"message":"The hash_cleanup_listdir function should only be called when necessary.\n\nIntroduced a conditional that will specify to _finalize_put() whether\nthe put() method is being called within the context of ECDiskFileWriter.\n\nIncluded some unit tests that will test the number of calls to\nhash_cleanup_listdir() during put(), commit(), and delete()\noperations.\n\nChange-Id: I287b2648591fdbf686898dc9047beda80a64da0c","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/c953e8cc9049e94cd01239d5e4a30a56f26a97d3"},{"sha":"11e5c4adf0e02c8fdab0d8584fc943d96e1165cd","author":{"email":"110f47070ba3ccc2d79e32307817c523b8a06186@cloudvps.com","name":"Koert van der Veer"},"message":"Allow default reseller prefix in domain_remap middleware\n\nPreviously, the reseller prefix needed to be provided in the host name\neven when the domain was unique to that reseller. With the\ndefault_reseller_prefix, any domain which matches in this middleware,\nwill will be passed on with a reseller prefix, whether or not it was\nprovided.\n\nChange-Id: I5aa5ce78ad1ee2e3660cce4c3e07306f8999f02a\nImplements: blueprint domainremap-reseller-domains","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/11e5c4adf0e02c8fdab0d8584fc943d96e1165cd"},{"sha":"037a0c5dbe01a57b741d34988e21f4f763b6204b","author":{"email":"bbc046deb6b230bf2fb4644ac0f266475e036fa6@gmail.com","name":"janonymous"},"message":"Object are sorted and concatenated by swift in cardinal order.\n\nModified values in example in decimal to be more precise,\nadded a small description on Object are sorted and concatenated\nby swift in cardinal order as a small phrase would be sufficient.\n\nChange-Id: I914ded4e5726e50bb93b05759c3bfb76edda53ab\nbackport: none\nCloses-Bug: #1383893","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/037a0c5dbe01a57b741d34988e21f4f763b6204b"},{"sha":"5ad369c1cb0f480b164a5b5e7eb194081835c266","author":{"email":"59dfe7e004393a2f89ad78e3a810b4cb6d0cd637@redhat.com","name":"Victor Stinner"},"message":"Add six requirement\n\nThe six module is needed to add Python 3 support to Swift.\n\nChange-Id: Ie637ed3458c7ff56c26834bca73203ed55604d74","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/5ad369c1cb0f480b164a5b5e7eb194081835c266"},{"sha":"8c954f9e7e4944420fc385759b4095b0af150abd","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"The hash_cleanup_listdir function should only be called when necessary.\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/8c954f9e7e4944420fc385759b4095b0af150abd"},{"sha":"47b611e1d03fed5474a2436a14bbdf3dcca5b331","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"Change usage help and Attention messages to warnings\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/47b611e1d03fed5474a2436a14bbdf3dcca5b331"},{"sha":"0a1a447b0198c3192810ab45b32fa41a27bd22bd","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"Add metadata size tests on the border\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/0a1a447b0198c3192810ab45b32fa41a27bd22bd"},{"sha":"0e22371cd0f9559277ce99d62b43ae0dd8a272fb","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"Add six requirement\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/0e22371cd0f9559277ce99d62b43ae0dd8a272fb"},{"sha":"09e7477a391e7425af62a6516f43c69bc98b7080","author":{"email":"bbc046deb6b230bf2fb4644ac0f266475e036fa6@gmail.com","name":"janonymous"},"message":"Replace it.next() with next(it) for py3 compat\n\nThe Python 2 next() method of iterators was renamed to __next__() on\nPython 3. Use the builtin next() function instead which works on Python\n2 and Python 3.\n\nChange-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/09e7477a391e7425af62a6516f43c69bc98b7080"},{"sha":"af72881d1dcc46486d8d652b92a0ec9aa9ca2bfa","author":{"email":"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@swiftstack.com","name":"Samuel Merritt"},"message":"Use just IP, not port, when determining partition placement\n\nIn the ring builder, we place partitions with maximum possible\ndispersion across tiers, where a \"tier\" is region, then zone, then\nIP/port,then device. Now, instead of IP/port, just use IP. The port\nwasn't really getting us anything; two different object servers on two\ndifferent ports on one machine aren't separate failure\ndomains. However, if someone has only a few machines and is using one\nobject server on its own port per disk, then the ring builder would\nend up with every disk in its own IP/port tier, resulting in bad (with\nrespect to durability) partition placement.\n\nFor example: assume 1 region, 1 zone, 4 machines, 48 total disks (12\nper machine), and one object server (and hence one port) per\ndisk. With the old behavior, partition replicas will all go in the one\nregion, then the one zone, then pick one of 48 IP/port pairs, then\npick the one disk therein. This gives the same result as randomly\npicking 3 disks (without replacement) to store data on; it completely\nignores machine boundaries.\n\nWith the new behavior, the replica placer will pick the one region,\nthen the one zone, then one of 4 IPs, then one of 12 disks\ntherein. This gives the optimal placement with respect to durability.\n\nThe same applies to Ring.get_more_nodes().\n\nCo-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>\n\nChange-Id: Ibbd740c51296b7e360845b5309d276d7383a3742","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/af72881d1dcc46486d8d652b92a0ec9aa9ca2bfa"},{"sha":"65fce49b3ba74827d4c378f128082cf99b0b5396","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"Use just IP, not port, when determining partition placement\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/65fce49b3ba74827d4c378f128082cf99b0b5396"},{"sha":"df134df901a13c2261a8205826ea1aa8d75dc283","author":{"email":"718d47fcd21c68c803076e61c9feca82594faf1a@swiftstack.com","name":"Darrell Bishop"},"message":"Allow 1+ object-servers-per-disk deployment\n\nEnabled by a new > 0 integer config value, \"servers_per_port\" in the\n[DEFAULT] config section for object-server and/or replication server\nconfigs.  The setting's integer value determines how many different\nobject-server workers handle requests for any single unique local port\nin the ring.  In this mode, the parent swift-object-server process\ncontinues to run as the original user (i.e. root if low-port binding\nis required), binds to all ports as defined in the ring, and forks off\nthe specified number of workers per listen socket.  The child, per-port\nservers drop privileges and behave pretty much how object-server workers\nalways have, except that because the ring has unique ports per disk, the\nobject-servers will only be handling requests for a single disk.  The\nparent process detects dead servers and restarts them (with the correct\nlisten socket), starts missing servers when an updated ring file is\nfound with a device on the server with a new port, and kills extraneous\nservers when their port is found to no longer be in the ring.  The ring\nfiles are stat'ed at most every \"ring_check_interval\" seconds, as\nconfigured in the object-server config (same default of 15s).\n\nImmediately stopping all swift-object-worker processes still works by\nsending the parent a SIGTERM.  Likewise, a SIGHUP to the parent process\nstill causes the parent process to close all listen sockets and exit,\nallowing existing children to finish serving their existing requests.\nThe drop_privileges helper function now has an optional param to\nsuppress the setsid() call, which otherwise screws up the child workers'\nprocess management.\n\nThe class method RingData.load() can be told to only load the ring\nmetadata (i.e. everything except replica2part2dev_id) with the optional\nkwarg, header_only=True.  This is used to keep the parent and all\nforked off workers from unnecessarily having full copies of all storage\npolicy rings in memory.\n\nA new helper class, swift.common.storage_policy.BindPortsCache,\nprovides a method to return a set of all device ports in all rings for\nthe server on which it is instantiated (identified by its set of IP\naddresses).  The BindPortsCache instance will track mtimes of ring\nfiles, so they are not opened more frequently than necessary.\n\nThis patch includes enhancements to the probe tests and\nobject-replicator/object-reconstructor config plumbing to allow the\nprobe tests to work correctly both in the \"normal\" config (same IP but\nunique ports for each SAIO \"server\") and a server-per-port setup where\neach SAIO \"server\" must have a unique IP address and unique port per\ndisk within each \"server\".  The main probe tests only work with 4\nservers and 4 disks, but you can see the difference in the rings for the\nEC probe tests where there are 2 disks per server for a total of 8\ndisks.  Specifically, swift.common.ring.utils.is_local_device() will\nignore the ports when the \"my_port\" argument is None.  Then,\nobject-replicator and object-reconstructor both set self.bind_port to\nNone if server_per_port is enabled.  Bonus improvement for IPv6\naddresses in is_local_device().\n\nThis PR for vagrant-swift-all-in-one will aid in testing this patch:\nhttps://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/\n\nAlso allow SAIO to answer is_local_device() better; common SAIO setups\nhave multiple \"servers\" all on the same host with different ports for\nthe different \"servers\" (which happen to match the IPs specified in the\nrings for the devices on each of those \"servers\").\n\nHowever, you can configure the SAIO to have different localhost IP\naddresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the\nservers' config files' bind_ip setting.\n\nThis new whataremyips() implementation combined with a little plumbing\nallows is_local_device() to accurately answer, even on an SAIO.\n\nIn the default case (an unspecified bind_ip defaults to '0.0.0.0') as\nwell as an explict \"bind to everything\" like '0.0.0.0' or '::',\nwhataremyips() behaves as it always has, returning all IP addresses for\nthe server.\n\nAlso updated probe tests to handle each \"server\" in the SAIO having a\nunique IP address.\n\nFor some (noisy) benchmarks that show servers_per_port=X is at least as\ngood as the same number of \"normal\" workers:\nhttps://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md\n\nBenchmarks showing the benefits of I/O isolation with a small number of\nslow disks:\nhttps://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md\n\nIf you were wondering what the overhead of threads_per_disk looks like:\nhttps://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md\n\nDocImpact\n\nChange-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/df134df901a13c2261a8205826ea1aa8d75dc283"},{"sha":"12d8a53fffea6e4bed8ba3d502ce625f5c6710b9","author":{"email":"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@swiftstack.com","name":"Samuel Merritt"},"message":"Get better at closing WSGI iterables.\n\nPEP 333 (WSGI) says: \"If the iterable returned by the application has\na close() method, the server or gateway must call that method upon\ncompletion of the current request[.]\"\n\nThere's a bunch of places where we weren't doing that; some of them\nmatter more than others. Calling .close() can prevent a connection\nleak in some cases. In others, it just provides a certain pedantic\nsmugness. Either way, we should do what WSGI requires.\n\nNoteworthy goofs include:\n\n  * If a client is downloading a large object and disconnects halfway\n    through, a proxy -> obj connection may be leaked. In this case,\n    the WSGI iterable is a SegmentedIterable, which lacked a close()\n    method. Thus, when the WSGI server noticed the client disconnect,\n    it had no way of telling the SegmentedIterable about it, and so\n    the underlying iterable for the segment's data didn't get\n    closed.\n\n    Here, it seems likely (though unproven) that the object server\n    would time out and kill the connection, or that a\n    ChunkWriteTimeout would fire down in the proxy server, so the\n    leaked connection would eventually go away. However, a flurry of\n    client disconnects could leave a big pile of useless connections.\n\n  * If a conditional request receives a 304 or 412, the underlying\n    app_iter is not closed. This mostly affects conditional requests\n    for large objects.\n\nThe leaked connections were noticed by this patch's co-author, who\nmade the changes to SegmentedIterable. Those changes helped, but did\nnot completely fix, the issue. The rest of the patch is an attempt to\nplug the rest of the holes.\n\nCo-Authored-By: Romain LE DISEZ <romain.ledisez@ovh.net>\n\nChange-Id: I168e147aae7c1728e7e3fdabb7fba6f2d747d937\nCloses-Bug: #1466549","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/12d8a53fffea6e4bed8ba3d502ce625f5c6710b9"},{"sha":"b32abab56c38ec72d290eda41eb19ca08d67c6bd","author":{"email":"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org","name":"Jenkins"},"message":"Merge \"Allow default reseller prefix in domain_remap middleware\"","distinct":true,"url":"https://api.github.com/repos/rackerlabs/swift/commits/b32abab56c38ec72d290eda41eb19ca08d67c6bd"}]}