{"push_id":686811212,"size":1,"distinct_size":1,"ref":"refs/heads/one-server-per-disk","head":"f345d79515f9710e6722094a18b48efe329fa187","before":"a5e1a9d9eee8daf8158329b0a898c46d39d04b6a","commits":[{"sha":"f345d79515f9710e6722094a18b48efe329fa187","author":{"email":"718d47fcd21c68c803076e61c9feca82594faf1a@swiftstack.com","name":"Darrell Bishop"},"message":"Allow 1+ object-servers-per-disk deployment\n\nEnabled by a new > 0 integer config value, \"servers_per_port\" in the\n[DEFAULT] config section for object-server and/or replication server\nconfigs.  The setting's integer value determines how many different\nobject-server workers handle requests for any single unique local port\nin the ring.  In this mode, the parent swift-object-server process\ncontinues to run as the original user (i.e. root if low-port binding\nis required), binds to all ports as defined in the ring, and forks off\nthe specified number of workers per listen socket.  The child, per-port\nservers drop privileges and behave pretty much how object-server workers\nalways have, except that because the ring has unique ports per disk, the\nobject-servers will only be handling requests for a single disk.  The\nparent process detects dead servers and restarts them (with the correct\nlisten socket), starts missing servers when an updated ring file is\nfound with a device on the server with a new port, and kills extraneous\nservers when their port is found to no longer be in the ring.  The ring\nfiles are stat'ed at most every \"ring_check_interval\" seconds, as\nconfigured in the object-server config (same default of 15s).\n\nImmediately stopping all swift-object-worker processes still works by\nsending the parent a SIGTERM.  Likewise, a SIGHUP to the parent process\nstill causes the parent process to close all listen sockets and exit,\nallowing existing children to finish serving their existing requests.\nThe drop_privileges helper function now has an optional param to\nsuppress the setsid() call, which otherwise screws up the child workers'\nprocess management.\n\nThe new class method RingData.load_devices() will load only the device\narray from a serialized, gzipped ring file, retaining a reference to only\nthe returned array of device dicts.  This is used to keep the parent and\nall forked off workers from unnecessarily having full copies of all\nstorage policy rings in memory.\n\nA new helper method,\nswift.common.storage_policy.all_bind_ports_for_node(), returns a set of\nall device ports in all rings for a server (identified by its set of IP\naddresses).  The function includes support for caching mtimes of the\nring files, so they are not opened more frequently than necessary (if\nno rings have changed mtimes, then None is returned).  This is used by\nswift-object-server when servers_per_port > 0 and takes advantage of\nRingData.load_devices().\n\nThis patch includes enhancements to the probe tests and\nobject-replicator/object-reconstructor config plumbing to allow the\nprobe tests to work correctly both in the \"normal\" config (same IP but\nunique ports for each SAIO \"server\") and a server-per-port setup where\neach SAIO \"server\" must have a unique IP address and unique port per\ndisk within each \"server\".  The main probe tests only work with 4\nservers and 4 disks, but you can see the difference in the rings for the\nEC probe tests where there are 2 disks per server for a total of 8\ndisks.  Specifically, swift.common.ring.utils.is_local_device() will\nignore the ports when the \"my_port\" argument is None.  Then,\nobject-replicator and object-reconstructor both set self.bind_port to\nNone if server_per_port is enabled.  Bonus improvement for IPv6\naddresses in is_local_device().\n\nThis PR for vagrant-swift-all-in-one will aid in testing this patch:\nhttps://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/\n\nAlso allow SAIO to answer is_local_device() better; common SAIO setups\nhave multiple \"servers\" all on the same host with different ports for\nthe different \"servers\" (which happen to match the IPs specified in the\nrings for the devices on each of those \"servers\").\n\nHowever, you can configure the SAIO to have different localhost IP\naddresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the\nservers' config files' bind_ip setting.\n\nThis new whataremyips() implementation combined with a little plumbing\nallows is_local_device() to accurately answer, even on an SAIO.\n\nIn the default case (an unspecified bind_ip defaults to '0.0.0.0') as\nwell as an explict \"bind to everything\" like '0.0.0.0' or '::',\nwhataremyips() behaves as it always has, returning all IP addresses for\nthe server.\n\nAlso updated probe tests to handle each \"server\" in the SAIO having a\nunique IP address.\n\nFor some (noisy) benchmarks that show servers_per_port=X is at least as\ngood as the same number of \"normal\" workers:\nhttps://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md\n\nBenchmarks showing the benefits of I/O isolation with a small number of\nslow disks:\nhttps://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md\n\nDocImpact\n\nChange-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6","distinct":true,"url":"https://api.github.com/repos/swiftstack/swift/commits/f345d79515f9710e6722094a18b48efe329fa187"}]}