{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/Hystrix/issues/131","labels_url":"https://api.github.com/repos/Netflix/Hystrix/issues/131/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/Hystrix/issues/131/comments","events_url":"https://api.github.com/repos/Netflix/Hystrix/issues/131/events","html_url":"https://github.com/Netflix/Hystrix/issues/131","id":12528923,"number":131,"title":"Explore \"Adaptive Safety Control\" Approaches","user":{"login":"benjchristensen","id":813492,"avatar_url":"https://avatars.githubusercontent.com/u/813492?v=3","gravatar_id":"","url":"https://api.github.com/users/benjchristensen","html_url":"https://github.com/benjchristensen","followers_url":"https://api.github.com/users/benjchristensen/followers","following_url":"https://api.github.com/users/benjchristensen/following{/other_user}","gists_url":"https://api.github.com/users/benjchristensen/gists{/gist_id}","starred_url":"https://api.github.com/users/benjchristensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benjchristensen/subscriptions","organizations_url":"https://api.github.com/users/benjchristensen/orgs","repos_url":"https://api.github.com/users/benjchristensen/repos","events_url":"https://api.github.com/users/benjchristensen/events{/privacy}","received_events_url":"https://api.github.com/users/benjchristensen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Netflix/Hystrix/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/Netflix/Hystrix/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2013-03-27T19:00:37Z","updated_at":"2015-04-24T15:23:26Z","closed_at":null,"body":"A good article by William Louth (Twitter @williamlouth) talks about adaptive systems versus circuit breaker patterns: http://www.jinspired.com/site/jxinsight-opencore-6-4-ea-11-released-adaptive-safety-control\r\n\r\nI'd like to use this issue to explore whether there are principles that can be applied to Hystrix since Twitter is very limited and that blog doesn't take comments.\r\n\r\n-----\r\n\r\nSome background for the discussion (brain dump so forgive bad grammar and stupid thoughts):\r\n\r\n**Circuit Breaker**\r\n\r\nThe circuit breaker patterns gets way too much credit in Hystrix. It is the concept people seem to grab onto but it is a very minor aspect of the Hystrix implementation and I have quite [publicly](https://speakerdeck.com/benjchristensen/performance-and-fault-tolerance-for-the-netflix-api-august-2012) stated that it is just \"icing on the cake\" and a \"release valve\" when the underlying system is known to be bad.\r\n\r\nThe reason (as William states in his article) is that it's an all-or-nothing affair and it is reactive. \r\n\r\nFor example, at the velocity of traffic for the most used HystrixCommands on the Netflix API system we would be dead by the time a circuit breaker can do anything about it.\r\n\r\nThe concurrent execution limits via semaphores or threadpools are the actual thing that prevents underlying latency from saturating all application resources.\r\n\r\n**System Characteristics**\r\n\r\nHystrix was obviously designed for the use cases Netflix has and doesn't necessarily apply to different architectures or scale down very well.\r\n\r\nSome of these factors are:\r\n\r\n- most backend systems are accessed via synchronous client libraries over blocking IO (some use non-blocking IO which Hystrix will better support at some point as they have different resource utilization and failure scenarios - https://github.com/Netflix/Hystrix/issues/11)\r\n- application clusters scale instance counts from ~600 cores to ~3200 cores each day (number of instances depends on instance type ... but it can be ~100 on the low end to 1200+ on the high end)\r\n- the backend consists of 150+ different functional services (modeled as HystrixCommand instances) in 40+ groups (representing backend system resources and modeled generally by thread-pool for isolation groups)\r\n- some functionality has good fallbacks, some can fail and cause graceful degradation of user experience and others must fail fast as they are required\r\n\r\n**Behavior - Adaptive, Proactive, Reactive?**\r\n\r\nThis depends on which aspect of Hystrix is being looked at ... \r\n\r\n***Concurrency Limits and Timeouts***\r\n\r\nThese are the proactive portion of Hystrix - they prevent anything from going beyond limits and throttle immediately. They don't wait for statistics or for the system to break before doing anything. They are the actual source of protection given by Hystrix - not circuit breakers. (see diagram for the decision flow: https://github.com/Netflix/Hystrix/wiki/How-it-Works#wiki-Flow)\r\n\r\nWe configure concurrency limits using semaphores or thread-pools based on simple math - 99th percentile latency when system is healthy * peak RPS = needed concurrency. Timeouts are set with similar logic.\r\n\r\nIt's to get an order of magnitude, not exact value. The principle is that if something needs 2-3 concurrent threads/connections let's give it 10 but not 50, 100 or 200. This constrains what happens when the median becomes the 99th and the 99th multiplies.\r\n\r\nWe have considered making this value adaptive - use metrics over time to adjust it dynamically. In principle though we have not found a need to do this as systems rarely change behavior enough to need a change in their order of magnitude once first configured and if they do it is an obvious change in a system modification (new code push etc). \r\n\r\nMaking it adaptive would have to take into account long-term trends (at least 24 hours, probably longer) otherwise slow increase in latency and concurrency needs could \"boil the frog\" and allow the limit to slowly increase until it has stopped providing the protection it was put there for in the first place.\r\n\r\nWe have found that after operating 150+ commands at high volume for over a year that we don't have a desire for adaptive changes of what the concurrency limit or timeouts should be as that complicates the system, makes reasoning harder and opens up the ability for \"drift\" to just raise the limits over time and expose vulnerability.\r\n\r\n***Circuit Breaker***\r\n\r\nCircuit breakers are reactive. They kick in after statistics show a HystrixCommand to be in bad state (resulting from failures, timeouts, concurrency throttling, etc). It's a release valve to skip what we statistically have determined to be bad instead of trying every time. It helps the underlying system by reducing load and gets the user a response (fallback or failure) faster.\r\n\r\nOn a single application instance this is not a very \"adaptive\" thing - it's an on/off switch,\r\n\r\nHowever, at scale it is actually quite adaptive because each HystrixCommand on each instance makes independent decisions. \r\n\r\nWhat we see in practice in production is that when a backend has an error rate high enough to cause issues but not enough to shut the entire thing down the circuits on individual instances are tripping open/closed in a rolling fashion back and forth across the fleet as the individual instances use their view of the world to make decisions.\r\n\r\nThis screenshot of one circuit during such a situation demonstrates this behavior. Note the circuit open/closed and how the different counts represent different types of throttling and rejection occurring while most is still successful:\r\n\r\n![Screen Shot 2013-03-27 at 11 58 30 AM](https://f.cloud.github.com/assets/813492/309966/f58eb8a6-970f-11e2-9b68-67a62c86a6e4.png)\r\n\r\nIt very dynamically reduces load so that a percentage of traffic can succeed and backs off, tries again etc.\r\n\r\nWe have many times considered if we should make the logic of the circuit breaker more \"adaptive\" like a control valve that constrains a percentage of traffic depending on algorithms and statistics.\r\n\r\nEvery time we consider it we decide not to because it makes reasoning about the system harder and because at our scale we already effectually get this behavior because of the large size of the fleet.\r\n\r\n**When Hystrix Doesn't Work**\r\n\r\nThe principles above will not work well if a cluster size is 1 or a very small number of boxes. In that case a more adaptive algorithm would likely be preferable to the on/off switch of a circuit breaker - or just turn off the circuit breaker and use just the concurrency/timeout protection.\r\n\r\nAlso, if an application only has 2 or 3 critical backend components without any reasonable fallbacks or graceful degradation then Hystrix won't be able to help much. Constraining the single service your app needs breaks the user experience - the only value it would then give is very detailed and low-latency metrics and quick recovery when the backend service comes back to life - but it won't help be resilient during the problem since there's nothing else to do.\r\n\r\n---\r\n\r\nWith the above thoughts on the matter I'm curious as to where more \"adaptive\" approaches would make sense.\r\n\r\nDo they provide benefit to a large system like Netflix or just make a more complicated version of the same end result?\r\n\r\nAre they critical to make something like Hystrix work on a smaller system?\r\n\r\nEven with an adaptive approach and a \"valve\" instead of \"circuit\" it still means it is shedding load, failing fast, doing fallbacks. Is that any different than what already happens with circuits opening/closing independently and rolling around a large fleet.\r\n\r\nOther than the circuit breaker (which is already a limited aspect of Hystrix) where else would this concept apply?\r\n\r\nThoughts, insights, (intelligent) opinions etc welcome ... I'm interested in whatever the best ides are to operating a resilient system.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Netflix/Hystrix/issues/comments/95965336","html_url":"https://github.com/Netflix/Hystrix/issues/131#issuecomment-95965336","issue_url":"https://api.github.com/repos/Netflix/Hystrix/issues/131","id":95965336,"user":{"login":"Petikoch","id":9936866,"avatar_url":"https://avatars.githubusercontent.com/u/9936866?v=3","gravatar_id":"","url":"https://api.github.com/users/Petikoch","html_url":"https://github.com/Petikoch","followers_url":"https://api.github.com/users/Petikoch/followers","following_url":"https://api.github.com/users/Petikoch/following{/other_user}","gists_url":"https://api.github.com/users/Petikoch/gists{/gist_id}","starred_url":"https://api.github.com/users/Petikoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Petikoch/subscriptions","organizations_url":"https://api.github.com/users/Petikoch/orgs","repos_url":"https://api.github.com/users/Petikoch/repos","events_url":"https://api.github.com/users/Petikoch/events{/privacy}","received_events_url":"https://api.github.com/users/Petikoch/received_events","type":"User","site_admin":false},"created_at":"2015-04-24T15:23:25Z","updated_at":"2015-04-24T15:23:25Z","body":"Hi @benjchristensen and @mattrjacobs ,\r\n\r\nthank you very much for your feedback. I'm very short on time right now, sorry for my short answer right now.\r\n\r\nI collected my thoughts today into here: https://github.com/Petikoch/java_adaptive_control_experiments\r\nIn the next week or two I will start to implement one or more of the experiments. It's a topic which is \"hot\" for a customer of me, so there will be soon results available.\r\n\r\nI'll keep you updated.\r\n\r\nHave a nice weekend and best regards from Switzerland,\r\nPeti"}}