Very good morning. It is Tuesday 22nd of June. Hope you're doing well as ever if you are watching this delayed on YouTube And don't forget to like and subscribe to the channel if you want to get this briefing early Don't forget to check out amplifier live comm but getting straight into it and Looking at the heat map on the clothes on the S&P 500 yesterday really strong rally and fat that we saw across the major indices Eradicating pretty much all of the losses that were seen following the hawkish tilt from the Fed last Wednesday and a follow-up hawkish comments and bullard that dented equity sentiment on Friday So the S&P was up around 1.4% the Dow actually outperforming Helped by the fact that energy financial industrial shares really led to comeback The Dow up 1.76% the Nasdaq was up just 0.6% Commodity stocks that were hit hard last week We're really seeing the best of those gains in the small cap Russell 2000 in fact outperformed all of them and was up just over 2% One of the major areas here kind of sweet spots so to speak is the energy sector You can see here glowing the most green so you can see the lights at Exxon were up 3.6% at the close last night Chevron CVX was also up around the similar margin And so let's take a look at oil prices and those continue to move higher So here on the intraday view you can see we had a bit of a breakout and a nice platform for price on the pullback It came back to that same level of what was the high point of resistance Back on Thursday last week initially in the APAC session last night Sunday going into Monday And then we just managed to continue to move higher here And now at the high finding a bit of resistance up at 73 28 for the moment But if you look at this on a daily chart you can just see where we're at on the bigger picture now with oil and Continuing to just move up at the moment and from a technical perspective You've got that high the top from the summer of 2018 as the next kind of target here We're trading at 73 56 at the moment that would be up at 75 28 And then the overall larger upside target the October 18 high would come in at 76 90 but oil then really quite a focal point at the moment in terms of advancing the most in a month On a couple of different things few people are looking at some of the recent dollar weakness that we saw particularly Going back to yesterday's session albeit I must say that the Dixie is just actually firming up reversing some of those moves for the moments So worth keeping that in mind But the other thing is key time spreads surged now what we're looking at here is a case of Backwardation meaning that the near-term contracts are more expensive than further out in the future But why this is happening is expectations of further supply declines at the biggest storage hub in the u.s And so data-providing Genscape so if you're an oil trader there's kind of three major phases of information you get on a normal weekly basis Genscape on Monday API is on a Tuesday and then you get the DOE's on a on a Wednesday and Genscape reported a 2.6 million barrel drop in stockpiles at Cushing in Oklahoma last week according to people with knowledge of the reports and Infantries at the hub already at the lowest since March of 2020 So a tightening market and remember the u.s. Is in full reopening mode at the moment Remember just last week we saw the likes of those highly populous areas like California New York kind of taking the lockdown shackles off and completely now reopening fully Given that over 70% of the people in those areas have now been vaccinated and so yeah continues to To to play out in a very positive fashion for for oil prices and of course all of this coming in context of OPEC Not budging at the moment and continuing to lend it supporting hand from a supply-packed point of view So yeah, definitely WTI spreads at their strongest in years now as Cushing supplies continue to fall Some of the dollar fluctuation as well particularly downside Coming yesterday just helping propel prices higher or once again. So Helping that energy sector in the S&P but having a look then at the equity market One of the things here that was quite evident. I've marked this up just to give you a visual cue of where what we're looking at and Here we are I mean these ellipses here mark where we were trading before the FMC announced on Wednesday and this is where we're trading right now and Despite the excursion to the downside that we have on the double whammy of the hawkish to hike 2023 dot-plot and then bullards follow up on Friday Monday's rally was pretty extreme in fact One of the best single-day rallies you had a couple of months and it's taken us all the way back to Basically flat and you know actually when you look at this the all-time high is here Only around 30 points above where we're trading at the moment So, you know despite people getting a little bit spooked about the hawkish tilt that we heard from the Fed You know equity markets still remain pretty robust at the moment and a lot of that does come on the back of More comments from the Fed we knew that Fed commentary I mean there's a lot of speakers this week and that's quite typical of a communication tactic from the central bank to try and Kind of just steady the ship after we saw quite an extreme reaction last week And we've got a bit of a fight breaking out here between the hawks and the doves at the moment because the text For the testimony to US Congress about COVID-19 response That's happening today in terms of power speaking live, but the prepared text came out last night By the major news wires. This is quite common. I was at my desk Maybe half nine ten last night when the when the news came out And basically there's nothing too substantial that I'd say that power has said but just to go over some of the main stuff He said the US economy continues to show Sustained improvement from the impact of the coronavirus pandemic and ongoing job market gains But inflation has increased notably in recent months And he continued to talk about the fact that he sees that dropping off and reinforcing that transit review separately though on the back of this the New York Fed president Williams came out this is an FT piece last night as well and said that the US economy does not Justify a policy shift So for me putting these two comments together and given the fact that these two are two of the most senior most important people on the FMC Both powers prepared testimony Williams remarks suggest that the top top brass if you like at the Fed are definitely much more cautious on the prospect of Quick policy changes compared to those of what we've heard from more regional level presidents like Bullard and also like Kaplan So the Dallas Fed president spoke yesterday. He's a well-known Probably the most outlying hawk Anyway, did say favored starting the process of reducing the bomb purchases Sooner rather than later, but again given his stance generally on policy that doesn't come as a surprise But I think tactically this is all Part of the plan for the Fed as the way I see it You've kind of got these outlying hawks Sounding hawkish and and rightly so because there are elements in the economy that would justify that and you know I was just having a look at the Bloomberg High Frequency Index in combination with the Atlanta Fed GDP tracker and Although we're going to see Q1 GDP Final reading this week. I mean who really cares about that. That's old news now Looking at what those trackers are saying for Q2 GDP in the US. It's going to be in excess of 10% You know, so there's there's these kind of reopening pains to go through which are going to cause pretty explosive growth in the short term At least before we average out and so yeah, some of the hawks are being hawkish and And the center ground is being held. So for me, this is kind of We're just leaning in that direction Which I think is probably appropriate But at the moment the markets taking a degree of assurance and relative calm having reversed a lot of these moves On the fact that look the status quo at the moment is inflation is transitory and we're not going to move anything very quickly But you know, the seed is being planted now And I think the hawks are doing that purposefully because change will happen in time and we still stick to the timeline of A more in-depth kind of discussion on tapering to emerge Come then the signals in Jackson Hole to be formalized in September for the whole tightening process or reduction of bomb buying to commence the beginning of next year So yeah, a couple things there But just just flicking back to the charts for a moment before we look at some of the other geopolitical news that I think it's worth touching upon So quick look at the currency pairs The dollar index right now as I'm talking is now up about two tenths of one percent So in fact, it's pretty much reversed the Dixie the losses that were seen during the US entrance yesterday So we're back to pretty much flat in that move now trading at around just around the 92 handle So as such then that's just providing a bit of a cap for euro dollar in the near term you can see here So still upside keeping an eye for a firm level of resistance at 1943 and a half any further pull back here and we are trading sub pivot in the euro future at the moment be keeping our 119 04 and a half and then down to that double bottom that was printed Friday Monday at 1 1867 so at the moment you could argue this is a range and we kind of midpoint of that at this present point in time The dollar strength just helping on that recovery in a Dixie Euro dollar just just grind back down For the moment as far as cable is concerned quite a nice technical Response that we saw on the daily chart yesterday So again pretty reminiscent of the euro in reflection of some of the dollar recovery that we've been seeing the last couple Of hours during a the Asia pack session on the daily chart though Nice bounce off that previous low that we had at the end of April which was around the 138 Psychological handle as well as that technical support area and you can see 138 has been an inflection point for price on a number of occasions through 2021 so it's a meaningful level and we saw quite a decent bounce off that for the time being but again short to term I'll be keeping an eye on the dollar for the moment with a lack of real sterling catalysts Meanwhile elsewhere Just a quick look at gold. I've just been having looking at this price pattern here in gold that I've been looking at Which is this on the downside this trend line has held so far nice test and Support being founded around that level and so we're in close proximity to the lower bound of this kind of channel for the moment So I'm quite keen to watch that got the pivot just below But we really see too much support on any breakdown in price until we get further down to This level from the Monday morning low and the Monday kind of US morning low Which would be around the 1773 and a half level in the futures if we break the The recent pattern here in the pivot level Otherwise on the reverse near term we've got to get back up and through this kind of range here you can see These areas of support that have played out quite nicely through Monday and also in the Asia Pact session to then Kind of print that that upside of that channel Which has been that rising trend line going back from last Thursday the test on Monday and the overnight session So it's kind of the setup I'm looking at in gold at the moment terms of the driving forces as to what could be the trigger points there again Dollar fluctuation and movement is going to be quite quite key On a geopolitical front, it's kind of the three Three forces at the moment. So you've got North Korea Russia in Iran These are the kind of main talking points when it comes to the US Present and starting off with this is Kim Jong-un's sister, of course And she's come out and dismissed the prospects of an early resumption of diplomacy with the US Basically just saying that their expectations and ambitions are misaligned essentially I won't comment as yet as to what I think of that until I run through all three and then I'll give you the summary Elsewhere the White House has said it has no timeline on reaching agreement on the Iran nuclear deal And it will continue to negotiate that's of course coming after as a rainy election We talked about yesterday and then the US State Department says it is preparing strategic stability dialogue with Russia And it is in the process of of scheduling those conversations at the moment So on all three fronts not really a great deal surprising I think it's way too early yet to be brought back to the table after the current Journey of those meetings after several rounds with Iran I think that's still got some way to go before we get to anything more of a definitive agreement As far as Kim Jong-un's sister is concerned, that's probably the least surprising of the three They're just keeping a fairly firm hand at the moment And I don't really think Biden's got too much appetite to Reengage with North Korea at this point when he still has to really deal with China first Then Iran kind of second on the geopolitical matters and then probably Russia's third So North Korea is kind of there and present but also tends to be Almost a proxy for US China talks more than directly specifically about North Korea So perhaps this is just part of that that usual political play from from China at the moment And then yeah, the Russia side following those meetings from Biden Putin last week. This is probably what you'd expect So fairly lukewarm But warm to a certain degree and so it's just about when they're gonna talk again So for all three of these I don't see any impact for that from the market open today But probably prudent to update you on those matters As far as the calendar is concerned It's it's pretty quiet in fact for today There's really nothing at all coming out in the UK European morning from a data perspective Therefore leading on into the afternoon We get the US existing home sales report and the Eurozone consumer confidence flash reading for June and API inventories come later on in the evening as per usual from a speaker's perspective Chief Economist of the ECB Philip Lane talks about the resilience of the euro at three o'clock There will be a text release in Q&A Feds Mester a voter next year will be speaking on monetary policy At 330 feds daily who's a current voter. She'll be speaking at 4 p.m. And he's to be schnabel will be speaking at 630 and again as far as Jay Powell's Testimony to US Congress on the COVID-19 response that does happen at 7 p.m. London time formally But again, we've already had the text out last night. So not expecting that to be a market moving event And then you've got 60 billion dollars coming a two-note auction just to wrap it up for any fixed income traders So that is it not gonna go any further gonna leave it there any questions at all Feel free to reach me in the Amplify live community or on YouTube Just drop a comment happy to help but otherwise have a good day ahead. Thanks guys


So you know that electricity is tantamount to modernity. Everything that we associate with the 21st century world is predicated on the availability of electricity.  Where you don't see light, there's one of two situations. Either nobody lives there or the place hasn't been electrified. And if it hasn't been electrified, from my perspective, there's no greater gift than sustainable, reliable electricity. It really unlocks the future. By the way, you know this is a collage, right? This is not a NASA image of the world at night, because when it's dark here, it's light over here. If, you know, the world never looks like this. If it ever does, it's a really bad day for all of us. I never see it like that. Electricity is very unusual because, you know, the electricity powering the lights in this theater was generated just moments ago. Because the way the grid operates is that supply has to be in balance with demand everywhere at all times. So you're looking at the world's largest supply chain with zero inventory. So imagine if every time you wanted to draw water from your faucet, the water had to come immediately from a spring, because there's no such thing as water storage. And then when you turn off the faucet, you have to cut back the supply from the spring. Otherwise, the water will continue to flow and build up pressure in the pipes and damage, pipes, valves. The same thing happens on the grid. If supply exceeds demand, voltage will rise, frequency will shift, both with devastating consequences. Imagine if every time you go to plug in a device, you have to ask yourself, do you feel lucky? Do you? So, you know, what's worse than no electricity is bad electricity. How do we deal with this balancing? We deal with it with overcapacity and redundancy, which leads to unparalleled inefficiency, underutilized assets in terms of generation, distribution, transmission, excessive emissions, and all at greater cost to us, the rate payers. And then add to the mix the environmental imperative, which is brought on by the climate change, which argues for a broad deployment of renewables. But the renewables are intermittent, so they themselves alone are incapable of being fully integrated into base load. And supply and demand requirement means that they're no help. Electricity at zero marginal cost that is out of balance is not a solution. It's a problem. So how do we deal with the intermentancy? Overcapacity, redundancy, all at greater cost. Storage is the key missing piece here. Batteries would do for the electricity system what refrigeration did to food supply, or water storage does to water supply. It's compelling. So what's the obstacle to the deployment of batteries? The obstacle is cost. And this is a semi logarithmic plot, so this is 1000X. And it shows various options for electricity storage. And because this is a semi log plot, all of this is meaningless. It's less than 0.1%. So none of the batteries are of any value because they're far too costly and they can't meet the long service lifetime requirements. The only thing that works really is pumped hydro. But pumped hydro is geographically constrained. You have to have a difference in elevation. You have to have access to large amounts of water. You probably couldn't get permitted to put in pumped hydro in many places today, even where there is that difference in elevation. The other thing I learned from looking at this graph, which I had one of my students construct, is that there is a wall here at about $500 per kilowatt hour. So cost is key here. And my message to you is that the classical approach in the university or in the corporate research lab doesn't work in energy. The classic approach is invent the coolest chemistry and then get it to the manufacturing people and they'll chase down the cost curve. And they will. But look at lithium-ion. It's 20 years old. It's still way too costly for this. In fact, it's even way too costly for automobiles. It's great for mobile devices. But it's not going to make it in these other markets. So if cost is the key, then cost has to be a factor in the discovery process. So I think about cost in terms of the chemistry that we're working on. It has to be chemistry that is abundant, earth abundant, and has to be simple to construct. So I refuse to allow my students to go to certain parts of the periodic table because the results will not scale. So when I started thinking about this problem, I decided I had to adopt a different approach. I disregarded everything we knew about batteries and instead looked for inspiration away from the battery field. In fact, I looked to a field that neither stores electricity nor generates electricity, but instead consumes electricity, vast quantities of it. This is an aluminum smelter. This is the cells here. It's about 30 meters across. It goes back maybe one or two kilometers. Consumes vast quantities of electricity, and yet it produces metal from dirt for less than $1 a kilogram. So I knew that if I could teach this thing to not consume electricity but to store it and then give it back at the end, I'd have something that's cheap. So I started with something that's cheap and figured out how to make it work to invent something cool and figure out how to make it cheap. This process was invented in 1886 by two people simultaneously working independently. Hall in the U.S. and he ruled in France. They were both born in the same year. They both died in the same year. 222-year-olds changed the world. With this invention, aluminum changed from a precious metal costing more than silver to a common structural material, thanks to a change in the process. But let's not forget one other important fact. Aluminum is the third most abundant element in the Earth's crust. If they'd invented this process for iridium, it wouldn't have made any difference. So that was my point of departure. And this was the invention. It's called a liquid metal battery. It's got three layers, all liquid. So I have liquid magnesium on top and I'll show you some other metals. And liquid antimony on bottom, and a molten salt electrolyte in between. And the metal is insoluble in the salt and the salt is insoluble in the metal. So they phase separate and sit one on top of the other like salad oil and vinegar, only in this case we have three layers. Magnesium has a density of less than two, antimony greater than six, and the molten salt is somewhere in between. So there's no need for membrane separators and so on. And this operates at elevated temperature. This is crazy. It can't work. Because it's elevated temperature and everybody knows you have to make a battery at room temperature. Why? Because you want to put it to your face. You don't have to put this to your face. So don't pay for attributes you don't need. And then people don't know much electromedalergy. That's my other field. And it turns out that when the current passes from top to bottom and magnesium alloys with antimony, you generate jewel heat and you trap that heat and it self-sustains. And then when you charge the battery, you bring magnesium back to the top and it generates heat and you trap that heat. And people said that heat will just consume all of the energy of the battery. It turns out the round-trip efficiency of this accounting for the heat loss is 75%, which is greater than the round-trip efficiency of pumped hydro. And I say that and people still say, it's crazy, it won't work. I love it. There's nobody else trying to do this. And now this was my team. There's one student. He was a physics graduate from Queens University in Kingston, Ontario. Actually, David and I were both born in Toronto in different years. And he's staring at this thing. He's looking at it. He's wondering if it's going to work. Actually, I'm not sure it's going to work, but I don't tell him that, because that's how mentoring works. I tell him, no, no, you'll figure out how to make it work. But he had no prior background in batteries. So he was bright, he was young, but undjaded. And I kept that going all the way through. And then I got funding from Total and from ARPA-E. And this was my team in the summer of 2010. Twenty people. And again, this is the only one that had some background in batteries. The rest of them were all novices. So I don't hire experts. I hire the anti-expert, especially in an area like this. And multinationals. She's from Spain, Korea, China, France, Trinidad, French Polynesia. She's from Poland, David and I were both from Toronto. And I hired a few token Americans as well. And that was the team. And I have to tell you that 20 times 3 is far greater than 3 times 20. So if I had the same amount of money spread over 20 years with three people, I wouldn't be standing here right now. And by the way, the first year, their results were terrible. And in fact, when the Department of Energy monitored David Danielson, who is now the Undersecretary of Energy, when he came to review us after the third quarter, he stood up and he announced that of all of the projects in his portfolio, he gave this one the lowest chance of success. And you can feel the mood sink in the room. But I was patient and they learned from each other and after two years, good things happen. After three years, they worked miracles because they didn't know what was impossible. They were novices and they were at the university. And electrochemistry began at the university. It began with Professor Alessandro Volta. And there's his first battery, a stack of coins, silver and zinc separated by cardboard soaked in brine. And with this invention, it immediately turned into not just a new field of electrochemistry, but new technology, electroplating, electroforming, and it served the basis for electrowinning of aluminum. The other thing that Volta's discovery did was for the first time it demonstrated the utility of a professor. Until Volta, nobody imagined a professor could be of any use. Volta showed that if you give a professor money and good students leave him alone, he's allowed to produce something of value. So sponsored research at universities today is in the tradition of Volta. So how did I get to the next invention? I just sat here with the periodic table. I didn't use density functional theory, no high speed computational material science. I was teaching freshman chemistry. I taught freshman chemistry at MIT for 20 years. Most of my colleagues would kill to avoid that assignment, but I went for it because it was high impact and I was not schooled as a chemist. I was a metallurgist from Toronto. And so as an error of East, I had to teach myself electronic structure and bonding and I can tell you my invention comes from that trajectory, that odyssey. And by the way, the periodic table was given to us by another professor, Mendeleev. The metals on the top had to be most electropositive and the metals on the bottom had to be the worst metals but still metals, so they would be the most electronegative of metals. And mercifully, nature was kind in this case. These are all high density metals and these are all low density metals. And so we've looked at over a thousand combinations of alloys and salts and so on. This is one of the papers that came out recently. I don't care if I publish or not, but I do publish in order to launch the careers of young people. And so this was an image that Felice Frankel made for us. This is an analog. This is liquid mercury. The electrolyte is sodium chloride in water and this is a nickel mesh that would represent the current collector for the lithium. So this is a discharged battery. All the liquid metal on the top is now on the bottom. It's a beautiful image. You know, there's actually an artistic side to all of this as well. And then we had to think about manufacturing because product to market is a long way and so we founded a company, David and I, and another person and we called it Ambri because Ambri is Cambridge. We invented this in the heart of Cambridge and it's a five letter pronounceable name and Ambri.com was still available. And here's the governor of Massachusetts at the time. This is Phil Judis. He was our CEO. And we invented all of the manufacturing capabilities for this. Let me tell you, it's a long road from the laboratory bench to the market when you're talking about energy. In fact, it's such a long road and it takes so much money that it's incompatible with venture capital. So all the talk about massive innovation, it might be fine for little devices and apps for the phone but when you want to take on combustibles, it's a very tough road. So in the background, I want to show you what we... These are the cells that we're building now. These are 10 centimeter by 10 centimeter. Each of these cells is 100 amperes and we aggregate these in order to make platters that can aggregate into modules. And then you'll see, I mean, this is the trajectory that we're on and we invented the robotics that will actually automate this. So this is 25 kilowatt hours. This is about the size of a large refrigerator. It's more than adequate for a single family home. And then we aggregate these.  So people who work in virtual reality are kind of easy. And by the way, power electronics, you know, battery management system, all of this, zero help from the industry. Siemens, General Electric, ABB, Schneider, no help, no help. So we hired our own people. We had to program all of this stuff ourselves. And this will give you a sense of what it's like. So these are the cells here. And I'll show you, this is the inside, one of the operations. So we invented all the robots and this is TIG welding and so on and so forth. It's primitive. But there's a chance it might succeed. So this would be the footprint of a 53-foot trailer. It's about 16 meters by about two and a half meters. This would be one megawatt hour. So it's silent, no emissions. And by the way, this can act as a load as well as a source. And you know, sometimes to balance the grid, the remedy is not more current. The remedy is to get rid of current. That's when they go to negative price and they call you up and say, please turn on all of your machinery. Well, this thing in one millisecond can turn from a source into a load. And we've abused these things at very, very high currents and they take it. The only question here is, where does this thing go? Because if this thing is on the right side of $500 a kilowatt hour, it doesn't make any difference. And our cost models indicate that we're going to be somewhere shy of $500 a kilowatt hour. So there's hope. We're on the trajectory. I wish I had the battery. I'd put it in front of you here, but it's a long way. So that brings me full circle to, you know, what have we learned from all of this? So high temperature battery, oddly enough, is safer than a room temperature battery. And then, of course, the whole business about mentoring and so on, I did not go to experts. I did not bring in consultants from the battery industry. Instead, I brought in young people, bright young people. And the other piece of it is that if, let's say, we want to bring electricity to places that don't have it, the vision of this is that batteries in Africa should be built by Africans using African resources. That way they become authors of their own future. Batteries are not built in one plant in China and shipped all over the world. So that unifies everything. And actually, there's this grand unification that I see because the other half of my research is in electrolytic production of metals. Imagine if you have a sustainable source of electricity that electrolytically produces the metal that goes in the battery that then stores the sustainable form of electricity. Because ultimately, it's all about electricity. And 50 years from now, we will be off of carbon. We will have renewable, sustainable electricity which means all industrial chemistry will be industrial, electrochemistry. And chemicals will be used as sources of synthesis for fabrics and materials, but not for sources of electrons. I hope I've given you some hope. This is a fantastic space to be active in. And with storage, we'll see the widespread deployment of renewables. And the last thing is the mentoring piece. This is what I tell my students, be realistic, ask for the impossible, and sometimes with enough ingenuity, the impossible becomes the inevitable. Thank you.


Hi, I'm Peter Robinson. I'm here to talk about who, what, when, where, why, how for Fedora or IOT on Fedora or Fedora on IOT, depending on which way you want to look at it. So the who. The initiative was started by me. The role was provided around three years ago to do it as a full-time by the Senior Vice President of rail engineering and it was my remit is to basically investigate IOT how it would look from a Fedora rail ecosystem perspective so that you know ultimately we're not starting from ground zero. We have a small but growing community which is awesome and I actually have my first full-time team member other than me starting on Monday so welcome Patrick. Yes, Patrick's role is basically a very generic IOT security so he's going to be dealing with a whole bunch of things funnily enough around IOT security so but Patrick has a talk tomorrow that where he'll be covering some of this which is very exciting. So who's the audience for Fedora IOT? Well the Fedora community in general because obviously we're just using Fedora. There's an initial focus on a number of use cases. Industrial IOT, the term which I absolutely despise is industry 4.0. Oil and gas industry and we have a talk about that tomorrow where we actually have a Red Hat customer that is working with us in the Fedora space and so Chad and I will be talking about how we're dealing with some of the problems that the oil and gas industry has with regards to industrial control and various other bits and pieces. We have a partner that's interested in smart cities, automotive and a bunch of the community including people like the FPL are interested in home automation and smart homes. I would just like someone to automate the packing of my dishwasher. And it's one of the things that's quite interesting about Linux about IOT in the Linux space is we're actually getting net new people. We had a docs writer in February for about a month or so who massively improved the docs for getting started and one of the reasons for that was I was getting people going I've never used Linux before. How do I do this from Windows? Which from increasing the Fedora community and the Fedora ecosystem is a problem that we've wanted to have for some time and we've not. And so it's an interesting problem to solve and of course there's existing Fedora enterprise Linux and related users. So that's who we're talking about. It's now an official well has been for some time an official council objective. We're moving to an addition very soon where I've got a whole bunch of list of things to do and there's a few other things for others to do before we make that. It's obviously based on Fedora standing on the shoulder of giants, the Fedora wider ecosystem. Using Red Hat technology such as OS Tree, SystemD, Podman, FW Update Manager, as I've said to a bunch of different people that I've spoken to at Red Hat over the time it's basically the same puzzle pieces building a different puzzle. So all the same technologies but just we're producing an IOT picture and part of my role is upstream ecosystem. So the ARM ecosystem through Leonardo engaging in standard initiatives across distros to basically help improve the slot or even a little bit help improve the dumpster fire that is generally IOT. And so it's a very wide-ranging when. I started in this role almost three years ago. I don't know where time goes. I think three years will be the single longest role I've ever done in IOT. I've averaged usually two, two and a bit years before I've got bored and moved on and I see Denise in the background shaking her head. Like my role now compared to three years ago doesn't look anything like it was three years ago yet it looks very similar to what it was three years ago. I don't know how I'm going to actually be able to get bored anytime soon and like every time I look back I go wow that was busy and that was fast. It's not possible to get any busier and any faster and it just seems to keep accelerating, accelerating, accelerating. So our first official release was Fedora 29 as a spin. We obviously released Fedora 30. We're promoting to addition really soon now. We're generally doing around monthly feature releases. We're starting, we've started off, it started off quite slowly. Like when I first started doing this I thought three years ago it'll be a couple of months and I'll have you know the first release out and holy hell I was wrong. And we had all these dependencies and various other bits and pieces I wrapped and unwrapped and repackaged and dealt with and we eventually got there. But it's like quicker and quicker and quicker and quicker. Why? A traditional RPM DNF yum distro for IOT is not that great. If you've ever lost power during a update and you've tried to work out whether you end up with a bootable system or not and if it actually boots how to get it back to a consistent state, doing that across tens of thousands or hundreds of thousands of endpoints, millions of endpoints is impossible. Like if you've got 10 million endpoints out in the field and you've got a roller truck to fix them, that's a problem. I feel Fedora is a good base for an IOT distro. A lot of companies are throwing like the baby out with the bathwater and not doing basic enterprise style security like shipping telnet and not disabling it and things like that. And so we've got all this institutional security for enterprise and things like that that generally mostly just works. So why throw it all away and start from the beginning again? Fedora moves fast, which is good. Like for IOT and things like that, we can get the latest features and functionality in system D, in kernel, in tool chain, which around some of the things like the kernel self-protection project, things like that is going to be useful from an IOT perspective. And yeah, so it's, but one of the advantages we have is it's a generally new use case. We don't have legacy users, so we can think differently and evolve stuff because we don't have J2E platforms and traditional database platforms and things like that that we have to care about whether we break or not. Like it is net new platform from a Fedora ecosystem point of view. And so we can change and evolve things and break things because we don't have a tradition of like 20 odd years of Red Hat Linux legacy. How are we doing it? So based on RPM OS tree and other related atomic core OS technologies, uses OCI container stack, so podman, scopio, et cetera, a simple compose process will be supporting the image builder technology, big focus on security, lots of stuff around TPM2, IMA, system D, SECCOM, SELinux, Secureboot and all of that sort of stuff. Ultimately, it's a similar OS to the data center, but without the physical security. So things like storing all the credentials in TPM so they're not recoverable even from a running OS is useful in the data center and a lot of like security focused companies would be interested in it, but probably not as interested as say the latest version of Kubernetes or OpenShift or whatever it happens to be. Whereas when you go and strap a device to a light pole or a oil pump or something like that out in the field, you don't have that physical data center security. So device security and things like that is of critical focus. Yeah, so we're currently at what I would consider a first phase minimum viable product and we're moving quickly. So some of the components that we're sort of building on top of this MVP layer, things like OS updates, automatic update, rollback, auto scale out of updates. So start to trickle out updates to devices and then scale up or scale back depending on what sort of success rate you get. OS config management. So people are like, well, just use Ansible and like, well, yes, but if you've got 10 million devices and that are not necessarily always online, an Ansible run could take months to complete. So you need a concept of eventual consistency. And I don't know that there's a server out there that Ansible could run on to hit 10 million devices. At management and updates around that, well, that's going to look something like Kubernetes to some degree, but Kubernetes running on a device with 2G of RAM at the moment is interesting. Some form of device management. So hardware failures, firmware updates, things like that and provisioning support and standardization. We're looking at ignition from CoreOS for this, but it's missing a lot of stuff that is sort of on the nice to have for them, but critical for us. So things like TPM provisioning, device encryption, stuff like that. And how do we support the deployment of millions of devices? And there's a lot of initiatives around that from companies like Intel and that, but none of them are standardized and none of them are open standards. They're all very much how do we lock you into Azure, AWS, various other bits and pieces like that. And that's great, but overall, there needs to be an industry-wide standard for secure deployment of that. And I mentioned, you know, eventual consistency for occasionally connected devices. Where? So the usual space, we have an IoT landing page, which links to basically all of this. We have a Kanban board, Twitter, IRC, all the usual Fedora communication mediums. And that's me. Does anyone have any questions? Software defined radio? Untested? It's certainly a use case. Like a lot of this stuff, like, I only have so many hours in a day.  And so things like SDR are like in the back of my mind that people would be interested in it. And so if there's people interested in the community, I'm around. Well, so it depends on the use case. So from an SDR point of view, I've not had, I mean, that there's, we've got obviously Wi-Fi and wired Ethernet, like in a factory, it's probably actually going to be wired Ethernet. They're like 5G is a topic I talk about quite a bit. SDR is certainly an option. It's not something that I've spoken to a lot of companies about. So from that perspective, it just comes down to focus and time. No, so we're not focused on MCUs. One of the use cases I'm working on with the Armour ecosystem is using, like, Fedora IoT as a gateway to MCUs. So, so Zephyrtos is like an open source, real-time OS for like a whole bunch of different MCUs. And I work quite closely with a number of companies that are working on that. So the idea is that, like, this would be a gateway to those devices, whether connected by Bluetooth Mesh or 802.15.4 or those sort of technologies, or in some cases, like, hardwired with CAN is another way of communicating. And then this would be sort of the gateway for those devices, depending on what they do. Yeah, and, like, I've had in the back of my mind that things like SDR and amateur radio would be a use case that a lot of people would be interested in. Yeah, and I understand that, but that is not the only bit of IoT.  So if you're, I mean, I've spoken with literally hundreds of companies with thousands of use cases. And, like, one of the companies I'm working with has, like, well over 100 different use cases, and that's one company. And so, like, everyone you speak to has a different idea of what IoT is. It's a bit like trying to define cloud five, 10 years ago. Everyone had a different idea as to what cloud was. IoT is basically that squared at the moment. Like, there is literally billions of use cases. Anyone else? Oh, I'm going to get off quite lightly with this. Excellent. So the question is, how much of the security work will go into the other additions? It will all be available. So some of the stuff we're looking at doing around, like, the system de-lockdown stuff and that will probably break a lot of stuff. I actively want to break that in the IoT use case. For the general use case, that's probably not such a good idea. But it will, like, it will all be going into... So some of the work that Patrick will be focused on will be around TPMs and stuff like that. And that will be going into Fedora and into the upstream community for any other distro. And so things like integrity measurement and astestation stuff in the kernel. Like, it's enabled in raw hide now and it will be a default allow policy in Fedora in general. We're going to have an active policy in Fedora IoT to enforce and measure stuff for a general purpose OS that will break shit. I don't care. Well, I do care, but I'm going to be actively opinionated on breaking stuff in IoT so that we can actively fix it and make things work in a much more focused manner. And of course, all of that stuff will roll into the standard distro. But there'll be different policies that people can turn on, enable and force what have you depending on what they want to do. Yes. Yep. Yeah, so we're primarily focused on running the application stacks in containers as opposed to on the base OS. Simply because they're more easy to update and that's the general direction enterprise is going as a whole. And, you know, a lot of one of the things that a number of, like, people are excited about is that basically they can use the same workflow they do in the data center on IoT without sort of changing things so they can have the same app dev process for containers on the edge or for applications on the edge that they do in the data center, say running on OpenShift style stuff. So you will be able to do layering and things like that as a whole. That can be quite problematic. It's randomly broken a number of times in raw height of late and needed to be fixed. And so, but, you know, so they're and it's interesting because there's at least six or seven start ups that I'm aware of that are actually playing around and using Fedora IoT and products, which is cool. And some of them, and like I had a question on the mailing list the other day, how do I do NPM install or PIP install on like a RPM OS tree read only file system? And it's like, you don't. So they're basically bundling it all into a big RPM and layering it on top. And I'm like, if you want to do that, it'll work, but it's not going to be our primary way of dealing with this. Because, I mean, ultimately, the advantage of containers is you can run four or five containers alongside each other and they're completely isolated from each other. So if some crappy IoT application decides to start doing bad things, it's isolated and doesn't affect the other devices on the other applications running on the device. Well, thank you, everybody. I'll be around. I have one other talk and we have a hack fest on Saturday, I think. Patrick will be giving his security talk tomorrow. And, you know, feel free to come and ask questions.


So my name's Alex Henderson, I'm here from Pushpay and I'm about to talk about NoPaymentLeftBehind which is sort of just a bit of a short journey through improving our payment resiliency over time as we've grown very fast as a company. So I work for a company called Pushpay, I'm a Principal Engineer there and we're a technology company that is working in the church sector. We're really sort of building easy to use software that helps churches grow their ministry. So that's through payments and engagement solutions. We have around 350 staff and we serve over 7,000 churches and we're based out of Auckland, New Zealand originally but we actually have more staff now in the US based out of Seattle. So in the beginning when I joined Pushpay that was 2014. I was engineer, hire number five, our VP of Engineering and in fact another engineer in the audience, Carl, joined about two months before then and things were pretty simple, we sort of inherited a very basic monolith.  We were dealing with multiple gateways, some of them we talked to via Spreedly but for the most part we were talking directly to the gateways and we only did card payments, in fact we were really simplistic, we would allow a user to have a single payment method at a time, there wasn't a wallet where they could have a different payment method, they would have to delete that one to add another one. And we're processing about $1 million a year so payment volume was very infrequent. We really were sort of kind of in a hobby stage at that point. Where we are today, we're now processing $3 billion a year so a lot more volume. We use Spreedly for all our gateway communications with credit cards. We also support a number of other different payment methods so we support ACH credit card, New Zealand bank payments. We do sort of process checks, check scanning that kind of thing. And we scale to about 30 to 40 nodes and we're hosted in AWS now whereas originally we're sort of in a colo. We're also a little bit different. So unlike many people where they're sort of able to sort of process funds and aggregate them and then distribute them as necessary for a payments company, we have thousands of merchant facilities so every single one of our customers ends up with their own merchant facility. This presents quite a lot of interesting challenges from an engineering perspective but it does mean that we're quite well suited to what Spreedly does by having sort of gateway configurations become to be registered as we go. Initially when we started growing in 2014 this actually presented a bunch of interesting challenges to us because by having a merchant facility per customer we were wearing a cost of trying to grow really fast but it took a really long time to onboard our customers. It would take four to six weeks to actually get through underwriting and in some cases some of our sort of prospects would lose interest by the time they completed that process if they'd even get through it and we needed to fix it. So one of the things we did at the time was actually become a registered ISO so that we could market merchant facilities direct to our customers and that allowed us to go get that onboarding time and underwriting time done to sort of one to two days. And then we're in the generosity sector and generosity is also just a little bit different. So we're primarily processing gifts. We're not exchanging money for physical goods or tickets for the most part or any kind of other thing that might be like traded. We're dealing with tithes and donations. If you're not really familiar with tithes this is where you give a percentage of your income to the church traditionally 10% but it kind of varies depending on faith and kind of how wealthy the church in the area which operates. And payments are a combination for us of one time and recurring so we have people that are processing one time payments they're sort of giving in the moment so if they're in church or when they're just feeling generous or they hear about a cause they're interested in they can just go and give straight away and we also have people that are doing this more sort of tithes or giving to the church on a regular basis where they'll set up a recurring payment often aligned with when they're getting their salary so that maybe the first and 15th they'll set up a payment for a fixed amount that'll come out all the time. And because we're dealing with churches our uptick is a bit different. Sorry, windows. Our uptick is a little bit different to everyone else so on a US Sunday is when we see a strong uptick in volume as people are sitting in church and getting out their phones to give. The other thing about churches is that they're a community and for a community, trust is the currency of a community. So the trust between the community members which is the church goes and the church is really important. We see that the trust between those community members and the church is also really easy to erode and we have to be very careful. So when a church picks a technology provider such as Pushpay, you might think say if Pushpay is having a bad day, a gateway is down we're unable to process payments that's going to put people off using Pushpay and the church goes will be like that Pushpay we need to get rid of that, that's no good. That's not where it really happens. They actually start losing trust and their church's ability to select an appropriate technology and they may just stop giving to the church completely. They're sort of put off by the experience. And that's not a great outcome and it can be sort of irreparable in some ways like if you have a really poor choice of technology provider you will see those people either sort of stop giving to the church or the amount they give goes down and doesn't come back up again. So we need to be sort of very careful with that trust. So it's 2014. It's a US Sunday. We're like a new fresh team, we're all excited. The gateway is down. Or a gateway we're using goes down. You can imagine we're sort of leaping on call we're calling everybody, we're getting our gateway on the line like get this stuff back up, this is terrible we're all going to die, it's down. Well that sounds great but in reality we just didn't even know we were down. So in 2014 we were really immature but unfortunately the normally when we'd find out we were down in the first case was when a customer would call us or even worse when an investor would call us to let us know we were down. That is not a place you want to be that is something you need to fix, we could do a lot better. And in our case that tied in really nicely with being a young engineering team and wanting to introduce a bit of extensional rigor and we did that through post-mortems. Well in fact plain list post-mortems which we sort of adopted from Etsy's process. So the process of a post-mortem is really sort of after an incident or during an incident you sort of collect a bit of a timeline you're trying to see what went well and what did not go well about your incident response and then sort of try and take some learnings from that and possibly even identify some mitigation so you can stop it happening again or mitigate some of the impact so that it has less impact the next time it happens. And for us we did something which was both intentional and not necessarily intentional which was we decided we're a payments company we need to do post-mortems when we are failing to serve our customers. So if we failed to process a payment for our customers for something that maybe we did from an engineering perspective that was a reason to initiate a post-mortem. This also applied to gateway outages. So even though it was something that was outside of our control in many ways at least that's how we failed at the time. We still sort of would instigate a post-mortem for that and it had some really good outcomes for us. It meant our incident response process we kind of looked at how things were going we would go oh maybe we need to deal with our gateways a bit better and we also would identify opportunities to actually sort of mitigate some of those potential losses. So early on in our journey we weren't processing a lot of payments so it indicated we were doing sort of about a million dollars a year that meant there could be hours without payments overnight it was kind of tricky. To deal with that we sort of initially started looking at connectivity as our first sort of low-hanging fruit for really kind of getting to be more aware of what was going on with our gateways so before we approached this was really simple we just had sort of like a pingdom so just some kind of ping tool pointing out our gateways externally so that we could kind of see if they were up or not and at the same time we introduced a health check-in point internally with our network that would when hit run a simple set of tests to kind of see from within our network can we reach back out to that gateway so that we could kind of test things like firewall configuration proxy configuration that kind of thing. The combination of those two things allowed us to kind of really narrow in quickly when we had a connectivity issue as to where it was happening was it something we'd done internally that might mean that we need to roll back some kind of infrastructural change or was it something external like the gateways down and then we need to start escalating. We don't really rely on connectivity tracking or testing anymore we have more than enough volume 24-7 that the actual payments themselves become the indicator of failure now but there was some really nice side effects of establishing this process really early on so for instance like sort of middle of last year I think it was July we had an issue with First Data where they actually messed up some DNS records for us because we had really good connectivity sort of tracking and monitoring at that point that happened we were able to jump in identify that it was a connectivity issue that it was a DNS issue look back over our records of what the actual IP address used to resolve to for that service endpoint and then reach out to Spreadly and go hey would you mind just sort of temporarily creating a post entry just to work around this issue because First Data has kind of not got it all together right now and thankfully that kind of relationship allowed us to save quite a lot of volume over that incident so beyond sort of just connectivity tracking the things that we need to do is sort of start dealing with the more nuanced side of classifying responses there's a lot of opportunity to identify failures that aren't classic just connectivity issues but are more sort of something going wrong with the gateway or the gateway's behavior and we classify our responses into sort of three buckets so we have those user recoverable errors those are responses or response codes that we we know we can build some user experience around we can have people sort of go through a flow of say I've got a bank declined or insufficient funds we can provide an appropriate message they can try the payment again and then the bucket we're really most interested in around incident response we have three response codes we have gateway incorrectly configured so if you recall I mentioned we have lots and lots of merchant facilities it's possible that one of those merchant facilities may say have an expired password or an incorrect login and in those instances where we we can identify that in the response and classify it appropriately that's really only going to affect one customer at a time so it's not necessarily the kind of thing we want to wake up our whole on-call team for depending on the size of the customer and then we have two other responses which is communication error and provider error so communication error is kind of is what it sounds like that would be an issue with us being able to either talk to spreadly, talk through spreadly to the gateway or even potentially some kind of upstream issue with say a bank or a card brand that then looks like a communication error presented by the gateway we also have provider failure so provider failure is really sort of the counter example to user recoverable there are non-user recoverable error where we need to escalate with the gateway to either go like hey what's going on there's something weird happening with the behavior of the gateway we're seeing persistent failures that are not our user's fault and we sort of have discovered those over time as we go and then last of all we have unknown error so those are where we actually can't classify the response because we've never seen anything like that before and when we get those we've just got a task of investigating them and figuring out how to classify them so when we think about classification of responses gateways return structured responses you're getting JSON, you're getting XML we're talking to quite a lot of different gateways unfortunately like no gateway is well none of these gateways have really agreed upon a good way to represent these things in a common way so you'll get an error code, you'll get some error messaging but every gateway decides to kind of group the error codes independently or different ways and unfortunately it's a really leaky abstraction we see in some cases gateways will group things that we would consider a provider error it's not user recoverable alongside something that is recoverable so say like CVV incorrectly entered and it can be quite challenging to kind of tease those apart if you're relying on the codes and we tried to look at this in different ways over time and we just kind of eventually came to inclusion no you can't do it, just treat it as text pass it as text this sounds like it shouldn't work it works really well and the other thing is it actually gives you some nice benefits in that you can use the same approach for classifying weird responses you get back from gateways when a gateway starts giving you HTML back for instance I won't name any names for us we deal with we sort of like doing declarative rules for these kinds of things the kind of code that we're going to present back to the business so they can also kind of check out our working check our thinking so we have sort of declarative rules for doing this it's going to be pretty simple we've got something we were looking for the upstream error code text we're going to map that to being a decline for comms error and this is a rule that's only going to be applied to the pin gateway through Spreadly when we first started doing this we kind of fell into this pattern of adding these declarative rules and we didn't really think too much about how we were evaluating them we stored them in a list and we'd start adding them to the bottom of the list it worked okay for a while until we started realising how much this code because no one felt safe adding new rules they had to decide very carefully do I add it at the bottom do I put it somewhere in the middle at the top because it was a risk of occluding an existing rule and causing it its own post-mortem just by adding a new rule as we kind of sat and thought about it it actually kind of occurred to us it's really quite simple what we need to do is just sort those rules in a way that goes from most specific to least specific and so we just sort them by the length of text we're searching for and then how specific the rule is say all gateways because it might be say we're looking just for the evidence of they're giving us back an HTML response we'd make that less specific than a rule where we're making it specific for a single gateway or a single car brand and then if we don't match any rules then it's classified as an unknown area and we'll sort of trigger off a process of investigating how to map that later so once we had these provider areas and communication areas it was a good opportunity to then start actually putting in place some alerting of like when they had elevated rates which is great you're kind of getting this alerting now we've got a lot more like idea of what's down we might know what's down before our customers do but we still have been stuck with the problem it's like what do we do we're going to sort of log in we're going to sort of triage maybe what issue it is is it a gateway down we'll start escalating the gateway but meanwhile there's just things are on fire we're just seeing payments failing this is not kind of a great outcome and one of the first things we came to was like it's really simple to kind of just turn off our schedule payment engine and stop processing schedule payments at least during a gateway outage and so we started with a manual command for doing this but as we kind of went we realized sort of with the manual command there was some operational concerns we needed to address and we really started like chat ops and so we wanted to know sort of who executed the mic command and then we really want to know like how do I know it's off like if I'm on call it how is the system going to wake me up to let me know and the way we did this was pretty simple so we have a manual command for turning off the schedule payments engine when we turn it off we end up with a slack message where we do our incident response it's going to indicate what the state was before and is now so in this case yep the engine is off and who did it so in this case I turn the schedule payment engine off and then the other thing we do is we have a health check endpoint again which we'll check if the schedule payment engine is on or off so in this case if the schedule payment is engine is off we count that as a loss and this health check endpoint starts returning like a status code 400 we're able to ping a point ping to matter and then wire that up to pager duty for our alerts so we'll then get a pager duty alert to say the health check is down this has sort of covers off both those concerns we had this is going to wake people up when we know the schedule payment engine is being turned off and then even better it's going to keep bugging us every 15 minutes until we turn the schedule payment engine back on because we forget to turn it back on that can have sort of real world consequences for our customers and especially their payers if we allow schedule payments to sort of run into the next day potentially people could be expecting that the money has already come out and then they go and spend money that then we later take out and they miss a rent payment or something like that that's not cool so we had this manual command and we were pretty happy you know we could turn the schedule payment engine on and off but there was a little bit of a problem with this and that it's 2 in the morning you get paid, you kind of get up, you get your laptop in you then have to go to a payments list sort through all the payments try and figure out which payments failed try and figure out which gateway it is or what the problem is and it's only at that point that you probably go oh I should turn the schedule payment engine off at which point we might have lost 10 to 15 minutes worth of schedule payments that we've kind of been plowing through just sort of throwing them into the ground we're losing a bunch of payments, we're really not happy about this so we needed to start automatically triggering but on what and in our case it was pretty simple we had these provider areas and communication areas that definitely needed an instant response so it made a real good case for turning our schedule payment engine off we also considered unknown given that those could potentially be a provider or communications area we get to classify but inevitably week on week we get new unknown areas so 99% of them are not provider areas or communication areas we're actually dealing with sort of things like unusual AVS areas we've not seen before things like that and so they're sort of really more user-recovered areas we have so if you love your own call team don't wake them up for unknown areas but we still needed a process in place to at least identify overall elevated error rates and kind of wake people up for that and so much like we do for classifying the rules for a response we do the same thing for triggers our rule for triggering looks like so we're looking for three comms or provider areas within a sort of five minute period in that case we're going to raise what we call a payment failure reaction type so we tried to separate what we do for identifying when to trigger from then what different parts of our system will take actions so the turning the schedule payment engine off would just be one thing that's kind of looking at their payment failure reaction type of comms provider area and then doing something appropriate once we introduced this we needed to kind of revisit what our on-call story looked like again because now we're waking somebody up and they don't know who turned the engine off or why and so we did a couple of things here we needed to put in why it was turned off so in this case we turned it off automatically say because of a communication failure and then more importantly we keep track of all of the payments that have contributed to that trigger and include links to them in our messages and this means when people get woken up they very quickly jump in there click on that link have a look at say the transcript and spreadly and within a two or three seconds they're kind of have got a keyed in as to what the actual problem is and can start deciding to escalate with the gateway or not so that kind of did it for schedule payments but for the first couple of years we're really sort of resigned to the fact that when we're processing these sort of one-off payments is people just getting their phone out and giving and church in the moment that they were just kind of going to fall on the ground and we're just waiting there watching the clock beating sort of our head against the wall kind of waiting for our gateway to kind of sort the issue out so that we can then get back to processing payments. It wasn't a great feeling we felt like we're sort of out of control and I've sort of mentioned initially that we were processing both through spreadly and non-spreadly sort of direct with gateways but over time we've moved all our communication over to using spreadly and as through post-mortems we kind of looking at mitigations or possible mitigations like actually if we're already tokenizing all of our payment methods through spreadly and you know we're getting to the point where we're about to process that payment we could just not process it and come back to it later. We're not exchanging anything for goods there's nothing we really need to kind of hold there we don't need to process these payments immediately and that's where sort of delayed payments as we call it came into our minds and what that process looks like is this so first of all we'll pick a gateway or really a gateway type we have a lot of gateway configurations but they're for a gateway type so like First Data, NMI, different kinds of gateways we'll pick one of those gateways and say let's set that to delayed. The next thing that happens is you're asked at the point that we're about to do some communications to sort of enact a payment with the gateway we'll check is that gateway delayed if it is we just choose not to do the gateway communications to kind of put that payment aside into a queue for later. The next thing we do is once the gateway issues are over and we're kind of happy like ah yep they've sorted that issue out we'll turn the gateway delay off that means any new payments coming in against those that gateway type will start getting processed so just we can kind of observe how things are going and then last of all we'll come back and release those delayed payments for processing once we're really happy everything's stable. To do this we needed to change a few things about our payment experience so this is our success messages for payments and you can see like the top messages what we normally tell users when things are going well if we've got delayed payments turned on then we're going to actually change the messaging up a little bit so we use the language authorized instead of success and we'll use sort of you know we'll tell you when it's been processed via email rather than we're going to send you a receipt and we were a little bit worried when we first introduced it there may be like this language would be confusing but it's turned out to be incredibly not confusing we've had basically no support tickets related to this kind of change in experience when we're delaying a gateway which has been really pleasing the other thing is we had to sort of think about the on-call story we now have a new queue so instead of turning the schedule payments engine off we've got these delayed payments sitting there that we need to process within a certain amount of time otherwise we start running into issues and we have a bot that takes care of this for us so it's just observing the delayed payments queue and it's going to then run out a message on a periodic basis kind of like how many payments we got in the queue for each of gateway because we could be unfortunately having multiple gateways having issues at the same time sort of overlapping where we may be trying to process the queue for one while sort of accumulating payments on the other because the issue hasn't been resolved and this is kind of working really well for us now processing delayed payments is interesting one of the things because we sort of we put this feature in manually and started using it and automated it is we were not quite ready for the somewhat obvious realisation that once you're queuing all these payments up you don't really know when it's safe to turn it back on I mean most of us are used to kind of a gateway outage but you're relying on seeing actual payments succeed to trust that the gateway has fixed the issue you're not going to rely on the gateway going yeah it's all good just because it never is it's always kind of like yeah in about half an hour it'll kind of even out so we need to come up with a process for this this is kind of pretty manual for us but it's not really an issue we'll go and find one of the churches is impacted by this outage we'll make a $1 gift to them and then we're able to just go in and find that payment within the queue and just release that one payment we'll monitor to see if that succeeds if that's looking good then the next thing we'll do is just underlay that gateway so that new payments coming in we can kind of monitor those see how they're going once we've seen a period of stability we used to be very cautious about there was like an hour we've got a lot more confidence in the way things work now so it's normally about five to ten minutes of stability then we'll start releasing the payments that we've got queued up but again we've really really risk averse so we'll always go and sort of sort those payments by dollar value first and pick a small batch of low dollar value payments so really protecting our customers from any additional risks if that gateway is actually kind of still flapping release those first and then if we're happy there we'll just release the rest and the other thing is we're fairly conservative so we're not trying to rush these out the door we kind of trickle them out one at a time so we sort of we just want to eventually catch up we're not going to try and put ourselves at risk by processing them really quickly have another failure. So it brings us to the sort of elephant of the room, like what if Spreedly's down? Well for starters if it was I imagine we'd see some people exiting the door, then very quietly. Yeah yeah that's fine, you don't know it's all good. But no it's all serious like we've again very risk-averse we kind of thought this room was like you know if Spreedly's down unfortunately that means we're not going to be processing with any of our gateways. It's the one sort of risk of picking one of those providers to kind of fend all your payments through. At the same time we won't be able to tokenize new payment methods. So it kind of initially sort of sounds like a game over. If Spreedly's down we've put all our eggs in the Spreedly basket we're not going to be able to proceed. But that's not really true because in the audience we've got a people who are churchgoers they're coming to church every week. Most of them are going to end up with an account they've got an payment method they've captured from the first time they did a payment and those people can just proceed through our experience without actually needing to do tokenization. So we're still able to sort of save those payments. And then the other thing we do is that we don't actually sort of mark Spreedly as an overall gateway of being down and then just not try to tokenize. We'll keep trying to tokenize with Spreedly on the hopes that it may succeed. If it fails we say like I get a communications error with Spreedly what we're going to do at that point is actually just change our user experience. We go hey you could use one of your existing payment methods or why don't you try this kind of groovy thing called ACH for this payment and you can kind of then fend them off towards a different payment method type entirely so that we can still complete that transaction. And this is working really well. To be honest Spreedly doesn't really go down so it hasn't been a problem. So I've been calling this delayed payments all the way through. That's definitely not what we call it. Do not quote me on this. We actually call it assured payments. So this originally came out as a feature that we identified as sort of on the back of a mitigation. So we feel a lot of pain as an on-call team. We sort of pride ourselves in how quickly we respond to these payment issues and it hurts us deep in our heart when payments don't succeed for our customers. It's not good and more importantly the impact analysis is horrific. So you want to just not do impact analysis on these incidents at two in the morning. You just want to go back to sleep. But as we sort of work through this feature we call it delayed payments because we're developers it's like well we're delaying the payment so it's delayed payments. That's just what you call it. When we're getting there to releasing it a sort of VP of Engines and some other people were kind of going actually you know what like there's an appetite for this. Our customers really really want this. It really plays into this theme of trust. They don't want us kind of failing their givers and then kind of that becoming an erosion of trust with the church. Why don't we start marking your customers and at the same time why don't we think about renaming this a little bit because we don't really want this negative term. And so we pivoted and decided to call it assured payments was really sort of identifying it based on the duty it fulfilled the customer. And it taught us sort of a really valuable lesson I think which is like we hadn't really think thought about marketing these internal engineering features around payment resilience to customers. It turns out it's what we do is what we're good at why shouldn't we be sort of marketing it. And today in the last quarter we've saved about four million dollars worth of payments and somewhat sort of ironically or awesomely like in the last talk we actually had assured payments go off for the first data and we sort of saved a few thousand dollars of payments as well just then. So how many 25k. There we go. So the system works. So just in wrapping up like I guess I've got a few takeaways. I think when you're starting out especially it seems like payment processing issues are really sort of out of your control. They're just not and it ties in really closely with I think establishing a process to learn from your payment outages. Once you have that process and once you broaden the reach so when you're conducting a post mortem it's an opportunity to share with a wider engineering team. People who were not involved at all in the incident response are going to come by and kind of go hey you know I've got some ideas about how we could actually make this better how we could save some of those payments. And then I think on top of that when you start thinking about these mitigations definitely employ them manually and learn about how to safely operate them manually before you then automate them. And this has two sort of good really good outcomes. I think you end up with a much safer command to operate manually and you get like a better outcome from a chat ops perspective. But also it means you can manually employ them because what we find is like triggering covers the basis or the scenarios you know. But inevitably gateways are going to throw up situations you aren't anticipating where you're triggering is not going to be capturing that and being able to just manually turn those mitigations on or off for a period of time is actually really cool. And it kind of helps build your roadmap for what you're going to actually build for your triggering functionality in the future. That was all I had to say. Thanks for listening. And if you have any questions find me during the break. Thank you.