Hi, hwnnw yw hwnnw yw hwnnw yw hwnnw yw Arbys TV yw hwnnw yw hwnnw yw hwnnw yw Caren Frankel yw'r West Australian Mixed Media Artist yw'r Educator.  Her experience comes from people of her experience with people of all walks of life has kept her passionate about bringing creativity to those who think they haven't got a creative bone in their body. Now in 2003, Caren started teaching small private drawing classes that grew into a full art school, Galleria Art Studio, running 17 classes a week for adults and children. She talks all about this in the interview, how it happened and how she ended up going to this next phase which in 2020 her long held dream of becoming a published author and this fabulous get drawing book even if you think you can't was born into the world. So in this interview of course we're talking about how to get your book published. So let's go ahead and dive straight in. We're going live into our business for artist Facebook group and today I am super super super excited to be doing this very first live interview actually with Caren and we hope that it's all going to work. We hope that the stream is going to hit the group and you can all see it. So while we're waiting for people to come on board, we're just going to have a bit of a, we're just drinking our tea aren't we Caren? Cheers everyone. Cheers. What have you got tea coffee? Coffee. Coffee. Perfect. Perfect. Okay. So. Okay. So if the first people I can see that somebody somebody can see us. Can you very kindly let us know in the chat that you can see and hear us. Okay. Are we appearing on your screen? I can see I think is it. Oh, there we are. Look, I can just see us. Yes. Brilliant. Okay. But we're all good. And I can see a cheers from a Facebook user. That's from GoT so I can see. Yeah, that's great. That's great. Lovely. Thank you. Okay. Well, I guess we will. Well, we've got one minute so we'll have a chilly chatty and then we will begin on the hour because I like to be punctual. So somebody says in here and see brilliant. Unfortunately with the software we've got there we actually can't see who it is. It's just I can look if I look on my phone. So if you're going to ask us a question, we might just ask you to just pop your name because the software we're using doesn't tell us the name of the person. So it would require so if you keep refreshing her phone, which is not a great idea. It's not ideal in an interview. It's not ideal. It's not ideal. All right. Okay. I think it's four o'clock. I know that both Karen and I have been super super super excited. It's Joti. Yes. Hi, lovely. I can see awesome. Welcome. We've been so excited to do this interview. And well, first off, thank you Karen for agreeing and for being here. Oh, you're welcome. You're welcome. It's a favourite subject of mine. So I have no problem talking about my art in my book. Awesome. Awesome. Okay. Cares. Hi, Cares. Hello. Welcome. Welcome. All right. We're going to officially start as people come on. It's wonderful. You're watching the replay. Of course, you know what you got to do. I always love you to just hit the comments below hashtag replay and let us know that you are watching the replay. Wonderful. Okay. So our topic for today, of course, is how to get your book published. So I guess for some of you watching, perhaps this is something that you might be interested in. It's something that you've maybe got half a book written or you're thinking, oh, if only I could put that book together, I'd love to do it. And when I first met Karen, it was kind of what you were saying. I've got this book. Yeah, indeed. Indeed. I've got this book a couple of years ago. Yeah. Oh my goodness me. So it's been quite a project. It's been quite a journey. So let's just we just this is quite informal. We're going to dive straight into views. Everyone's ready. We're just going to go straight in there. So Karen, just tell us a little bit about your background before I met you before we knew about the book project.  Well, as you can hear from my accent, I'm originally Zimbabwe in South Africa and answer to both. Came to Australia in 84. So quite a while ago, always been creative. My father was an artist so grew up in that household. I did not study art beyond school. I did a BA, which was commonly known as a buggerall. Love it. So my art was always my hobby. It was never, it wasn't something I studied. However, I needed it more when I came when I migrated and I started doing courses and, you know, various classes, et cetera, et cetera. I have, I guess, always wanted to share my knowledge, particularly about drawing, because there's this crazy idea that, no, you have to be born a drawer and all of that sort of thing. Even before I knew I wanted to share it, I wanted to share it. Does that make sense? Yeah, that makes sense. That makes sense. You've always been an artist really? I've always been an artist. Have I always been a teacher? I did a teaching degree, but I never taught in a school officially. But I think I've always been an instinctive teacher. And one of the things that really started the ball rolling for me to be an artist, because prior to that, I was a computer programmer in another life in a business with my husband. In 2000, I had a mental breakdown and part of my coming out of that was getting back in touch with my art in a very, very big way. And that was, yeah, the catalyst to starting up my art again. Wow, that's quite a change, isn't it? But I guess it's those skills then that have you used those skills, because I know that you've also built an art business, a literal physical business. Did you use, tell us a little bit about those, about that business and maybe have you integrated some of those skills in there? Yes, so as a result of looking after myself, shall we say, I volunteered to teach small groups. I volunteered at a community centre, and so that's what got the teaching part started. And one of the places where I volunteered was for an adult community education for UWA extension. And I was approached by someone to say, do you want to come teach for us? And I went, really? You're going to pay me to do what I love? And so that happened and then those people said, resonated with me in my style and they said, we'll come to you for private lessons if you want to do that.  So it wasn't an intentional business. I fell into everything. And in terms of what I learned through my own healing, that came out through the art as well and the teaching of art. Because people are so self-judgmental about their abilities when it comes to art and drawing. That they think they should know how to do it and that should word is so terrible. That self-esteem issue, leave your judgment at the door, all of that comes into teaching art. And it's become your brand, if you like as well, hasn't it? Yes. So tell us, I pick up on that five people in a room. So many people say to me, oh, I'd like to teach, but I don't know. I don't know if anyone's going to show up. But it's the number one worry, isn't it, for anyone who's going to do a workshop?  What happened after that? Because I happen to know that you've just sold a very successful business. So take us just briefly from the five people to selling the business. Yeah. And let me say that I didn't hire the whore until I had the five people. I didn't do it as a bad person. I like that. Yeah. And in fact, nothing in the growth of my business was just build it and they will come. They came and I built it. Perfect. I love that. And I wasn't even aware that I was running a business, to be honest. So I did that for two and a half years. People approached me to say, do you teach kids? And I started teaching kids in another community centre. And I got involved with them. This is a very helpful thing to do. I didn't have it so feedback then. But what I did have and they still exist is the small business development corporation, which is free advice on all sorts of things business. Perfect. Yeah. And so I used their workshops. You know, what about renting? And I decided I did my homework and I decided to rent a studio. And in the lease I rented it for a year by year rather than five years and five years because I thought at the most I'll lose 10 grand over a year, which is not a small amount of money, but it's the most. It's better than five years worth, yes. Indeed. I like your thinking. So that happened. And everything about the business was people requesting. Do you teach watercolours? No, but I'll find a watercolour teacher. But now I was paying for a studio that was empty if I wasn't using it. So you brought in other teachers, didn't you? It wasn't just yourself? Yes. Absolutely. And then I needed help with admin.  And eventually I turned around to her and said, I'm running a business, you know, because I was running my org and I was running, I was advised by a mentor that I was seeing freely through the business corporation to get a system that looked after my students and the admin of my students. So it was a bit of a shock to my system that I was running a business. I wonder how many other people find that, you know, you just start organically somebody else, you do it. And then suddenly it's like, oh, actually, this is a business. Yeah. I didn't sit out to run a business. It was very organic. Yeah. And I ended up with 17 classes a week. Three different kids age group classes. Watercolours, acrylic drawing. So I had other people coming in. 17 classes a week of which I was running three. And that's the key, folks. Anyone who wants to do this is right. OK, Karen's not running 17 classes away. She's running three. And that's it, isn't it? You know, you've built a brand like that and you've got followers in the tribe and people who came to that business. So what happened? You got to a point where you decided you wanted to make a change? The first change was it used to be called Karen Frakel Art Studio and may I advise anybody out there not to, well, it's a bit cheeky to say don't put your name on it. If you're running an art business and you're selling your art, absolutely it has to have your name on it. Yes, yes. But the fact that I was running an art school meant that it didn't really need to have my name on it and I wasn't teaching all the classes. So about five years ago, I had a plan, I had an exit plan basically, but five years ago or longer I decided to change the name to Galleria Art Studio, which was an excellent move because what it allowed me to do was separate myself as an artist, Karen Frakel Artist, and Galleria Art Studio. Absolutely. And I took a term's leave, I gave myself long service leave. I love it, I love it, it's brilliant. And I wanted to see if the business could run without me. Yes, because at the end of the day, the real business should run without you, that's class technically, that's what I learned back in the day. Yes, exactly, and I worked very hard to make it so. Yes. And my admin lady knew everything, so she ran everything. I got a replacement to teach my classes for a term. All the other classes were being taught by the teachers anyway, and it worked like a charm, which meant it was saleable. Yes, and that's what you went on to do then? Yes. So how easy was it to sell an existing business? Did you find that process easy? I found it easy because the right person came along serendipitously. The right person came along at the right time, and she's worked out to be... It was a win-win situation that was, if we told you the full story, you'd think that only happens in the movie. It's that good. It's that good. It's fantastic. Although I did engage somebody to help me, but in the end didn't need to use him because this person came along to find out about running an art school. She lived in Port Headland, and I said, oh, it's a pity you don't live in Perth because I'm selling my business. And she said, really? The whole thing happened. That's wonderful, isn't it? When it all just falls like that, it's the best possible way. I love that. So having sold the business, what was the plan? What was you said you had an exit strategy? You exited from the business, and so where now does the book come in? OK. So the book has always been there. I've always wanted to take my teaching, which seemed to resonate so well personally. I wanted to put that in book form if possible. And somebody, a friend suggested, based on her son wanting to draw, that I try and marry written words and video somehow. So we weren't watching a YouTube video, but we weren't just looking at a book, and I thought that was a brilliant idea. So long before everyone became OFA with QR codes, like they are now, I had professional videos made to link with the book. So I'm not answering your question. That's all right. I've always wanted to do it. The man had just started seven years ago just before my dad died. He said, where are you going now? He asked me exactly the same question. And I said to him, I'm going to write a book. And he looked at me and he said, how can you write a drawing book? And I said, just watch me.  So how easy was it to write a book, Karen? Not as easy as you would think, Soph. Although most people don't think it's easy, I know. And on the other hand, easier than one would think. Because I'm a natural talker. I found that if I wrote, you just have to read it all out. That's the first thing. I did have some strategies, but in terms of how to set it up, because I taught it for so long that I knew how I take beginners through. And that was what the book was going to be. I wanted myself in a book, my classes in a book. Yeah, in a book. But at the end of the day, I worked better when I'm talking, when I'm demonstrating. And so the way I wrote it was this, which has its own idiosyncrasies. The other thing that I was aware of was that I needed pertinent sketches and diagrams, pertinent photographs. And I'd chosen to go the video route as well. And I wasn't, you know, now being of faith with all of the stuff, maybe I would have done it myself, but I wasn't then. OK, so you wrote the writing if you outsourced the other bits? Yes. Yeah. And if you were doing, I'm not going to ask you that question later, so I'll hold that question off. But OK. So what was your process? So the only output was the video and the design. Yeah. The final design. And how long did it take from kind of, OK, I've got the concept. I'm going to write this book to, I'm holding it in my hands. Seven years. Oh, OK. So because I was running a business, I didn't actually dedicate myself to, OK, every Monday you're going to write your book. It was when I had breaks, when I had the motivation to do it, and the best motivation of all was going away. I had to remove myself from my home. Yeah. Even from my dog maybe, or sometimes she came with me depending on where I went, but I had to, and then I dedicated the whole time from waking up in the morning to going to bed at night was book. And that got me through well because I'm a big procrastinator and I'm a big, I might sound bolshian confident that there's a little person in me going, ooh, I'm writing a book and it's not going to be good enough and it's got one and why am I just trying, you know, so that self-doubt obviously creeps in. So OK, that's a really good point because I think a lot of people would be hampered there, it's a lot of drawing books out there or a lot of painting books or whatever book you're going to put together. How did you think yourself out of that process? How did you get out of that process? Or didn't you? I did get out of that process, but I had to continually convince myself that when I had those thoughts, I had the arguments ready which were, no Karen, your book is different because you've got the video. The other reason why it was different is because of my teaching experience meant I had, I'd met everyone before, I knew what mistakes they were going to make and most drawing books, you have no clue why your work is going to be different to why it doesn't look like what your exercise is telling you but you haven't got an answer and I hopefully in sections called Watchpoints gave them the answers to did you do this, did you do this, did you do this, did you do this so I knew that there was also something different so I had a unique selling point shall we say. That's the most important thing isn't it? Did you do any research before or did you just go look I've taught this, this is my thing, this is what I'm going to do, did you look at what people had written or did you decide not to do that? Absolutely, no I did and sometimes I found a book and I thought to myself why am I bothering, it's all there it's beautifully written so sometimes I did that but I have to say which I feel quite chuffed about is that most of them the majority were not aimed at, they were aimed at beginners but they were not written for beginners so they were assuming a certain level of knowledge assuming a certain understanding of jargon and I don't teach like that and I don't write like that and I think that was quite refreshing. If you look at the standard which is drawing on the right side of the brain which kind of has become the standard and I don't want to diss her because I wish that I had that at this but it is so boring and belaboring going through her screams and I didn't want it to be that. You wanted it a bit more hands-on a bit more inspiring, a bit more real and down to earth and I know you've got your copy there and I thought I was like ready to see I've got my copy right here so we'll talk about where other people get their copy at the end. If you were to do it again would you take seven years what would you do differently what would you say to someone who wants to write their book now? If you were involved in other stuff I think mainly the seven years became seven years because I was doing other things and I chose not to put my priority into the book so I knew that once I sold the business and I wasn't thinking about the business 24-7 I would have much more time and so I probably did more work in the last three years than in the previous four I marked about so yes I might take seven years I don't think that was a definitive thing of success or not success what really did it for me was the COVID lockdown last year Interesting isn't it is that a enforced time? So that's when I completed it the things I would do differently which I'm happy to talk about now or later depending on what you No let's do it now let's do it Change is I went through a massive learning curve and therefore I would change some things based on that learning curve The most important one I reckon is the editing of the book so in terms of I paid an editor somebody that I'd met many years ago and I thought I would give her the book she would send it back to me all edited and I'd send it off to my designer and bang bang the book would be done and that was a very naive thought so that you don't have to send it to a professional editor you do have to send it to a professional editor but not first you need to get other people as many people's eyes on it as possible people in your field people not in your field I had fantastic one of my ex students went through it she was an English teacher my best friend was an English teacher she went through it checking grammar I'm a writer in that I don't have to check my grammar when I'm talking easily and if I did that for the book I needed to check into it even though it was in my voice so I was very careful not to take me out of it I didn't want to correct it to that that was the major major major learning curve I gave my designer way too early not for other people to look at it before it goes to a professional editor and how did you in my opinion so how did you put it all together so we've got the writing which I guess you know literally let's be practical here so you've got the writing what on word docs this is the stuff people want to know you wrote on word docs you had it already so here's my menu I printed this up at at Officeworks and I bound it and everything so that was my very first book right yes in word I did try an app called Scrivener which seemed fantastic but in the end I was playing with the bells and whistles rather than just writing the bloody thing I knew that I was going to get a designer to actually do the physical layout of the pages etc so I had to give him placeholders for where the photographs were can be so you do have to pay a lot of attention if you're writing this sort of book to labelling your every image you've got have labelled what it is, where it's going to go otherwise you're going to go wrong especially if you're not if you're not designing it if you are designing it in something like publisher and you're of that nature and I am of that nature but eventually I said to myself Karen you cannot do everything great I love it it's so important to learn to go what are your key skills your key skills you've got the knowledge you're the teacher you put the information in you do not want to be not wasting your time but you've got better things to do with your time than to do that I love that so that went to somebody else very lucky in that I could afford to pay him I chose to pay for exquisitely designed professional I wanted a provisional result I didn't want to go what's it called print on demand which was my other option alright so you've got you get the first draft you look at it all and you're like okay we make some tweaks and changes how does it go from the designer to being here and being able to buy it what was that process like my designer has a relationship sorry I was just reading a a message my designer has a relationship with book printers so he got me a whole lot of quotes I looked at quotes I belonged to some publishing Facebook things that I was looking at lots of different options I was looking at ingram spark and I was looking at there's lots and lots of print on demand things even for Amazon it was massively difficult for me to get a print on demand for Amazon as well because of the images it was not suitable for a completely different ballgame isn't it to just write in a book and getting it on there I was so interested to hear the process because wouldn't it be great to put something like this but how do you so he found some suggestions and you did the hard labour and had a look at them all yes pretty much and another thing that I wanted because I wanted it to be what I wanted and again I could afford to pay for it so I take less profit because of that I wanted it ring bound right so some real work I wanted a cover so when you take this to libraries to book shops and I've taken it to book shops they all look at this and they go hmm now I can put that on my bookshelf whereas if it's a pure ring bound book that they cannot put on their bookshelf interesting so that cost me more money I have to say I chose to do that but that means now you can't get it I want this ring bound to go flat absolutely use your hope taking a lesson and the last thing you want is some book flat I've got a spine your attention to detail there is really really good I'm hearing that you've outsourced a lot and I'm not going to ask you what it costs but you've got a substantial investment into this my daughter came with my hobby because I took you money at it absolutely hopefully at some point they'll come back to you so how did you make the final decision with the company that you went with what was your criteria for that um cost they were all around the same cost for this level of quality it was made easier because not a lot of people offered this binding so it was made easier and quite frankly my designer is a professional designer who designs the docker's logos and he's up there and I took his advice I was happy to take his advice it's brilliant and it turned out just stunning because he's a stickler for detail and if they would have screwed it up they would have had both of us to deal with because I had gone through so much trouble that I wasn't going to lose it at the last minute and I've heard horror stories of people getting their books from the printer and going through and it's been crooked and it's been this and it's been that and I didn't want to go there so experience was worth every cent so we're saying if you want to put basically if you want to put an art book together a book with pictures with anything whatever you're teaching doing you want to put your own book together somehow A you did a lot of outsourcing which I really love and B you've really gone and got top quality you've taken people who know what they're doing and there advice and gone that's it I'm going with somebody who recommends that and you've ended up with this beautiful quality product and I think so like I said your attention to detail has paid off in abundance it's a stunning workbook I appreciate that What are your future plans for this lovely book? It is available as a pdf online as well as the physical book my plans ultimately are to turn it into a course that's what I thought you were going to say course so my thinking is Karen it's all you've got the videos, you've got the images you've got it what is there to do however there is quite a bit to do isn't there so just a little bit just a little bit so now that I've been working with Kajabi for a year now because I signed up for Kajabi specifically to launch this book for anyone who has no idea what that is that is an online software where you can have your website but primarily we're using it for launching courses and memberships and that sort of thing and other courses as well but specifically the book and also it's about introducing people and I can see from the person that's put that lovely message up I've got a get drawing Facebook group which is the community and that's one of the things I really wanted out of this was both before this for people to buy the book obviously but a continuing community of beginner drawers learning with me and that's something that we didn't talk about and I sort of looked through it and I thought that's interesting if you've mentioned beginners and you mentioned the other books we didn't specifically actually talk about the audience that you targeted this to as it's completely and utterly for beginners for those people who think I can't do it I can't draw or I'll never be able to draw well hence the title get drawing even if you think you can't even if you think you can't and when I thought of that I thought don't be naff Karen don't put a stupid that's just silly and then I thought no that's who you are go for your life absolutely how often do we get put off by a little voice in the head going don't do that don't do that but the fact that I came out straight away that was the right title it's your title it belongs to your brand your book I love it I'm so pleased that you just went with that you didn't allow yourself to go oh I need to have some other title yes yes yes thank you it's very powerful so you've got the group so people can can because I know that you do weekly videos you do weekly videos they've got the book and then they're going to be able to get the course yes hopefully so the big secret which is no longer a secret is that I this in fact next week is my last week after 18 years of teaching in the business I have now retired from teaching I will still be doing some physical workshops and relief teaching in that and it's very bittersweet because I leave some of my students who've been with me for 15 years and even the beginner students are going oh but we love you which is very flattering and lovely so now I can put more attention both into my art but also into doing developing this into a course I've got another course on watercolour pencils that I want to develop and change pivot shall we say absolutely that's what it's all about isn't it I mean your journey has been so inspiring I think not I mean the book yes but also building the physical business you know going through that whole sort of chopping and changing and now you know I'm really excited for you to see you know the next phase I think it's incredible so many people dream of having something like this and you know they don't make that happen so so what would you say to somebody who is sitting there thinking oh I don't know you know I've got an idea but just like you thought but I think there's a lot out there I'm not sure what would you say to someone who's sitting kind of on the fence they're not really quite sure I would say do it just do it even if it's for your own satisfaction but I do think I guess that you need to at least ask yourself why so if it's for personal gain fabulous if it's to make a big profit don't do it I hope I make a big profit but that's not the reason to do it to be a best seller yes of course we all want to be a best seller but that's also if you think you've got something to say like I had to play those two heads find that unique selling you know why is my book different to the others absolutely very very important there are other reasons for writing a book even if it's not different your voice has to be different just because it's you so it's not hard to find a reason why it's different and it's as simple as a notebook you know writing it physically or a word processor and can I say something that I haven't said yet which is a very big thank you to you Sophie because I met you at the right time in terms of my journey it was just before I sold the business and having a mentor having someone who kicks your back when you need it kicked is very useful and important so I highly recommend for people to do that for themselves and hold them accountable I think there was a bit of butt kicking I do remember in the process I think just whoever it is choose to do it make sure that you're speaking your voice and that you have that unique angle whatever it is and find someone to hold you accountable because if you're sitting on turrets trying to get it done there are easier ways reach out and get some help it's just been amazing amazing amazing so I think we need to let people know where they can get their hands on this book when can we buy it if you physically want to go into a shop to buy it I'll give you my address no I won't give you my address you're welcome to come and buy it from me it's available in stores it's available in a planet books it's available at Diabolic what's the name of the suburb specifically what if we're outside of Australia if you're outside of Australia you have to get the pdf I'm afraid you'd like to pay the enormous costs at the moment but within Australia I actually pay for shipping you can go to my website which is simply caronfrancal.com we'll put all the links to your website or your social media all the stuff will be we're watching this we'll put as much either above it if we're on Facebook or below it on YouTube but you can buy directly of my website pay it by credit card and I'll ship it to you perfect and if you're a long way off you can grab the pdf version unless you want to pay for the pdf for the postage so someone was just saying that she's overjoyed to be a member of Get Drawing and learn from Karen excellent artist, dedicated teacher Karen is an eternal inspiration I love that, thank you so much and I agree I've known Karen for a little while now and totally totally inspiring lady I say both as an artist and as a teacher and now as an author I think you know no matter where we are in our journey it's an inspiring book anyway and just to know the person that's put that together I think is magical so have you got any final thoughts anything else that you would like to share to our aspiring writers or aspiring art book publishers don't beat yourself up if you are procrastinating it's not a train rig I took 7 years and so if it takes 7 years it takes 7 years but do it that's the key isn't it that is the key it's the doing and there are lots of trial and errors and I've done pictures and I learnt to edit the photographs or I drew photographs drew sketches specifically for it and I drew them 3, 5, 10 times because they weren't good enough and then I scanned them so it is time consuming it is relentless it is nitty gritty stuff that just needs to be done not the fun stuff you just have to get through it I think you know what I've spoken to various authors over the years that's what I hear you just got to get over it I met somebody who said I went to a foreign country sat in a cafe for 10 days for some people that will be one way of doing it others will just spend 2-3 hours a day over a certain period of time it depends on your disability it doesn't matter does it matter to you you just got to do it the couple of other people on the book is definitely fantastic and happy to have it somebody says by direct if you've seen Karen at an art show as I did that's fine that's great as long as you're in North Australia but if you're watching this from I know the US it's available at the Mandeirin Art Centre and Midland Art Centre so it is available around Strat it's available if you're in Perth WA and then you can come and meet Karen as well so that's a double bonus thank you listen Karen thank you so much it's been so much fun it's been totally enlightening and I wish you ever every success with your course and of course ongoing with getting this book out into more hands and growth of the group etc etc and don't forget to follow Karen on the good old socials as well and we'll put links to all of those above and below these videos has anybody got I should have asked has anybody got any questions for Karen I should have said that at the beginning really if you've got any questions type them in now or if you're watching the replay of course post your questions below I'm very happy to answer questions happy to come back and answer them because I think it's such an inspiring topic I should have asked that earlier I should have said if you've got any questions just post them below and Karen's happy to answer them at any point thank you so much thank you for asking my pleasure alright thank you everybody for watching live as well you're amazing I can see we've got some Karen fans so thank you so much take care and have a great day bye bye everyone and get drawing bye bye thank you


This 2018 Ram 2500 Mega Cab Short Box is stock number 12367Z. We're here at Summit Automotive in Fond du Lac, Wisconsin, you're new and used heavy duty truck and RAM headquarters. This 2018 Ram 2500 has the 6.4 liter V8 Hemi engine, pumps out 410 horsepower. This truck has been fully safe and inspected by our service shop per the state of Wisconsin. Inspection process has a fresh oil and filter change. All the fluids have been checked and topped off and this truck is 100% ready to go. Bright silver metallic is the color. We shoot all of our videos in 1080p, 60 frames per second, so if you have HD capabilities on your computer, tablet, smartphone or television, I highly recommend turning them on right now because it is your best way to check out the quality, condition and options of the truck before seeing it in person. And if you like the video and want to subscribe to our YouTube channel, in the upper right hand part of the screen is a subscribe button. Click that and then click the bell notifications, you'll get updates on the videos we do each and every day here at Summit Auto. Comes with the 20 inch painted alloy rims and it has Nitto Terra Grappler G2 AT tires. They are 35x12.5 R20 LT's and they have right around half the tread left. Has the Bilstein shocks, part of that sport package and it does have a leveling kit in it. That's probably about a 2.5 inch to 2 inch leveling kit. Frame and underbody is in really nice shape, didn't see any major corrosion on there. Front fender looks really good, didn't see any dents or dings on that. Does have the blacked out headlight bezels, part of that sport package. It's not a rock chip, that's a fly. Front bumper is in excellent condition, no dents or dings on there. Has the front bumper parking sensors and the hood is in excellent shape. I didn't see any dents or dings on there. Looks really good. Passenger side front fender is in nice shape as well and the passenger side rim no scuffs or scrapes on there. As you go down this side of the truck, take note of how clean the body is, how reflective and mirror like that paint is. We take these HD videos so if you are far away or even if you're close by and just cannot make the trip down but you're still interested in purchasing the vehicle, you can see the truck, hear the truck and have confidence in the vehicle that you're looking at before you even get here. So when you do get here there's absolutely no surprises and you can make a smart and informed buying decision from wherever you're at.  Back rim is in excellent shape and the back tires have just as much tread as the front tires. The frame and underbody looks really good. Get the Bilstein shock back here as well as all the remaining factory exhaust. Low rockers and cab corners all look really good as well. What you'd expect on a 2018. Do you get the fender flares on here, part of the sport package. You also get the blacked out 4x4 and Ram logo back here and the painted rear bumper. No dents or dings on that rear bumper and it does have the backup parking sensors. You do get a full towing package on this truck with the receiver hitch 4 pin and 7 pin wiring and the tailgate is in excellent condition as well. Does have a locking tailgate, backup camera, spring and bed liner. You do get the 5th wheel and gooseneck prep package comes with 7 pin wiring back here as well and you do get LED bed light. Those lights turn on with that button there. You can see you get the lights up there as well as on this side and you also get the cargo cam up there which we'll check out in just a little bit. Tailgate shuts nice and solidly and as you go down the driver side just as clean as that passenger side. This looks really good on this side and for full disclosure there's back rim, excellent condition as well. Rest of the cab looks very nice, doors look really good, it does have the heated fold up tow mirrors, they fold up like so, they have directional signals in them as well.  There are no rips or tears on those seats, really nice condition. Power driver seat, both of these front seats are heated. Auto headlamps, power windows, power locks, power mirrors. You do get the leather bolster on there and these mirrors do power fold in. I always like showing both sides so that you know that both sides are working properly. We'll hop inside, check out the miles, radio and everything that this truck has to offer on the interior. You can see that this one has 29,826 miles and we do have a little over a quarter tank in this truck so I don't think we need to add fuel right now but it says it's low. Anyways, digital spinometer, compass outside temp display, range to empty and you get the time in the corner there as well. Instrument cluster is very nice and clean, comes with the leather wrapped heated steering wheel, cruise controls on the right, bluetooth and information center controls on the left. This one does come with the six speed automatic transmission with the optional tap shift. You get AM, FM and Sirius XM radio on here.  You can do your heated seats and heated steering wheel buttons there as well and you do get your backup camera there as well as your cargo cam. This one also has the Uconnect apps which includes projection manager where you can project your cell phone to the screen via Android Auto or Apple CarPlay. So if you have a navigation system on your cell phone projected to the screen it's like you have nav right on there such as Waze or Google Maps. We actually did a demo on Android Auto if you want to check that out it's in the upper right hand part of the screen but a nice alternative since this radio doesn't have nav.  You can see just how nice and clean this interior is. This thing is like a new on the inside, passenger side seat, no rips or tears on there, smells very clean inside of here. You do get speakers in the ceiling, part of that Alpine premium sound system, map lights, power sign, rear window button and home link buttons for your garage door security systems and lighting systems. It does have SOS and assist buttons on there as well. We'll take a quick look at the back seats and then we'll start it up and take a look under the hood. The back seats are just as clean as the front seats, no rips or tears back here. This does have the latch child safety system for any child car seats you may have. Power slide and rear window, once again these seats, excellent condition, no rips or tears. So the thing I like about the mega cabs is you can put these seats down two different ways.  What's nice about that, put both sides down, you can have a nice flat place to lay or sleep or if you have a night or if you have pets you can have a nice flat place for them to lay down as well. The other thing is there's storage back here, but the other way that you can put the seat down is just the backrest. So what that does is it gives you a higher load floor and it also gives you kind of an enclosed area back there to store extra things where they're not going to roll around. You have latch tether anchors back there and that sliding rear window. So that's two ways that those seats do fold down. One's very nice and clean back here, you get child safety locks on the back doors and the bottoms of the doors all look really good. We'll start it up and take a quick look under the hood. Starts right up, no check engine lights or anything like that. I would personally like to thank you for checking out the video today and hopefully from this HD video you'll be able to verify the quality and condition of the truck inside and out. Under the hood we have the 6.4 liter V8 Hemi engine, engine based, very clean runs, very smooth. Once again this truck has been fully safe and inspected by our service shop, has a fresh oil and filter change. All the fluids have been checked and topped off and this truck is 100% ready to go. Here's your mission sticker. I would highly recommend this truck to make quality and condition standpoint. And to see more pictures of this truck or one of our other 450 new and used cars, SUVs, minivans, Wranglers, halftones, recorder tons, one tons, you name it, we've got to go to that website right there, www.summitauto.com, full pictures and descriptions of every single vehicle from two locations all at summitauto.com. And if you'd like to check out more HD videos you can go to youtube.com slash summitauto. Remember to like, subscribe, and share on this video and all the videos that you see there. Click those bell notifications, you'll get updates on the videos we do each and every day here at Summit Automotive. In fact in a second you'll see a link to subscribe to our YouTube channel in the upper left. We have Ram 2500 truck videos like this one in the upper right. Link to this vehicle on our website in the lower left and a link to one of our latest YouTube videos in the lower right. Click those, check us out. We're super excited to help you with this Ultra Clean 2018 Ram 2500 Mega Cab Short Box, Big Horn Sport, and Bright Silver Metallic. Thanks again for checking out the video. Remember to like, subscribe, and share.


highlights RGO-slash-Mzine-slash-TO2-slash-V2C heterointerface porous microspheres prepared via scalable method to boost polarization, customization of hierarchical structure by precisely tuning 2D RGO-slash-Mzine intercalation, optimal reflection loss of minus 67.4 dB and EAB equals 5.47 GHz at low filler loading of 5WT%. Simulations showed the benefits of 2D nanosheets intercalation on polarization loss. This article was authored by Gu Wang, Chongfongli, Diana Estevez, and others.


Ahead of the forthcoming election, Ogun said governorship candidate of the New Nigeria People's Party and NPP, Olufemi Ijadi, has a short Nigerians of a better nation. Ijadi disclosed this during a New Year celebration held at Ibar for Ogun State, while urging Nigerians to collect their PVCs to exercise their franchise. He therefore called for a change of heart on the part of Nigerians, leaders and followers, to enable the country to move away from needless killings, corruption, fraud, kidnapping and selfishness, and embrace security of lives and property. You have both hands at this time. It is time for you to come and change the narrative. You are the only person that can change it. Politicians cannot change it for you. That Nigeria is going to be better again. Nigeria is going to be great again. Nigeria is going to be a great country.  Do not lose hope. This is not time to lose hope. Nigeria is a great country, which you and I can make it great again.


You can help a child improve their math skills with an activity called I add you add To do this We just need some small objects that you can easily move around and count For example, I have some glass beads here You could just as easily do this with some rocks or beans or any small objects you have around So first we're going to move them into two groups One group will be for you and then the other group will be for the child And you can start and you can say okay I'm going to move a few of mine out into the middle here and then let's count how many there are and The child can count them one two three and you can say very good there are three Now you can invite the child to add a few more so the child might say okay I'm going to take these two and put them out there and you can say okay Let's count them again and the child can count one two three Four five and you can say yes very good. There were three and you added two more and now there are five Then you can add a couple more. Let's see you might say I'm going to add two more here And let's count them again Then the child can count one two three four five six seven Then you can continue taking turns adding a few more and counting how many there are each time Doing this activity will help to introduce the idea of addition


Is it a sin to ask God for a sign? Often, many will quote the following verse to imply that asking God for a miracle or a sign is wrong. Only an evil, adulterous generation would demand a miraculous sign. Context is key. Here, Jesus was rebuking the religious leaders who in the state of cynicism were testing Jesus and demanding a sign, even though he had already performed many notable, credible miracles. It's one thing to demand a sign from the place of unbelief. It's another thing to ask for God's guidance from the place of sincerity. If signs were evil unto themselves, they wouldn't accompany the gospel message, and Jesus said to believe him based on his works. God has given us signs of his very existence. There are several instances in Scripture where signs are not just tolerated by the Lord, but highlighted. It's not immature or selfish or evil or demonstration of a lack of faith to ask God for guidance or for a miracle.


So, mae'r byw chrys ac yn amlwg chi i gwn i'r bwysig o ddullus. Yn y cwmhoedd blynedd, roeddwn i'n ddatblygu'r gweithredu cyfaint o'r cyfrannu gocardlus yn y cyfrannu cyfrannu cyfrannu. Yn cyfrannu gocardlus, mae'r api o'r fforddau gwybodaeth, mae'r gweithredu i gwybodaeth http, yn gyfrannu llwyddon cyfrannu, dweud 100 o gyfrannu, Those allocate money from their customers and we transfer that and everyone is generally happy about their lives. Because of what we do we place a high value on that request. It's not something like a social media post or something similar where if that iss not the end of the world. About people's money here so they tend to foresee completely if they are working properly. Now. It can cause emergence to loose customers if we fail while they are in the middle of check out.   You're probably gonna have different concerns to us. Like everyone in this room works at different companies with different restraints and different things that their customers expect of them. But I think ther TRus ym comen ideas that we can extract from the times we failed to be up and available that probably apply to a lot of peoples different situations. Dyna, gallwn i'n gallu'n gweithio'r system dylau, gallwch i'ch gweithio'r cyfrwng, gallwch i'ch gweithio'r cyfrwng o'r cyfrwng, ychydig i'r cyfrwng sy'n gweld ardod i'r llangos. Mae'r cyfrwng yn y ddweud, mae'r cyfrwng ymryd cyfrwng yn y baeth cyfrwng, ac mae'n cyfrwng yn cael eu bod yn cyfrwng ymryd yma roedd, cael y 5 o 5 o 5 o syml. Mae'r cyfrwng yn cyfrwng, a'r cyfrwng yn cael eu cyfrwng. ym mwy o'r pwysig iawn, yn ei wneud i'r wneud'r rai'n hynny, pob edrych i'r pwysig iawn, i mwy o'r rai cyntaf, ond ei wneud i'r pwysig iawn fel y dyfodol yn arwag, ond mae'n rhaid i'r cyntaf ym mwy o'r pwysig iawn. Byddwch chi'n pwysig iawn yn y rherod o'r ddataeth a'r mewn ni. Rwy'n rhaid i ddechrau, rwy'n ddechrau, y dyfu'r pwysig iawn a fyddai sy'n hoff wonnag wedi aralu. Fy fawr, mae'n gwrs gennym wnaeth gyda'n gwyrddio, ac rydych chi'n ymwyaf, mae'n golygu'r rhaid ond o'r bod yn gennym ei ddam. Mae'n gyda'r ddeudu Miol ac mae'n bydd mwy o'r ddod yn yr uwch. Oes y golygon ni, gawaynt mae hwnnw i'n rhai! Mae hyn yn gallu bod yn gweithgrifennu i gyd ymgwyllfa o'r cyflawni. Mae hynny'n gweithio i fynd yn yn ôl. A maen nhw'n ardal i'r fflygu poisoned o'r chweithwyr ond yn ôl. Mae cymdeithas, mae'r cymdeithas yn ymweld yn ymweld yn rwy'r cymdeithas yn y Gygo Cardlys, ac mae'n bod yn ffynwyr o'r gweithio'r storiaeth a'r cyllid yn gyfweld, mae'r bwysig yn cael ei ddweud a'u fwyfyn yn yr ysgrifennu gyda chi'n cymdeithio'r cymdeithas. If you're not familiar with postmortems, I recommend Etsy's article on this and I will be posting all the slides so you can grab the links later, but that's a really good description of how they run postmortems at their company.  So I'm going to give you a best of of our incidents, the five best down times we've had. They break into three categories. The first one is about running out of a resource on a box, CPU, memory, disk, whatever. The third incident is about weird interactions between your ORM or database driver and the database itself. And the last two are around running clusters of highly available databases. For each of those incidents, I'm going to talk about what happened. I'm going to pull out something we learned. I'm going to try and give you some ideas that you can take away and use. I'm going to summarise at the end with three things that I think you should definitely do and there'll be a little bit of time for questions and then I'll be around in the hall for lunch as well. So, let's get started by talking about running out of resources. Incident number one, the fast migration that wasn't fast. This was the first time that I caused a complete outage of GoCardless in production. So it stuck in my mind pretty well. For a little bit of background, we will make roughly 10 to 20 schema changes per month. Sometimes this dips down to like five or so. I don't think it's ever gone much beyond 20. And in this I'm including everything from creating a new table to back like a new feature like subscriptions or plans, adding a column to a table for similar reasons, or adding an index to an existing column to fix a performance issue that you've hit. It fluctuates, but it usually falls into that range. The problem with schema changes is that they typically hold exclusive locks. They need to modify the actual structures of the database and they need nothing else touching those structures while they do it. The issue is if you have exclusive locks and they're long held, nothing else can use your database. There's plenty of guides on the internet on how to not put yourself into that situation, how to make sure your database schema migrations are fast. There's three bits that come up all the time. The first, don't add columns with defaults. If you add a column with a default on a large table, Postgres is going to sit there holding an exclusive lock on the table while it rewrites the entire thing with the new column. That's bad if you've got like millions or billions of entries in a table because nothing else can touch that table for the time. The second thing is to validate any constraints you add after adding them. In Postgres 9.4 and above, this does not hold an exclusive lock on the table while it validates the constraints it actually holds. So it's generally something that you want to do. Lastly, add any indexes concurrently. What this means is that Postgres will hold a weak lock while it builds the index so the rest of your traffic can continue normally. Following these rules, this migration seems fine. We want to add a refunded Boolean on to payment so that our merchants can now issue refunds and send the money back to the customer. It seems pretty easy. We're adding a column, it's not got a default so this can't cause any kind of problem, right? Well, as I learned on that day, there's this thing where locks can get queued if they can't be acquired. It turns out that when a lock can't be granted, the query enters a queue to acquire that lock. Anything that would also conflict with that lock will queue up behind it. So, let's look at some sample queries. Let's say we have a query that's like select all the customers that have ever made a payment through GoCardless. This is like run-of-the-mill analytics. You want to see maybe how many customers you have, who it is, who your customers who sign up but never make a payment are. That kind of analysis. You then deploy your new feature, which is refunds, but this has to wait behind the analysis query, which is taking quite a while to run because it's a select distinct. What that means is that your regular API queries, such as viewing a single payment, no longer work, they have to queue up behind that thing that's trying to grab the exclusive lock on the table. What that means is that your API is down. In this case, we had a total outage of about 15 seconds, where critical parts of our API payments were no longer available. So, what was going on here? What was the core of the problem? The issue is that we were kind of mixing these two different loads on our database. The first one, that select distinct customers query, is something called online analytical processing. This refers to when you're performing like aggregation over larger datasets. Generally, these queries are slower. They operate over a ton of data, and they look a bit different to your regular API queries. Our regular API queries are generally in the category of online transaction processing. They operate over a small number of rows, and they don't try and perform aggregation functions on the whole. Something we decided after this was, don't OLAP where you OLTP. Move that OLAP load away from your primary database server that's serving API traffic. There's actually very little reason to do that sort of analysis there. You can easily do it on a replica, or you can move it out to some data warehouse, or Google BigQuery, or whatever if you're feeling fancy. It doesn't need to be load on your primary database server. We also learned that it's always wise to set appropriate bounds on your system. Now, this is a slide that we're going to get super familiar with by the end of the talk. Now, pretty much every postmortem that I'm going to run through, we run into something like this. It is better for you to set a bound on some part of the system, for that part of the system to fail in an expected way than for you to set no bounds, and the system still to fail, in a way that you don't understand like the whole thing falls over. In this case, we started setting these two things on all of our schema migrations, lock timeout and statement timeout. What this means is that if a schema migration is waiting to acquire a lock, as we saw before, it will give up after, in our case, 750 milliseconds, meaning that all the regular API traffic that was behind it in the queue can go through and there's no problem. Sure, your schema migration failed, but it is less critical for you to deploy a new version of your software than it is for your API to go to stay up. You can get your developers to deploy a bit later in the day or whatever, but fundamentally, you want your system to be there rather than your new version to be out. We went into way more detail in a blog post on this, so again, I'll be posting the slides later. If you're really interested in this one, we've got a full write-up of it. We've also open sourced that little piece of code that inserts those timeouts around every schema migration. It drops straight in if you're using Ruby on Rails, but also you can probably rip it off in any framework you're using because it's like 100 lines of code tops. We also have a wish list, so what we do there is we set timeouts on lock acquisition and on a statement level, but locks actually pile up if you begin a transaction. Every statement you run inside a transaction keeps the locks that it was granted. What we'd really like is a way of saying, set a timeout on this entire transaction because what could happen is you have someone that does begin transaction, then runs a schema change that's quick, then runs something that's really slow, and the exclusive lock on the schema change is still their blocking queries. There's something that looked a little bit like this in 9.6, but it's only for idle connections, which isn't exactly what we want here. Also, turn this setting on. Log lock weights. What this will give you is visibility into any time your database is waiting on a lock so you can get a general impression of how contended your database is. Cool. Incident number two. Do we just use one and a half terabytes of disk? Answers on a postcard. So, this one started off with an alert that really got our attention. For background, we do not have big data. Generally, as a payments company, your records are fairly small. You're not trawling the web for everything and building an index of it. You've got records of payments that happened. So, until you hit mega, mega scale, you won't have anything that I would actually classify as big data. To be clear about that, I'm talking about our data is measured in gigabytes, not terabytes. We had previously, to this incident, spent about two hours chasing what looked like totally random timeouts between our services and we couldn't connect a pattern to them.  When we looked at the graph, we saw this. At some point in the day, something had started cheering up disk on Postgres like nothing else. This is a really horrible alert to receive because somewhere a little bit up there beyond where the graph ends, is how much disk you actually have and we don't want to hit that. So, something wrong had happened earlier in the day to cause us to start cheering up disk. When we went back to the log files from Postgres, we saw some lines that we'd missed. Postgres was creating a ton of temporary files for some reason that we didn't know. Now, the reason we've missed these is that we were looking through the logs for stuff that we already knew about. We were looking for slow queries and stuff like that missing what the actual problem was because we didn't look at the logs as a whole. So, what was it this time? It was a runaway analytics query again. In this case, someone was trying to add a breakdown of how merchants were using our API. So, what different features they were using the most and it would generate an email to centre developers to let them look at that whenever they fancied. The problem is that that joined across many, many tables to do so. It was going from everything from merchant through their payments, through their agreements with customers, through absolutely everything. The result of that join materialised into memory was massive and therefore Postgres started backing it down on to disk in order to provide the result. So, once again, don't overlap with your LTP. We were bitten by basically the same problem and after this, we really started to shift more and more of our workload off to data warehouse instead. So, again, we saw this problem of queries operating over large datasets interacting poorly with queries operating over small datasets. We were lucky in that this was a near miss. We managed to cancel the query before it hit the line of doom at the top of that graph and then we got some time to really think about this and come up with something that would stop it happening again. Once again, we set appropriate bounds on the system. In this case, a flag called tempfile limit, which is a size in kilobytes of the maximum size of temporary files created per Postgres session or connection. What this does is it shields your service from road queries. It means that no one connection can just sit there and fill up the entire disk of your Postgres server, which is a good thing to do. But there is a caveat that comes with this. Generally speaking, your data is going to grow over time. I don't know of companies where the opposite of this is true. Please tell me if you know of one, because I'd be kind of curious. The bits of your system that are liable to break when you set this are things that aggregate large bits of data together. Now, in our case, this was our payment submission run, which happens at some point during the day and has to aggregate everything that wants to be submitted at once. Again, that's kind of an OLAP thing, because we want to do it over fairly live data, like payments that have actually been created today. We couldn't do that off a data warehouse. This was something that broke for us when we added our temp file limit. We had to bump our temp file limit accordingly and optimise some of those queries to not need temp files. Be a little bit wary when turning this on, like some stuff is going to break and some stuff may break down the line. What I would recommend is instrumenting your server for how much temp files are we using, getting that into graphs, setting an alerting threshold of whatever your temp file limit is, and then being a bit before it, and then you will know before this is a problem. Cool. You want to measure these things as you approach your limits. If you're setting a limit, you probably want an alarm at 90% of that limit or whatever. There's one more caveat. Index creation uses temporary files. Whatever you set as your temp file limit is instantly the largest index you can create, which is frustrating because you may have some fairly large tables. The way around this is to disable temp file limit during your schema migrations. Much like we wrapped timeouts around our schema migrations, you can set temp file limit to minus one to disable it, do your migration, and then re-enable it. This requires super user, so you're going to have privileged connections that your schema migrations run through just for this purpose. Cool. Our first two examples have focused mostly on Postgres behaviours. I'd like to move on to those interactions between ORMs and database drivers and Postgres itself. While this may feel like a little bit of a detour at primarily a databases conference, I think it's important to remember that our users care about the combined behaviour of the database and the app. Postgres might be singing along happily in the background, but if the app can't use it, users don't really care what's broken. We just know that your service is down. So, incident three, revenge of the ORM. Let's start by starting a company. Say you've chosen Postgres as your main data store, and you have a couple of clients, and they're like a Rails app or whatever. You don't have mega scale, so you don't need many clients. So your startup starts to have a little bit of success, and you have to add more clients. As this goes on, you realise that because Postgres uses a process per connection, this is quite expensive in terms of memory. So you want to get better utilisation out of your Postgres box, or otherwise you're going to have to provision one with, like, a lot of memory. The typical solution to this is to deploy a proxy called pgBouncer in transaction pooling mode. The way that mode works is it takes many client connections, and it multiplexes the queries coming over those down into a smaller number of connections down into Postgres. The guarantee it provides you is that if you begin a transaction, it will stick you to a back-end connection and not, like, switch you between them. Otherwise transactions will be completely broken. For everything else, though, there's no guarantee of which back-end connection you're going to hit, which means you lose access to session level features. There are bits of Postgres you have to stop using. The three main ones are setting session level variables. Setting transaction level is fine. Session level advisory locks, if they're something you use. And prepared statements. You have to stop using prepared statements from your app. Now, you still can use, like, bind parameters, but you lose the advantages of caching of query plans and stuff like that. So we went through and prepared our apps for this, made sure we weren't using any of those features, and in the places where we had to, for example, when we were setting those timeouts on lock acquisition, those have to go through dedicated connections to Postgres because pgBouncer isn't compatible with them. And we thought we were good. We switched over to transaction pooling because we were starting to run into a bit of a scalability wall with our current Postgres setup. And then the next day, we started seeing this in our exception tracker. So if you don't speak Ruby, this is a null pointer exception, and it's saying, I just tried to call the method fields and it was on a null object. Now, this was coming from deep down inside of the database drivers, like pg, which binds on to libpq. So something really funky was going on. If you Google for this error message, the entire internet tells you to disable prepared statements. Now, we've been through all of our apps. We'd already done this. We went through again and, like, double and triple checked. We were definitely not using prepared statements anymore. So this one was going to be down to us. We ended up syncing three days of three people's time into fixing this problem. And, like, that may seem like a lot, but we basically had no choice. We were either going to enable this feature or run into a scaling wall and start having to spend, like, inordinate amounts of money adding more and more RAM into our Postgres servers. So we had to find out what was going on here. It wasn't an option just to, like, pump this to the backlog. Digging further, we noticed that this was happening more when we deployed applications that were violated, such as a unique index or anything of that kind. We finally started to, like, lose our minds and we decided that we would locally boot up Postgres and our application stack while running TCP dump and, like, looking at it through Wireshark. And while we didn't strictly need those, we got our answer. There were a bunch of extra statements going down the connection that we weren't sending. This was what was going on. And those were running, like, just after the connection was established before we actually sent any of our app queries. Those middle three really stand out. For some reason, it's changing the level of error message that the client receives to panic, the highest level of error. Then it's trying to enable standard conforming strings, which, if you're not familiar with it because it's not really a thing, it's enabled by default now, is there's, like, non-standard conforming strings and you generally want strings that conform with the standard because that's good. So Rails is trying to enable that. And then it sets the error level back down to warning. So it's really weird that we're seeing these there. And the reason that Rails does those three statements is that standard conforming strings is only something you can set on or off in Postgres 8.2 or above. And Rails has this to be compatible with 8.1 and below. So it was saying, don't tell me about any errors. Try this thing that might cause an error. Okay, now tell me about errors again. Let's see how that interacts with PG Bouncer. Let's go back to our diagram and simplify down to a single client because that's all we need to trigger this. We're going to set that level to panic and then down to warning. The first statement goes down back-end connection number one. The second statement goes down to warning number two. Because we're not inside a transaction, there is no guarantee that you're going to stick to an individual back-end. So what happens if you leave client-min messages set to panic? It turns out a whole bunch of useful stuff that you want to hear about like constraint violations is no longer reported to the client. What this means is that the client connection is left in a bad state and the next thing that comes along is going to error out in a really super weird way like that nil error we saw before. So what was the fix? It turns out, by their own policy, Rails didn't even care about this anymore. They were not maintaining compatibility with Postgres 8.1 because it was 2016. That was not a thing they had to care about anymore. So we removed this code from Rails and they were happy about that and the bug stopped happening. So that was all cool. We also wrote a detailed write-up of this. So again, if you want to go and read in full depth and there's links off to all the different bits of stuff we had to look into, then go right for it. Something that was really cool was that Rick Branson, who used to work at Instagram, had seen this same thing with Django and PG Bouncer. Django was also issuing session-level things and when they switched to PG Bouncer with transaction pooling, they ran into very similar problems. So it's not just the Rails ORM but the other problems. It's like fairly widespread. So, what did we learn? There's a lot more to running a database reliably than user bin Postgres. Our stack is just Rails and Postgres, but in debugging this, we had to understand what PG Bouncer was doing. We had to dig through libpq. We dug through Ruby's PG gem, which is the bindings into libpq and we dug through Rails' ORM active record. If you're going to start running Postgres and understanding how it's behaving in prod, you've got to get comfortable with other people's code. You want to remember that it's just code. It's not magic, especially when things are open source. This is a really easy thing to do. You can go to GitHub or wherever, pull the code and start pulling it apart. If you're really stuck, you can probably ask the author for some help. They will explain things to you in general. I've had a lot of success with that. Also, don't trust the ORM. The great thing about frameworks is that they do so much about you. The awful thing about frameworks is that they do so much for you. There's never a good surprise when you find out that your framework is doing something. You usually mean, I was really surprised by that behaviour in a bad way. That's all I'm going to talk about Rails today and ORMs in general. The final section is about running highly available databases. Everything I've talked about before this applies to a single Postgres node. If you care about availability, you do not run a single node of anything because that's kind of a bad time. In this case, I'm going to talk about two incidents where instead of providing higher availability, our cluster actually reduced our availability. Generally, running highly available clusters is trying to answer these two questions. How soon are we back online and is all our data there? You want to be able to answer soon and yes. Cluster is a super opaque word. It doesn't really tell you anything about what's going on. I'm going to boil it down to four pieces that I think are essential to our clustering. The first is consensus. Some way of the nodes agreeing on the state of the world. The second is a state machine. Some way of each node modelling the state of the world outside of it. The nodes agree on that state by using the consensus mechanism. Health checks. We need to be able to understand if the other nodes are up and agree on whether one of them should be primary or not right now. Lastly, you need some sort of scripted actions that can be triggered whenever you have a state change. Say I have a state change from health data unavailable for a node and that node happens to be primary, I want to run the scripted action that is promote new node. I'll say this is a mostly true simplification. Clustering is way more complex than I can get through in those four points but it serves enough for what I'm going to talk about. For clustering, we use a tool called pacemaker. It's kind of a generic clustering tool that you can plug different scripts in to make it compatible with different bits of software. We use the Out-of-the-Box PG SQL script but there are also scripts for MySQL and a whole bunch of other tools. Incident 4. Sorry I cannot hear you. I'm kind of busy. This one starts with us getting paged. Our API had gone down and we'd been alerted to that. When we looked at the rest of our alerts of monitoring, we saw this. This is basically saying that the cluster had decided to demote the current primary and move it to a new node. During that time while we're moving to a new primary node, our API is unavailable because we can't access any data. The reason it had decided that the current primary was unhealthy was that it had replied, sorry I've got too many clients connecting to me already. Now this was kind of surprising to us because we thought we'd provision the right number of connections everywhere so that we couldn't run into situations like this. Especially not for something like a cluster health check which should be running through a privileged connection so it doesn't get affected by applications. We had about 30 seconds of downtime while the primary was moved over to a new node but then everything came back so we were in an okay state by the time we were actually reacting to stuff. In order to stop this from happening again we rolled back a change that we had just deployed to the infra. We'd provisioned a bunch of new asynchronous workers into a new app infrastructure that we were starting to use. So we figured, we've probably put something wrong there or something bad spin those down because it's async work it doesn't really matter if you're not processing it quickly enough right now and we started to investigate from there.  I mentioned earlier that some of our things need a way of connecting directly to Postgres and not through PG Bouncer and we have a flag that says use direct PG connection, true or false. We had accidentally provisioned all of these asynchronous background workers to go directly to Postgres and Postgres was mighty unhappy about this to say the least. As I mentioned you'll be saying that what about your super user reserve connections like why didn't that save you? Let's run with an example of 100 connections just because it's a nice round number. We need four super user connections to run our things. We've got a couple of replication connections one for health checks, one for backups. So let's say the apps are using 70 connections. There are 26 free in the pool and we've got those four spread across three reserved connections and one non-reserved connection. The reason for that is that three is the default number of reserved connections for super users and we hadn't changed it. Whoops! So under load once we told everything to go directly to Postgres you hit 97 connections on the app. Your three reserved connections to get used and then unfortunately the health check gets rejected because there are no more connections left. So the same lessons again. Set appropriate bounds on your system. In this case rather than having to limit a resource we hadn't thought ahead enough to reserve a resource for an important process. Now I think the only real way to foresee situations like this is to properly comb through the config files of anything you're going to deploy. Really this was because we'd gone like this default is probably fine rather than thinking about what are all the really important super user privilege processes that we have have we set aside enough connections for them. Planning pays off. Cool. So the next incident is like kind of intense so I'm just going to give you a like government mandated break and take a sip of water. Cool. Incident 5. What's in a health check? Our second clustering incident is much more complex than the previous one and it was way more difficult to debug. This time there were plenty of connections available and to figure it out we're going to have to look deep into what goes on during those health checks. If we massively simplify how the health checks run the way pacemaker does it is it uses su to change down to the postgres user on each node and it runs some sort of query against the node individually. It then aggregates those results and decides what to do. In reality it runs a bunch of queries that check things like is this node the primary or a replica and like individual stuff about replication state but we can reproduce this bug by just doing a select now on all of our nodes as the health check. So we have a rule at gocardless which is that for user accounts we put humans into LDAP because it's generally easier to manage than having to like reprovision everything onto your boxes but robot accounts such as postgres live locally on each machine. So the reason we do that is that we never want a downtime in LDAP to turn into a downtime of gocardless.com. Those two things should be like separate failure domains. So following that rule postgres is a robot account you never need to reach out to LDAP to resolve this account. So it should be fairly safe to just like restart your LDAP nodes at will right like your users won't be able to reach anything but you know it's fine the API is going to be up or not. We were rolling out some upgrades to LDAP and during that we said our postgres cluster hit a timeout on its health check and when it hit the timeout it decided to transition to a new primary node in spite of postgres being healthy. The way this works is also the way this command works is that the pacemaker demon runs it and soothes down from root into postgres. Now despite of everything I've just told you when LDAP became unavailable this command blocked and it turns out that the command itself didn't matter you can sue down into postgres and echo hello and it's still going to block. So what was going on? We turned to some of my favourite debugging tools Strace and Ltrace the way Strace works is it attaches to a process and prints every system call that process makes. Ltrace is the same but for C library calls. This may be a little bit controversial but I think these are probably the two quickest and best tools you can learn for your debugging toolkit. The reason I think that is that they're extremely general. They will follow you to any programming language you use they're totally agnostic of what you're writing but they tell you a lot about what your program is doing. If you're not that familiar with them I think a cool starting point is Julia Evans articles online the one I've linked to is a like zine that she drew of how to use Strace and how it's cool. So we ran them and looked specifically for anything to do with user and login calls and these ones stood out get UID returning 0 which is root and get login returning Chris which is me which seems like really unusual because I'm not the pacemaker cluster so that's okay. So what does get login do? Let's look at the man page for that. So according to the man page get login returns the user name associated by the login activity with the controlling terminal of the current process which is super clear and we all know what that means. The two interesting bits of information there are username so it's a username string rather than a UID and login activity. Now login activity is something that follows whoever initially started a session down through any sub processes they started and to show this I wrote a tiny C program which printed out the result of get login. When I run it as me it returns Chris when I sudo up and then run it again it still returns Chris because that's followed me through the session. Linux stores this in two places var run utemp and proctpied login UID and let's have a look at what that is currently set to for our pacemaker cluster. So we find the process ID of pacemaker D we look at its login UID and instantly that does not look like root and predictably we run getent on it and it's me like what like seriously it's me like this is deeply worrying so it turns out that if you log into your system and at least this holds true for Ubuntu and upstart and you restart a service such as pacemaker your user becomes permanently associated with that service until it is next started because of this whole login activity thing so what is su actually doing here why does it use get login it turns out like the code is kind of subtle what it's doing is it's trying to get the past WD struct for the calling user so the reason it needs to know about the password struct is it needs to one allow root to su down to any other account and two it needs to authenticate you if you're not root why actually can you like supply like a password or whatever so let's step through this in sudo code I've replaced the calls with some things that are like I think are slightly more readable because like not everyone's going to know the unix login stack that well so first off let's say su is like getting the UID of the current running user so it's zero it's root get login returns me which is frustrating it then calls a method which gives you the password struct by name so it's passing chris into that then it's comparing the UID in that password struct with the UID it got from get UID which is not the same now the step before that password from name is where we blocked earlier so I can't really see when this conditional is going to be true particularly if I'm like the one who's associated with the login so what it ends up doing is calling a method that gives you the password struct by UID which is zero for root so what situation is the first side of that conditional useful in only if you have duplicate UIDs is this ever actually going to return you a different password struct which to me feels like a really weird way to set up a system like I've never in my life done that and I generally don't like things that collide on IDs it feels weird and like this whole notion of login activity doesn't feel like what Soo actually wants in reality so I'm not sure why it's using that it feels very disconnected from the thing that is actually controlling the process last time I gave this talk I had to wave my hands here and say like maybe there's reasons and then like peace out and leave the room this time I got to do some more digging it turns out you can trace the lineage of this code in Soo back to free BSD to at least 1994 the reason I can't go beyond 1994 is that that comes from an import from SCCS which I do not have on my laptop and do not know how to use so I would have to go down into the BSD 4.4 repose or before then to actually track down where this code is introduced so I went to the free BSD hackers mailing list and they were super helpful like I got a ton of responses on the same day I asked the question so it turns out there are reasons and we have some answers the first one which I didn't hit in Linux but makes a ton of sense in free BSD is that they have an account called tour which is root spelled backwards the reason they have that account is that lots of people want to customise things like the shell that gets used and whatever but they generally favour not doing that to the actual root account in case you introduce like a broken shell or something because you might totally lock yourself out of the system at that point so tour is also UID 0 but you can change its entry and Etsy password to say like use my fancy ZSH or whatever you want it turns out other people are doing the same thing or similar this one person responded to say that they are running like a bunch of mailing lists and they use different unix mail accounts for each user now they didn't want the faff of setting up like file permissions or ACLs or like a shared group or whatever so they just use the same UID all over which is kind of fair if this is like something that's an expected behaviour of the system feel free to use it I still find the whole like duplicate UIDs thing a little bit weird lastly someone linked me to this really cool repo that's been put together which details the history of unix I've not had a chance to go like looking through it yet but someone's gone all the way through the old old old unixes and like put together the history going forward to free BSD as of today and like I really enjoy geeking out on what I like to call code archaeology it's like I think it's super cool being able to trace what we have now back to like the 1970s and it's also can be useful in answering questions like the ones we hit in these incidents and a bit of me wonders if in like 50 or 100 years this will literally be a job title so we need to fix our setup and one more time the answer is to set the appropriate bounds on the system in this case we set an LDAP time out which was much smaller than our health check time out which meant that in the case it couldn't get to LDAP it would just fall back straight to the local user everything would be fine and we wouldn't falsely fail over the cluster we saw again that there's a lot more to our stack than user bin postgresql in this case we had to understand the entire unix login stack including LDAP so you've kind of got this weird like distributed system behaviour inside distributed system and it all combines to like take you down another cool thing I think is like running game days now if you're not familiar with game days they're basically trying to answer questions like what happens if I turn this service off running like actual game days where you go and switch things off will help you to surface problems that you basically won't find in any other way you have to run them on some sort of representative infrastructure now that could be that can be production generally so you can start with like staging environments and then eventually you can go like full netflix mode and use like chaos monkey and stuff like that to start taking things out in production as well again we have published a blog post about this very recently so this details exactly how we run game days at go cardless and hopefully it's kind of useful for anyone who wants to run similar exercises so that's all our postmortems I promise you I'd give you three bits of advice that you could take away and do something with the first one is check your logs there was a ton of interest in context in a lot of our incidents which we would have got if we'd have taken a look properly into our logs rather than just passing them by you probably have too much stuff in there to extract noise on a day to day basis but there are tools that will help you such as PG Badger and aggregation tools like Kibana or Elasticsearch the next one is set appropriate bounds on your system when you're configuring something new or if it's config file and look for anything that looks like you may want to set some sort of limit somewhere it's just generally a good thing to do lastly do not trust your ORMs and database drivers go ahead and fire that up locally in development against a Postgres database and just like see what it does because maybe it will surprise you like Django and Rails did you'll notice that none of these are about clustering I think it's hard to give generalised advice about running HA clusters at least in a little wrap up like this and frankly a lot of people now use hosted service for clustering lastly I've got a couple of thoughts just to sort of close up operating a database is about much more than the database itself we saw this slide a couple of times and I think it's so true you never want to think of Postgres just by itself you want to think about it in the context of the distributed system that you're actually running lastly this is one of my favourite tweets ever it's from a guy called Scott Andreas he's saying that not once have I regretted spending unbounded amounts of time investigating something fishy I think if you care about reliability of your systems you generally want to have this attitude you find something fishy and you dig into it you will learn things you will improve your knowledge and you will improve the reliability of your systems thank you so just before I launch into questions we are hiring for site reliability engineers so if you want to do this kind of stuff and much more and you like building reliable stuff come chat but now the real thing questions how many transactions are we talking about here and was any of it at big time so can't give exact sorry the question was how many transactions are we talking about in these less than a minute outages or exact numbers on each one what was the second part of your question sorry how many transactions do you process a second okay so our transactions per second number isn't public at least as of yet but yes oh sorry to the second bit which was is this at peak times yes it has been at peak times question at the back so the question was around have we considered like using manual failover in two of the incidents we had auto failover actually taking us down there are like empirically more situations where the auto failover has saved us and like we've stayed up or we've been back up more quickly than we would have been if we'd like page an ops person in the middle of the night to do the failover so we're going to stick with it but I totally get the point of like this is a risky thing and you generally don't want auto failover to be too aggressive it's something you want to tune is like your timeouts on when it considers a node to be unhealthy for example so yes it is a balance I agree with you but I don't think going back to manual failovers is suitable for us I can't see any more so if you want to chat just like more relaxed I'll be out in the hall anyway so thank you all very much