Hello Thomas, but I got a far better talk off of this morning, which I couldn't pass up. I met Paul in Shenzhen, China earlier last year and we both had really interesting experience over there. So thought it would be good to hear from Paul about some of the things that his experience looking at Chinese supply chain. Well, mostly this is a talk that I gave from my local makerspace after I got back, because I had such a great time. I'm sure you did too. How many people followed dangerous prototypes? There's a website, great open source hardware website, you should all keep an eye on it. It's fun stuff, run by people who do the stuff and they've moved to Shenzhen. And they've started running courses in Shenzhen, let me start this, which is just across the border from Hong Kong. And I mean just across the border from Hong Kong. 30 years ago it was a fishing village. Now it's a city of 10 million people, something like that. It depends how you define the border. It gets bigger every day. It is a very clean city. It's growing like crazy. It's got brand new subways being buried as we speak. And it's where an awful lot of the electronics that we work with comes from. I build consumer electronics in my day job. We manufacture in Shanghai because that's where you go when you want to build a million of something. But Shenzhen is where you go if you want to build a thousand, because you can get stuff cheap, really, really cheap. So how do you get there? Mostly you fly into Hong Kong and come across the border. There's lots of different ways. I took the subway and if you look down the right hand side there's a grey subway line. That's literally the train in the subway in Hong Kong.  And then you get back on the subway on the other side. And if you get off, where is it? Here, you're at the markets where all the good stuff is. It's literally the subway line, the green line. It's that simple. Christmas Eve, I was traveling with my family in Hong Kong. I went shopping. Now, one of the real problems with visiting China is visas and stuff like that. If you want a tourist visa, you can go do this. It costs you $150 in New Zealand. You have to send your passport away to the Chinese embassy, who will put a thing in it. You'll lose it for two weeks. It's a slightly scary thing to have to do, give your passport away. I presume it's similar in Australia. Anyone done that? Yeah. If you want a business visa to visit China, you need an invitation from a company. But there's one really cool thing you can get, and it costs about the same as an American visa. Sorry, as getting a single Chinese tourist visa, it's an APEC business card. Now, it's $150 in New Zealand. You don't have to send your passport away. You have to convince the government that you travel in Asia for business. This is a free visa to all those countries on the back, including China. You don't have to pay anything more than that, $150 for three years. Very great thing if you want to go travel. You don't have to do any deals with the Chinese government. You just show it at the border. It also gets you through the crew line at San Francisco Airport and LAX. So you don't have to wait. It's not a visa for the US, but as Australians in New Zealand, you can get a three-month visa anyway at the door. But it avoids that to our line that you often find if you get there during rush hour. So there are issues with travel, but once you can get into China, you can have a great time. The trip we went on was organized by the Dangerous Prototypes guys. Some reason SVG and there's other people from SVG and I open office don't have all that sore backwards for some weird reason. Ian at Dangerous Prototypes organized this for a long time. He lived in Amsterdam. He moved to Shenzhen a few years ago because of all the cool stuff that's there and he now lives there full-time. The main reason to go are the markets. There are electronics markets and galore. You have never seen anything like this. You can walk through.  There are other ones where they sell by the thousand off or a hundred off. So you have to go buy the right things. They'll offer you different prices in different places. Obviously and they'll probably tell you to go away if you try and order too little at the wrong place. There's a whole street of markets. This doesn't show the markets on the next streets over. I think these are Bunny's pictures actually and this is his hand-drawn diagram of where to go. There are LEDs. There are parts. There's this enormous tower. This is the SEG market which is the most famous one.  I've been buying at these guys on the left which are where SEED buy their parts, I'm told, reliably told. You can buy reels for $2 each. So that's a reel of $5,000. I brought back a quarter million resistors for $50. Look down the bottom here.  You want to buy switches and you want that little switch that feels just right. You can go down and you can ask to pick them up and push on them and check the feel of the switch. I go in with the part I want and walk around with it and say I want to buy a thousand of these. You can do that. I did that with USB stuff last month. None of these pictures really do justice. Apparently they don't let you take pictures in there so I didn't try. These are other people's pictures. They're kind of suspicious about people who are taking photos. They're security guards who apparently get stroppy. I didn't really caught things. We also visited the dodgy cell phone market which my hotel was close to. It's a very interesting place. It sort of starts at one end with people selling bits on the street to try and sell them into the market. Dead cell phones go in one end and they get knocked down into parts. In the middle, people sort the parts and check them to see what's good and take them apart. You get the idea, right? Then further down, they've got all the parts knocked down and categorized.  All these guys here are putting them all back together again. They're re-balling the BGAs and at the other end, they're selling cell phones. Now, you might want to buy one of these cell phones because they tend to be bizarrely re-branded with different cases and there are people selling circuit boards and it was built that morning. But if you're in a third world country, it's not just China. In India, people fix phones. They don't just throw them away when they're dead. If you throw your phone away and someone says they're, quote, recycling it, chances are, it's going to Shenzhen or somewhere like it and your phone will reappear in pieces in three or four other phones somewhere else. So this is what recycling is. This is not necessarily a bad thing. You have no idea of the quality because you don't have no idea of the guy who actually put it back together, right? He might have been an expert or it might have been his first day. So a lot of what we were there to do was to learn about doing business in Shenzhen and we learned a bunch of little things from them. Carry a calculator. If you're going to do business in the market, they may not have a lot of English. You may not have a lot of Mandarin or Haka or Cantonese or whatever the person you happen to be talking to what speaks. But you can write a number on a calculator. You can point at something that you're interested in and you can say, I want a thousand and they'll pick their calculator up and give you a price and you can say yes or no and say yes and they'll go off and get them. It's very simple. You don't have to have a lot of language. You just have to be able to get up there and walk around and see what you like and what you want. I talked about visas already. They told us a lot about starting companies in China. There's a bunch of different options. I won't go into them because I haven't gone that direction yet. I'm going to go and do a build in probably next month or the month after. So I'm going to have to probably not start a company but at least have my company doing some business in China. There was a lot of street things. There are electric bikes and taxis everywhere. The electric bikes sneak up on you when you can't hear them. There are no motor scooters, no motor bikes in Shenzhen.  It's a cloud of black two-stroke smoke. There are motorcycles everywhere. They have banned all gas-powered motor bikes in Shenzhen and scooters. There are only electric bikes and they have about 10%. I think maybe 20% of the cabs now are electric. There's a tax on petrol-powered cabs. You pay less in an electric cab. So you catch people fight for the blue ones, I'm told. There's a great subway. It's white and clean and new and cheap. Really cheap. The traffic is amazingly polite. We have the stereotype of Chinese drivers. I have no idea where it comes from because the drivers I saw in Shenzhen give way to pedestrians all the time. I think it's a... In China proper, they have these are first-generation drivers. They were all pedestrians a generation ago. So people stop for you in a way that people are maybe you're not used to in other Taipei or Hong Kong or somewhere like that. The food is wonderful and I had this bizarre... I have a beard, right? And people were continually stopping me and taking photographs. People would sneak photographs in restaurants. I'm strange. I would buy stuff in the market and I would have my photograph taken with the store holders because of my beard. People just don't have beards and I was unusual. I had the same experience by the way on Christmas Eve but with kids who thought I was Santa. Now I've trimmed my beard. It was actually a little bit longer but it's really bizarre. There are other things you have to get used to. If you do go to China you should carry, if not toilet paper, something that will substitute because it's not necessarily provided. In public places you're expected to provide your own. It's just a different way of doing things. And if you're not prepared you will be disappointed. We also went to a soldering class where they taught us how to re-ball BGAs. There we go. There you go. This was just amazing. These are the guys who teach the people downstairs and we basically started off with... I took some little pictures here. We started off with some cell phones and we took the BGAs off. And then we scraped all the balls off the BGAs and cleaned them up and got rid of all the epoxy and stuff.  You don't have to buy a re-balling machine or anything like that. This is how they do it. Someone has figured out their own way of putting balls on BGAs. So I tried to take a picture of the master doing it where he actually took a... Didn't he take a... He took a CPU off on iPhone 5 and put it back on. This guy can remove CPUs underneath shields without removing the shield around the outside and then put it back on again. There's no microscope. Not even a magnifying glass. It just doesn't make fair eyes. Yeah. I mean, I do most of my...  I do most of my day work under a microscope. But absolutely amazing. Oh, by the way, Ian is going to run another set of these early this year. I'm sorry. I think April, May, if anyone's interested... How much did it cost? I can't remember. It wasn't much. It was in the order of hundreds of dollars. Yeah. I mean, the plane... The plane trip cost more. The visa almost cost more. It's really worth doing. You end up meeting an amazing bunch of people. The food is wonderful. We also went to make a fair Shenzhen, which I think I have pictures for here. Oh, so this is what I brought back with me. As you can see, I bought a bunch of resistors. A lot of resistors. I bought a whole lot more while I was on... And I bought a bunch of... Just a whole bunch of other bits and pieces. And we got given a bunch of things as well. I mean, I bought static-proof bags. I know that's a silly thing to buy, but they were stupidly cheap, and I was going to be shipping some stuff and need a bunch of static-proof bags in my storage, you know. This is a tray of USB connectors. I think they were six cents each. Something like that. These were $2 each for $5,603. So you bring an extra suitcase? Well, buy one there. So Shenzhen Make-A-Fair was on afterwards. I went there for a day. I'm still limping, and I broke my killies a while ago. I was just mobile for this, so I only managed to make it around the market, as it was. It was raining, and I didn't do a lot there. The Make-A-Fair was just also pretty amazing.  That was the other amazing thing. They had just grabbed some outside space and put up canopies, and it rained, which was sad, but there were lots of kids. Lots of grandparents and kids. Did you notice that? Do you guys go? There was lots of grandparents taking their kids there. And just all sorts of things. People showing off. You name it. People were showing it off. I think I have one more picture.  He had announced it while we were there.  So that's what I have to talk about that. The other thing I thought I could talk about if people are interested is I'm going to talk about this thing on Thursday, which is an open-source random number generator that we're currently kick-starting. But is anyone interested in doing kick-statters and what that's like? Because I've just gone through the process. I don't really have a lot to say other than I chose kick-starter. I probably would choose someone else next time. Not because kick-starter is particularly bad. They really hold your hand. It was really quite easy to do it. Making the movie was the hardest part because it turns out that Ubuntu broke all the libraries for outputting movies from any of the Linux movie makers. I couldn't get my movie out of the sound without finally rebuilding my entire laptop in the process. The main reason I wouldn't use kick-starter again is that they only let people buy one of something. You can't have someone who says, I really want to buy that thing and that other thing you have, those other three or four things. I got a lot of grief from people who wanted to buy more, which is not a bad place to be in. I did get to add extra multiples. Kick-starter won't let you sell more than 10 of anything. I understand why. They don't want to be in the production business. But we're all kind of a little bit more in that world. On the other hand, I'm bringing out a new product. I probably don't want to give people more than 10 of anything up front. I want them to try it out. I want to use kick-starter to get them to try it out and love it and then come back and buy more from me later. Our plans are to build a small amount of our product up front. We structured it with two levels of rewards, one of which was... How do I explain it? One of which was basically we're going to make them in a month or two. I'll order the parts at the end of next week because that's when the kick-starter closes. We're still running. The other one, we're going to go to China and we're going to manufacture probably 1,000 units and probably about 500 of those will go to the filling kick-starter orders. And we're going to use all of our profits, such as they are from the kick-starter to fund building extra units so we can bootstrap a business out of that. I'm not going to quit my day job. So how are we doing? Just a minute. I can't find it. Here it is. How are we doing? There's my dashboard. So we're about two and a half times what we ordered. So what we needed to be viable. Just to give you an idea, this is going to be hard. Here we go. This is what kick-starter let you play with. You don't get to see this not as a... There's how our sales have gone. I don't know what you're laughing at, Oliver Strong's, yeah. We made our... It took us six days to fund a product. We announced it. KiwiCon. It's a security product. We actually did a product announcement at KiwiCon, which is just weird. And we funded it in six days, and we were just completely blown away.  We kind of figure at the end of the month when we stop, it'll be at about $30,000, which is more than enough for us to bankroll a product build in China. So we're really quite really happy about that. More importantly, does anyone have questions? The back. Pardon? Your $30,000, is that... Oh, a great question, yeah. We're kind of ignoring that issue. To be fair, we are building a product that basically makes noise as its main product.  But you have to work on it. You have to work on it. It's going to be much harder than you think. But you have to worry about keeping it inside the box. One of the points of our product is if you want people to be able to open the box and look inside it, that's a tinfoil hack. You can take off, you can look inside and make sure you've got the circuit I shipped you because there are a few three-letter agencies who want to change what you get. So we're selling a slight, slightly apparent equipment for questions. I was just going to get back to the Shenzhen thing. I was just going to second your recommendation if people visit Shenzhen. Yeah, he's just saying, go take the instructions in. It's a real blast. It's a point that's just for the food. Your product is a bonus. We also have a small collection at the dot g chain's market, novelty cell phones and some Red Bull BGA chips. So I'm trying to bring them to the box if they start to take them apart. It's a blast. Anyway, so I think there's someone else who's more of the insulation. Okay. So DPD is up next. We're going to be talking about LEDs here and being coming from a manual waste company that means nobody's talking about. I hope I do. Do you believe in that? No. That's the LED side dot. That's the example. Yeah. The horrifying thing is all this stuff is full sweep. Most of you are probably familiar with LEDs as a getting to the other end of the market. Have a bright light. That was part of the sample. We didn't quite get bright the next time we did. What doesn't get bright is chuck them into the dumpster. We were diving in dumpsters. So let me turn on the meat. It's over there. Wonderful. Can I see the bit down in the bottom corner? It's being obscured. There's nothing there. What I'm trying to do is get to these. What resolution is that? Now you're asking me a difficult question. It just worked. Let's give up on the mouse. If you go in 1024 by 7068 it will show the whole thing. If you use larger resolutions you can get its color.  Right, I want system. That beast here. Right. System settings. Displays. Put this back in. Yeah, we'll use this second display. Just click on that one. One bit in time. Okay. That means I can still display that one. You can see that down there because this is actually a second display. It's to the right. So you can actually drag this window here. So I've picked that one up. Pick that up and take it out there. I'm over there.  It's been on this screen. Okay, go to the right. It's slide show. Start from the first slide. All right, Blanc. Now just right key go backwards. Never used this before on my knife. If. Okay. I think. I'm Digby Turner. Hella who make lights. Mostly for aftermarket vehicles or navigation lights. And the electronics factory that assembles all this lives in Wahee. Two hours that away. Nice little country town. Not even a set of lights. All right. Most of you are probably familiar with this sort of LED. You're normal indicators. If it's on the circuit boards those are 0805 chip LEDs. But if you want something a bit brighter these are what they call a piranha shape. They're a through-hole LED quite a bit brighter. Normally running it somewhere is between 30 and 50 milliamps. So you've gone up by factor of at least 10. Then you move on to it. As I said. The reason I bought this lot was that you can see the real thing somewhere in one of these containers. There's your piranha LED if you want to have a look. There's floor sweepings. If you've got a real use for them help yourself. These ones surface mount are about yay big square. Reasonably bright. Running again something less than 50 milliamps. You're probably familiar with what they call the Star LED which is these beasties on a little aluminium base. It's a K2 LED no longer manufactured. Instead one of the things to deal with when you're dealing with LEDs you find a data sheet this one's useful. It has all the useful information I want to look at on one page. It's actually a fairly old blue LED 3mm 1.2 candela. Only 1.2. Not terribly bright. But the important bits of information are maximum current. Exceed it and you start killing the poor beastie.  What I didn't manage to do was draw myself a graph showing your typical turn on for an LED of voltage versus current. Whichever way. Nothing happens until you get to the knee. It goes around the knee and then the current goes through the sky. So you want to for controlling an LED you're actually controlling the current rather than just the voltage. As long as your voltage is turning on, so blue LED is similar to white LED it's just that it's got a lot bluer and then they put the same phosphors over it as they put in fluorescent lights. Now we're moving into the next generation. These little beasties, I mean there's some of them over there are called a golden dragon and they are typically running at 350 mA. So we've gone up by another factor of 10. They're fairly bright. These things are very small but also very bright. Give a quick demo. This is a reversing light. That's the guts of a reversing light. This should be able to work. Where's me lead gone? Fairly bright. Then you put a lens in front of it and concentrate the light and you've got a good backing light. Up. Dang. Stay. Put that into there. Now we move up to the next generation. I mean you can see the bottom of one up the top there. Your contacts are on the left and right. The center is connected to some heat sinking. You're talking about 350 to 1 amp for these beasties? How big are they? Have a look at them. They're in one of those jars over there. They're about 5 mm on a side. Fairly small. I just said damn bright but those ones the first one I showed you is the next generation again.  One of the things you've got to deal with when you're dealing with bright LEDs is you've got to get rid of that heat. They generate the regional amount of heat or kill them. If you look at the specifications for any of these LEDs the specification is at 25 degrees C. Sometimes you're lucky they'll tell you what sort of light output you'll get at at 80 degrees C. But the difference comes from but the difference can be the difference between one of the LEDs I haven't got to yet is 240 240 lumens as against 280 lumens when you if you take the temperature up you've come down by 40 lumens. The higher the temperature the shorter their life basically. The cooler you can keep them the longer they'll last. Keep these beasties cool and 50,000 hours is what estimating you'll get out of them.  Put the lights in your house and yes, you try and keep the base of it below 80 degrees internally it might be upwards of 150. But as long as you keep it as cool as you can keep it basically and this is one of the reasons where you're controlling current not worrying about it. As long as you've got enough voltage there control the current because one of the problems is they have a positive temperature coefficient. The warmer they get the lower the voltage and if your voltage controlled the more current they draw the sky is the limit. This is that beastie that I was showing you that I lit up the first time. That's the footprint. It's about a year. Here's a couple of them you can pass these around. Look at the circuit board you'll notice that the front is connected to the back with shut them both ways with a whole pile of wires the board is then mounted on a heat sink. That's part of a prototype that we had something like a 20% failure rate on the first time we'd mounted these LEDs they were a pain. We've since solved that problem but again you're looking at something that's probably 5mm on the side and there are your power contacts and that's your heat sink. The heat sink is isolated from the rest of the LED I have seen a product that they were putting this thing in 12 LEDs in a row with lenses in front of it they made one to see if it would work and decided that it was too damn bright because this was what they were going to call high beam. Shine it 100 meters away and you've got a patch of light 10 meters across and about 3 meters high it's a tad bright and half of that are the lenses that are in front of it what are we trying to launch this one's a little bit bigger there are some couple of samples in the little red box that you can have a look at so he's 8 or 9mm on the side maximum current 3 amps it's generating a fair bit of heat but we test them at 50 mA they're normally built on circuit boards which are aluminium not your standard fiberglass you've got an aluminium substrate insulation, tracks, printed all on it because you've got to get rid of the heat but are they bright enough of looking at the pretty pictures if you want to have a look at some of them there are samples of all that stuff there now controlling these LED's simple way constant current source simple constant current source use an lm317 regulator any of you not familiar with it, most of you probably not it's a voltage regulator it's a requirement between the input, the output and you've got a sense line the voltage between this point and this point wants to be 1.25 volts so normally you have resistor from there to there resistor from here to ground and that sets your output voltage or you pass the current through it when the current across the resistor is 1.25 volts you've got a constant current source nice simple, easy and if you look at this one that little extra board there is one of those, in fact that gets the hottest of that particular combo but it was just to demonstrate how bright it is now if you want to control your actual brightness so you've got a variable brightness lamp you need to add an FET into the circuit and pulse width modulated or you can actually go backwards in technology simple transistor circuit the bias for the transistor is set by two diodes the reason being that's one diodes worth there so the voltage across that at the current you want is equivalent to that second diode 0.6 of a volt some voltage current through there to turn the base on now you can feed your pulse width modulation in here as long as you've got enough voltage you can drive however many lamps you want to it will vary I mean for white LED's you're looking at anywhere between 2.8 volts through to about 3.2 volts also it gets the load of the voltage across the LED so the reason for the resistor is this is actually if the current through this resistor drops more than the equivalent of one diode you start turning the transistor off so you end up with a constant current source as I say because you're controlling the base of it pulse width modulated that and you can vary the brightness just to a faster moment or we can move up though when I was sort of throwing this thought together I thought ok how can I talk about the switch mode power supplies and most of the LED lamps will go into your C and then on cooking and hacking space one morning and somebody is using an admiral, tiny admiral to drive his string of LEDs he's only going to be running oh well no he's not he's running half an hour of LEDs here what he's got is what's called a boost circle so you can play around with this one actually having seen this I thought oh I don't know how to hack that so I started playing around with it so let's look at the classic switch mode power supplies this one's called a buck feet of oxygen at the top when you turn the power on the current starts flowing in the inductor your inductor values such that you start to store magnetic energy in there because inductors don't like sudden rises in current they'll allow it to rise slowly and they store energy so you turn it off and the inductor says you can't do that and keeps pushing current out that magnetic energy goes away again and so you get a current through the net through the LED which you can now sense here volts drop across the resistor to control your pulse with modulation so you've got a feedback loop to keep the current flowing it sort of goes somewhere so it comes back through a shocky diode so the current flows around that loop when that's turned off when that's turned on it's providing the power don't turn it on permanently you'll just let it smoke out of the LED but that's your classic what they call buck circuit it drops voltage it doesn't get hot because there's no there's nothing near dissipating heat you're converting the current pulse into magnetic energy and then giving that energy back again makes sense one of the fun things I discovered when looking around was would you believe that's the same circuit but where I got this one from is there's a nice little chip about the AB with five pins on that has the sense resistor on two of its pins drive your LED, the inductor hang on under a couple, the switch is built into the chip now the nice thing about that is if you go back to that previous one it's using a prefect driving prefects to the pain because they're a high side switch and you've got to have the voltages right if you do it wrong you can try to fit the end vets are an awful lot easier most of the common ones will turn on with three to five volts so you can just feed the output of your micro mic the microprocessor is straight to it if you're doing it this way actually it's easier to apply the chip the chip does it all if I could figure out how to switch from this to my other screen I could show you a data sheet for that chip but this makes a nice easy system in fact some of them are much more junk I'll actually mash one together and have more vets and I have a lot of fun with it I discovered the problem I had was that the micro was running only ahead of six the crystal on the fastest pulse with modulation like the Crank Gallop out on the chip same as the Arduino was 23kc and the inductors I started playing with were in micro Henry's because that's what most of the stuff we used to work with but the stuff at work is running at hundreds of kzs up to mega bits I eventually got some mini Henry jokes and hey it worked the way it was needed to I had a lot of fun playing with this I mean I know all the theory I don't know how it works I'll actually sit down and put some together and see how it works it was fun having run this is for running single LED you're dropping voltage by using that inductive store energy so that you're taking away that extra voltage now we go to the other way this is your classic boost circuit instead of one LED I should have drawn 10 of them there what you're doing is turn on your FET in the input easy to drive current flows through the inductor builds up the magnetic field turn the FET off it's got to go somewhere so it cranks up the capacitor and as long as it hasn't exceeded the voltage required to drive your 10 LEDs at 3 volts each are you 30 volts nothing happens keep pulsing away that energy builds up on the capacitor to the point where you turn the LED off keep thrashing it along you're driving 10 LEDs with 5 volts it's your standard boost circuit most I mean that reversing light is using a boost circuit there are strings I think 5 LEDs strings in and out so from a voltage using a circle switch mode you can generate as much voltage as long as you're controlling you've got a sense resistor in there you've got a feedback to control the current so your standard chips which are plenty of low quite a range of them got to grab the right piece of paper that I haven't had a file listed on but they're that common if you're into driving LEDs you'll find standard LED drivers if you look for LED driver and automotive you'll get ones that can handle the voltage surges that vehicles produce and vehicles are nasty things you think it's a 12 volt battery it's a 12 volt system no you've got to be able to handle 40 volts and I think that's that pile now how do I get it escape gets me out of that one yep we can stop there so any questions I mean basically I'm just giving you a bit of a run down on what's required to drive bright LEDs driving with a simple indicator yeah resist will do the job you only want 3.5 milliamps to it and you want to bribe lights to light up the house the house truck whatever you're starting to move into a switch mode power supplies and all the hands will beat around the ear because you're generating all this noise the NJK gets strips of LEDs in a meter long or so and they run off 12 volts they've got resistors in series oh yeah so you're taking a couple of 3 LEDs there's a string of maybe 3 LEDs you can chop it with pieces there's probably 3 LEDs in a string so 3 wide LEDs that's 9 volts you're on a couple of resistors you're wasting energy using a resistor but yes they keep you very simple don't suppose in your travels you've come across a genu 10 LED where the power supply doesn't suck no that one doesn't ring a bell at all most of the stuff we're building is for vehicles for the marine world they came out with a new product part of my job is testing the stuff to make sure what we're shipping meets specifications so you've got a nice little switch mode power supply and an LED and there's a micro-computer hung on the end of it with one connection to it it's an output on this micro it sits there and this is a nautical light it sits there and counts 20,000 hours of use and flashes the LED when you first turn it on for a few times when you get 30,000 hours it just keeps flashing continually which means basically replace me nautical lights have got to meet a certain specification of light up over time LEDs fade very slowly so current them, they fade a little bit faster do you have any experience with infrared LEDs safe for computer milling paper or what no, not to the level of power that you would need I mean at some stage I'd like to take some of these very bright LEDs and focus them with a with a lens and see just what sort of burn capability they've got one of the nice things we have at the factory is that we identify all the PCBs manufactured by laser marking them with an ID number for the day week, year and what the order they belong to so if we do have problems we can trace back and figure out what caused the problem we've also found that the LED this laser marker is capable of producing logos very nice then now up to 100mm by 100mm take a piece of PCB paint it, stick it in take it out burn off all the paint where you want to forget the laser printers to produce your negatives for edging circuit boards this is fun on the side of it any other questions so let's say you wanted to know why a headlight rebuilt and converted from a halogen to an LED in Australia if you convert to say HIDs you need to change the optics of the image as well do you have any LEDs that could just be dropping replacements of a no I think you'd be looking at a set of LEDs as far as I know we have not made any light capable LEDs the only one I've seen was this test on the belt and it was just a tad brighter in the trucking market as an extra headlight they built I think two samples and decided that it was just a bit dangerous trying that in somebody's eyes I mean these LEDs, especially those bigger ones at the end are capable with a suitable lens of being a little wet flashing from somebody's eyes they're not going to see for a while flashing close and they won't get to see again they can be but don't look at the lens don't look at the lens when you lift it up we test them at low currents it's up to you right don't give a full current it's bad for the eyes in fact we've taken to testing these with the LEDs shining downwards and fitted through a fire rock so you can say yes the LEDs live it's at full current you don't have to fry the operating designs that's all we've got time for next one thanks to you alright thanks everyone for coming so today is an open hardware mini conference while I was doing some research for this I came across this picture here because everyone talks about the internet of things which is one of those terms that I sort of don't like because it existed before some marketer discovered it and decided it was a good idea but in some ways they had a point five years ago the internet of things if you call it that was routers and maybe a few industrial devices but in the last couple of decades the last couple of decades the last couple of years it's amazing what they've come out with one that caught my eye the other day is an a frying pan you can't see very well on the screen unfortunately but you can now buy a frying pan with a computer in the handle that you can connect to your mobile phone I guess you're going to smell it burning anyway but who knows so the disappointing thing about this is a lot of this stuff is hardware that it's not open I guess I want to get one of those fitness things a Fitbit or something all of them want to send your stuff back to the internet which is really annoying it turns out there's one called an angel which ran a Kickstarter that I just discovered after it closed so I can't buy one yet which is designed to be open and accessible which is really cool so there are people that are getting into this market that are thinking about people that want to actually use it to do more than sell companies their data so back to the slide however all this stuff wants to connect to the internet and everyone's probably heard in mainstream media there's everyone heard of things like shell shock or the Apple problem that happened last year there's been a few major ones that get in the news because they affect all the people that use all the i-devices but your home router runs a little embedded sock and a lot of the gear that is open hardware related and used that runs Linux they run little socks little system on chips there's a processor maybe an FPGA a whole bunch of IO there's actually malware that affects some of these devices some of them the first start of being discovered in about 2009 I think it was the Hydra one so there's a whole bunch of names they really like having good names for this stuff I'm going to post the slides later so you can follow up if you're interested but basically there is now malware that will go from router to router and by implication internet of things device to internet of things device so there's malware some of it is self inflicted manufacturers release this gear with backdoors accidental ones they're just in there for testing they've been left behind it was at the end of 2013 if you had turned on your router access from the internet to monitor it someone sent the right string which was some guy's name backwards to request even though it may have been HTTPS it would still open up a port and let you do stuff and things not very good and there was another one that came out just this week I discovered there was a new one on ASUS I haven't got it up unfortunately ASUS have a backdoor these backdoors come out so often it's just ridiculous so we've got all this hardware and people are just not paying attention now it's a router I mean it's a thing you're building this stuff you're building gear to do all sorts of stuff that's not routing do we need to care about this if you want to do really interesting things with small devices like Arduino but if you want to do real processing you need a bigger computer with a bigger chip there's a lot coming out now that are actually open the one I've got here is called a Karambola II the computer itself is actually the silver bit in the middle, the shield the rest of it is just a dev board so that I can get to the IO and everything they publish the schematics it's designed from scratch to run OpenWRT I guess the WRT is another one by its name OpenWRT is a Linux distribution the USB Armory is a new one I discovered very recently I think it's still in Kickstarter, I'm not sure it's all open it's got the whole logo this thing runs an ARM type processor it's got 512 meg of RAM it's designed and intended initially for security applications so you plug it into someone's USB port and it can pretend to be anything it's got half a dozen IO which means it's like to me a Leo stick on steroids an ARM thing running a normal Debian not an embedded one and doing stuff so I reckon go have a look at that so if you're developing open hardware on something a bit beefier you need the software the usual way to do it is a Linux distribution there's existing ways to do that my slides are off to the side a bit unfortunately you can grab Debian if you can make it fit OpenWRT is very common people roll their own using this thing you can actually get free BSD running on boxes I've had free BSD or net BSD one of them running on this it actually was not too hard to do which is quite interesting Debian was really a pain on the Carambola because I had to use an OpenWRT kernel and boot it off of the USB stick to the user space and AppGet would just kill it I don't know what they're doing in there so it's really only good for Debian on some of these things if you've got an image that runs off a RAM and you don't need to install it you're building open hardware you want to connect it to the internet to do stuff with IEI which is what I really want to do but somehow I ended up going down a security rabbit hole you want to connect it up you've got to think about software especially if eventually you want to have lots of other people build these things that don't know about computers and hook it up to the internet I want to build a weather station for example but I don't want to get it hacked so I'm just going to use OpenWRT it's a bit of an example but all the principles apply to any Linux OpenWRT itself has a pretty good guide on securing basics and I believe I've got that one up there's a Carambola 2, the centerboard up close so it's running an atheros chip that's the Armory Kickstarter, how are they going? 88,000 of 65K so they've met their funding goals too and it's still going if you want to order one and this is the OpenWRT security guide so if you're planning on deploying OpenWRT on something you're building read this stuff I'm not going to go through that today because this is just an overview of what's out there so just apply the basic stuff even if you're not rolling your own image for OpenWRT you can go in and do all this security hardening so when you hook it up it's a bit better than everyone else to a certain extent I think security and this stuff you just got to be almost better than the next person in some way because new malware's are always coming out so you don't have to be perfect you just got to try and be one step ahead and a big part of this is minimizing the attack surface so out of the box OpenWRT runs everything as root and it describes a bit of that and there are links to that from that security guide I showed you ideally you want to be logging into your router not as root and then switching over so that way you can connect it with SSH with a normal user and then switch over and avoid accidentally doing stuff as root that could compromise it the other thing with that it means if someone is able to get into your router and you're running a web server and that's not running as root and your web server has an exploit it's a lot harder for them to get a root shell on your box turn off things you're not using there's no point running a TFTP server on your internet of things gadget TFTP from it one of the main reasons for this is a lot of attacks now happen they use gadgets as pivot points couple of years ago someone worked out how to get into a printer and use that to pivot back into a network and go around people's firewalls by using a gadget like that and one way to make it hard as well is run your image in RAM now this is not too hard to do with OpenWRT you actually go into the menu and you just say give me an init RAMFS image instead of a JFFS2 image what that means is that when the computer boots linux it boots the whole user space into a RAM disk so obviously your device needs enough RAM but these run in as low as 16 meg depending on or less depending on what you want in and out of your image it means when you cycle the power anything that got into it will disappear the downside of that of course is you have to manage configuration if you can isolate that to a separate USB stick or a different part of the NV RAM then at worst you lose some configuration information and you don't have a binary that someone's planted on there the other thing you can do is make it really hard for malware to run because things can still go wrong with your other hardening people are always finding vulnerabilities in web browsers and other tools nothing will ever be foolproof even if we try and make it hard but we can just try and make it harder so that we're less likely to be the one that ends up on the news because of bash so if you're planning on writing software to run on a linux for open hardware there's a few things you can do there's a thing called address space the linux kernel seems to handle that in itself already I'm not going to go into that one today because I'm still learning about some of these things myself but basically it moves stuff memory chunks around in memory so that the code sits in a different place so malware because a lot of malware takes advantage of the fact that every time a process runs it's always at the same address so it knows where to look for it so every time you run it it moves it so it's harder for them to find it and do stuff stack smashing protection I'm going to go into in a minute and something called executable page mapping protection there's a whole suite of mitigations that are designed to knock out whole classes of malware it doesn't have other types of attacks but again we're just trying to do our best effort so stack smashing protection it's an option to GCC where you just tell the compiler to do something called a canary where you write special memory at the end of the stack so if you know much about C programming whenever you have a temporary variable it allocates down something called the stack the way a lot of malware and viruses work is they send inputs to your program to deliberately attempt to overrun the end of that usually by means of a programming error where someone hasn't correctly checked the length of their buffer and what the compiler tries to do is put code into the tech that someone has actually done that it does depend on tool chain support and also the build system support for your package OpenWRT does have it but it's mostly off by default which is really annoying but you can add it in by especially if you're building your own package for OpenWRT you just make it a dependency and leave SST make sure your build system is using fstackprotector and you will get that in there no no exec is a means where the system software can allocate in Linux types of memory some are designed for reading some for writing and some are for executable and for a long time there wasn't much of a distinction malware writers used to use that to their advantage by getting data and turning into code and running it it turns out now that they've been addressing that for a while in Linux and in other operating systems all the major operating systems have these type of mitigations where you can mark the memory as saying I'm actually never going to execute that as code a lot of hardware x86 has a lot of support for this at the hardware level if the code is written properly then you can't just turn data into code unless you can get a root exploit so to use this one it's another option to the compiler again you'd have to go read the manual to work out how to do it in particular if you're writing assembly code sometimes you have to take special handling to make sure that the piece of memory is marked correctly there is a program called checksec which you can use to check it produces a table of whether your binary has the SSP it has various other mitigations in it so that's worth Googling for x86 is pretty getting has mostly got hardware support for that and ARM a bit one problem there's the URL one problem I've discovered is MIPS doesn't have hardware support for some parts of that I'll post the slides up later anyway so many of the kind of gadgets that you want to run in open hardware like this Karambola they run a MIPS SOC most of them have NX hardware support not present you can get it in the 64 bit ones some of them but it's not there software support can be added but that's tricky and that's what I'm going to talk about the rest of this sorry yep I might have to just do the demo later so basically there's a thing called GR Security which is a massive patch that a bunch of guys wrote that adds a large number of hardware improvements and I've lost that one too it's grsecurity.net it adds a large number of hardware a large number of security mitigations to the Linux kernel and it will help support systems that don't necessarily support some of the out of the box you can use it to do stuff like blocking programs that try to allocate data and make it executable when they're not supposed to you can log behavior like programs mounting things when they shouldn't and various other features I would like them to implement NX for stacking for the stack protection in software but they haven't it's not supported on MIPS unfortunately what is supported on MIPS is protection most of the other protections if you're using openwrt however it's a problem if you're rolling your own distribution using build route you can just patch it over the top because it's designed to patch on a stock Linux kernel it's not possible on openwrt because there's about 600 patches for hardware and for memory footprint it turns out you can do it with a bit of effort and I spent rather a few late nights, late last year working on how to do that because openwrt finally lined up their kernel with the long term supported kernel for GS security so if you've got a 3.14 kernel in openwrt and gr security you can do it so just very quickly cut a long story short I get a stock I get the gr security kernel sorry I start with openwrt I start with the openwrt kernel I apply gr security, fix the patch rejects, fix some build issues turn on gr security kernel config, test it lots and that's what I'm in theory demoing but the gods have not been smiling on my computer very nicely today so this will be interesting so so like most of these dev systems there's a serial port connection and you can do stuff one of the other parts of this demo is at the same time I'm demonstrating running linux from ram instead of off of the flash so tftp boots demo A and just to check beautiful, oops what did I get wrong there it's a good start and ah come on where's the other tab there we go awesome so I'm loading now I'm going to run it so this is a stock openwrt that's just running from ram that I've loaded it hasn't got gr sick applied let it boot so pretty standard linux embedded firmware on this kind of thing I don't know I'll answer your question after because that's actually a good point as in I think I've got a slide so what I've got is a programming user local bin that I wrote sorry yep this program called mwrmapx that just ran and succeeded and what it's actually doing is allocating some memory and telling it that it's executable now we actually want to block that because it's not something you normally want to do so what I'm going to do is reboot and this time I'm going to load an openwrt that's been patched with gr security sweet and there's a few things that I can demo in this waiting so can I get the right one? yeah so the first thing that it's doing before it scrolls off is it's logged and demessage everything that mounted so if someone was to try and sneak something in and attempt to mount the usb stick when they shouldn't or mount a loop back or anything it'll get logged so you can catch that in any logging system it's also logging time changes the way I've set this one up you can see that time set so that's actually ntpd and busybox changing the time so you would get a log if someone else that wasn't busybox ntpd changed the time and then if I run that other program I ran before all the source code for my demo is actually on github so you can try this out for yourself it's basically saying I can't try and allocate and memmap for that's executable marked so that's grsec doing that if you've got an x86 system you're an embedded one say using one of those intel is it Galileo whatever it is you can just apply it on a stock or get all this stuff without having to go through weeks of trying to make it work like I did but the patched one I've got all the way I did it on github so I'm going to go back to my slides and finish off because how much is left alright zip zip oh demo where's the next one so it's not a magic bullet though but research all these methods for doing hardening and security is a heart is a bit of a mind field I even got stuff wrong as part of this I thought you could actually use grsec to get stack protection over and above SSP but it turns out it doesn't add software support for that so I made a bit of fooling myself in one place but if you're watching this and you're an infosec expert you can take my github and make it actually do things properly for openwrt because I think they need it and they don't have time to help deal with this they're busy just making it work on stuff so thank you very much


So, I will talk about what observability is, why we need observability, the observability based on monitoring, because it's a common question, right, is observability the same as monitoring, or like what are the differences, right? The pillars of observability, observability with Kubernetes built-in tools, and also CUBE CTL, extending the Kubernetes cluster to use external observability tools, some best practices for observability in Kubernetes and share some links and some insight on like how teams, cloud native teams, observe their clusters. So, what is observability? So, the English word, right, observe observability is basically to know what's going on, right, on something. So, observability is a process of gaining insight on your application, the behavior, its performance, right, in order to like identify issues, right, that may resolve, or basically to know what's happening, let's say bedside view, right, in order to make the system more efficient, or know also when you have issues to diagnose and to resolve them. And in order to, for Kubernetes to ensure the stability of workloads, reduce downtime and Kubernetes is a lot, right, the cluster system with pods, nodes, there are several moving parts, so that's the essence of observability. Why do we need observability? So, for years, software have changed, right, the way we build software, from monoliths to service oriented, so now micro-services are like some other stuff. So, monoliths, everything in one place, deployed on one server, service oriented, maybe you can have your, the web application, the database, so all separated into maybe three layers or four layers. Then micro-services, companies on two services we use, like Netflix, Uber, there are hundreds of micro-services deployed across several clusters in different cloud environments, some on-prem. So, if you can look at it in like in the real world, right, so I have currently a PC, it's easier to look at a PC, right, then if I have three PCs, you can look at the three PCs, but is it a lot harder to look at three pieces at the same, three laptops at the same time than the first, than when you have one? Then, when you have hundreds of services, right, you can't really like look at every service at the same time, check each log at the same time, manually, per se, and that is the essence of observability, right, bringing two services, techniques in order to get insight on hundreds of services at the same time, setting up alerts and notifications, dashboards and all, to make sure your system performing as it should be.  I didn't really see, I wasn't able to get who exactly said this, so I hinted someone. So whoever said this, you are listening to this talk, yeah, thank you, this is good. Observability-based monitoring, so I don't know if you read the introduction or description rather of my talk, right, I talked about like some confusions and struggles and hard-wired understanding of observability in Kubernetes. So and this is a popular one, right, and if you check a lot of articles that have been written on observability, you see that they are differentiating between observability and monitoring. So in short, right, monitoring is collecting data from a cluster, it's component nodes, ports, containers, to ensure they're all performing as expected. Observability, on the other hand, is a broader concept, right. So you can see monitoring is under observability. The goal of observability, like I mentioned earlier, is to understand your system, to know what state is in, what's currently happening in your system, in order to, let's say, do things better, right. So when your system is using more CPU resources, right, or memory resources, so okay, the system is working, the users are using it. But then again, you think about peak periods, right, let's say during, for USA, an e-commerce application, during Black Friday, so what can you do to make your system better in order to save costs at another time and also to give your users better performance. So, yeah, so everything that comes together, both debugging, right, maintenance, and everything that comes together, making your system better, can be put under observability. So both use the same type of telemetry data, telemetry basically is taking data automatically, right. So these data are known as the pillars of observability. So next pillars of observability, so I kind of like didn't put in my description, I say I mentioned four pillars, right, but I kind of didn't put it in my slides here because initially, most people say three pillars, right, but there's a new addition profiling. And also, if you look at it, some teams right now maybe using more than just these four, right, like per se in the development team, how they practice observability, because they are teams, they are observability teams out there. So hence, I just made it like a blank, an open page, right, in the sense of whatever you want to add to it, whatever you want to call it for your organization, for your team, and yeah. So looking at the current for the new addition of profiling, logging metrics, tracing, profiling. So metrics in Kubernetes, right, is a type of telemetry data to know what's happening inside your pods and your nodes in your cluster as a whole, your namespace depends on how like the particular aspect of your Kubernetes components you want to look into. So Kubernetes components emit metrics in primitive format, right. So primitive format is text-based. It's machine, it's text-based. It's easy to read. I have a screenshot of an example. It's easy to read and easy for both humans and machines to understand. So let's see if for you to create dashboards, right, a set of alerts. So there are some issues that like your system, let's say you're using Grafana, right, with primitives. So there are some issues that your system may not catch or the setup you use may not catch that you'll be able to read, right, while debugging or at certain use cases, right. So in most Kubernetes components, metrics are available with the metrics endpoints of the HTTP server. So this is an example of primitive metrics. Like I said, it's text-based and it's easy to read and for one to understand. Then logging in Kubernetes, logging, collecting and storing data about events to diagnose issues and when stuff occur. An example is like there's an error in our application. The first step is to look at the logs, right. So and also you look at it depending on industries like in FinTech industries, in FinTech industry or financial industry, you have to like kind of like for compliance purposes, you have to keep logs for like a few years, right, in order to know what happens. And if you have some live scale issues you need to resolve, you need to be able to submit these logs for compliance purposes. An example of logs. So this is an example of how a log might look like. So breakdown, timestamp, log level, name of the component, so easy on our app or database and all the unique identifier, then message, right. In Kubernetes, you see starting up container filling, there are different like kind of logs that you might encounter. So tracing in Kubernetes, tracing basically is to know like what's what's happened, right. So when a user makes tries to log in, where each request is sent, we had the server that the user tried to that the request was sent. Was it this my replication in, let's say the initial application was deployed in the EU year, right. And someone is trying to access it from, we are from Nigeria, right. So we had the request, let's say the content or the feedback was given to the user. Was it from the servers they have in the local zone in Lagos or the ones in South Africa, less if you're using AWS. So to know where the origin and the entire flow of request, right, in order to know when you have issues and also to see bottlenecks in your application, do I need to use a better CDN, right, to not deliver this content or this particular service to my user. So the tracing system, tracing tools like Yega or Jega, however you want to call it, Zipkin to enable distributed tracing in Kubernetes. So yeah, this is an example of what I just explained, right. The user makes a request, front end shows the user. So like, if there's an error or something occurs, right, as this is happening, we'll be able to know like what happened, where it happened and know how to make it better. Then the next pillar of observability, which will see the latest addition is profiling, profiling to analyze, to know what the particular, let's say resources, right, each section of code, each section, each pod or each deployment user, right, now let's make it better. So there's CoupFlame, right, by Yahoo. It's a QCTL plugin for you to know. So it's just with a few commands, there's little to no overhead. You don't have to build any infrastructure from ground up to know exactly what's happening and profile your system. So generating Flame Graphs. So Flame Graphs, so this is an example, right, my SQL database, CPU Flame Graph. So when each SQL query was run, what happens? Like, is it the performer, does it use more CPU, right? How can I optimize that SQL query?  So this helps you make your system better. This helps you prevent issues in the long run. So then observing, observability with QCTL, with Kubernetes built into QCTL. So implementing probes, health checks, using QCTL tools to monitor Kubernetes application. So probes, right, so the English word probes, so basically means to investigate, to ask questions to know what's happening. So in Kubernetes, there are three types of probes. We have liveness, readiness, startup probes. So readiness probes to determine whether a particular container is ready to serve traffic. So, you know, when you deploy an application on Kubernetes, it starts from container creating, then there's an entire container lifecycle, right? So once it gets to the running stage, right? So, okay, for a pod, right, once it gets to the running stage, it's possible for your pod to be running, but a container in your pod is not running, and it's not ready to. It's less, it's an internal fail state, right? So that's possible. And you will see that by just running QCTL, get pod, right? So how will you make sure that, like, there's no, an issue that, oh, your application shows running one-on-one container running, right? How will you make sure that it's ready to serve traffic? It's ready to be, it's ready to go live as per se, right? So, using readiness probes, startup probes to determine where a container is set up successfully, and also, live readiness probes. So, okay, this is just, so I'm not going in depth, right? This is open source on Ramp. I'm not going in depth to explain each particular, to each particular feature because of the, the goal of this particular section of this summit. So, this is just to give an, give an understanding, right? So, an introduction level into observability in Kubernetes. So, this is a manifest file for, let's say, demo deployment with the readiness probe. So, if you look at the readiness probe, you can see it's run the command core, look at our street thousand, which is the part of the container, right? So, it pings it. If this return and exit zero means the container is not ready to serve traffic, right? So, it has an initial delay of five seconds, period seconds of ten. So, after ten seconds, it tries again. So, when that request is successful, right? Then, you can see that the container is ready to take traffic. So, now, implementing this in as like the first step, right? To deploy your application. Well, you're sure that if this port is running, means that every container in the port is running. So, I kind of like prepared a demo. So, let's observe a cluster with kubectl together. So, I don't know if you can scan this for, to go to your KIDACODA playground and also scan for the GitHub with some commands. So, I'll just give you a moment to scan while I, okay. So, I have a kubectl playground here. Okay, let me start a new session. So, this is free. I think you just have to log in to KIDACODA, KILACIO. So, K get ports. So, K is set areas for kubectl or kubectl, however you may call it. So, this GitHub repo, I've set up some commands. So, in order to know, or you can use this, let me start from the scratch. So, now, see all the namespaces in this particular cluster. Then, to know what's happening in each of these port, you can use the describe command or to see the logs, you can use the logs command. Then, the goal of this, right, is to want to see the resource utilization of ports and nodes in the KILACODA Kubernetes playground. So, to see the monitor node and port utilization, we use the top command, right. So, let me go. So, this command allows you to see the resource consumption of nodes and ports. It requires the metrics server. So, like I mentioned earlier, Kubernetes emits metrics, like third metrics endpoints. So, and it says this command requires the metrics server. So, if I try to see the kubectl top node, if I try to see, oh, I will get metrics API nodes available. So, this process, right, this entire process is kind of like debugging, right. I want to see the resource utilization of the nodes in my cluster, but I can't see that. So, it's giving me an error, metrics API not available. So, I added some commands also, some commands, so we need to first install, deploy the metrics server. So, copy that, then deploy the metrics server. So, you deploy the metrics server, it's RBAC, robust authentication, service account and everything it needs to run for you to access the metrics of nodes and ports. So, if I run this command again, I would still get the same error and that's because, okay, this is a playground cluster. It's not really like a reward cluster, right. It's not meant for you to deploy applications. So, the metrics server is designed on the Kubernetes repository. It's designed to need secure access, right. You know, the Robes authentication hours are there. So, what next I need to do is to take away that, for this demo, I need to take away that, I need to make it possible for me to access data from, for me to call the metrics server in security or unsecurely, rather. So, this is just for demo purposes. So, I need to edit the deployment and, okay, then head over to the container and add this argument. So, you know, with Kubernetes, with YAML configuration, right, this is a top level, how do you call it? So, the fact that I don't put this dash, right, before the command means that the following flags I will add would be under the initial arguments. So, let me add this. So, if I try to call, to run the command again, okay, so now I see it's not metrics API not available. So, let me check why I use K get ports, dash n, namespace, keep system. So, I see, okay, it's dominating the initial deployment of the initial port of the metrics server and it's deploying the new one. So, if I see, so it's still terminating. Okay. So, now I have the metrics server running. So, let me try again. Okay. Now, I can see this, the control plane node is using 14% of the CPU and I can see the metrics of the nodes in the cluster. So, if I go for ports, I can see the metrics of the nodes in the system. So, now I can see for ports too. So, moving from just the metrics server and implementing prometheus, right, you can then have a complete monitoring pipeline. And from then you can now build dashboards, right. Let me go back to the slides and, yes, then extending the next part of this talk, extending a cluster to use observability for metrics, right. There's prometheus, victoria metrics, inflows dv, graphana. Graphana is not really just for metrics, but it's mostly used for creating dashboards, alerts. Graphana basically for logs, you can also make dashboards for logs by implementing graphana. Also, pyroscope for profiling is built by graphana. Or you can use an all in one or basically a non open source, observability tool like Diana trees and there are several out there. For logs, plain D, log stash, like I mentioned Yeager for profiling city of flame pyroscope. So, for metrics, right, like I mentioned earlier, every component in most components in the Kubernetes, then prometheus, right, deploying prometheus, it extracts those metrics, then connecting graphana, you can be able to create dashboards and display those metrics, right. For next, you have extending a cluster, external logging, right. So I wrote an article on sidecar container, it's on this space, I don't know if you know the company split space for IEC deployment, but there's no I'm talking about today. So basically, multi a port can have multi multiple containers, right. So the sidecar container is for logs, streams the logs to flu and D. So the exits are flu and D. Ideally, you can use like writing a shell script to stream logs. But the essence of flu and D is like is a unified login layer, you can perform more complex actions, right, on your logs and more performance, then you can study logs, you know, an external log back end, maybe Amazon history or maybe deploy mini or on your own on-prem or your personal server. So for tracing, so you know, we talked about Yeager. So this is implementing tracing with Yeager open telemetry. Open telemetry is like a standard for standard and open standard open source for how telemetry data is being taken and transferred. So the open telemetry collector takes the data from the application, then Yeager collects the traces stored in an in-memory database, and you can then know what happens with the data, which request or that happens in your system. Then profiling, right, like you saw for cube city of flame, in order to claim frame graphs, you can install a Grafana agent or Pyroscope SDK in order to profile and know what section of code or what deployment is taking more CPU and able to determine performance or to resolve issues. Then best practices for observability, a lot of best practices out there. But this I would say is like the most basic and things to keep in mind, consider your specific telemetry needs, right? So when you first deploy your application, what telemetry data do you need to look at first? So if you look at it, right, like I mentioned earlier, profiling was basically was done now. It hasn't been this kind of new addition to the system. So do you really need to profiling, right? You know, Kubernetes is complex on its own. So adding motoring increases complexity to your system. So you look at what you need now, the basics like we did was metrics, right? So you need to know what's using what. So that's the basic keep things simple, right? Depending on your own team and the application being deployed. So keep things simple. From there, you can scale as your application scales and your team skills. And an important note is, and like Abubak had talked about in the earlier talk here, security rights, permissions, user access, access control. So you shouldn't be using it to that grants everyone access, right? So the metric server, you know, the demo will be looked at. I couldn't, it didn't deploy the it failed it because of the cluster, the playground cluster I deployed it on doesn't have security set up. It doesn't have user access and rules set up. So I couldn't really. So I had to make it insecure in order to access. So that's not what you should do in the production grade environment. So it's important to whatever tool you're using, make sure like, can you can give this privilege is right to each user to from the engineering manager to administrator to the coenities developer, everyone should have like precise access and just have access of what they need to do at a particular time, right? And yeah, how do cloud native teams observe their clusters? There is a pretty, it's an entire blog highlighting several user stories. I highlighted it here. I've also added my presentation to the schedule. So there are several use cases. One can basically look through read and learn in order to implement in your application or your platform. But one one thing you want to keep in mind is like no application is the same, right? So no application receives same amount of users from same same country. So an application deployed in Nigeria shouldn't use try to replicate the observability of an application that was deployed in Spain or in the EU year, right? So you need to know your platform, know your application in order to deploy deploy your observability tools and techniques. So this user story just serves as a guide, right? So that's where for you to learn and know what you should try and implement on sandbox environments, right before going into production. So melt what are you adopted to the system? I don't know if you know, sirium. So sirium is basically container networking interface. But sirium does basically everything on the networking aspect from observability to security. So I think sirium has the has the future for tracing packets be a BPF trace. Can we recall exactly I worked on sirium last year during the Google season of dogs. So raise up a raise up is an Indian company, right? So someone trying to deploy an application is in India should okay, maybe you're more interested in checking out what raise up a deed, right? Are you doing something fintech related, raise up it will be your your best pet, then who is shooting interface. And also these are several examples one can read and understand how do these things in the observable classes. Some of them use the tools I mentioned. Some of them build their own tools on scratch, like Netflix, I think it's Netflix builds its own from scratch. So it's just for you to learn makes much test at sandbox environment and be sure what works for your application. Then conclusion, like I mentioned earlier, observability is broad. There are a lot of things to consider. So in from match in metrics in profiling in logs, a lot of things to consider, but starting small keeping things simple and scaling and learning, right? I wouldn't say myself I'm an expert in observability, right? But is those learning that I'm able to understand like, Oh, this is used for this, I'll rather use this for my application or use this. So you just want to keep learning and trying and implementing within your your organizational environment. And like I said, more about that in sandbox environment before moving into production. So if things break, or if things, if you're deploying your observability stack, but it's in turn having effect on your application, that doesn't make sense. So you know that in sandbox environment before going into production. So like I mentioned, Kubernetes is complex. Nodes, networking, Kubernetes has a flat networking. I think topology does the word. So there are a lot of moving parts, a lot of things are happening. So in a team, maybe only two people might know some stuff about a particular Kubernetes cluster that have been deployed. So more tools you add, the increase in complexity. So start small, keep going, but trust in the point of starting small and keeping things simple in order to scale. And yeah, that's it. Thank you for listening to me. And yeah, I'm on Twitter, LinkedIn, my GitHub. I try to write as much open source code as possible. And yeah, I hope this was interesting and insightful. And if you have any question, you can ask and I'll try my best answer to the best of my knowledge. Yeah, thank you. Is there any question? Absolutely. Yeah. Thanks very much for the presentation. I was curious about why you think that there is a new, I would call it a new fourth pillar in observability and profiles. And maybe are we ready to obliterate the pillars and just call them something new? They're all one thing at some point. Like what's the difference between a log and a trace? And a is it just an event at this point? Okay, so I would say like it's been new doesn't mean that it wasn't companies and organization weren't doing it before, right? When I say new is like, okay, if you're talking about observability, it's been like categorized, right? That's why I would say when I say new is like being categorized. So hence why I didn't give it like, oh, just three pillars or four pillars because someone on the organization can call it a different thing, right? So I don't know if this answers your question. Well, I think I want to keep pushing and say, okay, if the categorization is the only thing that's happening, let's stop categorizing it and just say observability is built on events. Oh, yeah, that makes sense to. So like I said earlier, right? Depends on like what you need right now. Observability, like I mentioned, observes what happens in the cluster, right? So you can decide to categorize it. You guys decide to not categorize it. Hence the confusion, right? Hence the confusion for me, like I mentioned in my description, trying to understand is monitoring different from observability. Is it? Well, it's actually the same thing or you can say observability is built on anything you just need to know, like mentioned, events that are happening across the cluster and having an idea in case of issues, know how to resolve issues or to make your application better. Yeah. Any other questions? Yeah, thank you very much for that. Yeah.