My name's John McNabb presenting on vulnerabilities of wireless water meter networks. I won't read all this but the quick summary is I come from this from a water-centric position. I ran a local water department outside Boston for 13 years and one of the last things we did when I was there was look at wireless water meters and to be honest when we were looking at them and I was pushing adopting wireless water meters I never thought of security. So after I left I was thinking gee maybe I should look into these things a little more. Last year I gave a talk here on cyber terrorism and the security of the National Drinking Water Infrastructure which is a basic overview of physical and cyber vulnerabilities of the entire drinking water infrastructure of the country. Short story is any individual water plant is is vulnerable for a number of reasons but the entire infrastructure isn't because it's fragmented they're not connected like the electric grid is. But one potential area of entry is of course any wireless networks that they may have and this will focus on the wireless water meter networks. I presented this at Black Hat a few days ago and my white paper part of my white paper is going to be published in a book Weapons of Mass Destruction and Terrorism second edition when it comes out next year. The book is a collection of papers on that topic. And interestingly enough Hacker Japan was very interested in my talk last year interviewed me for three hours after my presentation and I was the cover story in their March 2011 edition. So I'm very popular in Japan apparently. I tried to read the article but I don't know what it says about me so presumably it's positive. So what this talk is about water meters obviously what they're used for it's not going to be as much as a deep dive as I said in the abstract I'll get to that later but it's but I'm looking at what they are what they're used for what they could be used for what the potential avenues of attack are potentially what the concerns are the issues involved and the need for this is that while they're not a major part of water water departments across the country right now they're they're they're increasing the use of them is increasing and I'll get into that as well. I give a very detailed analysis of the overall drinking water infrastructure in my Black Hat white paper which you can email me my address is in the first slide it's also the last slide it's on in the PDF in your in your CD and I'll send you a copy of the white paper. Okay the water meters the cash register to oversimplify it's money to the water department so it's for you as the homeowner it's it's how they determine your water bill from the previous reading you know the difference between the previous reading and the current reading is the mountain gallons or cubic feet and then it's applied to the water rates and that's how they determine the usage part of your water bill overall annually it's about 40 billion dollars a year that water utilities take in mostly from the results from water meters not all systems are metered but in the U.S. most systems are strangely enough in the other parts of the world water meters are not a common occurrence but in the U.S. almost every water department has water meters to determine the usage the alternative be a flat rate or fix your fee which is a flat rate applied to how many faucets and toilets and so on you have in your house the disadvantage of that is it doesn't properly account for the usage and so that it's harder for the water department to recover the cost of producing the water which is what the water rate is really all about and I can get into further analysis which is very interesting when your water commissioner trying to figure out how to set the rate you have to set it to not just cover your day to day expenses but also for your capital maintenance on an ongoing basis and that's a big problem nationally as well average monthly water baits water bills range greatly and there's this it depends upon local practices what the local governmental authority decides to adopt and there's really no consistency in what rates are across the country so what could go wrong outside of the fact that meters sometimes just break they get clogged over years with what we politely call sediment which could be any solid matter in the water pipes and and for certain types of meters that that can lower the readings that can slow down the process of accounting for the water but it's also theft the electric utilities assume about 10% theft or loss from theft or bypassing meters and there's no similar statistic for the water industry and again it would depend upon jurisdiction is jurisdiction so it's theft of services in general which whatever utility deals with is is a serious issue because any doll they don't collect to help recover the cost of operations and capital maintenance they have to get from somewhere so it increases the rates for everyone else and and also bad practices of meters can cause a problem Brockton Massachusetts for example just conducted the city council had an audit of their water department after the water department realized that over the last 15 years they hadn't been reading the meters properly and that they've been undercharging people and so they unwisely decided to give people retroactive bills going back 10-15 years which I went through that inadvertently in my town and it's not fun people do not like retroactive bills and I think the legality of it's questionable wherever you are and they were handing out bills like $90,000 $97,000 so as you imagine the citizenry got up on arms and got the city council to look into it and the audit revealed that for 15 years most of the meters hadn't even been read you could read them you could send the meter read around and read them but they weren't reading them they giving estimated bills that had no relation to how much water was going in and gee that's why they had fiscal problems and they weren't using the billing process properly so there's a lot the water department can get wrong without you know a malicious third parties getting involved clearly clearly that's not just the hackers of the malicious third parties but we'll look at all those issues actually in terms of water meters very briefly water meter I want to do some technical stuff here I guess but I won't go through all this in detail the most common type of meter there's displacement velocity ultrasonic electromagnetic which are my favorite we were trying to adopt those in my town because the water just flows through the meter it's not stopped by a disc that turns around or something that that detects the velocity of the water so that the electromagnetic flow meters using Faraday's print law of induction are less susceptible to being clogged as you might imagine but the most common type use in the United States is a positive displacement meter that uses a new mutating disc which is just a disc that spins around and so they know how much is in the vet in that area of the water meter that the disc turns around and every time it turns around it registers a certain amount which appears on a dial and there's many different kinds of registers we call them this is one common time that they type that basically is an odometer type readout and then a dial and after the dial goes around one turn then it clicks on the on the far right part of the of the odometer that's the old this is the old-fashioned historical one that's still out there in millions of homes across the United States that uses a mechanical process for both detecting the amount of water and reporting it this is you know generation one perhaps but now there are you know we're up to generation three that I'll get to later and some improvements so data collection methods I could go into great detail but we don't need all the detail here but to make a long story short eyeball there's an eyeball right there so you send a meter reader in they knock on the door they go in the house they go down the basement they find the water meter of course they've been there before hopefully so they know where it is they get through the stuff on the in the basement and they and they put a flashlight on it and they read the register and again that's still what's being used in millions of homes across the United States so the so the the meter reader has to write that usually it's handwritten they have a sheet and they hand write it down and then when they get back after they do all everything in the route they go back to the plant and then someone types it in or hands writes it down and then the type it again so you get lots of opportunity for human error you can also have computer error but this is mostly a human human error type of problem then to get more to computerization you've got to walk by so you attach even on these older meters there are like two or three wires you can attach a reader to it that then drill a hole and goes to the outside of the building and we call it a gun it's a handheld computer that's only purposes to read the meter so you put the gun on the the device at the at the outside of the house and it electronically downloads the information and so after the meter readers done his entire route so we still have to walk to the house but he doesn't have to get in so that reduces the problem of having estimated readings when you can't get into the house which still occurs some people don't want to have the meters read for some reason then they get so what we did in my town is we gave them estimated readings of like five thousand dollars and that got them to come to the water plant to say hey what are you doing and we said well we just need a reader meter and then they say okay fine otherwise we keep them we can't charge them five thousand dollars but that's a way of getting their attention so the town council might have got involved in that but we didn't have that many and it worked so we didn't so we abated the five thousand and then we you know give them the proper bill then there's drive-by which is getting closer to what we're really talking about so instead of having the meter reader go to the put the gun it sends a signal out that can be reached from the street or there are varying ranges for these things and it's just automatically downloaded as they go by so that's that's getting as closer to the ideal of a smart meter or meter network which is our next slide is a fixed network and this is what you hear about on the smart grid the water like the water smart grid if you can call such a thing or the water smart meter is the little sister or brother of the electric smart grid which is moving faster and being developed more frequently across the country and across the world so actually to research this there wasn't much out there on the water smart grid so I'm adapting a lot of things from the electric smart grid and fixed network architecture which is basically being model being copied for the for the water and for gas and anything else that has you know is a utility provision so instead of having a drive-by the signal of course goes out everywhere and depending upon the topology and the size of the community if you're lucky they can all be received at the Senate at the at the administrative headquarters or where the billing center is located and all be collected there in most cases even in small towns you get geography problems it's tough to depend on the signal getting there all the time you can set up repeaters you can do a mesh network in the long run or you basically set of aggregators so that it Washington DC does this they have aggregators so the water the water meter sends the signal out using RF radio frequency transmission and then it goes to an aggregator in the neighborhood or that area of the town of the city of Washington DC and then it's sent by cell phone to the water plant to the billing system there's a lot of different means for for sending the signal automatically RF is the most frequent but it's also cell phones power lines are used by electric utilities obviously but for some reason are not used in water in the US is a means of sending the signal that's another possibility cable is another one so without getting into the technicalities of the different means I'm going to be focusing a little later on in the most common one which is radio frequency which is also one hopefully we can we can do something with the other key thing about the fixed network is that it can allow two way communication now with electric systems smart grids that gets very interesting when you also add to it a home area network which might have been talked about in the talk previous to this that I wasn't able to see because I was getting ready for this but home area networks are very interesting because every it's the internet of things everything and every electric appliance in the house has an IP address there's connected to a central controller that the homeowner can can presumably control but also the utility can control to turn things on and off to you know to deal with power surges or reduce reduce the use of things that aren't actually being used when when they're when they've got a heavy load things like that no no one's really talking about that for water for obvious reasons because basically it's mechanical issues you can't really do the same thing but it it gets into interesting questions that I'll get to a little later so the topology isn't very interesting it's probably conceptually a star and but the key thing is two way communication so suddenly your water bill can be read almost any time instead of every quarter with the drive but with the walk but the eyeball of the walk by or even the drive by it can be monthly it can be daily it can be hourly every five minutes you can you can put any household any any anybody on your network on any schedule you want basically the trick is you know how much information you want to get and you really know what you're going to want to do with it so so that gives us a lot of advantages in terms of information collection the benefits of the a okay these terms AMR automatic meter reading AMI advanced metering infrastructure and smart grid are sort of fluffy they they go it's you could be talking about one when someone else thinks you're talking about the other automatic meter reading is mostly could be could be the drive the walk by that's not really advanced metering infrastructure but we're going to focus from now on on the fixed network system which is smart grid or AMI for water so it lows them low as the meter reading cost it theoretically owes better accuracy even though that's what they say in many cases you'll see if you do some research like I did that a lot of people have higher bills or the bills don't make any sense with the electric with the wireless readings it's better for the water utility in terms of helps resolve bill disputes because the computer never lies you can just see if that's what the computer said and better customer service in terms of if someone comes back from vacation and they think they had a leak the someone at the plant can actually look at here's what the readings where you're why you're away or whether home it's more information that can be used to help the help the consumer better understand what they're using for water maybe you know help them do better water conservation it helps the water utility keep better track these these two charts on the right are very interesting in terms of showing examples of how the water utility can use the information even on a daily basis on the top graph so the water the water utility has an outdoor watering restriction and see that black line that's when someone evaded it by turning you know by by putting your automatic sprinklers on in violation of the restriction so right before you have the automatic meters you'd have you could find that out if you drove around and you know saw that that that was on and you could do something about it but that's you can't catch them all that way this is an easier way of doing it you could presumably set up the software to give you an alert for that and then leak detection is the main reason why I was pushing for it in my water department years ago was that if you set up for daily or hourly or you know pick an interval at some point if there's no leak is going to be it's going to go to zero at some point because there are many times when no one's flushing a toilet no one's turning a faucet you're not running the washing machine or the dishwasher so if it never goes to zero you suspect there's a leak and you can go further from that so it's got a lot of uses to the water utility and the growth right now going to the to the second to last bullet only about 7% of water utilities in the United States have what we would call smart meters the AMI the wireless water meters that we're talking about but looking at the studies that have come out looking at the entire smart grid for water gas electric it's it's it's projected to explode to make major increases in the next 10 years and and that's mainly because California is ahead of the curve because they have a state mandate to reduce a water consumption 20% by the year 2020 which is quite a chore actually I mean after you eliminate water you know unnecessary water uses like watering lawns it gets tough to fine tune the water use and so some of the so this this is a useful tool for the water utility and for the homeowner in California to to keep track the water use and you know reduce it as much as they can so looking at the information that we're talking about you know what what are the benefits digging deeper for the information that the water utility can get so obviously it's part of the billing record that's what it's for that's the major reason it's for leak loss analysis but also allows them to look at the the usage so when's the peak usage what part of town uses the peak does a peak usage come from maybe there's an issue there they can look at maybe it helps them plan the running of the plant in the in the withdrawal of water from the source if they know better with the peaks in the valleys are our our minute to minute throughout the water district conservation I mentioned a few times feedback directly to the consumer so there's a potential benefit with the consumer and this is from a report from California and it's it's almost essential in California for the utilities to tell the state what they're doing or not doing to meet the mandate so something at all up the wireless water meter is an embedded device it's a node in a sensor network I'll get to it's a data information collection device that's why I'm talking about it at an IT security conference it's not just a mechanical device that talks about that deals with water it's it's an information collection device it's an electronic cash register now instead of the old metallic one that I showed an earlier slide and it allows allows for better conservation and of course could it be big brother so it's hard getting a handle on you know looking at a water meter because there's so many different kinds there's about 20 over 2 dozen manufacturers that pie chart is from a study looking at some estimates of market share itron and Neptune are the biggest ones there's also census Badger hexagram those are ones that I've done some research on but it's a big field to try to get a handle on because there there's at least as many manufacturers they have different models they would different types of meters some of them are the transceivers that can go on an existing meter some are ones that were the transceivers built in already to the meter so there's a lot of variety here because there's no standards really they they they all the standard they need to meet the American Water Works Association standards for proper operation of a water meter but as far as I know that doesn't address the wireless portion or a lot of the issues that I'm talking about and as I mentioned earlier they they can deal with a lot of different transmission method methods phone lines cable power lines radio frequency or a combination of the two and again the most prominent one prominent one is power lines checking my time so looking into you know how how are they put together what's the tech technology and the engineering use and again I won't read all this but patent information is very very useful so I bought a couple of water meters on eBay and transceivers and one of the ones I got was a census MXU model 550 and they're very helpful in actually printing the patent number on the outside of the of the casing so I took the thing apart and I'm afraid to say I didn't get very far but I took it apart identified you know as much as I could what the different parts were and here they're described we've got an Intel 8050 family integrated circuit we've got a one kilobit e-prom it gives the description and third bullet for what the interrogation mode is what information is sent out and received so I'm still going through this patent and and some other ones and trying to pick a number of water meters to really dig into but you can see it's mostly off the shelf components no one not many people are designing specifics hardware software for the water meter they're using off the shelf chips processors e-prom and transceivers as it turns out so looking generally part of my research I realized that a water AMR AMI smart meter smart grid is a type of wireless sensor network which is a large network of resource constrained sensor nodes with multiple preset function such as sensing and processing so the sensor node of course is the water meter and the major characteristics of a water meter and a sensor node in a wireless sensor network is low prop power processing processor and that's true in our case low energy they have batteries that are usually good from five to 20 years there are some models coming out now that actually you know plug in and use AC power but for the most part they use batteries because you never really know where it's going to be placed when you produce it and even if it's someone's basement you can't guarantee there's going to be an AC power power near it so they depend on battery power so it's low energy and small memory one kilobit you know sometimes you get a four kilobits of e-prom so not much you can do here with software perhaps so looking at wireless sensor networks in general there's a lot of literature on this you can get off the web a lot of computer scientists looking at what are wireless sensor networks what are the potential attacks what are the potential defenses and it's really a taxonomy of all the you know everything that we're used to looking at for networks active versus passive outside of versus inside mo class versus mo classes attacking from an actually sensor node or water meter so I think a laptop class is probably more useful where you're using a laptop with the right receiver and software and you're getting into the network so the things of bad things you could do or interrupting the signal in again this is in general for water sensor networks interception modification replay and the countermeasures again in general but applies here as well basically boils down to authentication encryption which you don't see in a lot of low power low memory low processing ability wireless nodes and again that's the case we see in water meters as well so why would anybody want to hack a water meter I mean you know that's really what started my questioning you know looking into this is you know what what how could you abuse this it's maybe not as interesting as an electric meter or some of the other things many of the other things that people at this conference talk about but there are things that need to be looked into could you reduce your water bill could you increase someone else's water bill could you steal water the theft of service which right now is being done so well in many utilities by just mechanical means I mean the good old fashioned method you put the meter in backwards we see that occasionally you bypass the meter so you get someone who knows how to do plumbing and you just go around the meter you find that a lot out there but this is more sophisticated way if it can be done to actually you know do a man in the middle potentially and and and give the wrong information to the water department evade water restrictions very important thing in my community when you had water restrictions people get very hot under the collar that they can't water their lawn and they some of them think they've they've a God given right to water their lawn and use the water and the water department usually doesn't mind because they're paying for the water and they can afford to pay for the water but again if you have a water restriction because you've a low you have a problem with the supply you don't really everyone's got to obey it so so if you want to evade the water restrictions and you want to gamble that someone's not from the water department's not going to drive by and see that you're doing that with the outside sprinkler you just you know if you can do it you you you change with the readings that they get from your wireless water meter and they're none the wiser could it be used for surveillance of human activity in the home and I deal with with the top ones in some some degree a little later in here the unanswered questions are could you use it to get into the SCADA system the control system that runs the plant that's a very interesting question since we know from talks at black hat and other talks here and stuff in the news and and and viruses like Stuxnet that that the control the control systems and that's what that the second that's what that picture is that's a control system at my plant which was running on an XP SP two machine I believe unpatched and that's usually what you find because they they they think they're off the internet so they don't have to apply you know patches and virus protection and so on that would just get in the way so it'd be interesting if so the so the wireless water meter network only goes into the building system right and the building in the office system is never connected to the SCADA system right well actually about half of them are in polling it's been down the last couple years or not so it's but it's I don't I don't know the answer to the question that it's a further further research into whether that's a potential entry into that that screen right there and changing the operation the plant could you use it to get into other smart grid networks there's a move to integrating water and gas smart grids into electric and you could get a lot more bang for your buck if you're hacking into the water system if you could also hack into the electric of the gas and and do something nefarious there so again that's another question that is further down the line so what could the evil consumer do theft of services so we all know about the legal cable the scramblers which bypass the the the legal system that you're supposed to pay for also there have been news reports that in China people are getting arrested for creating a device and I can't get much information on this if anybody knows how to get one I'd like to see one but it's presumably a box of some point that that either goes on your electric meter in China or there's a man in the middle they're not they're not describing it very much and lowers your water lowers your electric rates so she could somebody you know attack the wireless water meter system and and you know presumably with a box that someone like us would build and and lower their water rates or change or give you control over everything that's being told to the water utility so you could evade the water restrictions or you know you know again low you know steal money steal money from the water department and steal water it's both so the effects of the water utilities is bad of course they especially if they don't notice it that suddenly the revenues are going down their their their master meter at the plant shows they're still sending out the same amount of water more or less but they're getting less revenue for the same amount of water going out she may be there's a lot of leaks out there maybe there's a lot of legal connections so it's it's a possibility of course it can be detected because even the wireless water meters have a register that you can usually it's an LCD at this point for the more modern ones where you can go down and actually physically read it and then presumably you could and some of the models you can go to it and download directly from the meter what the readings were and get better idea so there's a way around it for the water utility but if a lot of people are doing this it's it's hard to hard to find out and control could it be used for terrorists or adversaries trying to do damage so I'm just speculating here could it be used to collect information to help them build a hydraulic map save the water system because of my talk last year and on my white paper the current state of analysis of potential physical security problems for water utilities are that the most vulnerable part of a water utilities is the distribution system where it's not monitored really you got hundreds of miles of pipes you get hundreds if not thousands of fire hydrants in a big city you got hundreds of thousands of connections and valves and blow off areas where someone can get into the system and it's relatively easy if you know the hydraulics of the system you know you know so you know what's going to happen that you can inject poison into the water system a sufficient pressure to overcome the pressure of the system and then spread poison throughout the city now so that's a question that's still a question more information is always useful if you're going to do an attack obviously so that's just a question disruption is more likely in a lot of places like California that are serious about water conservation but but also in other places that want it want an easy way to turn water on and off remotely say when apartment water comes and goes or seasonal somebody has a seasonal rental and they and you need to turn the water on on when they someone comes and turn it off when they leave there's something called water cop is one brand name it attaches near the meter and it can be remotely turn the water on and off so that may or may not be on the back hall in the same signal as the water meter but there you go there's a whole other area of of attack so for example you know what if you could shut off the water that's not poisoning the water but it's definitely a serious disruption now the major threat that conceptually that water utilities are facing from terrorists are poisoning the water supply al-Qaeda has threatened many times to do so people question why they have the capability to do that and whether it's high on their list but it's definitely something they've they've threatened water meters don't really help them in that regard unless there's a recon component to it but the smart grid for electricity is very warmable as somebody said I don't know if that's a word but the NASA's Gia net so I gave a talk at black hat Spain last year where he developed a wireless sensor attack tool called senses for again for electric grid primarily and it's for 2.5 gigahertz of transmissions and very sophisticated tool that that he demonstrated to spread a worm through an electric grid I are active did something similar they presented a black hat USA 2009 and this is a picture at the bottom from that you can go to YouTube and you can see their demonstration so all the green dots or nodes are the unowned nodes on the electric grid and the red ones are the ones that are owned and it's great you can run the thing and it starts all green and then it turns all red it was simulated he didn't do it they didn't do it in a real environment obviously so no one's done that for for the water grid yet but it's a lot of the same off the shelves hardware and software it's a lot of the same techniques so it could be just as warm a bowl but the biggest concern that I've seen out there that we need to consider is big brother the evil water utility so even if there's no adverse third parties out there no one's trying to poison the water shut it off some people are concerned that now the water department is going to know more about what you do I know is formally running a water department I don't really care what you do I want to know how much water you using so I can charge you the right amount but this these quotes are from the Kerry North Carolina Watchman the newspaper that's online and so they're concerned that information about our private lives will be known to the water department staff you know when you take a shower when you water the plants wash the dishes honestly when I started this research I didn't think this was a big thing but when you look into it there are a lot of people perhaps legitimately concerned that private information is going to leak out now the water utilities going to know it obviously I don't know you know why they'd use it except the course to enforce conservation so if they see hey Joe you're you're flushing the toilet five six times a day I don't want to know why but you know can you cut that down maybe they do know why so so it's it's it's a concern that people have however legitimate it is but it may be legitimate so this chart is developed from a chart in a report on privacy with the electric smart grid so I just changed any any work anytime I said the word electric to water and again this is speculation I haven't looked into each of these things in detail but so what what is the water meter data used could it could it be used for so here's the hacker thought we know it's you know we know it's supposed before to you know the primary purpose is to determine the bill to raise revenue to be the cash register that's the first line here water conservation so what if a company that sells water conservation devices gets the data and they find out that this area the city just isn't doing it for water conservation whatever reason they're just using too much water the city is getting down on them they're finding them and the city just isn't doing it but they get the data and they get all the names and addresses of everyone who's violating the regulation it just isn't listening which is just using too much water so that's your marketing that's your mailing list or your email list you send it out and say hey I got this device that'll let you comply with the city's regulations insurance companies marketers I just mentioned law enforcers I was discussing with somebody black hat about you know could it be used for like a search warrant like if you're making meth in the house that uses a lot of electricity and water I can't find any examples that where that's been done but he said he heard about something in the Bay Area of San Francisco where that or California where that may have been used it did okay okay yeah so it's it's happened there's cases where electric use has been used by by enforcers and and also thermal imaging is a big case that's talked about a lot landlords private investigators so what if you're in a custody battle you've got your daughter you know you've kidnapped your daughter and you're I'm not condoning any of this you know you're you know you've got her in the house and you're telling everyone no she's not here and they look at the water use and they realize he's just too much water for one guy you know and they use that as evidence I'm just speculating here but the thing is that the wireless water meters are about 7% of the utilities of something California so a lot of this hasn't happened yet but in but it will especially if this information is created and is out there and is stealable like almost other like credit card information creditors criminals so again I think I think in most cases that the electricity information is much more useful but what if you don't have that but you got the water information so you look at so you're you're casing out this house this this rich house and you and all you can get is the water information you're stealing it off the wireless water meter and you see okay shower you know you're seeing 100 gallons per person a day then it goes down to zero of course you there's other ways to find out if someone's not there but you never know for sure so if you see the water use go down to zero or near zero then you have a clue so I don't think we're being paranoid I mean a lot of us here are concerned about security and and and privacy and this is clearly a potential privacy issue where it's implemented but it gets better there's a device called HydroSense that researchers in Seattle area University of Washington have come up with which is a simple cheap low power device you just attach it not necessarily not on the water meter but on any pipe in the house and it's wireless so it could go over this back all over the same signal the water meter uses or not and it can detect it can find out after like a you know after sinking it and checking it over a couple days it can tell what fixture was turned on and on and on and off and how long fascinating there's a whole body of stuff out there I could give a whole talk next time about remote sensing of human activity in the home there's all these researchers who just dying to find out how can we find out what you're doing in your house without actually entering it and as far as I can tell they're being funded by the government it doesn't appear to be the CIA at the NSA but who knows so this so since these guys came up with hydro sense a few years ago other enterprising researchers have been improving it so there's one called water which is self-powered by the water going through the pipe I guess or the vibration of the pipe then there's one that I probably don't have in the slide that that actually attack that is you uses the water meter because the to plus I can't pronounce that that law the equation they use it parts you know it uses the amount of water being going through the system the flow to identify it can go either backwards or forwards with the flow to the pressure hammer to the change in pressure from turning something on or off so they take so that so this new generation of hydro sense takes information from the meter and a vibration sensor on the pipe nearby and then can do find out the same information so all these people worried about privacy from the water department just knowing how much water the whole house was using well guys they can find out what you're using and when and that could be potentially very interesting so there's a lot a lot of rich research in this area they're looking at it for gas which is obviously a more limited amount of appliances you know stove and you know water heater you know dryer perhaps other things but also electricity which you don't really need if you get the home area network because that's already doing all that but what if you don't have a home area network and there are things there are devices you can attach in Europe for your electric grid in the house they do something similar so it's just fascinating how much development there is and trying to find out what we're doing in our houses without us knowing it so that really blew my mind when I said oh well people are worried about privacy from just knowing how much water they use well now they should really be worried if these things are attached these things can be attached surreptitiously to I mean they're not they're not you can just if you get into the house you can attach this set it up power it on and it'll be working so in general though other vulnerabilities looking at wireless water meters in general one one vendor tells us what frequencies they transmit on and tells us that there's no frequency hopping sped spectrum which is their their usual method of encryption or security which breaks up the signal so it you know one one some packets will go on one for one frequency than another one and there are different spectrum schedules that they go out on that are pseudo random that only the the sender and the receiver are supposed to know badger gives out its default network username and password in its wireless key so I might go after them first transceivers can be purchased on eBay I've got five or six at home I'm working on so you can I'm trying to actually find the same one that I have at my house so I can take one apart and then you know apply the information to actually the sniffing but most of them don't have encryption so most of them don't have frequency hopping sped spectrum which I briefly described or DSSS which is a similar technique to obscure the message which again was both of these would develop in World War two as a encryption quote-unquote type method but the but the good news is well most of the manufacturers don't have encryption or good security there are now more and more that are coming out with it for example one manufacturer refers to this third generation and here's a diagram of them here where they don't need a battery they reduce the amount of electricity to the minimum to only when they not when they read the signal read the amount of water but when they transmit it to the register which then goes out over wireless and they get it from the AC current in the house and it's all built in it's all one unit a lot of water departments now are adding on the transceivers and it's always a trick you know what model you know getting the models to match up and so on and instead of having a new skating disc that mechanically you know goes around and around and then you know goes to the register they have a purely electronic system and there's at least one manufacturer I found that actually says they have AES 128 bit and 256 bit encryption that's not common at this point but it's we're seeing them now for like the first time there's still a lot of the out there to say frequency hopping spread spectrum is for security and I'll get to that in a minute most of them are 900 megahertz the 902 to 928 ISM scientific band which makes it hard I was hoping I'll get to this in the second to last slide to show to sniff a wireless water meter and I haven't been able to do it because it's kind of hard to find off the shelf or even things I can put together easily to sniff 900 megahertz mainly because that cell phones and I don't want to do you know wiretapping so I'm working on it but most use 900 megahertz and frequency hopping sped spectrum okay so there are two researchers who have done proof of concepts or better for sniffing frequency hopping sped spectrum transceivers alo was atlas cut away in queue I believe they're from in guardians at Schmuck on 2011 talked about this in great detail and showed how using a CC 1111 you know 900 megahertz model kit they were able to crack the frequency hopping sped spectrum and and read the signal and Rob Havalt who's speaking tomorrow morning I guess on spiders taking over the earth is what the title is black cat Europe cracked it as well using a universal usrp software radio peripheral use the peripheral to that he told me he said to the listen in all frequencies at the same time and because in this in the signals he was listening to they use the same management frames that you see in 80 to 11 he was able to wait to find the management frame that told him the schedule for the frequency hopping which is in the clear of course and then feed that back to some software he wrote to to intercept it so I haven't ponied up for a $1500 usrp v2 yet but when I do I'll talk to him and and replicate his method that's not the easiest way to do it I'm trying to go for the low-cost way and but it's least way to start to actually see what what's going on there and I I alluded a couple times I was hoping to come here and say here's how I sniffed my water meter but I couldn't do it I tried some what looked some ones that look easy first like the amtel rz600 which actually is advertised to listen in a number of frequencies and to have sniffer software for it and they don't so the next one is funcube dongle pro which which comes out of England and that's a software radio peripheral for like 64 to 1700 mega hertz but I got that like a week ago and I haven't tried to configure it yet so I'm looking at a number of other things as well and and hopefully that'll be that'll be my talk next year or smook on earlier in the year because it's it's possible to do this but because it's 900 megahertz it makes it they make it really hard but the point is it's not impossible people have tracked fhss before and of course there are there is hardware to read 900 megahertz so I'm going to be working on putting it together and show proof of concept hopefully for a man in the middle attack and wrapping up I got the the hook a few times water meters are integral part of the national drinking water infrastructure I won't read the rest of it but the bottom line for my research is the major concern is privacy and surveillance whether or not you have hydro sense if you have hydro sense that ups the ante quite a bit tampering with water meters is already something going on this gives the bad guys or the homeowners yet another way to do it and so again the purpose of this talk is to reveal what I know about vulnerability so hopefully the manufacturers and the industry can can use better practices and secure them so we won't have these vulnerabilities turn into real problems and and final this is a growth industry for water utilities as well as for hackers and evil doers and if anybody wants my white paper where I talk more about this up in depth email me at john mcnevic comcast dot yet dot net thank you


Okay for the record it is definitely not limited to those exclaimer disclaimers All right, so this talk is web scale is dead long-lived post-crisque well or The drinks are on me. We got some McClellan 15. We got some red best 12 and Some hibiki 12 so we've got the Scottish the Irish and the Japanese all known for their Very lengthy love of all things that make us sane and crazy Here's the deal with these At the end of this talk, there's going to be a series of three trivia questions You are not allowed to use your phone If you get the last one you get this one Because this one is the one that matters. I was gonna get the 21, but I'm not getting paid to be here So I and then the other two are for these. I'm also peppering questions Improv throughout the rest of the talk If you get one of those questions right your first drink is on me, we'll hand you 20 bucks Take whatever shot cocktail whatever you want a couple of beers if you're a hipster and really have bad taste PBR Okay, so this is web sales dead long live puzzle scale.  That's your problem This talk is not wholesome. It's not organic. It's not sweet. It's not raw honey This is 100% highly processed high fruit toast corn syrup. Where's the next look? Thank you It is rated PG 13 I actually don't swear all that damn much but Occasionally it comes out and especially when I'm improving if someone says something that particularly sets me off like rails is a good platform I might say something.  It's not my day job. This is not JD the professional I'm not here trying to get your business. I'm here to educate you a little bit. Let you have some fun Enjoy each other. You don't enjoy my particular form of comedy. I don't care.  I Hope you laugh and learn and yes announcing. I'm offended is basically telling the world you can't control your own emotions So when everyone else should do it for you What does that mean means this talk is for adults if you're not an adult leave Okay, these are adults that I like to listen to not because I agree with all of them I don't but because I find them funny. I find them funny in their offense We've got Bill Burr who is probably the angriest comedian.  No him is fluffy We have Eliza and I have no idea how to pronounce her last name. It's it's Some long schligion egg olaga thing But she is a fabulous comedian. We've got Robin Williams Unfortunately passed recently. We have got Bill Maher We've got John. I'm sure everybody knows John Stewart poor guys retire.  It used to be that you were allowed to just be fun and not worry about your precious snowflake And this comes from activist morality if you tweet, I don't know if a lot of you do tweet tweet whatever That's my hashtag for this talk Okay, none of this activist morality crap if if you believe in something Great believe in it fight for it.  No, one two three It bummer. I was really hoping to have some offensive culture thing going on All right. Who am I? I am at Linux hiker Okay, if you want to tweet about this talk and you want to reference how badly I'm doing There you go. I actually do use Google plus.  It's just active with stuff. That's not all crap, which is why I don't have Facebook JD a command prompt if in case you decide. Hey this guy. Yeah, he was you know off the clock and on vacation But I wouldn't mind talking to him again. That's my commercial professional. I will bill you for email No title applicable to command prompt ink any of you are here as a founder Know that titles as a founder really don't mean a damn thing because you do everything you need to do to make sure everybody gets paid Including yourself. I am a director for software in the public interest. Don't let my talk reflect on them They are a great organization. They're the nonprofit between behind Postgres org as well as Debian Libre office in the United States a bunch of great nonprofits to do or a great software projects to do a lot of good for us and I am a director and founder of USPG until this last election. I was president It is great that Robert is now president because now he gets to worry about all the crap and I just got to have fun Pint. Ah, yes private invitation parties are for hipsters.  This is not my party. This is the conference party where everybody should actually be It's the only party. It's at Ramscale Studios 8 p.m. Midnight at the penthouse. It's all barbecue, right? barbecue booze fun hanging out be a good time Helping control license class. We'll get to the talk here in a minute If you are selling licenses of software, you are not helping to control license costs Be honest to your customers and say we are still charging you for licenses even though what we make is actually available for free The world is not flat no matter what Fox News is any Fox News Honest watchers here is not like for the comedy of it really Jimmy. Well, you're from Jersey. I'm not surprised But I actually don't watch Fox News because the only TV I have comes from the internet so I don't have to be inundated with information from a company That actually went to the Supreme Court for the right to lie to people But I'm pretty sure they have a couple of people that are flat earthers that are on TV Before we get into this, let's have some fun. This is not for this. Okay Who won the two four twenty-four teams are anywhere? Well, hold on. I know you know Andrew Stonebreaker did 20 bucks Who is the who is Stonebreaker in relation to Postgres QL? Hold on come on someone. That's not a community like Maven here you Bingo, he's probably kind of the founding developer really but that's close enough all the way back to University Ingress in 1974 yes, he's older than dirt Where does this person currently work? MIT he is an adjunct professor. I don't know the man I tried to meet the man, but because I don't know the man. He won't meet me I I tried to went back when I used to run these conferences way before they were this cool I Tried to get him to come speak kind of as a you know Hey, you know Postgres is doing this stuff you should come back and he wouldn't have it It's unfortunate because I think he would have had a lot of fun But he's obviously out doing more interesting things like getting a million bucks from Google. So who am I to judge? What are we talking about yes, I actually add for those who have seen me talk I normally don't do these animation things, but I was revising this talk at a really late hour last night because For some reason I just didn't like everything about it So I added these little things just to make it you know seem like I might wear a tie someday We're gonna talk about a little bit of history. What is web scale crap? Web scale is dead Postgres kill is king and the conclusion the conclusion will have our little fun time with these Hello, Alexei Alexei is an awesome developer out of currently Germany, but he's formerly from the new state of Russia, Ukraine If you want to hire one of the best and you're willing to pay for it because he deserves it You should talk to him. He's got a great job.  So if any of you got a little extra I'd say euros, but the euros are not worth anything anymore. So if you got any dollars for him, he might be interested Little history Postgres is the sorry. Well postgres too because there's Ingress postgres postgres 95 postgres ql postgres The oldest open source databases one is it the oldest that still use I believe it is I Mean, I don't know anybody else.  I mean that's fair, but does it pre-date? Oh? It is derived from University of Ingress 1974. That's actually where I start I my original database career Started I was working at Powell's books And they were running D base and I learned D base and then they just had University Ingress sitting on the shelf And so I compiled or I don't think I had to compile Yeah, I had to compile it compiled it and played with it and that's where I started But the first actual postgres that I used was postgres 95 and I say actual because that's when I started making money from it It doesn't matter before that The most advanced open source possibly of closed to databases and why do I say that? The project says the world's most advanced open source database, which it is there is no database or Want to be database mongo that? Can compete with postgres, but the reason I say the most advanced is because yes There are things that oracle can do if you're willing to spend a million dollars that postgres can't there's no question about that They got a million dollars. They got it, you know huge budget that said if You take everything that we can do out of the box Plus everything that you can do if you just add an extension Plus everything that you can do from any number of any other things There is nothing no database not from Microsoft not from well There's really only three people now Microsoft IBM and Oracle and they can't touch us They wish they could but they can't So that is why the most advanced open source database and you can see it in the marketplace everything from Apple shipping it by default with their workstations Even though it's using it as a very minor thing It's still there all the way up to people using it to create their billion dollar companies It's fully acid compliant. It's relational. It's object relational. It's document if you're into that new thing That's not a new thing, but they wanted to be a new thing because they put on their plaid shirt modern scalable database Hey Kevin come on in man. Not you're at the door. You got to come sit. Come on. This is Kevin Kimter He's from consistent state. They are another consulting company. They are awesome He kind of went quiet for a little while he used to run the PG days in Denver, but recently I've started seeing Some tweets promoted tweets on Twitter for his company which Notes that although I know yours at least as old as I am someone much younger than you is starting to drive the marketing sales Yeah, oh And I need to point something out this Michael masks. I get that right Mike and Dave Kramer they're from Creditive I point them out because when I gave this talk at scale I forgot to mention Joe Conway who's also a creditive and He was very hurt and I felt bad Because Joe is awesome. So what is web scale? Web scale is not this. I'm not gonna read this whole thing I'll let you take 10 seconds to read it on your own This is from MongoDB is web scale and they're actually talking about my sequel, which is crap, too But it it's not this a lot of people think it's this But it's not is this is just what? Fanboyism creates, right?  I'll get to why that's the case here soon I they do have a good platform and I know they're a sponsor here. So Amazon really I understand But there are some specific things that just don't belong in a shared tenancy Network latency environment regardless of provisioned IOPS One of those would be where you keep all your damn data, which should be post-cris if it's anything else migrate We'll help you we started a hundred ninety five an hour Web scale is I found this graphic just and I I appreciated this graphic the whole deep impact kind of thing All it is is an architecture based around a philosophy That's it. It's not technology. It's not new. It's certainly not new It's as old as I am at least It's just now we have a different inner tube to work with What is the core at it core of it? It's consolidation on x86 It's intelligence and software distributed everything including storage sales self-healing API based automation and rich analysis. Well, it looks like Amazon and GCE This is sourced from a white paper on that link what's interesting about this white paper is that It's from new 10 newtonyx or new tonics or whatever who surprisingly became a sponsor of this conference I I'm sure the product is great, but what like most marketing material.  Does anybody here have a MIPS machine?  Who was the big purveyor of a MIPS chip?  I mean, I know you're like 14, but what about you Andrew SGI SGI indies these were the apples of the day except that they were cool and They were these blue or purple boxes They shipped with ISDN on the motherboard. Who knows what ISDN is You don't care you know Mike Michael What is ISDN sort of it's not phone is data need a different answer Yes Think about ISDN back when the internet was first starting to get commercialized everybody was stuck on like 2400 and then forth 9600 and 144 and then you got 288 and you thought you were a god But there was something that came along they had been around a long time but Just started to get popular because people wanted to be able to download their Art from the internet faster And you went from 288 or 56k you all of a sudden you got 128k and at 128k in the day of those resolutions you got moving art from the internet at fast speeds Okay, and that's what you would download. I mean that's what you would buy is this ISDN connection little routers There's company name is send that made this great router and what was cool about it is if you were an ISP ISDNs could also answer analog So you could have a what was called a PRI which would have 23 channels It would come in on one port you have 23 chance to get answer 23 calls of either ISDN at 56 or 64 or 112 and 128k or good old-fashioned analog Whatever modem was dialing up. There's a really efficient way to handle your capacity Anyway, I don't know how we got on that oh MIPS with ISDN on board It's a good idea. It's sad because I mean once upon a time I had spark machines I had I even had deck alphas. I had SGI indies in my closet fire them all up I try to install every version of Linux on them. I even had a deck alpha running Windows NT And then I grew up and got rid of them all because I didn't have more room in my closet Yeah, so now we're all on x86 x86 is one yeah arm is out there, but for the most part I mean anything. We're running to some kind of x86 chip Intelligence and software there's no intelligence and software. Can you imagine this? Software is binary. It is a true or false condition at all times The moment it's intelligent I Want you to look at the person next to you and find the first flaw.  These people are not perfect. They're dead There's no intelligence and so but the art of it I mean the community has even come out and said we we need to protect the human race from artificial intelligence Well then stop research in the damn thing Okay, no hardware dependency for scalability or availability resources should be able to be dynamically added without hardware upgrade or replacement We've been able to do this for years. This isn't new. It's called proper provisioning distributed everything Bullshit I Almost brought that button. I almost brought it, but I really was afraid I didn't think the TSA would understand They would find it offensive and be afraid like push the button and it talked to him I'd be like on the ground getting touched everywhere. I just didn't want anything to do with it All data metadata and operations are distributed across the entire cluster. Okay, that's cool What are I running to problem is there's one word can someone tell me what the one word in there that is Terrifying to have distributed across a network There's only one Hold on locks Locks what is a database administrators nightmare?  There's gonna be whatever and then all of a sudden they bring you down Do you really want network latency added to your locks? It's it's crazy. It's not a good now.  That's fine But when you're doing the real deal, you don't want your walks networked in any way Self-healing assumes failure and that any failure will not jeopardize the health of the cluster. Isn't that called high availability? I mean to me that I mean when my ship breaks it fails over and keeps working. That seems like what it's supposed to do I mean, that's why you have two kidneys, right? Anyway, who can tell me who that is this not the character the person Robert who said that? Nice 20 bucks Okay, Robert Patrick Can anyone tell me something he's been in that is not that role? Was he in the X files? I didn't want to okay cool All right, he's also he was also in He's been in burn notice true blood Plays a werewolf in true blood. That was kind of cool API based automation and rich analytics. This has got to be the most hipster thing I've ever heard.  I know how to run a query now I know this is important to some people in ties You know, they need to have their pretty canvas they need to be able to click and actually that's really cool I'm not arguing it. I wish my guys would actually turn on monitoring something that was useful like that because I'm tired of Parsing data. Yes Awesome. Okay, first off Google translate And I think that's Italian is that Italian? Yeah, okay. Okay. It is grandma. I apparently I do not grammatically understand Italian That's true. I don't argue that But it was it's close enough 1980s movie watched it with girlfriends got a lot of points Web skills dead. I feel like the dead horse still needs to be beaten.  Yeah, that's not gonna happen Why can't we just build out an infrastructure and Not have to sell that infrastructure. Why does it why why do we always have to sell up, right? Even to our even if you're in a data center and you have bosses. I Understand having to say here are the facts of the situation But it's gotten to the point now where you can't present facts.  Prove it. I plan on being this man Michael might beat me to it, but I plan on being this man I'm gonna be the guy that stares all the people down Just angry because I've been through this for so many years Web skills a marketing term That is all it is. So if you get people telling you the whole postgres isn't web skills Hey, thank you for letting me know I should never hire you For everything that professionals most of us here are at least in our 30s 40s here some of us are I'm gonna leave that number alone I'm Young enough to take me down, but old enough to be hurt in the next day I've been doing for decades. We just create highly scalable systems. It's not new on Amazon Amazon's not new They just figured out how to a way to charge you twice what you should be paying and I can prove that if you really want to know Postgres kill is king king Don't teach your children to read just teach don't teach your sorry. I'm not from Louisiana Don't just teach your children to read teach them the question what they read teach them the question Everything that also applies to your staff And especially your boss in first class in laydown sleets on a domestic flight. Thank God. That was such a nice flight But I don't read CTO magazine a lot of these people do I Had a customer and I'm not gonna name him because he's a great customer, but you know who you are Who went to an Amazon conference right before this? Some Amazon event and I'll tell you Amazon knows how to put it on We've got we've got miles to go before we're competing with Amazon and giving a conference And I got an email and I'm sure all the consultants here have got this email Amazon just released all this new stuff and I just read it and watched it and they were really fancy And I want to migrate to them now. I said, okay It'll cost you twice as much. Oh, no, it can't be they can perform just as well as bare metal. I'm an auctioneer Okay, so I built him to actually go and provision the whole thing in Amazon Compared to where he is now software rocks use it and it would literally have been twice as much and And they would have been stuck because they would have had to use the highest level of EC to Available that was that they would have been done They would have migrated and then ran out of capacity and had to migrate right back off Now that doesn't mean that Amazon doesn't have cool stuff for certain workloads Databases that are of high velocity are not it so let's talk about why post-crisis cake Let's you want to replicate we got hot standby and cascading replication. This is native course stuff I know there's a lot of other bastardized things out there like Sloney And don't get me wrong.  It's okay. I won't pick on you for it. I promise Okay, okay hot standby is when you have a readable Up-to-date copy of the database meaning as changes are being made to the master, which is the big happy They're automatically transferred to the little hapies Now this can be done synchronously or asynchronously. I highly recommend you keep it asynchronous unless you know Exactly what you're doing Because post chris will go down if one of the slaves are synchronous and stop accepting changes It does not literally go down not like crash But it will stop Receiving or allowing you to process because it's waiting for everybody to be caught up if you know what you're doing and You know how to manage that and you know how to make post chris work with that synchronous is great But otherwise keep it asynchronous. You'll be able to sleep at night We have cascading replication takes a village make a family and daisy chains those nodes You are able to say here's my big happy I want my big happy to replicate to my little happy and I want my little happy to replicate to his own little happy and What's great about that is that the bandwidth between the big happy and the first little happy is probably really really good But it's possible that there's no link between the big happy and the last happy But there is a link between the second happy and third happy And they can all be hot standbys Which means you can do things like never back up your master and just back up your replicants your clones Your DNA samples. I want to distribute Distribute via hot standby cascading replication and my personal favorite PG pool, too. I wish they were here. I Wish they were here. They are such great guys They're in the Japanese the primary developers of PG pool to are in the Japanese community, which is an awesome community. They're huge But they don't get over here very often What does PG pool do does load distribution does query cashing and connection pulling? Talk about load distribution and it's screened so far away. I should have got a pointer Dancy, okay, so the P is PG pool. This is where all your queries are coming in Okay insert update select delete anything that's a right is going to go right to big mr. Happy Anything that's a read will go to the little haps automatically transparently as if it's not even there Along the same lines you can have the big happy also replicate to the red happy, which is your disaster recovery And none of the none of this stuff here that I'm One of the big arguments that I hear about say a mongo or I don't hear this from Cassandra very much But from a mainly from a mongo is that well, you just turn it on and it works Usually I find that that's because they're not doing anything with it But this actually isn't that hard to set up You can set this up if you know what you're doing you can set this up in a couple hours Assuming you don't have like a terabyte database to copy over or something if you don't know what you're doing You're really talking about a weekend of just understanding what's happening. This is not complicated stuff So we have our slave one which is read only our slave zero which is read only our master which is read Right, and then we back all the way over to our disaster recovery, which hopefully sitting on the other side of the country Away from gas leaks in New York Do you guys hear about this? We lost two buildings yesterday while we were all in here. Thankfully nobody died, which is great Query caching PG pool again with query caching their P is our PG pool or queries come in we got slow-ass queries that we cash That way they don't have to be slow-ass queries We just connect and there's our data and obviously you want to set some different parameters. They refresh and things like that No, no, no, no we that postgres proper has materialized views I'm actually talking about PG pool has a cache that will cache the results of specific queries And when those queries come in it will grab deliver those results and yes, we do have materialized views They're much better in 9-4. Oh Here's a good question. What is the minimum version of postgres acceptable to use? Come on somebody Think about it Steven. You're too easy. I can't give you all my money 9.2 I'm not knocking the hard work that was put in for years and years and years below 9.2 But there's one very simple reason 9.2 is the minimum It's not just faster than every single other version before it. It's It's like what the hell is that hipster car Prius Versus Ferrari Okay, I mean that that's what we're talking about here If you any benchmarking you do what you'll see is all the way back to if you were crazy enough to run postgres 95 It goes like this 7172738 you know and so on and so forth then it goes 9.2 okay Nobody answered that I'm very disappointed Okay, next connection pooling we are all about the renewable energy Who can tell me without me going to the next slide why connection pooling is important? This is worth 20 bucks folks. You don't get to answer that You new guy who what's your name? Trey Trey Trey is the man Trey. Let us let us have it. Why is connection pulling important? That is the basic result in postgres making connections is expensive This is what happens without connection pulling to your load That's hand-drawn because I'm an expert Okay, this is what happens with connection pulling to your load The other thing it does is much better not just resource utilization for processors But it manages ram and a whole bunch of it helps manage ram and other things a lot better the cat Colonel cash file cash all that Colonel cash file cash All right, I want to scale. This is the badass technology of the next century. It's all about second quarter second quadrant I'm not disrespect. I'm I run a postgres company. Okay Creditive their postgres company consistent state their postgres company enterprise DB their marketing company I'm kidding. They're also a postgres company But the honest the truth and they're a great community member guys I'm just teasing them because they're the you know 800 pound gorilla Second quadrant is doing the most badass stuff with postgres right now They are it is awesome. They're doing the most interesting technology They are providing the most interesting tools and keep in mind. They're a competitor of mine and there's no doubt I love using my competitors software and it's all open source. So I'm allowed to do that It is bi-directional replication no million dollar license fee no million dollar support contract Although I'll be happy to sign you up for one with asset compliance proper controls scaling it up to 48 notes Run that through your head 48 Rack you got to pay what a million bucks for two of them to sit next to each other and not work This just works. This is not yet in core. You can download it as a distribution They are working to get it in core. Excuse me in core. The hope is that it will happen This is water by the way We're hoping to have it happen for 9.5 slash 10 I don't know what we're gonna call it yet It is eventually consistent. I've had people kind of Say well eventually consistent doesn't work and we're like well Banks like it and they seem to do all right. Yes If Simon wearing here, I would ask that question. You got it. Go for it It's an honest answer basically Okay, it's all too much. I just want to shard Postgres XL apparently the talk on this was yesterday I feel bad because I wanted to pimp it Mason is the lead developer of that Jim in the back here used to have a a failed company That was based around it. I just tease him because I was one of the initial investors But very cool technology all open source And it allows you to shard and horizontally scale and it's postgres compatible in the whole bit and again, it's open source. I Want foreign data? Link tables. We're not gonna go through all this because I know I'm running out of time Or I am out of time Um Everything there Yes, we actually can query mail chimp. I don't know why Or Twitter I actually kind of understand that Or s3 why the hell anyone would use that? I don't know But this is all the people we can just talk to natively from within postgres you can install an extension and query the remote source So you've got a head everybody knows you got heterogeneous networks. It's very rare It's very rare That you have a situation where I all I have is Linux and Apache and Python and postgres. I'm in heaven Usually it's I've got Linux and some jackass installed rails And I got postgres and then I got some queuing mechanism over here and then some guys still using like Java I don't know who uses Java anymore. Not that I have a problem with Java I just I don't run into it anymore And it's you're all over the place. You might even have to do Microsoft SQL Maybe, you know, you got your accounting people thinking that it's a good idea. You can talk to it right there This is the picture. I'm trying to paint Postgres understands that you need to be able to work So we make our database, which is your database because you're now here Able to work with whatever it is that you need to work with Nobody else does that. Everybody else is all Let's just use me. I Once more Functions and do imagine a world where you don't have to cross millions of rows in PHP I actually have recently been told that PHP isn't used all that much. It's still on my website Instead you let the database do it Think about it.   Hey, dude, I got this. Hold on a second We're good. Here's your result You can do it all right inside posters and I'm not suggesting that you process credit cards in Postgres And I say that because I had a customer doing that and I Know don't do that. You can but don't and do by the way. Do I talk about do? And I didn't say that either It is my professional recommendation that you do nothing with PHP unless you have to not because PHP is bad But because it is bad But and do functions and do I think I actually talked about this so functions I hate PLP. I don't know anybody likes PLP. Just you all. Okay.  It's based on a something that is as old as I am ADA But we offer you the ability you can write your functions in Pearl Python PHP if you are one of these no GS guys, we have a JavaScript capability to write our functions in V8 of all things Which is a remarkably fast interpreter plus many many more Ruby.  I'm fine.  I let them know I was gonna run late Two-minute warnings are kind of like football, right?  Okay, at least it's Python You call out to the custom handler so that you can run custom scripts without running through their ORM or whatever You can write your PL Python code Write the do statement and it will execute right in Postgres like it's a normal function and send things back You're still processing in the database But the function's outside of the database there's a lot of good reasons for this one It allows you to manage it better because one thing we don't have very good right now is the ability to manage like Functions with revision control and database revisions and things like that. There is some programs for it, but it's not integrated But postgres is old Look you retro plaid shirt shirt wearing hippie We may be metal age, but we have NBCC and transactions caught everybody knows who this is is he still here? Come on, buddy Come on. I want to give props to this man.   We really need to do what Postgres has been doing all along They're actually working on things like in VCC now because they realized that their paradigm is broken Did you know that you can use Postgres QL functions to transform your proper relational data to that Jason thing? So you can keep your data sane and make your developers happy by letting them just call a function and we'll return Jason for Right This is this is my favorite slide the next slide is my favorite slide I spent a long time just loving this slide I'm gonna use no SQL My daughter loved fluorescent ponies when she was six, too Okay Eric Evans, I don't know the man. I'm sure he's very smart Because that's the problem is that when you get really smart people that aren't properly managed you get new SQL Reintroduced the term no SQL in early 2009, which is what six years ago Great slide. I just love these May say I'm a dreamer, but I'm not the only one I hope someday you'll join us and the world will live as one now I know you guys think I've off my rocker, but again, I'm not getting paid for this and you came and sat so Here we go donate We prefer dollars now. We used to prefer euros. We now prefer dollars Okay, and we don't take pesos I post show that org about donate I would point you just a PG us because that's a nonprofit that I'm directly associated with except I'm also directly associated with org through SPI so Go to about donate pick your favorite organization whether it be Europe org or US and give all your money over and above your liquor and mortgage and We will take care of you Not like the Europeans do we don't strike like them For the following questions do not shout out. Please raise your hand. These are the bad boys Okay Please raise your hand Who is the original singer of the lyrics in this talk and what is the title of the song? You were first Michael boom 40 bucks and That oh, yeah, okay now This is for the Japanese whiskey Who was the what was the name of the 80s movie cover in this talk? You were first It was Hibiki 12 year You're welcome to have it. Thank you. You're welcome for the red breast for the record cast strength, which is 116.4 proof Hey, I almost came with bookers, which is 136 proof But it comes in a box with a glass front and it was just a little too white trash For the red breast Irish Who's the original director of the original boom? You got it. I hope you okay. Don't answer if you don't drink whiskey. Okay. That's just not cool Yeah, that's right No, come back Sweetie, I have no power. Why do I have no power? There it is. There it is. Okay. Okay. Yeah, that's right. Okay. I don't know how long this is gonna last. Oh, here it is. Yes No phones. Oh You the fact Got a daughter Zane Okay, fair enough. That's right the fat. No, no, no come on. Come on The fact that this gentleman was willing to remove all of his masculinity in front of us He gets the scotch Now, why don't I have power? I have other things to say It's not cool man here. Hold on. Hold on. I'm almost done. This I gotta have power. I Have no power. I Hope you didn't pay for this power. All right. Well, does anybody have any questions real quick any questions? All right, we're done. Have a great day


You see, he working his way over this way. Bring the ball back, boy! Hey, boy! Don't go in there now!  Oh, Joe, but what you doin', dawg?


Good morning world, it's day number 43 on our round the world trip and we are here in Morocco. Morocco! We camped here last night at a rest area just outside the town of La Rache, five euro for the night or 50 dirham and we camped here on the grass. So we had two birthday parties yesterday here. Good night. One was just here next to us and the other one was over there and then we had a lot of activities going on until like three in the morning. Yeah it looks like the locals are using this place quite a lot which is nice to see. It's not just a rest area or a camping area, it's a local hang out and there's like a bar cafe behind us and yeah it was a hive of activity last night. Eventually we had some peace and quiet and managed to have a bit of sleep which was nice. So we've packed up and we're ready to hit the road but let me show you where we are going to go today. Okay so we are here just outside of La Rache and here is where we went to yesterday to try to find the camp. This is definitely not the campsite. It is two camping, yes, two both clues. Okay and then we went back up to La Rache where we found this place and today we are going to be heading cross country into the heart of North Morocco all the way to Fez. So it's about 140 miles to go today and we are heading inland which is really exciting and I can't wait to see what Fez looks like as a city that looks like it's going to be absolutely an incredible place to visit. It's already 10 o'clock so better hit the road let's go. 21 degrees at the moment. It's not too bad eh? Very pleasant. Very pleasant. Goodbye camp here. Goodbye little camp. Goodbye little doggy sleeping in the leaves. Au revoir. Salam. Salam.  It looks like this place really comes alive in the evenings. Really to hit the road. So we just did the U-turn because Lavie said she spotted a turtle on the road so we are going to check that out because I want to see what's going on with that. Wait oh it's still there. Where? There straight ahead. Is it dead or? I don't know. Just park here on the road.  It's alive. It's alive no? Okay. Whoa look at that. Do you think they just have native turtles or do you think it's like a pet? Lavie's going to put the turtle over there under the tree in that little garden area. Some sheepies on the side of the road. Yeah the landscape starts to change. We can see some hills. Yeah it's going to get more and more mountainous I think as we go along as we get more and more down towards Fez. Look here's a Tajin massive mount of pottery. Look at that. Wow. So we're just stopping here at a rest area along the way. We've done about 30 miles 35 miles so far and we're going to have some eggs. Lavie's having a Moroccan salad. Yummy. Bumblebee's chilling over there. Very nice little place hey. Yeah totally. Then there's like a hotel and restaurant, cafe, petrol, kind of everything you need all in one place. Refuel and get on the road again. Yes this lunch cost us three euro. Yeah 30 dirham. I think the three euro is not going to break the bank. Just about. Staying in budget. Obviously every day we can spend less is much better for us because we don't have unlimited money. When we have been three months on the road we will break everything down and see where we're at, how much we have spent for accommodation, food groceries, transport, all of these things. Yeah at one point we're gonna do a video and break that all down. Look at this little place here. There's something going on. Yeah we're coming into a small town here. Got some tagines on the go over there. Some sunglasses. Looks like we are heading up into some more hilly terrain. I can see that the road gets a little bit wiggly up ahead. So it should be quite a nice road through the hills. Wow look at that. Yeah getting into the hills. Starting to get higher up into the hills and the landscape is just opening up around us. There's just vast vast vast planes all around still sort of cut up into sections. So it's like they're still farming up here or they're still using it for pasture or for growing. Yeah looks epic. It looks epic but you can still see that there's the there's sand in the atmosphere. So we don't really have like the clearest view but still really epic. That's to get really cool up here. Wow very nice. Also the temperature has dropped a few degrees since we've climbed in elevation so it's like 26 degrees now. There's actually like still cooler than where we were in Spain. So not so bad so far. Oh another onion truck. That's quite a nice smell when you're stuck behind them though. It's just this nice aroma onions coming backwards. Inside that truck must smell really nice. So it's now three in the afternoon and we're getting much closer to the Fez now. We're about 20 miles away but it's starting to get much drier around here. We're really seeing a change in the environment. Yeah it looks spectacular. It's really spectacular landscape. Yeah look at this and you can see it's just getting more and more barren and rocky. Yeah but the color of the landscapes changed quite a lot and it's more in this yellow red light reddish colors. It looks nice. It looks like a painting. A beautiful palette. Okay we are coming down out of the mountains and now we can see the city of Fez laid out in front of us. We can start to see this big load of white buildings all in front and that is the city of Fez and that is where we're going to be spending a couple of days. Yes and we booked actually a room in the old town where we can't really drive bumblebee too so we have to park bumblebee up somewhere secured parking and then we have to walk in. It's a really nice place it's like a traditional riyadh in the old town in the Medina or near the Medina. Yeah we found the room on Airbnb and it was I think 12 euros so a pretty good price. The pictures that looked amazing so I'm super excited to see how how it looks for real. It's cute. They are cute. Okay all right we're going. We've got just over six miles now heading in into the city and so we will meet our host actually in front of a hotel where they have parking for our little bumblebee here. It looks like you've got a bit more of the old town coming up in front. Oh my lord. He's coming out. Yep yeah the lanes and indication and this type of thing is not uh it's not too clear. It's kind of everything happening. You just got to sort of look around you and try not to bump into anyone I guess. Is there a taxi just there? Yep. Oh look there's like an ancient gate that we're going to be going through. Yeah I think this must be the walls of the old city. Oh wow. Because you can sort of see this like old walls all the way along. Yeah. So I think once we go up through here we'll be heading into the older part of Fez. Wow. There we go. Made it through the gate. Not bad. Whoa look at this. It's just like a sprawl of buildings on this side of us and so many taxis. These red taxis is the taxi of Fez and they look like they've been styled on the Fez hat because the Fez hat is red with a golden tassel on the top and these taxis are red with a golden block on the top. Oh yeah look we're going just to the right here but then we're going around and we're going in there. That way? Yes. Okay. These taxis aren't going are they? They're all waiting. And then the hotel is just here. Right here. Hotel Bata this one there. Ah okay okay okay. Hotel Bata I see. Okay. We are quite into the center of Fez now waiting for our Airbnb host to come and tell us where we can park around this area because we've got to carry all of our motorbike stuff all of our gear up to the room. I think it's going to get really sweaty. Okay we're in a little parking garage oh and he wants me to go in between the cars here. Oh my god. Good good good. I just followed our host down into this underground parking which is underneath the hotel that we were just in front so this is a nice secure place for mumble bee to stay so now we have got to take off our clothes because I'm boiling hot and unpack the bike. Just entered the old town here and there's no cars and no bikes on this streets in here. Oh it's I'm sweaty. Oh here we go. Look how narrow these streets are. This place you can't enter like Frigida TV something like bigger. You cannot you have tickets. Yeah this is like this. No way you can't say look. Yeah see. If you are fat you can assign them. Wow that was some kind of journey in. We were already like so hot on the bike and then jumping off the bike and then we had to unpack the bike and it was just really hot and we were wearing our trousers and boots and then carting all our stuff through into the old town through the streets. Oh my god just need to have a lie down now. We are dead but luckily we made it alive safe and happy to the right place and carrying the host is just amazing. He was carrying our stuff he was helping us so I'm really thankful for that. Oh we have to show you this awesome re-add that he has apparently just started up only three weeks ago I think they only just opened but we have to show you this place. We'll show you the kitchen. We made lunch just here and then they have a little kitchen area and it's all centered around this sort of massive atrium which stretches all the way up and there's not even a roof on it. It's just straight out to the top but we'll go up to the top now I'll show you the view. That's definitely the best part for me anyway because I love nice views. Look at this all around us it's just like roof tops roof tops roof tops roof tops and then there's like an ancient wall of the city all the way along and then the mountains in the background but it's super cool to be here in the old town like right in the center in the Medina area of Fez this is it's absolutely stunning here. So that's it from us today we really hope you enjoyed the episode if so please give us a thumbs up subscribe to the channel comment below please share with your friends and family we will see you next time


Tlo wasp We are on twitch. We are not live, but you can leave a like comment subscribe turn on your post notification bells Let's continue to grow the family from Chicago to the UK If we do go live and you happen to miss it This is where the highlights would be as you can see I have started reposting on this channel There four little videos for my new tactic. No, excuse me. Don't forget. We do got the discord as well and we also got the That's not it the patreon if you're looking for any of my other social medias and you can follow me on It's on the link below In the description and the link tree below. Dang. I can't even get my wordage right and nothing Hold on this is Liverpool this is from city life Liverpool after dark Y'all know Liverpool is where I leave Liverpool and Birmingham and Manchester Now, let's let's let's let's settle down for three for one number. I used to do the nightlife city walks Liverpool leads leads was number one Birmingham, I think Birmingham was number two and Liverpool was number three That's what the women Leads look like it was popping the most fun then Liverpool was number two Let's check this out though This is Liverpool after midnight I mean you are you and you out there at crunch time, ain't you? Okay, we're almost not off to a good start. This is this is this saved us Start to pick up Now don't don't don't don't come at me. Yeah, it's on the internet. I'm reacting to I'm drunk And he just he's not answering the phone is 4 a.m He could be on the game. He could be sleep, but I know he's cheating If he's not cheating real life, he's cheating his dreams. That's what this conversation is. I already know it Why are these Air Force one look at what she's doing one? I think as a whole no one should be allowed to where you shoot Air Force ones in the UK I'm tired of it.  This might be one of the best nightlife videos I've seen and we're only 55 seconds in Yeah, we getting right now He can he got angles It's her heels bogus or she or something like or do she got a handicap sticker in her car I'm asking the question. I'm not making fun of nobody. I want to know because I'm not sure what's going on right They drinking hula hooch. That's what that's called hula hooch I got a lot. She kind of looks she looked like a sleeper build Good Howdy Hey, we picking up traction right now you guys I don't see a lot of W's Look at them coming around a corner, bro. Let's make sure we get each other home safe like focus focus If you outside like this with your guys you I understand y'all bros, but at some point you've lost the plot And she looked good too, that's tough. I knew she was gonna look good by the color of this dress You can't pull that color off unless you look good Damn Listen y'all might think I'm I don't know what y'all think attribute when I be watching these videos How y'all be looking at me like he's such an a-hole. Look at him judging.  She's drop That means you look good.  What's the word? Boy capitalizing off your greatness He's got to be some of the greatest camera angles. I'm talking.  Yeah, I think everybody's out here dressed to impress even Brody. He got I don't know what shoes these are They look like B 22s. We could not be them, but This is one of the best Representations of liver one Liverpool women That I've ever seen on a night walk. I've done two of these. This isn't third one night walk in Liverpool This is third one this this might take Liverpool over the top I might have to switch my order around Liverpool looking like a top notch out here I'm already on a rare scouser it was for other reasons other than this but hey listen Yeah, get me lad Yeah Can I see a light off Yeah, light off. I'm talking about looks wise for Liverpool women. I'm not talking about how they acting or anything They're having a night out. It's good times This is it was a good night everybody was out W tattoo This is W tattoo I like the glasses Don't go see if they're gonna show my girl again, man I gotta go remember where my pick of the night was Gotta go back real quick. Let me see so far my pick of the night. Dang. I can't find it Get me. I can't find it. That's tough Wait, was that it?  I like tattered women man the one leg the arm Just W but your whole leg gotta be tattered I've noticed one thing about the UK man when it's about that time and everybody's drunk like they love sitting on the floor They don't know that's dirty I She's finished She is cooked take her to go get a kebab and take her home There you go, y'all got the right idea. I don't think I can take any more Air Force ones being this terrible Why are you looking at me? What's up? It's them Air Force ones you got on it better not be What them is them if them is Air Force ones they don't look like them anymore See my boy out here feeling like the man ain't he you got four of them with you salute Everybody getting progressively drunker this is crazy What's this called maca maca maca maca Happy birthday.   Do you go other places?  Say no or keep walking So what? You'll never see them again They look too young like all of them look super young but like They look too young Like she looked young but she looked like up above where I'm I'm sure everybody's above 21, but They should be tired at the end of the night. They do this and they do this in any city All the women do this they want barefoot bring extra shoes I Go home lost the plot sitting by yourself. Where are you going buddy? They got a Manchester one too. Oh, yeah, city life W's man W's W's Cuz I used to do a lot of these city walks with like they wasn't doing it right It's the only one I seen do it 100% right. There's nothing I could change about this This is it This is the pinnacle of greatness of what will a lot of if you really want to go party in the UK This is the channel you should go to to watch it city life W till I leave a like comment subscribe turn on your post is we done They're done live