Hey coach, welcome back to the channel. If you haven't yet, make sure you subscribe to stay up to date with all the latest content. At this point, Ben and myself with pretty much good content going out every 24 to 48 hours, so don't stay stuck with your business and if you need more help, where this video is, visit the description below. Click on the description. There'll be a number of ways that we can help you. You can either reach out to us with questions or if you're interested in working with us in our sports accelerator coaching program, then you can apply through there as well. So today I want to talk about how to make money coaching sports online. Now, a question that Ben and myself get asked on a regular basis is we get asked how can I make money coaching sports online or how can I start a remote coaching business? Now, something we do with coaches in our accelerator program is we do help them to implement an online business or an online coaching program with and on top of what they're currently doing in person. So what I wanted to share with you today is five different steps that you can take in order to start a business online where you're working with clients, not just in your local area, but you can work with clients all around the world. So the first one is you've got open and in-person training academy business first. So as I mentioned just a minute ago, majority of the clients that we work with that have online businesses are all clients who have in-person training academies. We don't work with coaches who just want to start an online business from scratch because it's very difficult and it does take a lot of time and consistency and we know, Ben and myself know that that is something that most coaches out there don't have. They don't have the patience, they don't have the consistency and they don't have the time to be able to grow it from scratch online. So what we like to do is we like to work with coaches who currently have training academies, might be in soccer, might be in basketball, whichever sport they're in, it might be baseball and they're currently working with customers already. They have in-person training clients that they're working with and now what they'd like to do is offer an additional service which is the online coaching. Now once you have 20, 30, 50, 100 or 100 plus customers that you're working with and you sit down and you think to yourself, do you know what? I want to now implement an online side to my current training business. The next step which is number two is create a survey with your current customer base to ask them whether they would invest into your online training. Now the reason why you want to do this and the reason why we work with coaches who already have in-person clients is because once you put your online product or your online service out there and your in-person clients start using it, you can use them as testimonials and proof that what your coaching does online actually works. Your coaches out there, they want to start these online businesses but they have never worked with clients, they don't have client success stories, they don't have testimonials, they don't have proof that what they actually do works, right? So for example in our company if you scroll down to the description and you click on where it says apply for our sports accelerator program that's going to take you to a landing page that you will find hundreds and hundreds of case studies, testimonials and proof, social proof of coaches that we have worked with, who we have helped and got results with. So it's the same concept with if you're starting an online coaching business and you're working with athletes, you need to have proof that what you do works and that it can get results. Right the next one is set up online platforms. So how are you going to do your coaching? Is it going to be just an ebook where clients purchase it from you? Is it going to be an online course? Is it going to be live training sessions where you're for example you're using YouTube and then you might create a YouTube channel to upload all your recorded sessions so clients can go back and do them after they've trained with you, right? So you've got to set up an online community and an online platform that you're going to use in order to run your online sports coaching business, right? And that also includes taking payments, right? But if you already work with customers in person then all of that should be in place and you can just add the online stuff as an extra product and use all the systems you have already with your online stuff, okay? That's why again we encourage get out there, build a business, build an in-person academy. Then once you've maxed it out then we look to add in the additional stuff which is the online. Okay now the fourth one is now you've got to build your coaching program and your price point. So building the program essentially means what problem are you looking to solve, okay? Is it a confidence problem? Is it a college recruitment problem? Is it technical training? Is it extra supplemental training that your clients need? What is it that your online coaching business is going to solve for the customers that buy your service, okay? What is the commitment that is involved? If it's an online program, right, it might be like a 12-week online program that you create that your clients can watch through it whenever, okay? Now if it's a live session where your clients are jumping on Zoom, you're on Zoom and you're doing the session together then what is the commitment for that? How are parents getting billed for that? What is the problem you're looking to solve? Is it a three-month program? Is it a six-month? Is it a 12-month? Right, so you've got to treat your online business the exact same way as if it was in-person. That's a problem that a lot of coaches have as well. They treat the online stuff with less importance than the in-person, right? You've got to remember the online is exactly the same as if you were to have an in-person, right? You've got to have systems. You've got to have marketing. You've got to have a coaching program in place. You've got to be charging parents, okay? So what is your coaching program? What is it you're looking to solve? What is the value that you add? Why should people come to you? Okay? And what is the price point? What is the commitment in terms of payment that parents or customers have to pay in order to be part of your online sports coaching program? Okay? And then the fifth one is personalize the experience. So once you start getting a certain amount of clients, you start working with them. You see what problems they're having at the moment, and then you can start building a business around solving those specific types of problems and then improving your service by maybe adding additional services to that.  Maybe you might decide to add an additional service where it's just an online program they can do any day of the week, but they've got to follow a 12-week program. Okay? It might be an e-book. It might be you offer workshops or you do webinars. Okay? So what would be additional services that you can add on top of your live sessions that you run for either Zoom or Skype? Okay? And how can we improve that? And then the sixth one, right? This is a bonus. It's not on the screen, but it's a bonus is now how can we market and sell this online sports coaching program so that you can work with more people across the world? A lot of coaches like this type of coaching program because it offers that opportunity for a lot of people to maybe work from home and also travel the world while also working with clients. That can be done if you have the right systems in place, but it comes down to how well you're marketing and promoting your business so that you're able to do this from wherever you are and it's an additional service that supplements your in-person. Okay? Now if you want to do this full-time then you've got to make sure that the online stuff pays you enough so that you can leave the in-person stuff and just focus on the online stuff. Okay? But that's something we can talk about on an upcoming video. Okay? So thank you for watching and if you have any questions description below that's the best way that you can get in contact with us with any questions that you may have. And remember if you haven't yet subscribed to the channel so you don't miss a single video that we post.


hat we create some kind of a CDN network and I'm not saying like akamai level basically an API driven mirroring network similar what we're doing with our mirroring now but it's API driven similar to S3 I think that's important for a lot of projects and we also want to improve our infrastructure security whether it's our security or the project security the weakest link and anything that we have hosted is that one machine that was maybe had one log in but it's been running for so long somebody forgot about it that still had access to other things on the infrastructure that gets owned and then owns the rest of the infrastructure you know having people dedicated to looking at that all the time and working through that is really really important another big idea that I've had is building an OSL university network university because well we're the university we have really good connections plus provides that neutral layer we've had a lot of I've had a lot of people from various universities come up with us it's like we love what you guys do with the OSL but how do we do the same thing where we're at and we're like well the first thing you got to do is build a data center and they're like no we can't do that so we've we've been working on this idea for a long time and we're finally getting to a point where we think we can implement it so basically we want to start collaborating with not only North American universities but global universities essentially we just want to be able to host a half rack of beer so it'll be standardized we'll have some geographic diversity it'll probably be just running some cloud services we're not going to be doing remote vocation at least not initially but just ability to have that has more hand support in my prior life and being a Gen 2 Linux infrastructure lead I had to deal with this where we had servers from all over the world and having to deal with those various IEPs was a pain in the rear maybe I just like doing that but I think for our point of view we can help with that pain on the rear and work through all of that and the projects just have to come to us to deal with any issues I have already mentioned bringing cloud services so if we had that you know CD and network a part of that would be hosted in that half rack and the other thing that's important for us at the open source lab is mentoring students so there's been a common theme that I've heard over and over again at least at the sessions I've been to where how do we get interested in doing this stuff or how do we even get some access to do that back when the day went out in university you never would have root on anything at the IS level if you wanted to work on information services and most of the stuff I learned on my own because I just tried doing stuff and at the lab we basically have scaled that up to work at OSU but that's great for us at OSU but that doesn't help the other universities so one thing that we're working on is developing a way to basically have some find a university and a specific mentor or a person kind of to lead that up and we can help mentor those students remotely it also would probably mean we would send some of our students over to that university help them get things going on or maybe they come to Oregon State and come to us but the idea is is this provides a mechanism for a particular university to invest in the type of ideas that the OSL does without having to build a data center yes they at least need to have a half-rack ideally but we can grant the root access or whatever so that they can get the experience they need to deal with these problems from a day to day and I think that's really important and I kind of talked about that already kickstart the OSL concept to other universities I think that's really important I think we also need a re-engineer or backend services that kind of fits the new day and age of things so continue working on standardizing our server management any IIT organization really isn't homogeneous we always strive to be homogeneous but I think we really need to try and do that to make things easier on us and for the communities at large now this doesn't mean we do a lot of unmanaged hosting which basically means they can spin up VMs and they can run whatever they want I'm not talking about that I'm talking about our internal stuff of what we manage we want to standardize we need to catch up with technology trends whether that means with certain kind of platform with a service you know containers we need to think of we need to catch up with that we need to have fully testable infrastructure which we're getting towards we need to make it more robust to failure you know hardware fails software fails often how do you work around that how do you work with network outages and things like that we need to be able to deal with that in a lot more automated way and we also need to make it a lot easier for our new services to play for projects so if you have a project that comes to us and say hey we want you to host Europe for us can you do it you know currently it's like well crap we need to like figure out how we want to host it do we make a we spin up a VM for it and then we got to get it in Chef and all this it'd be great just to have a system that this deploys it as a service and it's there and it's done and hopefully it has some way of updating in an easier way we can scale it if we need to that sort of stuff so this kind of goes on to bringing creating an OSL platform with a service I'm not saying being the source towards version of it but it's more of a you know fitting what we need so majority of our hosting is a simple web applications whether it's PHP whether it's Ruby now it's getting in the node you know you name it it's it's pretty simple stuff usually it's just a V host with some kind of application that needs to spin up there you go maybe there's some shared file storage that needs to happen whatever but really what we need to have is something else that's scalable so some of these projects they grow pretty quickly right now if we scale it we got to spin up another VM and go through all of that that's really not how the day and age works we need to kind of get up to the times we need something that allows us it also needs to be API driven for projects so it can be self-service so all we have to do is just grant them access and maybe set a quota and they can kind of do what they want and it'll also speed up our and expand our capability so we can automate you know a bunch of these things and we don't have to do it ourselves and I think having some kind of a platform as a service at the lab is really important and historically we would have a project VM and it would host all the stuff on that one VM that worked great back in 2007 but it's not the best idea in 2016 maybe this were some things you know some projects that said what they want and I'm not going to tell them no to that I'm not suggesting that we want to do that but there's a lot of projects that they just need something that just freaking works and they maybe they don't care how it's implemented if it's connected to some Git repository and they can update the code that way if they're happy they can do what they need to do so something else I wanted to talk about Supercell it is a project that we started back in 2009 Supercell I came from the Midwest that was back when cloud was becoming a thing and Supercells are big thunderstorms under head clouds this idea was basically building a cloud infrastructure or a compute resource for projects to do testing of any sort so we kind of did something with Facebook back in a day they wanted to be able to do some testing with hip-hop at the time we had Gennetti there was no open stack so we used that as a compute resource we worked on creating a web interface for it because it was primarily a command line interface to kind of make it work we have a couple you know dozen projects using it it was useful but it was pretty much an ad hoc just compute resource that would spin things up so it was great you know but it was one of those things that we created and we just kind of let and set in the rot and it's still important so I still think it's viable and I think we need to continue doing it so what I want to do is rebuild it with open stack and just expand it we're working with a couple of companies that kind of help either with the hardware or with the software side of that and kind of make that happen it'll ease onboarding a project if they want to use this resource we can offer pre-built managed CI solutions hopefully as well that would be great and we can you know since we're academia you know we have PhDs and master students working on various testing technologies that are fairly new and maybe projects want to have access to that to maybe catch new bugs we have a master student right now working with Paul McKinney from the Linux or from from IBM he works on the RCU and he's doing some really interesting stuff on testing specifically just to the RCU imagine if you do that with any project out there it's pretty interesting the other thing that we're diving into is education and diversity I don't know why that's on the Supercell slide but we're trying to make an open source track at OSU in the Logical Engineering Computer Science Department we're going to be creating some online classes later this in the fall well we're going to work on the DevOps part a little bit I hate using the word DevOps but we're working on creating basically some courses that people in the DevOps world need to know whether it's Linux or some administration application deployment test infrastructure database server management or you know things like that we're working on creating curriculum around that we also want to diversify our workforce in general as we all know that's not the best right now so it kind of sums things up I think testing resources are really important for projects we need to place the host weird unique hardware projects need some kind of a managed hosting service whether it's a platform where we fully manage a VM or whatever we need more API everything needs to be an API and it needs to be some kind of common tool set that projects have access to to do what they need to do and us personally we need to increase our academic mission around DevOps open-source software and so forth so with that we need your help specifically I need your help so right now I kind of want to start with a discussion of the future of host hosting based on kind of what I've talked about more importantly what do you need if you're a project what is missing what did I not cover what's important to you you think we might need to look into what should we be doing or what should we not be doing and yeah that's what I have so with that any questions oh got one way in the back I'm interested in your implementation of OpenStack I mean you know everyone says it's got to be this massive infrastructure of machines and what happens to make it realistic you guys have that kind of scale or well right now we have we manage it all with Chef and we have a basically a tune compute node cluster we have a really simple setup we're using Icehouse right now we're using Nova Networking because it's just kind of worked at the time we're primarily using it internally for our Chef configuration management run so when we do test kitchen we'll run spin things up do tests and tear it down we're using it actually in this computer Linux system administration class we also have a cluster running on power 8 with IBM that we worked on so we have any projects that want to have access to the power 8 platform they actually just get access to an OpenStack instance running VM but that's how we're doing it and part of what I've I need to work on is like how do we go to the next version with Neutron I'm looking at either Kilo or Liberty or whatever the M1 is going to be and kind of go from there but my point of view is I want something simple but it's a complicated thing because there's so many things that work and then they don't work and then when they don't work how do you fix it the nice thing about Gennetti is it's simple and it works but it doesn't it's not as flexible as OpenStack so you know what's funny is is actually our controller nodes are actually VMs on Gennetti because those are our pet VMs and we put our pet VMs on Gennetti but you know I really want to that's kind of how we have it deployed we're still figuring that out is basically my answer as everybody else is so I'm kind of curious with the with the OpenStack deployment is the long-term vision you have for that to be providing projects with basically API keys and quotas okay yeah basically my plan is is basically API API keys and quotas is what I really like to do I don't want to have a ticket come in like hey spin up this VM and we go on the horizon or whatever we do it god no no no no that's why I'm talking like API's API's API's you know but you know most of the time we spend are dealing with tickets is to do simple system and tasks that we can automate so that's really what I want to do I'm not planning on killing our Gennetti stuff because it's still really useful we might find ways of making it more API happy and easier to use but that's really what we're thinking like you know that the open stack environment could be used either for testing or whatever you know it's kind of it's there we might not being what open stack is we can't give a good SLA if we have SLAs on it because it might just break so we kind of need a I'm nervous about that but I think it's important especially on the testing side sorry to bang on about open stack but have you try to want to install PPI's that are out there like we're running within Alex Day or on bare metal we're primarily a CentOS shop so not really doing much on the Ubuntu side of that unfortunately or horizontally however you want to look any other comments or feedback I'm curious what you're thinking when you were when you were talking about the GCC compile farm and onboarding a lot of those non-X86 architectures one thing we do in Gentoo with all of our architectures is actually with some upstream projects is I just give them accounts on it because like the GMP guys Strace lib tools these projects that need to work on pretty much every architecture I mean Gentoo included but we have the ability to get machines for it it seems I mean you guys are hosting a lot of our machines and I in turn and kind of doing like hosting for these guys just to give them accounts on these architectures I feel like the GCC farm kind of intersects with that mm-hmm is there any talk of kind of opening that up a bit more without the GCC guys feeling like you know these are our resources why are other open source projects getting CPU time on it you know what I mean kind of I've been trying to have a better understanding of how we fit in with that GCC compile farm that's a good question and I don't know the real good answer to that I mean I want to we can we have the capability of hosting it so in the case of the GCC like the power eight systems for the GCC we can manage the hardware for it and they can do whatever they want but we also have this open sec infrastructure so and I think we've given you guys access to at least spin up DMs and do whatever you want and kind of go through there and we've been spinning that up for any projects on that particular architecture but that works for that architecture how do you do it with other architectures you know not everything's going to work with KVM maybe you don't want to have you want to have raw access how do you deal with that you got to manage shell accounts and go through all of that so I mean what is what do you what would you like to see I guess really what I'm looking for is I don't want to say Travis but the equivalent of a commit CI system but on different architecture yeah and that was another but go ahead whereas I think the the CI at the or Travis at the moment I think it offers I don't know if it has PPC but I think it just has 32-bit and 64-bit x86 so if you want to do CI on any other architecture you're pretty much yeah screwed so these other guys are basically I'm giving them accounts just to do CI for their projects that's it so you're basically saying if we have the capability of at least running kind of a Jenkins or some kind of CI system to run on the architecture that's what you would like you would love for us to be able to do that yeah I think that would I think that would help the most the people that I talked to and what would be the best way they would have access to it like we there's a specific tool set the specific API they like or they just want to have a Jenkins account and do whatever the hell they want I mean still bought Jenkins like whatever I think if they have the way of submitting jobs I think one of the advantage of the Travis with the GitHub integration is that you do get to include the script that does whatever random crap that you want so having a Travis CI kind of system was probably the most ideal where you include it and it configures it and it does it yeah yeah it's good to know yeah that's a good view you did any other questions wondering about the impact of GitHub itself you know because it's become the like the mother of all project hosting are you talking about the impact on us or just projects in general well no you guys I mean I know in the Drupal world Drush moved from Drupal.org to GitHub and it seems to be a trend you know you don't exist unless you're on GitHub yeah honestly I think it's been a net positive for us doing get hosting in general manually sucks and they feel that naturally well they have their own issue tracker if you like that they have their pull request system if you like that obviously we could start offering get lab instances which might be useful for some projects or Garrett for some projects I think that would be beneficial for them but I think a lot of the projects just use that I mean honestly I pushed I held back so long for us to actually start using GitHub and I finally like crap this is just not working plus for our students it's the resume essentially so I think for us it's not a bad deal like they fill a net we fill a net I think you know someday they may be no GitHub maybe there's something else out there that replaces GitHub but yeah the only thing I would add to that is just reliability GitHub is becoming kind of a single point of failure so get lab and others can be essentially used to give us more control and risk aversion then get lab GitHub engineers yeah yeah something else that I've been thinking about doing on the past side is like I don't want to reinvent the wheel so you know whether that means we're going to look in things like Cloud Foundry or things like that you know maybe that's what we need and we just put that in there and we give API access to projects and that's how they manage things I think that's going to be beneficial but they probably don't want to spin up the infrastructure to do all of that yeah we actually have an internal get lab instance not running because there were some things that like we just didn't want on get or get hub private you know private stuff that we wanted on there and so we started playing around with that and it's it's pretty nice but it's like how do we maintain tens of thousands of or not thousands hundreds of those that people wanted to host that does anyone know if there's like an online get hub but for subversion isn't that well google code used a lot of subversion back in the day and web svn itself is usable as a web interface to svn so like it already has htp DAV access if you wanted to deploy it yourself you can as far as like hosted services launch pad probably only bizarre okay so for the few of you in here that actually we host stuff for is there anything specific that you think that I'm horribly wrong about or that I'm missing or you think I should focus more on who's going to ask what not yet so something like open chips or well it seems like a simple container would probably do some of that something that they're not off people they are aware how to write those that they have no idea how to deploy yeah they just assume to send mail yeah anybody here excited about ipv6 because I don't have anything to offer there but I'm actually one of the things that we've gotten a lot we've had a lot of requests for serving bits over ssl and so we've been going down the fastly route like working with them but we haven't gone full fastly if you will I'm curious how much you've like the challenge with the traditional mirror network model is the security the certificate aspect of it and I'm curious when you've thought about or OSAIL city and if you're talking bouncing like mirror redirection type stuff or mirrors.osu.osl.org being multi-homed across the university networks that you would be part of it I really hadn't figured that out far enough ahead so whatever makes sense you know I'm kind of open to anything that works for projects because before we have a minute I want to get feedback from people like you like what's really important you know we've just recently got a request to having an SSL certificate on our FTP mirror which the hardware we have it on now won't be a big deal but then it's like well what new problems will that create for people if they just so happen to try SSL on something on there so you know that's important to know yeah and our problem like on our FTP mirroring network is we'll have virtual hosts that are that are on O.S.U.O.S.L but we have a lot of other virtual hosts set for other projects domains so then if we do that then how do we deal with adding all the other domains and dealing with the authority yeah it just gets complicated like yeah we can turn it on but it's going to probably break other things you know huh it makes it better yeah well any of the questions otherwise I'll call it good contact info a lot of you already know how to get contact with me and thanks for coming out and then it is like the voice of Zeus himself okay cool how about that cool well folks thank you so much for joining me at this advanced stage of the of the conference I think we're really kicking it into gear by now and from this point on it's just going to get better I do appreciate the fact this is of course the last day on a beautiful weekend for everybody who's coming in from from the East Coast you're welcome it's a beautiful day it's the last session to the talk for me it's I want to be sure that this being the probably the last one that you attend hopefully unless you have a time machine where I've gotten the reading of the schedule completely wrong I want to I want to end this on a very high note and be something that you guys really like and appreciate and have fun with my name is Brian Reynaro I am an engineer at MongoDB I am a developer advocate which is it kind of an interesting space to be in essentially I am a consulting engineer an integrations engineer a kind of a jack-of-all-trades that has been let off the leash so what I do is any time that anybody is using MongoDB from debugging it debugging their implementation of it preparing to scale out getting their data model all set whatever they need anything that touches MongoDB I can help them with and often do but now I am here to to talk about the internet of things and using it with MongoDB and thank you very much for joining me so the first question that I have for you I did tweet out a couple of days ago I wonder if anybody saw this this tweet there was a little bit of a challenge it was trying to get people to come in because I thought it was going to be the last one on Sunday does anybody know what this thing is can anybody answer this question what is this mysterious Egyptian device is it Egyptian is it otherworldly is it an extra terrestrial piece of technology any guesses what does it look like could be a water pump but if not good guess I like that one points for trying what does it look like grain mill windmill what you are looking at is a is a first what I would typify as the first institutionalized codified organized permanent wireless network this is a semaphore station this is from 1796 1796 it was invented by a gentleman named Claude Chapp and they could send there was a number of these stations each across France they could sparse them out from they stretch them out in intervals of about 20 miles so pretty pretty far distant and the way it would work is this is exactly a semaphore that you've seen on you know old naval vessels and stuff like that you know guys with the flag same principle is that the arms would be moved into different positions to indicate a different letter in the alphabet okay 1796 there is a network of these stations around France they still exist and they could send messages from one side of France to another in a matter of minutes right in fact there is famously a battle that was fought in Alsace as the French in the Alsatians are want to do and there was a battle fought and they got a message from Lille to Paris within an hour using the semaphore stations and that was a distance of 140 miles in 1796 1796 in fact in San Francisco there's a very famous hill called Telegraph Hill and it's not a Telegraph hill because of an electric Telegraph they actually had one of these on top of Telegraph Hill where Coyt Tower is now and the idea was that you there would be a guy up there watching the ships come in through the Golden Gate and then signaling that the ship was eminent its arrival was going to be eminent within a couple of hours right depending on the tides and stuff like that and if they could identify the ship they could identify the cargo it was carrying so if you're down on the other side of Telegraph Hill in the old city of San Francisco Yerba Buena you would look up and you would see at the Telegraph was telling you the semaphore line was telling you what cargo was about to come in port potatoes, eggs what have you eggs were rarity in San Francisco at that time the price dropped out of that commodity because you knew that there was a resupply of that coming in so you didn't overpay anyways so the gentleman who created this was a person named Claude Chap right 1796 invented the semaphore line revolutionized communications interestingly enough actually the semaphore line was in place and operative it was only replaced until 1846 by electric telegraphs that his brother would say was why would you want to have an electric telegraph that's totally unreliable anybody could come along and snip the wire right as opposed to a cloud bank coming and obscuring your network and causing connectivity problems but this was a very successful system right and it's an example of internet of things out there deployed in the environment that are exchanging data over these wireless networks that have to deal with environmental factors that have to deal with loss of connectivity and all sorts of different kinds of Byzantine faults if you will interestingly enough kind of sadly enough Claude Chap didn't live to see 1846 he instead threw himself down the well of a Parisian hotel to kill himself in 1805 just a few years after he invented the semaphore line why I don't know but I would just say the engineers never make money on this kind of stuff okay so how are we going to make you successful all right so what we're talking obviously about internet of things and the idea here is that what is the internet of things as we earlier there was a discussion here that was having we were talking about what is the internet of things why has it suddenly gotten so super hot as a topic because these are concepts obviously that go back to the 18th century if not earlier but why is it now that we're we have so much interest in the IoT space I think people are starting to realize what what is potential here or what is coming it's like dawning on us that all of these systems are starting to come together an example of IoT I'm just going to go through a couple of examples here because it's not just personal devices that we're dealing with we're dealing with any kind of system if we're going to define IoT we're dealing with a system that has transducers of some sort has interaction with the physical environment in some way to collect data that's going to be transmitted back to a central system and that's where the trouble comes in right is getting the data back on the central system and dealing with it so here of course is one example of IoT that we've worked with this is an embedded induction loop when a car goes over this sensor this induction loop it induces a voltage in the wire and trips the sensor the transducer goes off and we know that there's the presence of a car at this stop sign right not the sexiest implementation of IoT but you will love it in its absence you notice its absence because you're always waiting for at a stop light you're waiting for the timer to run out before you're allowed to get the green light this is one example you can with these sensors obviously deployed across networks we did a little bit of work collecting data from for presentation that it did a couple years ago that was really interesting about the state of New York publishes its traffic metering system actually a lot of states do that by now but you can actually go through their API and pull out vehicle speeds based on different segments of the road and they include weather conditions too as well so you can actually do some pretty cool analysis off of this published data set and maintained data set to see how does weather affect traffic conditions on certain kinds of roads and characterize those roads based on their average times and how many and how long it takes to get from one part from interval to interval on the car road another one that we've done a lot of work with is with power companies Internet of Things also includes smart meters there's a big field here these of course are now excuse me these of course are now fully wireless systems they have their own NIC cards in them there's no more meter reader that comes to physically read the meter anymore they just drive a truck through your neighborhood and pick up readings off of these meters if they're not already transmitted through the the transformer the electric transformer that's at the top of your poles so constantly drawing information off of these off of these meters it's kind of a fascinating space because what what kind of data are we getting off of the system we're getting voltage and temperature we're getting amperage off of these things so the deal is is we're trying to figure out how not just basic metering of how much power a household is using but maybe how they're using the power for instance I recently got an electric car and I'm completely pretentious about it so I called up PG&E and I said can I have a special rate PG&E well growing up down here it was SDG&E when I lived in San Diego I don't know what they call it now but PG&E is the north north part northern California power they would you know I said like well I plug in my car and off peak hours can you give me a discount no yeah yeah I can see that you do like I already got the car and they could tell from my meter reading that I was plugging it in at certain times so part of the work that we've done is to analyze some of the data that comes off of smart meters and then we say hey guess what the the goal was with this particular company was sending real-time notifications to someone's account that say like if you you're in a peak usage time if you turn off your washing machine which seems to be going right now you will save X amount of money please do now there's all sorts of fraud detection with these things as well because it turns out that people like grow houses are out there I mean there are some malifactors out there not necessarily that anybody's trying to analyze this data to find out if there's a grow house that people are trying to evade having to pay for electricity all the time so they'll rip these things out turn them upside down to try to get them to run in reverse literally foil to block the transmission of its data they'll actually take a meter off of a derelict house and put it on their house so that the billing goes to someone who doesn't you know totally different house so there's actually a component of location of where these things are in the data wild wacky times okay and then I think that there was a gentleman talking about home automation is another big field home automation is also very important of course for I'm speaking about it now because I think to put that in here but home automation of course is a big field as well H-back costs trying to keep that down now there are green mandates again to lower the energy consumption of large managed buildings and let's say office spaces as well that's a big component of what we're dealing with I think that there's also another component here while Fitbits are not necessarily all that necessarily advanced but there's they're trying to come with this idea that there's connectivity between devices themselves that play a part in the way that we want to deal with these systems that is to say that you have a network of devices that might be communicating with each other before they come back and communicate to the centralized system and also this idea that we at these these kinds of data that we're getting off of these new instruments these new devices are ones about ourselves which plays an extremely important part about the kind of processing that we need to do on this data the way that we architect our system what we're dealing with today is almost assuredly to be something different in the next six six to 18 months right talking including about embedded medical devices which are evolving as we speak and what kind of data will we get from them the I think probably the most meaningful thing that I've heard people say about IoT in general a lot of people talk about data volume and that's indeed a big big part of what's going to be happening on on these systems is just a huge amount of data coming into a system the volume the need to retain it the bandwidth required for the ingress of all of this data the ingestion of all this data but another aspect that I think is really overlooked at this point but won't be for long the value of the data that you're collecting on all these devices really doesn't mean anything unless you can process it some way you have to gain insight in this case we're creating a social network a social graph off of data that we can derive off of the way that people are interacting their proximity to other people what they're doing at what time and what context in marketing there's a term that I actually kind of like called a system of engagement system of engagement takes all of this data accrued from all of these different places could be devices could be apis personal accounts coordinates it and is trying to create an experience for the user that gets you to be engaged and doesn't fight you like if you have a if you have a flight booked to go to a city you want to know that there's a traffic alert along your route these systems are context aware so to borrow a little bit from that idea of systems of engagement I think an important part of IOT is remembering what kind of meaning you're trying to derive from these systems what kind of insights you're gaining what are you doing with the data that's actually important okay so when we talk about high level architecture and this is extreme that this is just kind of like setting a bare bones you know most people think of system architecture services architecture in this basic way I've got clients with data services layer in a back end database of course there's a lot missing in this kind of a diagram but it's I just wanted to start off with that one to so we can build out there in discuss a little bit of its absence we don't have monitoring systems in there there could be several layers of data services there could be the front end clients that are micro services that talk to the actual application clients that we see out there before you get towards the database it wouldn't be just a single layer you have a data services layer in front of the database and then your application clients ahead of that and that are actually talking to the back end systems but there's also a need for analytics archiving and monitoring of these systems as well which we're going to go into in a little bit but keep that visual because we're going to start building out from there okay so if we're going to start off we need a data strategy if we're going to be dealing with these architecting systems we need to figure out what we're going to do how we're going to handle our data and not just physically but the actual data model together now some of the things that I want to talk about today they're buzz terms and one of them is how many people have heard of event sourcing or CQRS? yeah there's some we'll go into detail about what that is in a very interesting pattern some would call it like a pseudo architecture of CQRS that's why I find this term a little bit more applicable is data strategy because it involves those ideas involve not just how you what database systems you're using how they're arranged and the data model itself but a little bit of the processing and how the nodes of the database system are arranged themselves as well so when we talk about data strategy let's talk about first off the way that we're going to model our data and the tools are going to be using this is a little bit of paying the rent because I'm going to obviously be using MongoDB with these systems but there are some several advantages with this idea the first thing how many people are familiar with MongoDB? great how many people have used it before? okay so really fast I just want to go through here some ideas about a flexible schema I hope that doesn't appear too small on the screen if it's not you'll have to move forward so I don't let's do it everybody stop what you're doing move forward because my slides are too small anyways the idea here is we talk often about flexible schemas which is an important aspect for this domain the reason that's an important aspect for this domain is that we are dealing with a lot of different types of things in the internets right we're going to be dealing with a lot of different devices that have different characteristics they could be very simple attributes that they're met that they're metering on they can be very complex but there's going to be a myriad of these devices some are older versions of a device that don't have the same features as newer versions do but they're still out there in the field active and collecting data so we need a system that allows us to easily ingest that data into our database maybe we'll become more strict as we move on and our classification of the data and the schema but for the time being we want to maintain a flexibility to ingest new kinds of data now the reason that I make note of this flexibility and how it applies to MongoDB is that in MongoDB there's not a schema enforcement you can use this flexible schema that is to say that records that are put into the database as long as they're well-formed JSON they will be accepted and there's not a need to check the presence or type of a field gives you a lot of flexibility for instance these are two records in the same collection or table if you will and the idea here is that they're of two different types right these are two records but the thing that's different about them is that they're two different types of vehicle what I like about this particular kind of demonstration or this example is they are subclasses of a parent class a sports bike is a subclass of vehicle just as helicopter is but they're two fundamentally different vehicles which we may want to put into the same collection even though they're fundamentally different things they share common ancestor or common understanding we want to keep them in the same collection because we want to access them by the same kind of query patterns okay so you see that in this particular example I'm using a discriminator field to determine the difference between these two objects but they share common attributes and one of them is is that this company Augusta makes helicopters as do they make sport bikes right so maybe I want to search on all of the vehicles made by Augusta for some reason or maybe all of the transducers or devices made by Siemens or General Electric or what have you Arduino if I so choose but the interesting thing is where the flexibility comes in is that we have what I'm referring to is a polymorphic attribute rake and trail is an attribute of the motorcycle that has no relevance to a helicopter and therefore does not need to be in that individual record just in the same token a motorcycle with blades a four-bladed motorcycle would be interesting crowd-pleasing but irresponsibly dangerous for the rider so we don't include they're not built with these blades and and they're not therefore necessary in this document so we have flexibility about the way we can store our data okay so that's just a little bit of level-setting about this document model let's talk about this this data strategy how we're going to actually deal with our data these sensors these internet of things they're sending in in many cases most cases they're sending in a state what their state is at a certain point something some condition some reading some measurement what what is happening to them at that moment and that's being sent back into the system those snapshots or those instantaneous snapshots in time really don't have a whole lot of meaning for us until we start thinking about them in aggregate now one way that we can deal with this is that we say we have an object that represents the state of a device or a system right now in the database and then every incoming message that comes into the database updates the state okay seems reasonable like I have a I have a smart meter I want to know what its current voltage is so I have a I have one record that corresponds to that one smart meter the new reading comes in from the meter and I update the state of that meter it's voltage last time I checked the last time a message came in was 220 volts it was at a temperature of 70 degrees Celsius and there was an amperage of 210 amps or something like that right it comes into the system that's the current state the problem is is that's an extremely limiting way to deal with our data and it's not we're losing insights we're losing ability and robustness against our data because if we're persisting state only we've lost a record of how we got into that state so the the fundamental idea behind event sourcing a buzzy term is that we don't want to think in terms of state on the on the system that is to say that we don't we do have an idea of state that we're going to derive state but we don't want to immediately think of a record persisting a current state of our objects what we're going to do instead is we're going to go into the system and we're logging these events an event log or domain events right if our domain is is smart meters it's going to be something involving current current power temperature all that kind of stuff if we're if we're dealing with a system a mobile system a component of that data is going to be where the where the object is physically on the planet at that time but what we're we're logging here is not we're not updating a state we're recording or persisting to the database deltas or changes or logs this would be akin to saying like I want to find out how much I'm going to have to do expenses right if I do a trip so how do I handle my expenses when I go home and submit an expense report I collect my receipts and then I then I derive how much I'm owed for my expenses this is like recording the receipts as opposed to constantly updating what my account is every time I make a buy a coffee or something like that right so we replay this log to derive a state right so at time 12 tick 12 the the state of this system the application state is nine okay now there's a lot of advantages to doing this if I lose the application state if the system crashes what do I do I replay the log and I regain my state no big deal very robust that way but there's also another advantage is I can see what the time what the state used to be in the past right I can see how I got there or maybe what the conditions were at that time five or ten I was up down all that kind of stuff I can go to any time in the past provided that I've not deleted out the logs and regenerate state in the past so maybe there's something I've learned about processing this data like I might find that there's certain users that are that I can reach or let's say there were users that I found that they have associations to one another I want to know if they were ever physically in a proximity to one another I can go back and search for that in the past I can change my data processing model my stochastic models and apply them to the way the world would have looked like in the past so if I update like bidding models or anything like that I can go back and see how well that would have played in the past so event sourcing is pretty powerful and in the domain of what we're doing here it's especially powerful because we want to keep track of how we got into the state we might want to find out from our sensors was there like if a sensor is about to fail was there variations in its readings does that indicate that there's some kind of failure on the system in the case of the smart meters that we were dealing with they wanted to determine if a group of sensor group of smart meters together in a location we're starting to act wobbly maybe it's not the an individual meter that's the problem it's the transformer that's about to fail and they need to get out there really fast otherwise the whole neighborhood is about to lose its power or something like that so another way to think about this is the A application state is derived is the first derivative of the log I think that I just like saying first derivative of the log I say that because derivatives are we're thinking in that it's continuous function off of the system or that you could do an interval off of the log typically with event sourcing you need to find your true state you need to replay it from the beginning of time and move forward to get into your application state you can't just start in the middle because that would obviously corrupt the system so that's a little bit of difference but we'll handle that in a second the idea here is that particular problem if I have to go back into the past to derive my current state if I lose my state or it gets corrupted somebody attacked me and they changed my application state in some way maybe by inserting erroneous records into the system or maybe I just got it wrong maybe I released a version of my software that was making logical errors or interpretations of the log that's okay I can go back and regenerate my state but if every time I need my application state I have to go back and replay my entire event log that could be prohibitively expensive in computation in disk in network I have to go back I could have terabytes of this data so the obvious solution is to create snapshots at regular intervals so when I need to go back in time to a certain known state I just go back to the last snapshot let's say I wanted to go time interval five then I go back to the snapshot I stored at time interval four load that and then play the event log up to the actual point in time that I want to recover two okay so lots of different ways this is important actually the snapshots this is particularly important because they are a view of the data if you will very much a view of the data at those times they themselves are generated by replaying the event log and as I said I can replay the event log with new information or changes to the way that I I process that log so not only do I have an opportunity with event sourcing to create snapshots of a type of state I can make different types of these these views right I can have multiple views all interpreting the event log in a different way for some reason very important okay so let's talk about some examples it just came down from from san francisco and the idea here is that there's a published data set on crud ad that I'll have linked to in the last slide that you can get you guys can check out and play with if you like but it's the last 30 days of taxi cab movements around the city I thought that was really cool so they're tracking taxi cabs and if they have somebody inside them and it's a lot of data for the last month is over 11 million records 11 million position locations of these cabs so it's every minute they're updating every maybe not a little bit frequently than that they're updating where they currently are where that cab is so there's a lot of data to play with possibly before uber and Lyft there might have been a lot more data to deal with but that's a different issue okay so here's what one of the records what comes in as actually the data that they provide is just a csv actually a space the limited list to I had to be ingested flat file here's how I format it and then just it into MongoDB okay same document model pretty simple the idea here is that number one each of these records is going to have an individual key a primary key that identifies one the cab that the taxi that is that we're recording what this reading is for this position this GPS location is and I've I've concatenated it with a pipe to include the time stamp right so each record is a snapshot in time of where this where this cab actually is okay in addition to that I have additional fields of the cab itself that I might want to play with that without having to parse out the individual record but the real sauce on this one is this format this GeoJSON format which is basically coordinates of latitude and longitude coordinates of where this car actually is at this point and you can see it's longitude first latitude second in MongoDB we can actually we have a geospatial index that allows us to index these these elements and search on them quite fast which is a nice feature that's going to make everything much easier to do that we're going to be doing this example second is what is the occupancy of this car zero is nobody's in it one is there's a fare going in playing around this data set in New York they actually show how many people are in the cab included with the fare price and the tip amount which I thought was really cool and I was going to play with that a little bit too as well to see if people tip more for longer distances and that kind of processing is much much simpler and then the last thing is that I want to know there's an actual smart date value in in MongoDB that I take the timestamp and I interpret it as an ISO date type an intelligent date type that I can index and search on in order through the system as well so I know where the cab is when it was there and if there was a fare currently going on or not now what I did for the processing on the system is as the stream of these these records come in I'm keeping track of I use like in electronics systems you have an edge trigger like if the voltage goes up that you change state on the increase of the voltages the system if if I met a state of fare zero right and I'm not currently on a ride when when that fare flag comes up the first record that comes up as one means the trip the starting that's one way that I can interpret these records and so for every concurrent every concurrent record that comes in while the rate while the fare flag is up as one I know we're on a trip we're going from somewhere to someplace with the rider and when it goes back down to zero the trip is over on the next record that comes in I consider that to be a zero I can actually I ran the code through off of these records ingested them in the MongoDB and then ran the code off of these records I can send it just out on the code so you guys can play with it as well with the data set and then of course started spitting out not how much the the car was where it was driving around when it was empty but when it had a fare so here's one that's overlaid on a map really kind of cool we see that I'm going from somewhere around the opera house over here down to down to my old neighborhood and that's one cab ride pretty cool can anybody see anything interesting about this particular route yeah a little known fact there's a tunnel between oak and fell now actually this is this is one of the issues that we'd be dealing with why do you think this happened why do you why is it that they teleported this card you can't yeah there's a there's a loss of connectivity in the system which makes and we can expect that of course right a couple of things maybe maybe the record's gone it just didn't make it depending on the system that we're dealing with maybe the record's gone it just if it doesn't if it doesn't make it through the network we'll never recover it other systems like smart mirrors will say hey I'm not able to transmit my data so I'm going to hang on to it for up to 40 days until somebody takes it off of me because I don't want to lose this data this system doesn't have that so the way that we handle that kind of failure mode is going to be different on a device device basis another company that I worked with they didn't want to deal with the logic of whether the transmission would get through or not so they had these record these sensors transmit everything they ever recorded every time and so the BOD rate of what was required of the network would just increase over time because these systems would continue to and they were hoping they were relying on a the unique key constraint in MongoDB that says like if you've inserted a record with this identifier before I'm going to return an error that you can't you can't reinsert something with this unique key that was their data management platform little unfortunate easy for them to code though anyway so that draws in a fact about what's what we can deal with this kind of a system here's what that record looks like this is the aggregate view of of my raw data stream was where those records right of where the cab is at a point in time from that using event sourcing I go through the log and I derive this view which is that trip that you see on the map those are the geolocation coordinates of where it was you see that I have it defined as a route I see the starting and time about where the where the cab was how long it took me to go there if I want to get really sophisticated I can use a haversign function to determine between these geolocation points what is the distance and divide by the time and then I get how fast the driver is running too as well I'm not about throwing that in here because it's kind of cool but I can always regenerate that I can add that functionality in because I've stored the records in my database and I can go back against these domain events and reproduce and enrich this data that's the the advantage of these views so actually I started really playing with this data it was a lot of fun and I said like well let's see everywhere this cab has taken a affair in the last month bonk and that's what that looks like now I'm not the only one that's played with this data set it's actually pretty cool and you'll see people more sophisticated than myself that actually put this on an animation and it's really cool you see the cabs going back and forth and they actually remove the city map on it and the city becomes apparent to you as the cabs are going through there now when I zoomed in on this I was playing with this a little bit you see that this particular cab is on every street in downtown right like he he nails them all he or she nails them all the thing I liked about this one is that he cut he takes somebody up to across the Golden Gate of course and he's got someone he took to Oakland right to the airport and he has a lot of trips that he makes down to SFL which is off the map here but I thought that was pretty cool now there's a 11 million records wanted to collate what are condensed all of these trips onto the map and really see how they're going that'll be for a subsequent follow yeah there's a question how do they do it what software is it I don't know I'm not exactly sure if I knew that I would have done that myself because it's a crowd pleaser that's for this presentation dot 2.0 I'll come up with some interesting stuff about that what I wanted to do was a heat map about the actual like I said use the haversine function and find out where the streets are the most the fastest streets and the slowest streets and then really get into the propeller head and that's in both really going down the the rabbit hole at what times like you know like what times are the streets fast is there a cab that's a speed demon with a lead foot and there's another one that drives really slowly and which one do you want to hail depends on how where you need to get yeah oh oh I'm using I'm not using map reduce in this particular case I'm using like how many people took electronics or basic electronics in school maybe we're programmers and maybe not so much but you know like an edge trigger you know what an edge trigger yeah yeah that's what I'm doing is edge triggering if I'm in a state that I'm there's not currently yeah I've got a flag or like a latch and I say like okay the fair flag went up start when it goes back down I will realize that that last record was both was the one cool and there was one more question oh thank you so much yeah so the question was how do how do I know when this goes when when a ride is started when it has not using an edge triggering function and your question there was another question oh what animations do they use I don't know I wish I knew because I would have been you guys would have been totally impressed more than you are currently okay so let's move on geolocation with this data there's lots of things we can do one of them is defining an area of downtown Manhattan for instance which I've done here as well this is this is a type polygon polygon which is supported by this kind of industry format of GeoJSON it's also supported by MongoDB so I can actually index on polygons like I can define polygons in the database why would I want to do such a thing maybe this is a service area maybe this is a zone a toll zone that as a government entity when I see your cab go into that zone I can bill you automatically just for your presence there maybe I can derive from that zone that my rates go up on the consumer side not the government side and then the neat thing is of course is this could be a floating zone that I'm moving around I can find if a person is calling if it's a service area for instance someone requests a service are they in the zone that I service right there's different variants about this that are really kind of cool is this device being activated or turned on in a place that I wouldn't expect it to be and is that can I attribute that to fraud right this would be geofencing as they say right all sorts of interesting things that I can do with this kind of data but the point is of course that this in itself this zone or the fact that the person appeared at that time these are different views off of this this data stream that we're getting off of the system so it's a pretty important part that we're recording these domain events we're recording these logs event logs so that we can replay them or on either retroactively or on the fly to create an idea of what we're looking at in the world now that's event sourcing event sourcing is closely associated to CQRS some say the guy that coined event sourcing Greg Young Greg Young says that event sourcing relies on something called CQRS Command Query Responsibility Segregation I'd pretty much go along with that myself too the idea here is that the fact that we're generating these different views of the data based on the event log right we can think about this is that when we access the database we have this basic pattern of ingesting the log ingesting the data as a log into the system but then we have these different kinds of views stacked up against them different ways that we look at the data right it's hard to see but this is kind of a couple shades of gray right here each shade means suppose I mean this is a different type of view on the data okay these kinds of this kind of our this has architectural impact that you'll see in just a moment the idea here is that we have variant read models and views these views are fundamentally different than the the the data that's coming in they're built off of the data that's coming into the system but they are aggregates or their interpretations or their averages there's something that we've done with the data to interpret them and that has an implication that because the data going into the system is different than than the data that we're extracting maybe we shouldn't use crud how many people or a crud object create excuse me yeah update delete create read update delete so from the application side when we think of interfacing with the database we have an object that represents the main object and we we perform these crud operations back and forth from the database by this one object we get we use getters to retrieve the data from the database and retrieve it as this object or we can use setters to set the object and then push it back into the database in one way or another but because the data stream is different right we have these domain events coming in on one channel but the way that we're interpreting it or these variant models these objects these pieces of our code should be separate because they perform separate functions they should have they have perform separate responsibilities therefore that there is this idea of command query responsibility segregation and the idea here is that your reads are handled by different portions components of the code than your rights do in this case command means writing writing to the database we're going to mutate the state of the database even if it's only an insertion of a mutable log records we're changing the state of the database and that's interpreted as command we're changing something on the database but we don't expect anything back we're inserting into the database we don't expect data back except for metadata that says yes the insertion was successful on the other hand the query doesn't change the state of the database system doesn't change the state of the data store but we do expect something back we expect something back from our query right so fundamentally different pattern fundamentally different objects the reason that this is interesting this is a cool idea is that on these systems where I may be doing lots of heavy analytics off of the incoming event stream maybe I have a disproportionately or asymmetrically high read load in comparison to my write mode possibly and if I have an asymmetrical read load to my write load I can asymmetrically scale these objects that can have lots of readers as opposed to very few or one-to-one read writer object now you can start to see if these these components of the code are different right and they're presenting the responsible for different views of the data at least the queries there are this it lends itself quite aptly to another buzz term that you probably have been hearing a lot about can anybody guess where this is going if this is a service area and instead of one big reader writer I use lots of small reader writers in my service small service microservice circle gets the square so microservices this is a pattern that lends itself to microservices now an interesting side note about microservices a lot is said about them right and when you ask somebody well how do I define a microservice what are the edge of a microservice what I take right now is a lot of people talk about the idea of these separations of concerns separations of code for maintainability but when you say well how big should my microservice be people scratch their head and says it depends here's a way to define a microservice is what is the view of the data that you want to see out there and one of these little read objects or query objects constitutes a microservice as does the writer and it has a single responsibility ingestion of the log stream ingestion of the domain event right and then the interpretation of that domain event log stream whatever you want is right over here right so now we're getting into microservices so I think that we have at count right now currently four buzz terms going in this talk did really good number one iot is itself buzzy but I've also talked about event sourcing and cqr ask for command query responsibility segregation which you have to say 10 times before you invoke it in your own code and then and then microservices but indeed that's fitting because a lot of this stuff is boiled into the complexity of an iot system which is as complex as you want it to be right so that's where these terms this is the fascinating part to me this is where these terms start to gel and have meaning to one another and relevancy to one another and context to one another the impact that cqr ask has on your system architecturally because ostensibly that's what this talk is about the architecture of a system is that if we have an asymmetrical load of read or write or one pattern over another pattern in our application tier maybe that asymmetrical load can be serviced asymmetrically by our data store so in MongoDB we have the secondaries we have all the rights go into the primary and then we have a system of replication where you replicate out the rights or the data out to the secondary notes which you can send reads to okay that's called eventual consistency now it's not eventual consistency is a tool it's a pattern it's not perfect for everything there's some caveats that you need to be careful about when you're using this kind of a system but in this kind of a pattern if we're regenerating these views on the data eventual consistency might be okay for me to use because I can I may not need to know exactly what the current state in the event stream is I'm regenerating this to take analytics for instance or something like that that doesn't require the most up to date data you get one needs to be careful about that okay so let's talk about how these systems get tricky let me just do a quick time check okay cool it's great these talks about buzz terms they're always about how it's going to be transformative it's going to change everything in the industry that's great I think that anybody who actually tries to implement these things they're obviously good solutions these ideas there's real hay behind them but I find like well I have to think pessimistically I maintain servers I've been on pager duty and the one thing I've learned is to be cautiously optimistic and strenuously pessimistic so where what's the tricky part of IoT in these systems well mishapery like for instance these these are systems this this guy almost got clobbered by the camera drone right these these systems they're out in the environment and as we saw with the cab example they don't always work they could themselves be reliable but we can expect service interruptions of some type so we need mechanisms to identify this and take action and again as we go into the future there's all sorts of problems that we could we could enter we can encounter with these kinds of systems because they're dealing with environments that we've not put sensors before I mean like if you think about embedded medical devices there could be a number of reasons that you lose connectivity to the system okay so how do we deal with these systems this is another emerging idea is a service management system this is actually kind of an interesting thing to me and I think this ties into a lot of the discussion that we see about containers that we see about infrastructure as code this idea that I can code my infrastructure or infrastructure as a service how does that work right I think as these systems become more complex we're going to need systems that handle that complexity complexity automatically for us especially since these systems they're expected to be up and reliable so a service management system is a component of the architecture that's responsible for detecting failures and taking action to resolve those failures or resolve those systems now we've seen examples of this in components of our overall architecture and indeed in MongoDB we've got a the replica set is there as a redundant system if one server goes down you've got this redundant system that they elect a new primary and they carry on right we've seen this in subcomponents in this system there's a service management system and it says like well not only is your architecture the back end it's also the front end these deployed systems these these objects what does it mean if the transformer or the central node that I'm collecting all this data from these service module the smart meters they stop collecting data or something like that can I deal with that problem and maintain service so the idea here is part of this is that you have the service management system has these agents they get their configuration or the understanding of what the world should be from the the federated configuration management database the idea is is that the the database stores information about what types of sensors are out there where they are what the recovery protocol is who to notify if they come down and the agents use that data to update automatically like maybe a ticketing system that says hey we've detected a failure we are using monitoring systems like Nagios and whatnot we enter a trouble ticket for that and then once once we have a resolution of that of that problem we update the trouble ticket automatically and it's resolved the ticket is long and all that kind of fun stuff could be quite sophisticated for these systems but if you consider the amount of these these devices that are distributed out there this could be an interesting part of architecture in the future maybe it's possible to completely automate this based on the the idea and side of a configurated management database I think with all the maturity that's coming with containerized systems it's possible and and orchestration systems so like I said the idea that you're containing you have here is where these devices are this is a data set I imported this is all the free wi-fi spots hot spots in New York so if you need connectivity just give me just tweet at me and I'll send you this map but yeah if one of these fails where are they could where do I send where do I send my service technician to right the other idea is that part of the this service management system is it's necessary to have the right metrics and monitoring on the system so you can keep your system in flight super important right we have all these deployed systems if you don't know what the state of these deployed systems are it's going to start munging up you know you're going to get if we're performing analytics off of the data that we're getting off these sensors the standard deviation of our processing or results or averages is going to go wild because maybe a lot of these devices are failing and it's affecting adversely the accuracy of our metrics we need to know that so that we can adjust accordingly that we don't get into this negative feedback loop that we're losing sensors and it's affecting our analytics that's affecting the way that we deploy and use our sensors right so monitoring just like you monitor your back end infrastructure it's going to be important for monitoring the state of these deployed servers and then of course it's very important that if you're monitoring these systems is that you have an accurate system for alerting right and then letting that's the juror ticket that's letting people know there's a problem going on in this system I love human like non-technological well what's not technological without Batman but I love using analogies the most famous alerting system of all in the case of smart meters again where we've seen a lot of work the one of one of the records that we'd see here would be something along the lines of you know this component has a threshold like we have alerts that we want to configure that we're setting for this specific type of meter what are these thresholds readings does the temperature go out of whack that was what they were really they would see as the leading indicator of an imminent failure of the system was the temperature on these sensors goes really high and it's about to die I thought it would be something different like the voltage would go up or down and the amperage would go up and down now it's the temperature so using this configuration data I can go back and set my monitoring thresholds right so as I'm replaying the sensor log that comes in off of this system I'm comparing it to that threshold and when the threshold is is exceeded I trigger off I go into this alert state and trigger off a trouble ticket and bugzill it through the API doesn't matter of example now how do we how we actually do this this this is involved this is another part of trickiness with event sourcing when we replay the log we we exceed this threshold so it's pretty simple we send a notification to the subscriber service right this that takes an action the SMS takes an action to go start the trouble ticket raise the alarms dive everybody wake up you know all this kind of good stuff one thing that you need to be careful with using event sourcing on these external notifications is that if you are replaying if you've lost application state and you replay the log does your system know to differentiate between the first time you pass that threshold or is replaying the log going to re-trigger the subscriber service because the subscriber service doesn't understand the data model the same way that your your event sourcing system does so it's possible that you can re-trigger these states by replaying the log or regaining your state in fact to use another human analogy there's a very famous example of this how many people are over a certain age that knows know who these people are number one what show this is and what he's saying has anybody seen this yeah I'll let you do it cheeseburger cheeseburger this is from 1977 Dan Aykroyd John Belushi and a very young in the back Bill Murray right and the sketch is is that there's this cafe where all they serve is cheeseburgers and this guy Robert Klein is saying I don't want a cheeseburger so we're early in the morning for cheeseburgers I want I want eggs and John Belushi says look he's having a cheeseburger he's having a cheeseburger cheeseburger cheeseburger cheeseburger cheeseburger every time he says cheeseburger his brother interprets that as an order for another cheeseburger and he starts putting it on the he's replaying the event log and his subscriber service can't make the differentiation right that's the classic case of this kind of renotification cheeseburger cheeseburger cheeseburger and then he finally says cheeseburger cheeseburger cheeseburger Robert Klein says okay fine I'll have a cheeseburger and so John Belushi turns around and says cheeseburger and then his brother says no more cheeseburger it was funny in 77 anyways so what do they need what do they need to solve this system what is the problem what would you do any ideas who didn't put themselves through school by working in a restaurant I guess I'm you know well the answer to this is quite frankly they need a circular buffer right you've seen this right this is quite literally a circular buffer right and the way that it's you put the ticket on the buffer it's a queuing system right and who's responsible for taking the ticket off the chef the cook right it's a consumer driven queuing system it's not a push system it's a pull driven system meaning that the the person the subscriber on the system maintains their own state about whether they were seeing a message or not it's harder you could implement such a thing on a push system but it's much harder coming in on the on the wind I think we're almost out of time so within in this case the way that we do these external updates is that when the notification comes through we're gonna be pushing on to a queuing system like Kafka or something like that and then the subscriber consumes it off of off of the system that's how it affects okay so one last thing about gaining insight on these systems and I'll try and wrap this ooh I'm right on the money okay this is the good part so give me a few seconds I'm gonna run over a little bit this is the Gulfstream first charted in 1770 right how did they do that how did they know that there was the Gulfstream there how did they do this very ingeniously a guy came up with this idea that he would put a card inside of a bottle and throw it into the ocean inside of that card it says if you find this please write down where you found it your latitude and longitude and the date that you found it and if you'd like to tell us what the ocean temperature was and weather conditions were even better and then mail this card back to me so I can aggregate that data this is a deployed sensor system and they figured out where the Gulfstream was who was this guy that thought of this well yuck yuck yuck it's Ben Franklin right and he he actually cut his scientific teeth doing this everybody loved him famous commerce clipped two weeks off of an Atlantic voyage if you figured out where the where the these merchant vessels kind of knew that there was something out there and they were out running the U.S. battleships to get to England and stuff like that so he had to figure out where this Gulfstream was so this is the point of this particular foray into the past is your collecting data now you have to do insights with it right so in those systems we need we need to integrate our data systems with processing how many people have heard of Hadoop okay how many people have heard of spark okay yeah spark is pretty buzzy Hadoop is very buzzy or had had been still kind of is these are related technologies and the idea here is that they're going to be doing processing of this data beyond perhaps what you're going to be processing and just in MongoDB alone MongoDB is really good for grouping and averaging and aggregate functions with the aggregation framework but when it comes to machine learning libraries you don't really want to have to roll your own that's where these systems come in they're there to handle huge data sets and processing but also they come with a lot of machine learning libraries that help you to do some sophisticated stuff I'm going to zip through this but the way that these technologies work is there's this kind of concentric rings of abstraction at the basis of the distributed file system that these systems will perform the processing against the distributed set of nodes and then the the spark and Hadoop layer in the middle of there actually performs the processing of this data in a distributed way across all these nodes this gets particularly sophisticated and complicated to people because this is a whole architecture in itself and a lot of people get frustrated about these systems because they're so complex right in fact you can see that typically the way that people would process the kind of integrate their systems with this back end analysis is that they would have their data services up here where the data is coming from into the application database and they have to ETL it over into these HDFS nodes then to be processed by spark or Hadoop right this looks like crazy right I mean it's a lot of people deploy that way but there's a lot of overhead associated with maintenance of these servers as well as the seemingly trivial but deceptively difficult ETLing system from one server from one system to another system if that ETL system breaks your analytics break and if you're trying to do a really fast closed loop analytics system for real time or nil real time analytics if your ETL breaks you start losing money because you're not able to process this data so where MongoDB comes into such a system is that we replace the the HDFS layer or you can replace the HDFS layer with MongoDB so that you're performing these analytics against the database against the data in the database and you significantly reduce the complexity of your architecture down into something more manageable when this this idea here that our architecture is now kind of morphed is that we have these data services with them itself composed of microservices that are what the views are on that data or what those microservices are to do with the inbound log stream is influenced by the analytics that's occurring on these back end offline systems that you're integrated with this is how you get value off of the data right this analytics and systems of that so I'm over time it's just one last thing I wanted to say the connector is available on GitHub I'll put a link in there later on well here's the link actually it's in my slides there you go and you guys can play with that as well if you like I'll I have some examples up there I want to cook up some more that you guys can run off of the taxi cab stuff and play with one last thing I wanted to do is who can tell me what this person is this is Emmy Hennings she is a inscrutable data as artists from World War One she bears no relevance to this talk I just thought I'd throw in another stumper there thank you very much for your time enjoy the rest of your weekend if you guys want to have questions I'm perfectly happy to answer questions I'm five minutes over I don't know if that means that I've run into higher rates for this room if anybody says who is the speaker I'll point to somebody else and get out the back door thank you for your time it's much appreciated


Imagine thinking that's a good poster.  Everyone else is dead serious not only is Tony Stark engulfed in flames But he's got a bird man blowing out of his heart. I like the spider-man suit But it looks like he's wearing a Casio watch in the poster.  We can see shocker even made the poster He's he's so tiny and he's doing his little attack that he does precisely one time in the movie Happy suiting up like he's going to prom which I guess is maybe the most Relevant thing in the poster because they do in fact attend a prom in this