Hello, good afternoon everybody, I'm glad you all made it, I know it's the last session on a Wednesday, afterwards been a pretty busy week, so I'm kind of impressed to see there's many people here, I wasn't actually expecting that many people, so good job. So my name is Andy, I'm an engineer on the Rackspace private cloud engineering and development team, and along with my colleague Jesse, who's currently sitting, we're going to talk to you about logging, and how you can manage it in your open stack environment. The way that the architecture has changed for kind of cloud and cloud applications and infrastructure as well, means that the way we were doing things before with logging doesn't work now, and we've needed to address that, the issues that have arisen from that, like very recently for our team, and so we've literally just gone on this path in the last six months or so, and so we've learned quite a bit and we'd like to share that with you. So I work on a pretty big team of really good guys, and like we're all developers at the moment, working on the OpenStack Ansible Deployment project primarily, but what it does mean is that if I say anything that's completely wrong, or a bit weird, the shepherd cricks are probably going to come out, and the talk might end a little bit early, so hopefully that doesn't happen and I stay on stage the whole time, and then everyone can be happy. So why should we care about logging at all? Logging is kind of the ugly cousin of monitoring and metrics. You don't know you need them until you really need them. What we get a lot of the time is product comes to us asking for metrics and things that they can measure, things like monitoring and metrics are very tangible, so when stuff's down a customer wants to know that something's down, and that's where monitoring comes in, and what the product wants to show customers are graphs and pretty pictures that customers can look at and go, wow, this is awesome, and those are the metrics, but customers don't really care about logging in the sense that they never look at it, and why should I care? It's just a bunch of lines of meaningless text to me, and that's a problem for us because developers also don't care about logs in the same way as supports and operations do, and developers tend to care more about stack traces as my application running or things working, is it being deployed properly, but supports and operations need the logs at critical moments when things are not going well, and at that point it's too late to say, oh, you know what, our logging solution is kind of bad, your site's done, give me a couple weeks and I'll fix the logging and then maybe we'll get back to you. So logs aren't just for support people, other people should care regardless. A couple other use cases like auditors need logs, anyone in compliance needs to store the logs, you know, you can use them for, aside from troubleshoots, you can use them for trending, access correlation, there's a whole bunch of other use cases that like I'm not even going to go into really, oh, I forgot, wait too many, back, sorry about that, there we go, okay. So yeah, like a lot of the use cases we get are like, hey, show us metrics, show us monitoring, but you kind of need the logs and the trick is also that even auditors who need the logs and need to store them, they don't actually care what the logs say, they just care that, hey, I've got logs and they stored, like are these logs being stored?  And almost all of that boils down to logs. So we all have logs on our servers anyway, so what's the big deal? Like why do we care about how they're managed? And that's great, so if we look at the old-day solutions of like we have one server with a web server, an application, a database and they're all on the same server and we've got one system administrator and the system administrator logs on to the server and we've got a site down issue and he looks in the Apache logs and he looks in the database logs and maybe there's an application log and we'll say a system log as well. So he's got four logs to look in but that's you know that's pretty doable, I can look in those four logs it'll maybe take me 15 minutes, find out what the problem is, I fix the problem, everyone's happy. What if we had two servers now and what if we separate our app and our web servers out into separate servers? So now we've got four servers, one running, two running application, a web server and maybe a database server and each of them have system logs. Now we've got eight logs across four servers and now it's going to take me a lot longer than 15 minutes to log into all the servers, look at all the logs and find out what the issue is. Now we move that forward to what we're looking at now which is that the explosion of kind of containers and microservices has meant that your logs are all over the place. They're on cloud instances and applications, for infrastructure they're often inside containers, they're everywhere and some of these things don't last long so I can spin up a cloud server and spin it down and then the logs are gone and then how do I troubleshoot that and then on top of that the sheer volume of logs, like I've got 100 logs on 100 different containers and I need to look inside all of them to find the problem but this just doesn't scale and it's a pretty common problem for like most things in in ITN cloud is that you can't look at the problems that you have now and solve them the way you were but just more of that and so you need to get smarter and you need to get more efficient so it's kind of like in the olden days before cars and bikes and planes you could walk around your town because in order to get food and water it was only a kilometer away but as soon as you needed something outside of your town you might need to get a bike because it's maybe 20 kilometers away and walking's not adequate for that but then if it's 100 kilometers I now need a car and if it's thousands of kilometers I need a plane and that's kind of the way that ITN pretty much everything else is is evolving like as soon as things start to scale exponentially we just can't keep doing the same solutions and this is kind of how the error of the log gripper began I don't know I don't know if Siri's doing my presentation for me but it actually would be useful so if Siri's in the audience like feel free to come up so I don't know if there are any John Oliver fans who watched last week tonight but he's got a segment that's called how's this still a thing and so tonight I kind of want to ask the log gripper how is this still a thing so I don't know if you are aware but there are people in organizations all across the world whose job it is literally to sit there and grab logs and it's normally for like cloud infrastructure we've got massive public clouds in various companies and I'm going to sit there and I'm going to grab logs all day and it's also normally the junior guy like you can be the log gripper enjoy that like Jesse's actually our log guy so he's going to do the demo later so you can look forward to like first-hand experience from a log gripper so for those of you that don't know grep is a Linux command line tool that's it's a pattern searcher so it's commonly used to search logs and other files I don't want to give the impression that it's not useful at all as a tool I mean it completely is I use it on a daily basis I think pretty much any developer or system administrator does but as a tool to search through thousands of lines of logs at a time it's it's not adequate it's it's simply not I really don't think this should be a job in in the modern cloud era and I think that the time for gripping through logs needs to go back to you know seven years ago when I had a one server infrastructure or two server infrastructure and and things weren't distributed properly it was it was just I have a load balancer and a couple servers behind it and we're all happy now I'm pretty sure they don't recognize this on their LinkedIn profile or their CV but uh you know look out for some key terms and I'm sure you'll spot them in fact anything that says junior cloud something is probably the log gripper a friend of mine put it like this it's like a pigeon looking for a solid dirty seed in a pile of wood chips and as the infrastructure scales you could say it's like a pigeon looking for a seed in one of several hundred piles of wood chips the end result is that you get disengaged unhappy employees you don't like the jobs they're doing and look to go elsewhere so recruiters if you find a junior cloud person you should probably ask them if they want a job they almost certainly do we have to get more efficient we have to keep these people engaged there's like smart people doing stupid jobs and we need to stop that so a little bit of an ad break I'm going to do a bit of a plug so apologies in advance but it does relate back to the log talk so I'll dovetail it back in one of the projects that we work mainly on is the open stack ansible deployment project my colleague Kevin did a talk yesterday I'm not sure if if any of you were at it it was in the stream actually and I just wanted to to give a quick plug to it because as somebody works pretty extensively on it it's something that I'm pretty proud of I think the work that that has gone into it is is really great and it works a lot better than any of the other deployment tools that I've used at least as with a lot of great things it started off with rack space but we've now made it into a community project so I'm a little bit biased but definitely check it out we're on ISE on on hash open stack ansible and it's up on Stackforge on github so if you're interested in deploying cloud or want to play around with like a container architecture for deploying open stack then definitely take a look and the reason it's kind of important to the log talk is mostly because the change in architecture that we took was from everything deployed on metal to all the open stack services deployed in containers with the exception of compute and swift storage they're all installed on containers and that's important because the logs for everything are now distributed across probably 40 containers for an install and are and are still in in different locations so if you take nova as an example you've got a nova scheduler container you've got a nova api container you've got a nova metadata container and that means I've got three places to look in for nova logs you know I mean that's not even including like the scheduler and all sorts of other services for nova so it just scales and makes the log problem much bigger and and makes a solution for that problem a lot more important and that's actually the reason that we've now addressed this and that we're giving this talk is because before when we had all the services on physical boxes it wasn't smart and it wasn't good but the problem wasn't big enough for us to address it and this moved it it just it just made it so much bigger that we couldn't ignore it anymore so one of the problems I've been talking about is that the logs are distributed across many containers in in many many places so a pretty simple solution would just be to create a centralized log server and that's actually one of the things we have done we have a centralized access log server but if we take a centralized access log server as the example it's great right we have all the we have all the logs in one place and you know that we can now search them in one place instead of having to log into into other servers but aside from skipping the step where we log into a whole bunch of servers what problem does it actually solve like we still have the issue that we've got you know thousands of individual logs they're just in one place um and like what if that server were to die like if we've got one server and it dies we just lost all our logs I mean that's way worse than any problem we had before actually um and can we scale it like if I've got this many logs coming through to one server that server is probably gonna like fall over under the weight of all the logs that are pushing through it we could crash our entire infrastructure really and then like backing up the logs like how do I ensure that they're not going to just disappear so we we solved that which is with a pretty common application that pretty much everyone is using who's doing log management for open stack and well any large application really by using elastic search so elastic search is kind of like a database where we can send the logs and they're then searchable and they're indexed and we can search via tags and fields on the logs that we've sent through and you can use command line new tools or there's a GUI called Kibana which you can use to to search it the one of the benefits of Kibana is it also gives the the pretty graphs and the pretty pictures that product are after and it's also useful in terms of just showing the the kind of logs that you have in and correlating logs elastic search can also be clustered so that kind of handles the performance and resilience issues and you can also do some cool things with archiving for example you can archive off to an existing Swift cluster for example or another object store so why is elastic search not enough so we've got all the logs in one place now and it's not in separate log files they're all actually in the same searchable place so everyone's happy well it's not quite the end of the problem so when you put a whole bunch of logs from different locations in one place what you're really doing is comparing oranges to apples to beach balls they're all round and but they don't all fit in the exact same way they have different characteristics and different use cases but really i can't search them to be more concrete in terms of what you do in OpenStack the application logs the database logs the system logs they're all in different formats in fact the OpenStack logs themselves i think have around four to five different log formats so just in OpenStack itself we're talking of like four to five different things that don't match up if i were to just throw them all in elastic search this leads to loads and loads of entries in elastic search but they're not searchable so really what all we've done is create a more jumbled view of hundreds and hundreds of logs that isn't really useful to anyone because now i can't even really see where they're from or what what the message is or anything really it's actually counterproductive and like for that reason you need to have a system where containing all the logs just isn't enough so how do we make the logs searchable how do we put this together so one of the things that you have to do is you have to tag and format all the logs and filter out the logs that you don't care about so a lot of logs at the moment in OpenStack are not set to the right priority so there was actually an ops meetup yesterday for logs which was incredibly useful but one of the takeaways from that is that most people who are doing logging from an ops perspective are doing all their logs on debug mode now that sounds really dumb because if anyone's looked at debug mode the logs that you get out are ridiculously verbose there's a ton that no one really cares about but the problem is that there's some logs that people do care about that only come out in debug mode and so what ends up happening is I ignore all the logs I don't care about and I have to filter through tons to get the ones that I actually do care about but the point is you can filter you can format logs and you can tag logs and for this we use LogStash which is also really common I think most people are using LogStash to to do the kind of log filtering formatting and tagging and and yeah you can basically like tag and and format logs so that all the logs are of the same type so if I take a log from patchier I take a log from like the nova console for example I have I set them out so that the time field is the same so now I can search on time field or the the source IP or something like that is is the same so a couple of the things that we've learned so as I mentioned we originally started looking at changing the way we do the logging because of the change to our architecture so we like to do as much as we can open source so the whole OpenStack Ansible Deployment Project is is open source and even the add-on stuff like the logging which isn't technically in the project is open source it's available on our Rackspace RCB Ops it's on GitHub so there's a repository called RPC Extras which has all these kind of things and a couple other tools that we use specifically for Rackspace we move them out of OS Ansible Deployment because it's they're like Rackspace specific things and we think that our opinionated version of what you should do should be separate and we want the OS Ansible Deployment project to just be community driven specific to OpenStack but one of the things we wanted to do originally was look at what other people had done with this and use it or fix it or make it better or contribute to it but what we found was that there were three implementations that were public they were all really out of date or didn't work quite how they should work and and we're now not useless in terms of they had no merit but we're not fit for what they were originally purpose for and were not being maintained or updated and there was no real community around it so there's no benefit to use these the funny thing is though that as an example in the ops meetup yesterday there there were at least three companies that are all using elastic search and log stash and are all doing filters and are all doing tags and formatting and and all the other things that you need to do and if you looked at them they're probably almost exactly the same but they've all done them by themselves and that's a little bit weird we've obviously gone and done the exact same thing so you know hands up to that as well but we have put it on our RPC extras thing and we'd love to to have something where you know there's a there's a single set of like these kind of formats and filters and all sorts of other things for logs dash one of the issues again is that the devs see the logs differently to to like ops and it's a community problem so the people are going to fix the problem on the devs but they really need the ops guys to get involved and say these are the things we don't like these are things we do like these logs shouldn't be debug mode these logs should be important these logs aren't required at all they should just be removed and and various other things and i do want to give a shout out to the open stack community for that because when i first started working on open stack in in 2012 i went to my first conference in san francisco and there was a obsession on logging and the obsession on logging in san francisco was a little bit different to the obsession yesterday in san francisco there was room full of ops guys and a handful of devs and they were essentially hurling words at each other that were not quite so friendly so the the ops perspective was this stuff sucks and all this stuff is terrible and you guys have done a bad job and like this whole thing is rubbish and the dev response was essentially well patch is welcome like you guys can do this stuff yourself and blah blah blah whatever which is just like this is not healthy conversation right and nothing changed in the logging in the following couple months which is pretty much a takeaway from the whole session when i compare that to yesterday's session when we had dev dev people saying hey listen give us give us examples of what logs you think are not worthwhile give us examples of how we can make this better like what would be useful to you how can we make this better for you like we want to help but you need to tell us what we need to do and i think that that's been very prevalent throughout open stack over the years how the the dev community has adopted you know the requirements from ops and tried to to engage and i think that at some point it needs to be a 50-50 so the ops guys need to come to the table and the dev guys but they're waiting to help and and so that's one of the one of the main issues and even within our own organization there's resistance to change right like as a sysad i know how to do grep i've looked through logs using grep for you know years like this is what i do so you're not telling me to throw that out the window and use this new swanky tool that you've created or you know implemented like why am i going to do that like i don't know how to use this thing it seems that's efficient i really know how to use grep i'll just do that and so it's hard to get people to start using the tools even when you've built them or they think they know the answer to the problem that they have and so they're telling you what they want and what they think the answer is rather than what they actually need and and so that's a couple of of the issues that we've had and then another thing is that what tends to happen with ops is that they don't think they can talk to the development team well or they don't really have a good feedback loop and so they end up taking whatever the devs have built and just building some stuff on top of it to make it look like how they think it should look and that's not really an effective way to do it i mean a perfect example is the the kind of filters that people have in general to remove logs like the correct approaches to go to the dev team and say hey these logs aren't needed like let's get rid of them or these logs are needed and and we're not getting them in the right velocity so let's like add those ones in not to add filters that just hide the fact that the stuff is broken so that was a pretty just general overview of the the kind of things that we've done uh my lovely assistant Jesse is now going to do the hard part which is the demo so fingers crossed it goes okay um but yeah he's going to run through some of the the tooling that i've talked about and and just kind of reinforce some of the things that that i've spoken about hopefully i might go and have a drink though so i might not be able to see it hardy all right this is uh stuff visible visible enough so welcome to the world of the log gripper yeah the number of knocks i've walked into um where this kind of screen is showing up and there's just lines and lines of text going through every now and again there's a color here and a color there those are the smart ones that have figured out how to sort of focus issues but generally speaking the knock guys sit there and and when the lines start to exhibit a pattern of passing much more quickly than usual then they know something's wrong that's basically the technique which is actually just ridiculous um that's it's not useful information and only exhibits actually one kind of problem the other thing is that logs um the using logs and analyzing logs gives you the capability to spot things that can't really be solved by alerting alerting tell you the known tell you problems with unknown thresholds and known issues that you have whereas logs give you the capability to not only dig into issues and and find um problems that way but they also allow you to spot trends and trends are very key especially when you're talking about and looking at uh environments at scale so let me just uh think about where my next step is oh yes not that one let me try and go over here done right so a couple of things become important um if I step through like the log gripper's world is horrible and um that really should die um and so we introduce tooling like elastic search elastic search allows you to have something that can receive the logs um there are various ways of getting the logs into elastic search which I won't go into um we're taking a pretty simple less moving parts um uh mechanism which is literally just to ship logs with a decent log shipper that can do multi-line stuff at the source because that becomes a big problem and uh we put it in elastic search elastic search then does the indexing and makes the stuff available where you can actually search and look at it and there are wonderful little gooey tools which you can show to your managers because that's awesome and uh they do little things like this present pictures of your cluster and my huge cluster of one elastic search server um is happily behaving itself and that's wonderful but it really comes down to the same kind of thing how do I get my tail in a in a more manageable way and uh for uh for the sake of demonstration because command lines don't do very well in presentations we like to do uh we like to use something like Kibana which is um provides an interface on top of elastic search to do um the gripping and the tailing in uh in a very easy and very easy manner so what you're seeing here is um it's an all-in-one open stack server but it happens to be containerized so every server's in a container so there are a lot of logs going around we're not we don't have every single log on the planner coming through but there are enough logs as you can see what is this 1500 events um for each bar here so that's quite a bit of events coming through but this isn't exactly manageable either so if I stop it and I start to think about okay what am I actually looking for am I looking for something along the lines of um all logs so let's see if I can get this right no that's not gonna work so obviously one of the the common things you might want to take a look at is uh you know our script kitty is trying to get access into my back end I'm actually trying to um figure out what my passwords are and get in so doing this kind of thing is pretty much like a log rebus job it's just got a nice little bit of makeup on it this is not very exciting either and uh as you can see there's not a heck of a lot of metadata going on here and this is all that's like if you only have some sort of log shipping and you only have elastic search on the back end this is kind of the basically all you're doing is dressing up the log rep is job but at least you're doing it at a bit more scale and log stash puts you in a position where you're able to do slightly more funky things essentially you're putting log stash in the position of doing the log rep as job so let's find a more interesting log nope can you try and don't log today let's go back to your long list I'm sure we'll find one so there we go so what you're seeing here is all of a sudden the log has a lot more metadata log level the hcdp version given by the client you're looking at the receive date request time request or ip but more importantly for we're not going to see it here for for some sort of metric something along the line of hcdp time that's the kind of metadata that that might be useful but let me do quest I think it is yeah so in your typical troubleshooting process when it comes to something along the lines of figuring out what's going on in the event you can track it by request ID across many logs and this is where it really becomes useful because now I'm seeing everything every log and every event that happens to relate to your request ID so this is not just the load balancer queries that are checking whether stuff is still live these are actual requests being sent through and if i'm lucky I will find the request ID that also has there we go no it's not it there's another one over here I'm missing it let me skip that but what you'll end up is effectively the log rep's job is being done so your queries become a lot simpler a lot more accessible to people who don't know the complexities of regex and and you have the ability to get access to a different set of logs and a different set of metadata but then you can go one step further and you can do the really cool thing that everybody loves to do in their offices these days and you can have a screen on the wall that shows pretty pictures to the managers when they walk past and the managers love to see things like this which actually is a very very simple depiction of what's going on in an environment but it's also a really good way of being able to spot an unusual trend so if I do something like this and I do a couple of goodies just to create some traffic what you will spot all of a sudden is the response time goes up this is an all-in-one server with eight gigs of ram it's not a very efficient open-side environment but this is the kind of thing that you actually want to see and you're not necessarily going to pick up in your alerting infrastructure you want to see that there's a change in trend you want to see that all of a sudden a particular service is executing a lot more queries than it has before you want to see that perhaps some of your underlying infrastructure is doing something unusual you want to perhaps see that there are a lot of delete events and why is that going on some of the dashboards that I've looked at have built dashboards around you know what's going on with instances at this moment in time other ones have done what what's going on with the storage at this moment in time you know there are different focuses that you can do with little dashboards like this and this is this is information that you really can only derive from logs and this is why you use cabana instead instead of trawling through logs instead of relying on alerts anyway that's me thank you very much I think we have time for some questions um you can reach out to us on isc or twitter our details are up there if if you need to but yeah it's other time for questions or beer can you would you mind using the mic yes so I have one question or maybe one with a follow-up so one of the things that's interesting about logs is you know the information that you get out of them but one of the things that's painful about logs is saying how much information you want so in the sense that I you know often don't really care to run my logs at a high logging level but if I start to hear about problems I want to be able to tune the log file so just as difficult as it is to receive logs from lots of different places it's equally difficult to control logs that are in lots of different places so that I can turn the knob up and say hey give me a little bit more volume on my nova compute log show are you all looking at that particular problem or yeah so I'd say that problem is is handled by the kind of deployment mechanisms that we use so like we can control the verbosity or the the log level of the individual services based on on how we deploy using ansible so it is something that we're looking at I don't actually think the functionality's in yet to control on a per-service basis but it's saying that we would definitely handle with with the ansible deployment tools that we're using but there's we are as a as a team as the RPC specific team we're also looking at logging on the same levels that with the log working group we're taking a look at the problem from many layers so as for for RPC or or OS ansible deployment we're looking at what there is now and we're trying to work with what there is now but parts of our team are also looking looking upstream at the open stack projects and trying to also solve their problem at a higher layer and then as that evolves we obviously can evolve the output of the product as well okay thank you okay I'm not sure to have understood the complete picture so basically you are using tell me if I'm wrong um lock stash as lock shipper to to a central logings thing that after afterward goes to elastic search and have a nice dashboard with keybatter or I use using RCS log to send logs to a central location that is afterwards uh processed by uh lock stash sure so those are actually uh two separate things really so we we have a centralized RCS log server mostly because of one of the points I mentioned was that um when support comes to us they think they know what they want so they ask for a specific thing they don't say we have these issues and can you address them and one of the things they asked for was a centralized RCS log server um so that's reasonably simple to implement so we said sure we can do that for you and we implemented it um it it's not a great solution and we wouldn't advocate using it um but but it's there and then the the shipping to log stash uh the shipping using log stash and then to elastic searches is kind of separate so it doesn't take the logs from the centralized RCS log server um we are actually I think you just set up the Python beaver to use uh log shipping for the logs themselves um so we're using uh Python beaver to do uh kind of multi-line logging um for using the log shipping to do multi-line logging and various other things uh which then since it's a log stash that has uh some of the filters which we were actually able to cut down a bit because of the way we were using the log shipper and then through to elastic search yeah we're finding that uh RCS log doesn't behave well when you're talking large scale um for some reason um queues build up and it's queues build up on the sending side and on the receiving side the the server just can't keep up when you're talking large classes so um within OS sensible deployment we've provided the capability to do your syslog to whatever uh it doesn't have to necessarily be an actual syslog server you could be using Splunk or log stash or whatever you want um but um for for the derivative RPC product we're um we're looking at basically using beaver um not the uh not the uh log stash shipper um and that's because we want to be able to handle multi-line at the source multi-line stack traces um must equal slow logs, rabid logs, all that kind of stuff uh when you rely on the log stash server to do that you can only have one worker and your log stash server dies um there are other ways to do it you could be dropping everything in in readers and having your log stash sit on the side and do the processing um we're trying to not have so many moving parts um and beaver's really simple to to configure and it's working really well um but the way that we're doing it there is each each container, each host in the environment will run the beaver servers it does the multi-line processing in a in a few tags right at the source sends it through via TCP and JSON um to the log stash server and the log stash server then does a bit more analysis and drops it into elastic search so the RSS log thing is a different path so that we have two paths to also store logs okay thank you sure so um in your ELK stack what versions have you tried and what works the best this question one and second is I believe on the on the log stash side you mentioned some of the drawbacks but from the elastic search and the kibana point of view have you experienced the drawbacks um so the drawback on the elastic search and log stash side is really Java um elastic search is reasonably manageable um because there are plugins that expose a lot of the underlying stuff so you really can look at it but log stash is a bit of a black box so when it doesn't work it gets very frustrating um the debug longing is a little over the top um you know you see every event in troubleshooting it becomes a a right to roll pain but um and we've we've done a bit of research into replacing log stash with hacker for instance as an alternative um which can do the same job can can operate the same way but has a little more insight into what it's running for the moment though we're we're sticking with the ELK stack as it is um but yeah just even in install time running a jvm um and having to maintain the jvm and the memory it uses that that took a little bit of doing to manage it but it's it's working okay now and it processes quickly enough the other drawbacks really were around the the main issues we've had so far in full scale is log shipping it's getting the logs to the log stash server the other issue was handling multi-line because you really want those multi-line events that's one event otherwise it's pretty much useless log stash can do it but then you have to implement uh when you implement multi-line codecs within log stash filters you um you have to implement only one worker it can't it can't implement multiple workers so what happens is your log processing grinds to halt you just can't funnel a log through quickly enough so you can work around that by having multiple log stash instances and then push your logs to those instances but it it's a messy way of doing it and we've taken the approach of rather trying to handle that at the source it works a lot better python beaver you'll find that on github there is another one which is a fork from the log stash order called log courier it's also written in go does all the bits that you would ever want it to do and more but it's not as simple an installation you'd have to download it compile it produce packages yourself that is just an overhead that we don't want to handle so i think we're out of time now okay if you want to come ask questions off to us feel free to step up and uh we're hiring i'm obliged to say that


My name is Balakrishna Chamola, I work with TCS as a senior user experience designer. Me and my team provide data visualization design solution to all our stakeholders within TCS. And we take care of all the data visualization activity that is starting from right from understanding of its need, kill the conceptualization part of it. How do you do it? Yes, know it in a form of ingredients and recipe. People come to me with huge list of data and ask me to create visualization that should best represent their data. Some come to me with predefined list of reports and visualization and ask to create a very creative visualization out of it which is already been available, but it needs to be meaningful. So, over a period of time this became routine of mine and one fine day I was sitting and I was thinking about it. So, I felt as if I am a chef or I am a cook and they comes with basket full of ingredients and ask me to cook some delicious or wonderful food out of it. So, that is how I started relating food with data visualization, ingredients with the preparation part of it and the recipe with the method of it. That is how I derived to a conclusion or I arrived to my title like design ingredients and recipe for data visualization. A colleague of mine came to me one day, he was a non-UX guy and he got bored of his monotone stuff. He came to me, he said, hey, Bala, you guys do a great stuff and creative stuff, which I could be a designer, a data visualization designer like you people. I said, why not? You can be a designer. When I looked at him, he was, there was a big question mark on his face. He said, how? How is that possible? How can I be a designer? Because I do not have any background of it. I said, not a problem. Like to create a good food, you need to know its proper ingredients and recipe. Similarly, to create a good data visualization, you know to its relevant ingredients and recipe. He said, wow, is there something available, then please share with me. I wanted to be a designer. I said, fine, let me take you to our kitchen and I showed him the menu that we have for him. So, first you need to understand what is data visualization, why data visualization is needed and why it is important. Second, the ingredients, third the recipe and then I have shown him some of our dishes. So I gave him a brief understanding about data visualization, like things that you get from a textual format data or a tableau format data. You cannot, that, things that you cannot derive it from them. You can derive it from a data visualization. Like from a data visualization, you can uncover and you can identify more thing. You can derive more insight from data visualization, but that is not possible when you see data in a tableau format or in a text format. What data visualization gives you? It's a pattern, trend and correlation of data. When you see pattern, it's talk about the various behavior of data, like where the data have been cluttered together, where you can find some outliers of those data. So those are kind of pattern you can derive from a visualization. Trend, you see a behavior of data over a period of time, how they are performing. Third is the correlation, how the data are related to each other, how they are connected. If they have some relationship, they are connected or if they are disconnected, you will find those data away from it. So these are the key things. There are many things that we can derive from data visualization, but these are the main things that you get it from a data visualization and not from a normal format of data or tableau format of data. So to make this concept clear, I told him about, I gave him an understanding about a traditional chart, basic chart, and about trend visualization. So even basic chart represent data visualization, right? They represent data, how they are performing and all that. But things that trend visualization shows you, it cannot be shown through a traditional chart like, again, in trend visualization, you can interact with data, you can explore the data. Whatever a different individual want to see from the data, they can play with the data and they can explore it and they can drill down to more level so that whatever they inside they want to infer, they can get according to their need. But same thing you cannot do with the traditional chart and graph. Is it fine? Yeah, this is really cool. I wanted to know, now I'm curious to know about the ingredients and the recipe part. Said, let's go ahead. So I took him to the ingredient section. So in ingredient section, what do you need to know? When you get a requirement of data visualization, it's not necessary that you have to, whatever data you get, you start, directly start building up the visualization, it's not like that. You need to analyze the data first, understand whether there is a need of a visualization. If there is a need of a visualization, then you define an objective of this visualization. When you say objective, what comes under objective? In objective, you need to define what is the question that the visualization is going to answer? Who your user would be? Or what are the tasks they are going to perform? Whether they are going to monitor data, they want to see a performance of something or they want to track some information. So those come under the objective. So once you have defined your objective, you evaluate and identify the parameters. So he asked, what is parameter in context of data visualization? I said parameters over here means a list of data points, which club together or set of data points when they are clubbed together and it derives one inside or one part of visualization to it. It's a parameter over here. So if you need to define a parameters if they are not available, but if they are already parameter available, evaluate those parameters. See whether those parameters are meeting the, meeting user's objective or not, or meeting the visualization objectives or not. So once you have list of those parameter, bucket them and map parameters. And so bucketing the parameters which are correlated to each other. You can group them together and put into one single bucket. And if you found multiple bucket and you see that still there are parameter which can have some common connection amongst them. You can map those parameters in a different bucket. In this exercise, you might also see there are some parameters which are not relevant or which are adding noise to your visualization. They are redundant, so get rid of those parameters. But if you feel that at some point of time they might be useful, group it in a separate bucket. So again, it's eliminating cluttering parameters. Then you move on to create narrative part where you create a story out of this parameter that you have identified. It will help you to identify the problem statement that you, based on which you will create a visualization. And then it will also give you an overview to the detail view of the visualization, how it needs to be designed. So he was fine with it. He said, okay, this is fine, I understood this. Now, why don't, so he was more excited to know about the recipe part. So there are three steps in the recipe, like understanding your data measurement scale, knowing the right type of charts and graph. And third is the understanding the graphic variables. So when we say data visualization, the output of data visualization is a graphical representation of data. To a high level, there are many great definitions to it, but just for a quick understanding, it's a graphical representation of data, but it needs to be meaningful. So we have data and it needs to be represented in a visualization manner. So definitely, there should be some visual encoder, which will be related to the data. That's how we'll be representing those data in a visualization manner. And these graphic visual encoders have some properties, they have a property of qualitative and quantitative. So some graphic variables represent qualitative kind of information, and some other graphic variable represents qualitative kind of representation. So to know that, which type of data needs to come under qualitative and quantitative, you need to know its data, what are those behavior of those data, to which category they fit into. So that's why we start from a first step, knowing your data, to which category does they fall. So step one. So there are four type of data measurements scale, nominal, ordinal, interval, and ratio. So it has been also termed as not, that's why you can see an OI and initial letters have been kept as bold. So nominal and ordinal, they define the category of a data. This measurement scale have data which cannot be measured or which cannot be quantified. For an example, a male female, these are two different category, you cannot measure it. So it comes under nominal. Nominal are also called as categorical data. Similarly, ordinal, ordinal talks about the sequence or order of a data of a category, like a rating scale, or a ranking from low, medium to high. If these are the order of a data, that falls under ordinal, whereas interval and ratio, by the name itself, we can figure it out that it's a numerical kind of data. Intervals talk about the time interval of data, whereas ratio is purely about a number, a digit of numbers. So since nominal and ordinal represent the qualitative part of data, a category of data, it comes under qualitative property. Likewise, interval and ratio, since they talk about number, they measure numbers, it's talked about the quantity of a data, it falls under quantitative. So this guy was very happy. Now he want to move on to the next step. He was very curious to know how to design a visualization based on all this concept, but still there were two step to it. So the second step is the selection of chart and graph. So this is a high level view of the purposes of charts and graph. There is an extensive list to it, but I have just kept a summarized version for your quick reference. So there are five purposes. 98% in visualization, you will find these five purposes. It might be relationship, comparison, composition, distribution and trend. And sometime in visualization, you might find combination of all these purposes and at that time you might find only one purposes from a visualization. So for example, for relationship you can show it with the help of a network diagram. For comparison you can show it with a bar graph or column graph for composition pie chart and so on. Then comes the third step which I talked about the visual encoder of data. So when you need to properly decide what type of graphic variable you need to choose for your visualization. So for that you need to understand the basic there. Again for the graphic variable component or visual encoder, there is a huge list of graphic variables but this is again a summarized list which we have used, which I have chosen to be shown for this presentation. So from colors, position, texture are the main graphic variables that are used for this for showing a data visualization. And again for the graphic variable also needs to be mapped with their respective qualitative and quantitative measures. So you see color shape orientations, different colors, different shapes and different orientations. It gives you a view that they are not identical. Each and every element is talking about different things. So that's why it's coming under qualitative and whereas in qualitative quantitative if you see the size of bubbles from smaller to big you can easily relate it that a smaller size means a less amount of data and bigger size means big amount of data. Likewise it goes for other graphic variables in under quantitative. He said, fine, I can see some confident in this guy. Now he said, but I'm not sure how I will be able to cope up in a real life scenario. So come on, let me show you one of our list. So out of multiple use cases, I have taken one use case from bus service planning which is a kind of a smart city kind of project and I have used the simpler visualization to show him so that he can easily understand how we have to apply this ingredients and recipe in a real life project. So you see the insight that was needed from this use case is to measure the number of people, passenger waiting at a bus shelter and the average wait time of a particular bus shelter and the best and best performing bus shelters. So when we extracted that we did ingredient parts what we saw a list of we found a 17 data sets which were related to this visualization but not all the 17 would be showcasing a visualization. So you need to analyze this data and it got classified into three different category. At the bottom you can see these are the shortlisted variables which will represent your actual visualization whereas the top part you can see those are part of filters or some other kind of interaction. So it becomes a part of an interface whereas the insight variables becomes a part of your actual visualization which will represent your data. You have a question? So these were the data that were extracted from the ingredient parts the exercise that we did while analyzing the data and this was the final set of data set we have extracted from the requirement. Yes this data into visualization. So we classified those data not all 17 would be a part of visualization then we classified it. Then as a part of recipe how do we do it? Step one is to map this data with the respective data measurement scale and that's how it's fallen under bucket of qualitative and quantitative so that we can define what type of graphic variable needs to be chosen. Then what we identified that this visualization is needs to be shown trend comparison in relationships that was a part of a requirement. But when we thought of using a traditional charts and graphs we felt that it might add clutter to the visualization or we may not able to show all the insights together. So we thought of going with customized visualization. Then to show customized visualization we need to choose the respective graphic variables with the data that we have categorized in a step one. So here you see the green side, green is the qualitative and blue is the quantitative. We see the data that have been placed under qualitative which will mapped with qualitative and data under qualitative will map with quantitative. That's how you choose how to create a visualization. So this was the output that we came out of it and you can see the green color indicates the good performing bashelter, the red color indicates the worst performing bashelter as the bubble size indicate the size of the people at a bashelter and the arc shape is indicating about which is in a metaphor of time it is indicating the time. Now this guy was confident and I thought of showing him some more dish to build is more confident level very high. So this is one of the visualization that we did where you can see the social network and participatory surveillance app which we have and the sensory devices have contributed towards reducing the manmade disaster over a period of time and another visualization from a passenger boarding which was again related to that project. And third, where you can see a different merchant channel transaction has been happened through different channel like critical, debit card check and their subcategory. This guy was happy and he said, wow, I'm very much thankful to you and now there was a new beginning to his career and he started as a designer. That was just a designer. Thank you. Thank you.


When you discard the narcissist first, narcissists are very arrogant. They have an exaggerated sense of their own abilities and importance. So they are not expecting you to discard them. They're not expecting you to even be capable of leaving them. And it is difficult for a victim to leave a narcissist because even when you've figured them out you end up becoming the cause of your own problems. You have all of the evidence to prove what you've been dealing with but you still find yourself trying to explain and justify their actions.  They will use a detailed vision of the future to bring about bonding and connection in your relationship. They will lie and promise something about your possible future in order to get what they want in the present so that you no longer feel like leaving anymore because now you believe that things are going to get better because they've sold you a dream, they fed you a lie and you want to believe it. You want to believe that this time it's going to be different but when you get to the point where you don't want any more excuses or to hear anything else from them you start making your exit plan which will cause a narcissistic injury once they find out about it. They can't just let you go. They will be very angry. They will stalk and harass you because they never expected it to happen.  They thought you would always do whatever they want. They didn't realize what you were capable of. They didn't think that you were smart enough to figure out their game so they weren't expecting you to leave which is why they took you for granted and valued you too lightly because they thought you would always be there but when you do leave they will see it as though you have abandoned them and they will want revenge but for them to do that they need to find a way to come back because then they can strike while your guard is down so they will lower their dignity and importance. They will control their behavior. They will love bomb you again and they will tell you that they have changed.


Hello. Hi everyone. Hi. Thanks for showing up. This talk is called, what did I call it? Reverse engineering 4G hotspots for fun bugs and net financial loss. I'm a senior work avoider at Pentes Partners. I'm also do hardware or embedded work sometimes. No one cares who the speaker is. Let's go on. So the structure of the talk today looks like this. I'm talking about why I'm talking about cellular routers and I'm going to talk about the sort of attack surface you might see on them. Then I've got two devices to look at. One made by a Chinese company, one made by a US company to sort of avoid getting everyone all over excited about it. Then I'm going to go over what was fun about each one of them and the sort of bugs that I found in them as well. Then I'm probably just going to complain until my time's up. 5G is apparently imminent. I've heard it's meant to be operational in some areas. I'm not entirely convinced by that yet. It's meant to be very quick. Consumers and business users are probably going to start using it more for their daily TCP IP activities.  There's not been that much public scrutiny on non-phone consumer cellular gear so far, which is sort of weird because there's not that many vendors or OEMs actually doing cellular kit from scratch. Because there's only a few OEMs, there's lots of code reuse. So the label on the box doesn't necessarily mean that the contents of the box are actually made by that vendor. And also code on these sort of consumer devices ends up on very similar industrial devices as well. Cellular routers are sort of interesting. They're somewhere between phones and traditional routers. And because they're just embedded computers, there's probably going to be lots of bugs in them as well. I would say at this point the bar is still pretty low for finding bugs in these sorts of things. Not as low as it used to be, but it's still pretty easy. I'm going to do a really quick cellular network refresher just to get everyone on the same page. Some people think cellular networks are safer than public Wi-Fi. It's not necessarily true. Device is connected to an APN, which is like connecting to any LAN that you don't control. Some APNs are really well configured, some aren't. It totally depends on the APN itself. Because you're essentially on a LAN, you're not going to see a lot of cellular connected devices on showdown and things like that. But if you connect to the same APN, you can start to poke around and have a look and see who else is on there. There's a few security measures you might find on public or private APNs like client segregation, web filtering, IMEI, ICCID, filtering. It's sort of like any network. There's certain security mechanisms they can put in place. We have to take that into account when we're thinking about attacking a cellular device because we don't really know where it's going to be used and the APN configuration might be really weak. So when I'm looking at something like this, I have a rough idea of what I'm trying to find. I'm always looking for higher risk issues, actual security bugs. Since we're going in with a pessimistic view of the APN, you have to assume to some extent that you can touch the WAN interface on the device if there's no client segregation on the APN or something like that. So we can't rule out the traditional RCEs as a vector. You can also think about things like if you're much more worried about state-level stuff or very, very serious actors you might want to worry about compromise like GGSNs or NGCNs or whatever they use in 5G, these sort of things that handle IP traffic on these APNs. We also can't rule out cross-site request forgery on the router web configuration interface because if you're connected to a router, you're probably looking at stuff in a browser while you're looking at it. Sometimes there's all other kinds of TCP, UDP services running traditionally DNS, UPNP, other obscure proprietary stuff that the vendor makes. Not all of it's going to get blocked off on the WAN side as well. They try these days, but it's not always the case. There's also the possibility of more traditional cellular services like SMS or MMS being exploitable as well. There's also lower-risk stuff, so the sort of thing that might get you just a shell if you're connected to something with a piece of copper. It might give you some sort of firmware access or something like that. These sorts of things are not necessarily reportable to a vendor, but they're really nice if we're looking for bugs on them. Or if we want to just mess around with some of the lower-level configurations on the modem itself, like trying to change the IMEI to bypass APN security measures, which I think is illegal basically everywhere, so don't do that. You also might want to get a privileged A T shell to lock your device to a particular band or something along those lines. There's sort of a limit to that, in my opinion, if you can permanently backdoor a device in like 10 seconds with access to a USB port, that's probably reportable to the vendor, and you might want to try and get them to fix that. Whether they do or not is who knows. They might not care at all. I also prefer if a shell doesn't survive a reboot, but a lot of these things don't check for the firmware integrity. So when I'm looking at devices like this, I have a sort of really general methodology, which is basically asking what's been done already? What public info is out there already for the same or similar devices? Can I get firmware? What does it look like if I can't get that firmware? Can I get firmware for a really similar device? Can I get a shell and start playing with that? None of that is necessarily in that order, but it's just a rough guidelines for what I'm trying to do when I start reverse engineering these things. So let's have a look at some devices. The first one's the ZT-MF910. I also had a look at the MF65+, and the MF920, which both had really, really similar code running on them as well. I'm going for a deep dive on this one because it's end of life, easy to get, really cheap. So if you're interested in looking at these, it's a really good starter. It runs a Qualcomm MDM SOC, which are really common, the MDM series in modems. ZTE are also a massive OEM slash vendor, so you tend to find there hardware and software running in lots of devices that aren't badged ZTE. There's also a few caveats. There might be, oh days ahead, definitely in the MF910 because it's end of life, so they're not going to fix anything in that. But ZTE also pretty slack following up on the reports that we sent them. So there might be issues in really similar ZTE devices because they share a lot of the same code. We originally thought it was supported because it's advertised on their website, still being sold and nothing on their website says it's not supported. Then we told them what bugs we found and then they told us it wasn't supported. We asked them how we're supposed to know that and they just said, we'll only tell you if you ask, basically. So we asked them to check their other products themselves, they didn't do that, then we found essentially exactly the same issues in a different supported device, disclosed again, then they fixed them, they gave a couple of CVEs, linked us to an advisory which they don't link to on their website. I think they're doing that thing where they're checking issues per device rather than looking at the entire code base itself or the code base genealogy or whatever. So I'm just going to do this whole talk, approach the device in the same way that ZTE did. I'm just going to treat the MF910 like everything that I talk about only affects this device and doesn't affect anything else and whatever. Let's go. So let's have a look at the hardware. It's got this Qualcomm MDM9-225 sock. Some of the newer ones are maybe 9-6-2-5s but there's much of a muchness. They run this ARM Linux core plus a baseband, the baseband's this Qualcomm hexagon microarchitecture. Other people have done a lot more intense work on that. I actually wanted to start looking at that when I started looking at these but I got totally sidetracked by looking at the really easy bugs instead. It also has this like combination RAM NAND chip which you find attached to a lot of these Qualcomm devices. It's got this really nice test pad array. If you're looking at a modem and you see some test pads that are sort of square like this, there's a really high possibility that it was made by ZTE. I've seen it on quite a few devices that have either been made by them or that aren't badged by them. And you can do things like on some of the more locked down ones you can pull these test pins low and it will boot into different modes. Sort of like GPIOs or like the volume button on a phone. There's already a bit of public information about this particular device as well. There's two ways to enable ADB over USB. So you can just get an unauthenticated root shell immediately. It's only over USB so it's not that bad. You enable it through just API calls in the web interface.  It's really well known. I couldn't really figure out who found it already but it's a really well known thing that's out there. It works on loads of other ZTE stuff as well, apparently not just the MF910. There's another public way to do this as well which is mode switch. Just does exactly the same thing. You just send the word factory to the switch command parameter and it'll just start ADB over USB again. So we can get a shell really easily on this. Start pulling firmware off, see how the thing works when it's running, see what binaries are running, see how they're all talking to each other as well. So the MF910 is quite bloated which also means there's quite a lot of interesting stuff about it. The underlying system is probably going to be familiar to people who are used to working with embedded devices. It's just old Linux kernel running on ARM, Android-esque, not really full-fat Android. The default root password you can find in just one of the RCS start-up scripts in a function called set passwords. It just echoes the password to a temporary file, pushes it into a changed password and then tries to delete the temporary file to cover its tracks. This kind of stuff is really common on embedded devices. It's not really revolutionary but it's quite funny when you see it. It's not hugely useful out of the box because we've already got a root shell but there's drop-bare running but you can't get in because there's default IP tables rules. Some of them have telnet running, others don't. It really depends on who badged the firmware I think. There's loads of Qualcomm stock binaries. These QC map ones are Qualcomm mobile access point binaries and there's a few DIAG binaries as well for anyone who's interested in that. There's also GDB server, just stock on it. The IP tables rules just default accept. They block TCP and UDP on everything. It's TCP and UDP 22 on everything and then block TCP 80, UDP and TCP 53 and 1900 on the one interface. That doesn't actually block everything that's actually running on it. There's still a few services exposed like Syslog D, there's that DNS mask, DHCP port and there's a custom binary called ZTE whisper which is some sort of proprietary whisper protocol service on UDP 4500. I had a really quick look at that. It's definitely using lots of unsafe C functions but I'm not going to talk about that one today. I'm going to talk about this ZTE top SW go ahead binary which is really fun to reverse. If you're not familiar with embedded web servers, the whole web server stack tends to be just custom binaries so they'll be based on some open source web server and then they'll just attach stuff to it. This one's probably based off a web server called go ahead which is just a common commercial web server. It's not directly accessible from the WAN side so you can't do any of this stuff directly over the APN but you could still attack it by cross out request forgery if someone went to a malicious page. So we're going to have a look at it with that in mind. The web APR logic is pretty standard for a router because there's a large configuration table that holds all the configuration data for this router stored in non-volatile memory somewhere. All the configuration is written to or read from that by this binary so the web server is just a broker for reading and writing that information. There's a couple of API endpoints. Goform get command process and goform set command process. Goform get is just for reading data and goform set is just for writing data to the NV RAM. Goform get takes a CMD parameter and then an optional multidata parameter. Goform set takes this goform ID parameter and then custom parameters based on what that goform ID is. So without information you can load the binary up in either or Gidger or whatever. I did it in either originally but I think Gidger is a bit easier to see in really, really quick time. There's a lot more endpoint handlers in this binary than you actually see when you're using the device normally. Not all of them are that interesting but it just means there's a lot more attack surface on it, a lot more stuff to play with. In goform set command process each goform ID points to a different function in a big look up table in the binary. So most of these functions are just wrappers around a pretty safe ZTE library function that just reads and writes data to the NV RAM and some of them are explicitly whitelisted as well so they can be used before authentication. Not all of the functions do this NV RAM reading and writing. They also tend to run other sort of sys commands. There's one more ADB mode that I found that you can enable preauthentication. So goform ID takes this set device mode parameter and just takes a debug enable, sorry, goform ID takes a set device mode as its value and takes a debug enable parameter. Checks if it's 0, 1 or 2 and then just echoes it straight to that Android USB debug enable file in sys which just brings up ADB again over USB. It's no command injection because they're actually checking the input so it's not really nasty risk but it's a nice new way to get ADB access over USB on a device you don't have the password for. As far as I know this hasn't been public before but it's quite a nice one. So there's three different ways to do the exact same thing in one web server binary. Something else that's really nice is there's lots of verbose logging so it's got this ZTE syslog append library function so you know exactly where you are in every function in the code. There's also a way to leverage this from the web server so one of those undocumented endpoints lets you set up remote syslogging. So you send a request to this endpoint and syslog data gets sent back to you on UDP 514 of the requesting IP address. It's only available post authentication but it's persistent across reboots. The function itself does a system call but there's no command injection in this particular one. I think there was command injection in it in the MF920 or the 65 plus I can't remember but they're very very similar functions just with slightly different bits of checking added. So if you open up UDP 514 you get all of this ZTE syslog data sent back to you which is useful if you're looking at a particular ZTE binary and trying to hit particular parts of the code in it. You can dump logs to a file as well download them, enable kernel logging as well. There's also a bug that I can't actually properly test and again because anything that doesn't directly affect the MF910 ZTE aren't going to do anything about so I didn't report this to them. There's all these pre-authentication HTTP share functions. They don't work by default on the MF910 because there's no SD card slot but with a bit of basic reverse engineering you can see there might be a few problems in some of the code. So in this particular one you get a 1024 length buffer, you just sprint a touch command with the user supplied variable in double quotes and just pass it straight to system. There might be a couple of problems in there. I'll leave that as an exercise for the reader but I can't really confirm this because I don't have a ZTE modem with an SD card slot but theoretically if I did the exploit would look something like this. That sort of qualifies as fun because it's not technically exploitable on the MF910 but we'll never know, we'll never know. So there's actual technically exploitable bugs. This first one is a leaky API endpoint, the GoForm get command process endpoint. There's two checks, one for authentication and one for cross-site request forgery but there's no proper authentication check if the multi-data parameter is set to zero for some reason. So we can just read arbitrary values from the configuration table as long as you can pass the C-surf check. So that includes the unencrypted admin password value. So if you set the refer to the root-ride P or 127.1 you can read things just like the admin password backwards without a C-surf bypass you have to be on the same LAN. There's also obviously command injection with code like this. There's that USB mode switch function from earlier. The USB mode parameter is injectable so it just passes whatever it gets straight to system. Those two bugs are not quite a proper exploit because we can't really, and they're not really a remote exploit because we can't really access the web server over the LAN. So to turn it into another kind of exploit we have to take some of the mitigations into account. There's this cross-site request forgery protection so it just checks the refer aheader. You can't control the refer aheader in a browser so any request that we send to another page you're going to fail even if we try to log in we're just going to fail if we're coming from another page context. I think you might be able to guess how to bypass this protection. Cross-site scripting means we can just run JavaScript in the root of page context so the refer aheader is going to be the root-ride P address. There's a really nice clean XSS in this totally inexplicable page, go form slash form test, which by default just prints name Joe Smith address 1212 Milky Way Avenue.  So we can have an exploit that just goes like this. XSS gives us a clean refer aheader and we don't have to worry about the same origin policy so we can read the responses back as well and we can just leak the admin password, log in, exploit the command injection. The C-serve to XSS is just a form posted into an iframe because if you try to use XHR it will send this options pre-flight request and the web server just can't handle an options request so it will just fail. And the injected JavaScript just sets up a couple of functions, reads the admin password, logs in, executes arbitrary code. So let's have a look at it. So on the right there is the Windows machine just connected to the router and on the left is a sort of attacker machine that's going to open up a shell. Okay, case study number two, a high end route at this time, the Netgear Nighthawk M1. I wanted to see what an expensive 4G router looked like because the MF910 is pretty old and pretty cheap. The M1 is new, new-ish. It's been out maybe a couple of years. There's barely any public information out there about it. It's using a new Qualcomm sock so there's almost no public information about that. So it's a sort of challenge. There's also a really cool Russian subculture of just pushing them way too hard by putting heat sinks and massive antennas on them. It's also part of the Netgear bug bounty which they run on their high end routers. I absolutely hate the Netgear bug bounty so much.  If you do they'll be so-called entitled to some kind of performance instead of giving money back. The payout calculation is also really, really complicated. They seem to calculate the payout amount based on CVSS which is arguably not the best way to calculate risk in a piece of hardware. I had looked back through some of their recent disclosures. One of them gave a post-authentication command injection, this CVSS v3 score of 6.8 which is conveniently a medium risk. So a $300 payout rather than a high risk $600 payout. So they're also cheap skates. So I did manage to find command injection in this thing so I very briefly considered my options. Should I go through bug crowd, never talk about it in public and maybe get $300 from Netgear or speak at DEFCON, complain about Netgear in public and get $300 anyway. So yeah, I'm here. The hardware is really, really similar to the MF910. It's got a Qualcomm MDM 9250 which is pretty new. You can't really find much information about it. It's got this Nanya combination and RAM chip as well. These are really generic looking test pads. I'm sorry for all those scrapes and scratches all over them because I tapped onto them and tried to clean them up. So the main core components are pretty similar to the MF910 but a little bit more up to date. There's basically no useful information out here at all for this one. There's a really long thread on 4PDA which is this Russian phone modem enthusiast website. They all want to lock it to particular bands for some reason. No one seems to be able to do much with it. The thread was really helpful because they archived some older firmware update packages and documented some arbitrary firmware update process that I'll talk about in a minute. So thanks to them for that. There's a GPL tab all on the Netgear site that seems to be about 14 versions out of date. So there's not much else out there. This firmware on the Netgear site, but every file looks encrypted. So it's not trivial to get binaries off it and look for bugs. There really wasn't anything that obvious. There's no ADB, no telnet, no known backdoor. It's nowhere near as easy as the MF910 to get into. There's a web server on port 80 and then an AT shell on port 5510 but only if you're plugged in over USB. I could figure out it's based on Sierra Wireless technology. Sierra Wireless is just a big Canadian vendor who make really expensive industrial cellular stuff as well like modem modules. So knowing that gives us a little bit of a wider scope to look for more information outside of just the Netgear specific searches. So since the M1 was a bit more difficult to get into initially, the fun stuff was sort of figuring things out black box entirely. These test pads, I started off thinking they'd be GPIOs like the ZT stuff or on phones. I spent time pulling them up and down on boot and seeing what the USB vid-pid combos were coming up on D-message, but I put it into Qualcomm emergency boot mode a few times just because I think I went too hard on it. A little tip, I think if you treat these things badly enough, they just give up and go into emergency boot mode. It's not that helpful because we don't have any signed public load of stubs for the 9250. It turns out they're JTAG, so thanks JTAG and NUM. I really wasn't expecting them to be JTAG. It's quite rare to see something like this. That's the pin out there for anyone who's interested. You just connect as a generic cortex. I think it's A7 or something like that. But in the end I didn't really spend that much time on it. JTAG on a really complex sock with basically no data sheet is not really like a quick win to get into something. I think the memory protection unit was also stopping a lot of stuff. I figured out a few things like the boot loader starts at zero and ARM thumb gives you some insight into potentially interesting memory locations or register locations. But you can't really read those locations from JTAG because the NPU is blocking most memory reads. But it's there for anyone who's interested in JTAG on Qualcomm stuff. That AT shell as well is limited. You can see extremely small amounts of modem crash data and things like that. There wasn't this usual privileged command mode that you sometimes see on AT shells. I think the password is nulled by default, so you can't get into it even with the right password. There's nothing like this like AT bang enable or anything that you see on some other devices that would just pop open ADB for you. The only thing that came close to being useful was this AT bang boot hold. That just put it into this Qualcomm flashing boot mode. From this mode you can use this tool called FDT.exe, which is a Sierra Wireless proprietary binary. It's not that easy to track down. You can find it packed with other Sierra Wireless firmware update packages. You just sort of open them up in seven zip and the files in there. If you're used to the normal Qualcomm download mode, this one's sort of proprietary. It doesn't open up a comport. It just seems to do it all over USB. Maybe it's just some sort of Sierra Wireless specific implementation. This was a really nice tip from that full PDA.ru thread. It's still not that helpful unless we can manipulate firmware update files because they're encrypted. I used some encryption loosely. Netgear chose everyone's favorite encryption. Actually, depending on your definition of fun, this might not sound like that much fun, it's not just XOR. It's XOR and AES. The firmware follows this really generic Sierra wireless structure. The firmware is split into chunks and each chunk has a header and a body. Netgear added this encryption layer where all the chunk headers are AES encrypted and the actual data, the chunk body, is just XOR encrypted with a 1024 byte key. The AES sections also just encrypted in ECB mode as well. They're just repeating 16 byte blocks. They're really easy to spot. The general theory of it, you can derive the XOR key for these chunk data segments just by assuming that most firmware bytes are just going to be nulls. Then you can just XOR out all the data segments, get all the data, get the AES key out and just decrypt everything. I'll go over the steps so you can just sort of see really quickly. This is the nulls encrypted with AES and ECB modes. Each block is just encrypted on its own 16 bytes. It looks like XOR with a 16 byte key, but it's definitely not. When you line up the encrypted file with a generic non-encrypted Sierra wireless firmware update file, you can see the repeating blocks of nulls line up really nicely with these so-called ECB encrypted null blocks at the same offsets. The headers reshunker all the same size. You can figure out where the headers are just by looking for this known encrypted null chunk block. You can figure out where the headers end and begin just by searching for that 16 bytes. It's pretty easy to figure out the file structure and just decrypt everything. I wrote up a script to do it. You can read about it here. It's a bit too fiddly to get in too deeply right now, anyway. The script itself will just iterate over the file and just decrypt everything. Some of the chunks are nested in other chunks. The key shifts very slightly if the chunk data modulo 32 isn't zero, so it's all a bit convoluted and weird. Anyway, the firmware is pretty interesting once you've decrypted it. It's got literally everything that's running on the device in there, including the boot loader, the DSP, the Linux system, the web root, the Android splash screen, and some generic APN configuration data as well. I didn't take this to its logical conclusion. It might be possible to fiddle around with this firmware, re-upload it back and run arbitrary code that way to get in. I didn't get around to doing that in too much depth for reasons you'll see in a minute. There's other Sierra firmware file passing tools out there, so you can figure out where the checksums are in the headers and all whether size fields are in the headers and stuff like that. Anyway, Neger don't think this is a problem, so they're not going to fix it, so have fun. Okay, I found some bugs before I decrypted the firmware. These were just black box finding the bugs. If you decrypt the firmware and you might find more things. As is customary, there's command injection. I sound bored, don't I? I'm so bored of command injection. It works like this.  You send a post request with a parameter like webd.adminpassword equals password one, and it'll just change the admin password if you're logged in. The session security is quite good on the web application, so it's not that easy to bypass. I thought maybe someone else will find differently. Request look a bit like this with the sort of important parameters highlighted. I did a bit of basic fuzzing with stuff that I found when I was actually just going over the web interface totally black box, but I didn't really find that much. But there is an interesting web page at introspection.html, which is a Sierra Wireless specific page, which gives you a really long list of parameters when you're logged in, and it'll give you their expected value type as well, like Boolean or Integer or String, and sometimes the values that they're set in there as well, along with the permissions, the user permissions that you need to modify or read them. It's pretty helpful because it's just giving us a word list of valid parameters that we can try to fuzz, which we can use to find this command injection in ready.devicegear.removeUSB device.  Just log in with the default credentials. They're root OELinux123, and you get a nice clean root shell on it. This is my absolute favourite cross-site request forgery bypass of all time. So unlike the ZTE, there's no dynamic API to request data from. You read all the configuration and session data from these dynamically generated JSON files. That includes also the CSERF token. They're not JSONP, so you can't just siphon them off because the SOP will stop you. But there is another file called netgearstrings.js, which is also dynamically generated. And just after the to-do comment at the top is a big variable called netgearload data, which holds the user privilege level of the current session and the CSERF token. So you can just load this JavaScript file from any site, read the netgearload data.session.sec token, and you've just got the CSERF token. So it's the dumbest CSERF bypass I've ever found. So theoretically you can chain these in a really similar way to the MF910. I didn't find an authentication bypass, so it's not... I don't have such a clean exploit. You'd have to try and guess a weak password. After just so much hassle with it, I was just quite happy to get a shell on it, to be honest. If you just want a shell to get... So you can start trying to change stuff on the low-level configuration, you can flash any compatible firmware with that fdt.exe file, just grab a vulnerable version and get a shell by command injection. Netgear have been pretty bad to disclose to. Maybe because I avoided bug crowd, they were having a little tantrum. I reported the issues in late February. As of this morning, they haven't fixed anything, so all of that stuff was just O days back there. They're not going to fix some stuff, other stuff they're looking into. They might change the root password. Literally, who cares? Okay. It's the end. These sorts of devices are really fun. They're really easy ways to get into hacking, embedded stuff. You should go into it with a mindset that vendors are going to be really rubbish. The bounties are going to be rubbish, but you can start to at least learn how these things work on quite a low level, which is quite interesting. I've got a massive to-do list of things, because I started the whole project because I wanted to find bugs in the Qualcomm baseband, but I got obviously way too distracted by all of this stuff. I want to do things like download all the extremely dodgy looking unofficial Qualcomm tools that people have released out there, figure out how they work and what they do. There's so much stuff out there, so much work that's been put into reverse engineering and changing these things. There's just a goldmine of existing research. Vendors should be asking things like, where's my code from? What else is running? What's the root cause? How do I fix it? They don't really seem to do that. They just respond reactively to bugs that you send them and don't think about it anymore. I tried to coerced to check for similar issues and they just didn't do it. I think one of the excuses was I sent them a proof of concept that used Netcat and they hadn't installed Netcat on another vulnerable device. Despite the fact that there was still command injection in there, Netcat wasn't running, so they just gave up and thought it wasn't affected. Netgear running is slightly modified Sierra wireless stuff. I'm really not sure how much Netgear actually do on this or whether they just outsource to a third party. It's been six months since I reported to them, almost six months, and they've done absolutely nothing. I have no idea how in control they are of their own device. In all of this stuff, there's loads of rebranding, shared code, especially in some of these lower level Qualcomm libraries as well. These issues might affect other vendors or similar devices using a similar development stack. I can't buy everything and check it myself and I'm bored of a lot of this by now. That's it. Thanks to everyone who's up here. Have a look at the blog post up here. I've written up some of this stuff and another issue that I found in a TP link, pre-authentication command injection in a TP link as well, and just really basic stuff on how to use Ghidra to find stuff like that. Hustle me on Twitter if you want at no thanks with all underscores underneath everything. Thanks very much.


Welcome to Data Doctor's video, Suggestion 822, Heading Doesn't End with an Equal Sign. Causes for Suggestion 822 are usually typos. When a heading doesn't end with the proper equal signs, the profile's public view will show the code. The intended heading is read as regular text, and is included with the same. This is the Suggestion report for Josephus Bucklets showing only one suggestion for this profile, number 822. The Info column states that the issue is with the biography heading. Here is the public view of Josephus Bucklet's profile. As you can see in the biography section, there is code showing before the word biography, which is also on the same line as other text. Below that is the content index, which shows four headings, name, notes, sources, and acknowledgments. If the biography heading was correct, it would be included as a link, but it is not. That section would also be listed below the content index. Click on Edit. In the biography section, you can see the heading name has three equal signs on either side, indicating that it was meant as a subheading to the biography heading. To resolve this, add two equal signs after the word biography. Click on Preview to check your work. The content index now lists the biography heading with the subheading name. Also, biography is properly showing as a heading with its information below the index. Go to the middle of the profile page and explain your changes. This is strongly recommended for all Wikitree members and is required for data doctors. For me, I identify myself as a data doctor. Copy the suggestion number and name and explain what I changed. Save your work. Return to the suggestions report and on the far right, click on Status. On the next screen, the suggestions status report, make sure your Wikitree ID is showing. Of the five listed, choose a status, in this case, corrected. Leave a comment from the comment tense or make one of your own. Click the Update Status button. On the last screen, the status history shows that your changes have been recorded. And you're done.


The verses which are supposed to refer to a specifically threefold deity are particularly weakest proofs aimed at someone who is not already a Christian. Usually the argument is simply based on a fact that terms for God appear three times in the verse. And this is just not a persuasive reason for accepting as strange and difficult a doctrine as the Trinity. Now, needless to say, there are many more verses where terms for God appear only twice. A few verses where they appear more than three times and an overwhelming majority where such a term appears just once. The most commonly cited verse of this sort is a very well-known one. Here, O Israel, the Lord our God, the Lord is one. This is from Deuteronomy 6 verse 4. Now, the point is supposed to be that the threefold deity, the Lord our God, the Lord, is really one. In fact, however, this is a mistranslation. Since Hebrew omits the present tense of the verb to be, even the word is in the Lord is one, does not appear in the Hebrew. So the correct translation is, here O Israel, the Lord is our God, the Lord is one. Here the Trinitarian interpretation can't even get started. But the main point is that this type of evidence, from threefold repetition, is significant only for someone who already believes. The verses said to demonstrate plurality within God that deserves serious attention are supposed to demonstrate such plurality without pointing to a specific number. The passage quoted most frequently to prove this is Genesis chapter 1 verse 26, where God, speaking at the climactic moment in the process of creation, says, let us make man in our image. Now, the Bible almost always uses singular verbs in referring to God. And the plural here really does require explanation. But almost any explanation is superior to the view that the Bible has chosen this verse to teach us the doctrine of the Trinity. The verb may be plural of majesty such as, you know, the royal we, or a plural of self-exertation as in in the English, let's go, you know, even when someone is talking to himself, or an indication that God was consulting the angels at this critical point of creation. Some Jews have even suggested that God was including the earth in the task of creating man, since it would supply the body while he would supply the soul. It's worth noting that the earth and water were a command to bring forth other living things. Every one of these interpretations explains very naturally why the plural verb is used here and not earlier or later in the chapter. Only the Trinitarian interpretation fails to do this. Another verse often cited to demonstrate plurality within God is Psalm 110 verse 1. The Lord said unto my Lord, Sit thou at my right hand until I make thine enemies my footstool. Now the Christian view would have us hold that the second Lord should also be capitalized and that one divine being, that is the Father, was speaking to the other divine being, the Son. But it's far more likely, however, that the author of the Psalm is writing a dialogue between God and the author's Lord, i.e. King David, the King. Thus the Lord said to the author's Lord, the King, Sit at my right hand, in other words, under my protection, to David is the heading of the Psalm, which means dedicated to David and not by David. The verse then refers to one divine Lord and one human Lord and has absolutely no reference to plurality within God.


When you try to connect to internet on your laptop but your usual Wi-Fi network is not showing up, it cannot be frustrating. This problem could arise due to Wi-Fi problem incorrect Wi-Fi configuration on missing or outdated drivers. Let's not worry because I will give you some short short solutions for this. 1. Enable Wi-Fi Service The problem happens most likely because your Windows disables the Wi-Fi service. Check if your Wi-Fi symbol is showing or just coming with a cross mark. Next, right click the internet icon and click open network and sharing center. Click change adapter settings. Right click Wi-Fi and click enable. Restart your Windows and you're ready to catch the Wi-Fi again. 2. Turn on WLAN Autoconfig Service WLAN Autoconfig Service can configure wireless security and connectivity settings. For this, press Windows logo key plus R at the same time and type here services.msc. Then press enter. Right click WLAN Autoconfig and click properties. Select automatic in startup type then click apply and then ok. Restart your PC and reconnect to your Wi-Fi network to see if your Wi-Fi is now showing up. Next is to update Wi-Fi network driver. Wireless network driver allows your operating system to communicate with your wireless and network adapters. If your wireless network is not showing up on your laptop it is probably due to the missing, outdated or corrupt drivers. Now you can update the driver manually. However, we recommend you to use advanced driver updater, download it on your PC, let it scan the system and choose to update the Wi-Fi driver automatically. Easy process. Next is to restart your modem and Wi-Fi router. This problem can be probably caused by the internet service provider issue and restarting your modem and wireless router can help you reconnect to your ISP. So just unplug your wireless router and modem from the power source. Wait for at least 30 seconds and plug your wireless router and modem back into power source again. Reconnect with the Wi-Fi network once again. Next check the device interference. The Wi-Fi signals can be interfered by many household devices believe it or not such as cordless, phone, microwave, oven, bluetooth speaker etc. because they use the same frequencies. So either bring the modem and router closer to your laptop or switch off the cordless phone, microwave, oven, bluetooth speaker etc. So that's it for today guys, we hope that you could have finally connected your Wi-Fi once again and if yes, don't forget to mention it in the comment section below and don't forget to subscribe to our channel as well.


This study examined the effects of soybean meal on the intestines of rainbow trout. It found that the intestines of fish-fed soybean meal had increased permeability which could be measured by the presence of sugar molecules in their bloodstreams and the amount of bacteria present in their blood. Additionally, it was observed that the more soybean meal the fish consumed, the greater the amount of bacteria in their blood. This suggests that soybean meal may cause damage to the intestinal lining, allowing for the passage of larger molecules into the bloodstream. This article was authored by Payment Mosberry in Tanja, Margarith Verland, Thor Landsverk, and others.