Ie, roi'r bod ni'n gwneud o'r ddylch gwaith. Rwy'n gweithio'n gwneud o'r bwysig, a'i ddweud o'r rhaid i'r ffordd. Dwi'n fawr yn cofnodd i mi. Rydyn ni'n ddweud o dweud o rhai a ddweud i gael cyfnodd gyfwngolengol ac yw unrhyw unrhyw o'r cyfwngol o'r ddweud i gael cyfwngol o'r cyfwngol. Yn o'r ddweud o ddweud o rhai cyfwngol o'r cyfwngol o'r cyfwngol o'r cyfwngol. ac mae gennym yn ymlaen cael ffordd yn Ymlaen, i ddweud o'r wych yn digonol a ymlaen o ymlaen ymlaen ymlaen, yw ymlaen mewn cymfer yn cyntaf mewn? Ond yn ddiddordeb am ystod, mae'r ddiddordeb yn ddiddordeb. Maen nhw'n dod i'r cyfrifiadau yng Nghymru, mae'r ddiddordeb yn un hwnnw. Mae rydw i'r ddiddordeb yn fawr, yn roi'r gynllun o'r cyfrifiadau yn ddiddordeb, mae'r ddiddordeb yn yn niferfynion. Oherwydd mae'r ddiddordeb yn ymdigiddordeb digital designer doing embedded software and digital design but what I found myself doing is spending most of my time installing the CAD tools, debugging embedded builds, reading other people's scripts and so I left that area and founded Alexis. Alexis is a company that specialises in tracing Linux applications and profiling file IO in order to solve installation problems, build problems, dependency issues and so on. The main thing is we work out why your application doesn't work where it should, i.e. works fine in your environment since you deploy somewhere else something goes wrong and we try and take the guess work out of working out why it's gone wrong. What am I going to talk about? I'm going to talk about profiling and debugging technologies so I'm going to mention a few tools but I'm going to focus more on the technology underpinning them. I'm going to include a bit about our tools and how they work as well and I'm going to talk about gaps in the tool ecosystem so what is missing, what do we need and hopefully by the end we'll have a bit of an idea about where we need to go in the future. So this is what our tool looks like. We pull out tool hierarchies, a program called hierarchies. What you're seeing there is actually a trace of Firefox. The root node is always shown in green. We show scripts in blue and programs in orange so what you're seeing there is that Firefox is actually an alias for a script. It calls two other scripts and a bunch of programs in the system.  So this was originally designed to help solve problems with scripted flows particularly in the semiconductor industry but really anywhere where you find yourself reading scripts and executing them in your head to find out what's going on. But it's been extended over the years to look at more system level problems such as library dependency issues and so on. This graph really illustrates what I mean by application level tracing so we're not looking at debugging any process in particular. We don't look at the algorithmic details. There are many, many debuggers that can help you solve algorithmic problems out there. It's more about when we pull the whole application together whether that's the installation process, the startup procedure and all the tools around it. That's what I mean by application level debugging. The way breeze is normally used is in comparing two applications. You would make a trace of a tool on a system that works, a trace of a tool or a scripted flow on a system that doesn't work and you get to compare the two. That could be working out what's different about a particular machine, what's different about a particular user. It's so common that you put together a scripted flow handed to someone else and it doesn't work. You need to work out why it doesn't work, whether that's a missing dependency or whether they're just using it in a different way. Are the arguments are passing in the same? Do they have some user settings that interfere with the problem? There are lots of tools out there that can help you make sure that your Linux installation is the same every time, same packages installed, same libraries and so on, but nothing can really prepare you for changes such as a user misspelling an environment variable. We help you take the guess work out of working out what's wrong by looking at the application instead of looking at individual programs or just the system. In 2012 we also added file.io profiling for remote file systems. That means that you can not just work out what your dependencies are, but you can also work out how often different areas of the file system are being accessed. This graph here is a build with multiple targets and you can see each orange spike is accessing an area of the home directory. Each green line is accessing areas of slash temp so you can very clearly see the build cycles for each target in the system. In this way you get a very holistic overview of what's going on. I've mostly tracked slash temp and slash home in this graph but you could set up a breeze to look at your database installation, your tool installation and track what it's doing in those ways. This really highlights where we are going as a company and where we see the industry going in that increasingly we're being asked to match up physical resource use with application status. That's a theme that we see time and time again and that's really what this talk is about. It's about making sure that if your file system is being overutilised, if you're seeing spikes in the network then trying to get back and work out which application is actually causing that particular behaviour. It could be something as simple as a global pattern matcher in a script can actually cause applications to trawl the entire file system looking for matches to that regular expression and it's one of those things it's very easy to do, it's very difficult to actually look at the performance of your file system and work your way back to which script, which line of which script is actually causing that to happen. I've talked a bit about our tool breeze but I want to talk about what else is available and how does it work. We've had instruction level debuggers for years, some of them are really excellent, many of you will be familiar with GDB. GDB is actually quite funky in that when you set a breakpoint it actually swaps in a special breakpoint instruction into the code so that when in hardware when you hit that breakpoint instruction everything halts and GDB is alerted and you can take it from there. That's an example of hardware and software code design. The software wouldn't work without the hardware support but for that reason it works very well indeed, you get very good performance but it's still not really any good if you're trying to work out what's going on at the system level. Once you've got it working and sent it to your customer if it suddenly slows down at one particular point GDB is not really going to help you, you can't really go into the customer site and debug it on site, you probably don't have any debug symbols in there anyway. Valgrind is another tool which is extremely powerful for doing instruction level debugging. For those of you who don't know Valgrind is a tool although I'm reliably informed it's pronounced Valgrind actually. It's a tool for spotting memory leaks in your application so in in C that's as simple as making sure that you've got a free for every malloc but it can do a lot more powerful things than that and Valgrind actually works by rewriting your binary and rewriting the allocation and free statements so that they can actually track all your memory use throughout the application. It's a really phenomenal tool give it a go if you've never used it before but again it's more at the application development stage when you're up to elbows in code it's not really about working out what goes wrong once you've actually got your application running in the in the wild. So there are various challenges as with any tools as well it's not only is it difficult to deploy this in the field but it can be difficult to make sure that the debugger doesn't change the application as well. I'm sure most of you have been trying to debug a race condition which disappears as soon as you attach debugger to it any kind of change even with the most minimal overhead can potentially cause a problem and I've just talked about some some very old well-loved instruction level debuggers but there are a few new ones on the market I'm not going to list them all but just a couple here for example undo software take snapshots through the running of your application so you can in fact run it backwards no you no longer have to sort of hit a breakpoint and and try and guess what scenario led up to the state you can actually use their tool to step back in your c or c++ debugger similarly a linear can debug parallel code across multiple machines so if you have an embarrassingly parallel problem with many many processes running your algorithms then you can optimise your functions using using their tools so the the tools available have really come on they're innovating all the time in this industry and there are there are dozens out there to choose from but it's still it's still not about debugging the system it's still not going to tell you why your database is a bottleneck one um or or why your your network that you so carefully carefully planned out isn't isn't up to scratch um a lot of these tools focus on embedded but these days when my phone is about as powerful as the computer I took to university um but what has embedded me in any way so uh many of you will also be familiar with a tool called ptrace uh ptrace is built into the linux kernel um and gives you access to to system system level library level calls as the application is running um l trace and s trace are both built using using the ptrace api and they they show system calls and library calls respectively so here I just want to have a small pause and clarify exactly what I mean by system call and library call um the the terms are often often used interchangeably with good reason because a lot of the library calls have the same name as the system calls are mapped directly but many of them don't when your application runs irrespective of what it is there's a very good chance it links into the glibc library in order to access to the next kernel so that will be for everything from opening a file to connecting to a network to spawning a a child process everything has to go um to into a into the kernel via system call and most of the time that's made easier for you um using the glibc it's um much easier to call um some of the some of the high level constructs rather than work out um some of the low level uh mechanics of of the i o but they're still in there for for you if you want to want to use them and l trace uh gives you library calls s trace unsprizingly gives you system calls both these tools dump out a huge amount of information they're often used to solve system level application level problems but it's it's really like sipping from from a fire hydrant when you when you try and um sort out what's going on with these applications and they don't cope very well with very very large applications so it's very very quickly you end up with gigabytes and gigabytes of data trying to sort through and find out which process you want to look at and uh which call is interesting can be very time consuming so now i'm going to talk about another technique um breeze our tool and others like it use ld preload instead of p trace um this is just for a show of hands um how many of you are familiar with ld preload techniques excellent excellent so ld preload is a technique that allows you to override library calls it's most commonly used to override um glibc or to replace a system library but you can actually preload any any library you want and override any any library in the tool so if the if your application has got proprietary application specific libraries you can even override those if you know the if you know the um function signatures and the way it works it's very simple you set the ld preload environment variable that's why it's always in capitals it's not that we are so excited about it we need to shout um the then once the the uh linker sees sees your library in the environment it will actually preload that ahead of any other system library so that your application is linked against and by preload i mean exactly that i mean any calls that the application makes go to the ld preloaded library before going on to any other system library and it's really up to you whether you in fact call into the glibc library um or return something else uh so this is this provides a very very powerful interface um and it allows you to manipulate the application so unlike using ptrace or strace it doesn't just give you data it allows you to handle all kinds of scenarios that uh strace won't won't cope with because uh you can't inherently change the application so this is what um a preload library signature might look like uh you simply match the applet the function signature um so here i'm overriding the read function in glibc and the first two lines might look a little bit like gary pokery but all we're doing in the first line is defining a function pointer and using this special function dl sim to get the next read function in the in the chain so in my illustration here we only have um uh the breeze preload preload library and the glibc but you can chain together a pretty much as many as many applications as you like i'm not aware um that there's any limit if i'm sure i'm sure some of you can find it um so all you do is ask for the next read function um do whatever it is you want to do in your wrapper so here we're just printing information that this application has read um and then finally we return uh we call next read um which would be the real read function as it were so here we're not changing the behaviour of the application materially we're just adding a print statement so similar techniques can actually do more powerful things such as swap file systems or suspend and resume a job so here we have um another uh another library wrapper um this time i'm wrapping the open call and what i'm doing is uh replacing uh prefixing all the path names with uh slash alt meaning alternate file system so in that way whenever your application tries to um access a file in the home directory it will instead be diverted to a file in slash alt uh slash home for example um so this is not totally unobservable from the point of view of the application because if it reads slash proc to find out if it's open the right file it will find it's it's actually open something else but not many applications actually do this uh so most of the time you can very very safely just swap um one file for another and the application will continue to run provided it sees the data that it's expecting so here we just return next open with the alternate um path name and so we can you can use this to to hot swap a different file system in in at runtime and uh many of you may be familiar with the yokto project uh the yokto project actually uses a tool called sudo um in order to do just this so in this call graph you can see that um in the in the yokto build they use a build system called bit bake uh which then makes a call to sudo which is just a wrapper strip that sets the ld preload library so this is used to fake root access um the during the build the application thinks it has root access to the file system in fact um the file system is swapped out for a database so here you can see that whenever the uh whenever the application tries to access a file it in that open or read write call is in fact redirected to the database and the data it's expecting is returned so that means that you can actually do a yokto build without being root and now this all sounds sounds very simple i've touched on a few things that can go wrong um but in fact it's a minefield of things that can go wrong um newer applications actually are much easier to trace than older applications but we have a legacy of older applications so i don't think there are very many applications that don't use for example bash or make um uh in order to in order to indeed start um and you have to make sure you get absolutely everything right so for example if you're trying to run breeze and sudo at the same time breeze needs to be first but sudo needs to not know that it's not first and you also i've talked about just overriding one function in the glibc in fact the glibc has got multiple versions of each function you need to get the right one it's no good if you start returning the wrong value the api is mostly clean but some of the data structures returned have changed over the years so if you get the wrong one it's a complete disaster so from from that point of view it can be very difficult to get right so it seems like a sort of a silver bullet solution but in fact you need to be just as careful as with anything else so why should you care about application tracing system design is getting more and more complicated we heard in the keynotes um today all about cloud um i'm sure if anyone doesn't know what that is then i'll suggest you're lost um it used to be that we had a single application um running on a single operating system on a machine that had local storage but that's not the case anymore now we have the application and the operating system sitting on top of job scheduling virtualisation on top of a host operating system as well as a whole bunch of other things in the stack designed to give you more options and abstract away from the complexity but at the same time it breaks us further and further apart from the real system resources and the applications so if everything goes right it's amazing when something goes wrong it it just makes it so much harder to work out what the problem is we now have multiple applications running on the same machine that have to be um separate from each other they may share they may have their own core um but they're still ultimately running on the same ship and have to have to share i o resources for example even if you've abstracted your applications so that they run on separate machines they may share a network connection to a shared file system so it's very very difficult to keep everything separate and keep this abstract bubble of virtualisation running um in an ideal world we would also have homogeneous hardware so that it doesn't matter where you deploy your application you get the same performance every time but realistically that's not cost effective most companies will buy a bunch of machines and then upgrade at some point by buying some more machines those machines will not necessarily have the same uh applications stored on them they won't necessarily have the same performance and at that point we start to see performance bugs such as um race conditions appear on some machines but not others um and i just want to put it down in the bottom corner because actually this is a company mainly working with windows and mac at the moment but this gives you an impression of just how complicated it's going to get um bromium is a company that's working on creating a virtual machine for every single process in your on your computer so for example if you're editing multiple word documents or browsing multiple tabs each one of those run in their own operating system stack um so from a user point of view this is amazing you can click on absolutely anything and you know that you can't infect your computer with a virus because it's that tab is trapped inside the operating system um in for just that just that tab you also um because you're customising the operating system stack you can actually avoid many of the many of the problems that arise from having a very predictable stack it becomes it very very difficult to write viruses as well so i just wanted to talk to them talk to you about them today not because this is a technology that is coming to Linux immediately but because it's the kind of thing we can expect in the future basically um we're just going to have the most unbelievable amount of complexity in the software stack and everything is going to get harder so now i've got a couple of case studies to show you what can go wrong when you have too much abstraction so we did some work with ARM recently um on profiling their file system now they have a huge cluster for compiling their chips they run a lot of verification computationally it looks a lot like mapping the genome or or doing financial trading it's lots and lots and lots of of operations done over and over again in order to verify their system and performance is key the uh they have a complex file system hierarchy in order to make sure that all their needs are served some super backed up some super low latency and it's critical that users or that their developers use the right area of the file system so their applications um are largely speaking uh third party um CAD tools that have been bought in in order to do the verification and then they put their own scripted flow around that like like many others in that industry and others so that means they introduce a whole new levels of application dependencies that are designed by some local engineer and not necessarily controlled by a third party tool um they have license settings so the license server is also critical because if you're running lots of short jobs high license latency can be a bit of a disaster i'll come back to that again um but their main question was is everything being stored in the right place and unsurprisingly the answer was no it is not always being stored in the right place so this is actually some result i hope you can all see that it's come out quite well on your screen look terrible on my screen um the this is actually the first job that we traced um with them uh we looked at two areas of the file system slash temp where you hope there is loads and loads of activity because you want to be doing all your temporary calculations on local storage you don't want to be hammering the network and sending sending data back and forth unnecessarily um scratch was a area of the file system used for results um so fairly low latency it's just designed literally to spit out for the application to spit out a log file and it hangs around for some time um but you can see from this trace uh we looked at the number of rights to different areas of the file system you can see that slash temp is hardly used at all whereas scratch is is used quite heavily so in that sense they were they'd spent all this money on optimising their applications optimising uh their file system buying the best machines they could and then the whole thing was being tanked by some script uh putting the putting the data in the wrong place so that's an that's an an example of how abstraction doesn't always work you do really need to know what's going on if you're going to get the best performance it's far too easy to to tank performance with uh just one simple mistake um here's another another example um um the uh this was a uh company that we worked with uh again very short running applications um and they had a bunch of different license servers listed in their environment variable very easy to set up um and the license software just goes away and finds an available license but if you look at this particular trace um the red lines here I hope you can see the uh the connect line across the middle of the trace the red lines are actually failed connections so you can see here it's the application spends 200 milliseconds out of a run that lasts less than a second spends 200 milliseconds just hanging around waiting for failed connections before finally getting a license that's a disaster you've just thrown away more than more than 30 percent of your performance so there's an example where just really something simple is reordering the license servers in the uh in the the settings for for obtaining licenses um can very quickly give you a lot more performance and here we have another case study this this one's a bit different um this is an example where one user couldn't type in anything into their login systems um so the it manager traced a star tech session for that user um and he saw that uh xm bind the key binding uh application was sourcing the wrong key bindings so here we've got a picture of that we've got a trace of star techs at the top uh you can see xm bind being called and you can see that it's accessing the uh bind aliases in slash grid um now that's an application specific area of the file system there's no way the the system should be getting their tool bind key bindings from there um they were overriding the the local key bindings and preventing the user from typing anything so that was another example of just simply swapping the environment variables around so that the system key bindings were sourced first instead of these ones in slash grid um so this time it was easy it was obviously due to a lot of system level knowledge from the it manager in question but how long can we can we do this what if it's not a problem with your application it's not a problem with your virtual machine it's in fact a problem with the host operating system if you're running your application in some third party cloud how on earth are you going to debug anything from there so where do we go from here one of the problems is um how do we trace a distributed application um i've the examples i've looked at here have largely speaking been applications running on linux systems what if you have um a linux processing system with a linux database you've got some java application which could be running on anything to um to be honest and then you've got the user accessing this system through the internet from any device they want how are you actually going to profile that it's you can potentially set up a few uh load benchmarks um in your in your own system but once it's out in the wild how are you going to cope with unexpected drops in performance you can use a whole variety of system level monitoring tools in order to see which part is being hammered the most but how are you actually going to tie that back to what the user is doing how do you know how much of the um when you've spotted a bottleneck for example in your database how do you know which calls from which users actually caused that that spike in in performance it's not just a question of uniform load obviously under under some uh system with a very simple api um you know it's just a case of lots of traffic but that's not what real applications look like in real applications you can have some some operations causing very little load on the system at the same time as some few application few uses causing a very high level of load on the system and so how are you actually going to create that story from the user right through the whole system accessing every part of your of your application back to what they actually experience so uh even if we could create um a tool that would do lots of different tracing with lots of i know there are there are lots of good tools out there with open apis that let you create and share information across different systems easily but how do we actually marry them all together because it's not necessarily going to be fit for android or a mac front end um as well as a a java application and the second problem is of course what shall we measure um it's hard to combine different datasets even with open um open open data and open apis for different tracing tools and debugging tools we're never going to align the whole community so how do we actually pull out the bits of information we need um and we certainly don't want to measure everything so how do we work it out so that we can pull out just the bits that we want at that particular time obviously if we measure everything we're just going to tank the performance of everything that we've been uh we've been working for and there'll be an unacceptable delay when you click like on uh on your on your girlfriend's kitten pictures so um we don't want to reverse engineer uh every system that we need to profile we need to find a better solution for that so one solution is designed for profiling we've all heard of designed for test um and hopefully we all design effective testing methodologies um whenever we go about architecting a system we all know how to create correctness in our code make sure that our functions can cope with with all the possible available inputs without throwing unnecessary exceptions but what about design for profiling because performance bugs are still bugs if it doesn't work at the end of the day it's a bug um so we can build programmes with debug symbols and debug code um but often turning them on um is a uh is a measure everything scenario and that's not always what we want we actually usually need something highly configurable um and that becomes another engineering challenge in itself um it's normal to have a design team for the application and a design team for test but maybe we need to start having a design team for observability as well so critically profiling frameworks should really be chosen at the start and not attached later on so i wouldn't be doing my job if i didn't tell you what we offer um these solutions are available for creating custom profilers and custom debuggers um but i'm not going to pretend that this is some um panacea for the for the problem um our breeze technology is based on top of a tracing framework called star trace um we can make custom debuggers and custom profilers using that and we also have um uh just started to support an open source uh solution for doing similar levels of inspection to Java applications so by combining the two you do get a lot of visibility into the system matching up for example calls in the application with uh network level information and and so on um but we're still a long way from creating a framework uh and an ecosystem that can tell us anything we want to know when we want to know it um one solution is to try and monitor everything and correlate to actually expect that a large proportion of the performance is going to be given over to measurement um and observability that means that we focus more on meeting deadlines rather than getting the best performance for our money um it's always going to be difficult to get fine-grained coverage without harm harming performance um but with enough options in there then hopefully we can we can gather the information that we want to see um and this gives um an opportunity rather than to go in for pinpoint debugging approaches i.e. is the correct number being read from that file the fifth time is it is accessed that that level of debugging we can go for a more holistic view of debugging where we can solve performance problems without looking at the application architecture in detail so um trying to work out where we get the what kind of behaviour leads to um spikes in the network or spikes in in memory use and perhaps one day we could actually start to predict what our applications are doing i know a number of the larger vendors are already looking at that i passed a um a big screen on my way to the station um to come here which is all about microsoft's um efforts in uh correlating uh data within their cloud by measuring and trying to predict what's going on and and i think the there are a number of people doing similar work in in the Linux community so now it's time for your ideas now it's time for your questions so yes um so you're you're asking about whether um breezes open source breeze isn't an open source tool um at the moment we can make open source ld preload libraries four companies if they need an open source solution um but of course it won't be the it won't be the full technology um uh so so yeah the the um the configurability in our ld preload library at at the moment it's largely been on a consultancy basis but we've got plans to open up the api so that you don't get the source code of our um ld preload technology but you can use it to make ld preload libraries yourself as in you can uh essentially supply your code to to a build infrastructure and then and then make your own so from that sense um we are opening it up to the community so yes you're looking there uh the existing tracers uh i'm the outer one of the main outer of ld i mean we've solved those problems solved those problems already i mean i'm wondering about the overhead that really the approach has uh because we're basically doing system problem tracing in ld from within the terminal we are in perfection within the applications with user-level tracing uh we have uh uh cloud uh cloud support basically some synchronization of traces across an entire cluster we are also at that synchronization of traces between user space in the kernel so pretty much everything that you're listing here as work that should be investigated are actually things that have been solved a couple of years ago so you might be interested to look into those solutions mainly for the racing back end because i think what what's interesting here is that you bring interesting analysis on top of the the data extraction but a lot of work has been done already on the extraction yeah so the the question was why on earth why didn't i why didn't i give a talk on lttng is is largely your question um there are a number of talks later on where you can learn about lttng and the marvellous work that um that has been done in filling this space it is one of the tools um that does solve a lot of the problems um that i've been talking about today um but it is again by no means a solution for for everything um it's a very powerful tracing tool but it doesn't necessarily give you the same flexibility to uh change the application so for example it's very difficult to um uh manipulate an application and uh change the argument so we can set up tracing differently on a job that's been launched remotely depending on what the parent did um we don't just monitor the whole cluster um because we don't want to monitor the whole cluster we can't monitor the whole cluster because we're monitoring applications that are running against running in systems with applications that have uh that can't be traced because they're they're in production they're in the middle of taping out a chip for example um so a combined approach um would be a very good solution for a number of problems but again there are still problems out there uh which it doesn't solve um yes yeah it's certainly a technology that is um working very hard to fill a number of gaps but but yeah as I say please go to the talk on LTTNG if you want to know more about that um any other questions yeah oh wow what a question um there are a number um of issues I suppose that my developers would uh would come up with and say that we could do this and that's to to make life easier but by and large um the um uh it gives you enough rope to hang yourself basically so you can pretty much do what you like as long as you know what you're doing and that's um that from from our point of view that's great but it also means that we've needed to produce the star trace libraries to allow people to pick up the technology and run with it quickly so um yes and no no system's perfect but it's certainly it's certainly fine fine ease but by no means that as we've already said they're the only technology in in in the game and interesting even more in the Java part of your work how it's progressing as I mentioned you are starting to do this in progress yeah so we're starting to support an existing open source technology um it's been developed by a research group in Switzerland and at the Italian University of Switzerland and um again it's a case of their system is brilliant um it's very very powerful it gives you a loss of control about what you monitor in Java applications um but it gives you enough rope to hang yourself basically so what we're doing is offering support so that people who want to develop something quickly and easily um can ask us to help them do that um but yes it's the the technology is fully available for anyone to download um and take a look at um if you want to look at the the technology itself it's called uh diesel d isl um and will be adding links to their commercial support later on this year uh on our website so yes uh yes there are a number of security um technique um uh problems that can prevent it from working and um we've circumvented most of them and we're not going to tell you how um so uh yes it's it's an incredibly powerful technique because it lets you change the application so of course there are security measures built into Linux to prevent you from doing that but um if you think hard enough most of it's not not not a problem so one last question okay i think oh yes so um the uh question was about uh visualization and the fact that when you change from instructions tracing to system level tracing you change your visualization and it's a very clunky transition i think that's that's one of the one of the problems most of the talk has been about the tracing technology but actually visualization is just as hard a problem um and uh i know lttng has got some uh very good visualization tools with with open apis but um uh what we find is usually when it's it's a mixture so some data can be easily represented by by tools that are already out there but some data isn't and it's just a question of having that expertise at your fingertips and just being smart about it thinking about this comes back to the uh design for profiling think about what you want to see and how you're going to see it um i think it's the only way so okay thank you very much


Okay, we're here in Palo Alto, California at theCUBE, CUBE Conversation. I'm John Furrier, the founder of SiliconANGLE. I'm joined with Dave Wright, the Vice President of Worldwide Solutions Consulting with ServiceNow. Welcome to this CUBE Conversation. No, great to be here. I mean, fantastic to be in the valley. ServiceNow leader in IT, service management, you guys are taking the next step. One of the main topics is ERP for IT. Talk about what that means in context to the transformation as you guys go to the next level and roll out more stuff. So maybe the easiest way to think about it is the history of the company. So we started out as a platform with an ITSM solution on it.  And over time, we realized that needed to evolve. So we evolved to what we've termed ERP for IT. And that's where we're not just addressing the people who are dealing with the problems that come up in IT, but we're addressing the whole of IT. So people from inception through project management, right through software development lifecycle, and then all the way to getting applications or infrastructure rolled out and being able to manage, monitor, maintain those applications and the infrastructure. And then we found customers were starting to take the next step themselves. So customers decided that what you do in HR or what you do in facilities or what you do in marketing in terms of linking requests together is exactly the same way that IT manages its infrastructure. So what happened then was people started to use the ability to build applications on the platform. And we're actually now starting to see a migration beyond ERP for IT into what we're called in service relationship management where people are managing the different service relationships between the different business units. So it's an exciting time. Yeah, I mean, it's exciting. Certainly IT is transforming and the investment cycle is coming in huge. People now have money to invest for transformation. You get cloud mobile and social. But IT has cobbled up all these tools over the years. I got a tool for this, I got a tool for that. And it's become cluttered, it's become complex. So can you talk about the dynamics that you guys see out there around the complexity? And certainly when you talk about vendor management, you gotta have some standardization. So I mean, that's a topic that comes up all the time on theCUBE. What is the challenge around one? Is that happening? Is there too many tools out there? Not necessarily a bad thing. A tool for the job is kind of the philosophy we hear, but it can be complex and talk about that standard issue. I think one of our philosophies was the legacy systems have kind of been around for a while and grown through acquisition, but you never really got any assimilation. So there's two parts to the complexity. The first is people had all these multi-tiered platform apps that were trying to hang together and needed to be implemented in separate ways, configured in separate ways. You have massive complex data chains between them doing data transformation. So the fundamental concept behind our architecture was to have a single system, a record. So everything sits on one system, sits on one data model with a common code base. And then all the functionality that you get, the ability to have orchestrated workflows, the ability to do alerts, escalations, you could build that into any module within the system. And it meant that you could not just implement one system and have it in place, but you could also deal with upgrades in a much more simpler way than you ever were before. You didn't have to upgrade multiple components. So that's kind of the first element of complexity. The second element is what you were talking about. You've got relationships with MSPs, you've got relationships with different vendors. So people wanted a way where they could compare, contrast different vendors and actually produce like a data model that says, how well are these vendors performing? So one of the new things we've released now is vendor management, which allows you to look right across from incidents, problems, configuration items, and start to map what value you're getting from your vendors and then be able to use that value in negotiations when it comes to contract renewal. Before we talk about the three apps that you guys are announcing, I asked you, you brought this up just earlier, single system of record, what does that mean? I mean, that's, Dave Vellante talks about the single version of the truth that kind of kicks around and that's a big issue, but define what does single service of system of record mean? What does that mean? So for us it means having everything in one data model. So you imagine, whenever you do anything, the amount of data that you reuse, so you might have a list of locations that you've got, you've got all the people in the company all the assets in the company. Now, typically in legacy systems, they're pulled in from various different components. In our system, they all sit in one centralized data model. So as soon as you build in your application, if you build the facilities application, you don't need to re-import all the buildings and locations and assets into that. They're already there. As soon as you want to access a file, you just point to that file and the data's there. Okay, so if I just tie this together real quick, the tool sets are complex, they're not going to go away necessarily, you guys are just abstracting away the complexity with a single interface. Yes, correct. And so that's what your secret sauce is. Yeah. That's what you mean by single service. You've got the underlying data model, it's got the consistency, but then you've also got the top level consistency. So everyone's using the same UI. Everyone's got the ability to use an application without training. Sounds like some sort of catalog might be on topic. But we'll get to that later. Let's get into the apps that you guys are announcing. So there's three apps, vendor performance, resource management, and the third one is the password reset. So talk about what each of those mean. So how would the vendor, perhaps? The vendor management side, that's been a challenge people have had for a while, but now it's getting more and more complex as more and more vendors are coming in. And you see in this layer of abstraction where people are starting to use SaaS based solutions or people are using cloud based solutions. People need a way to understand how those vendors perform. So what vendor management allows you to do is correlate all the different incidents, problems, outages that you've had with a vendor and monos of the SLAs that you get with a vendor. So you can then compare and contrast all your vendors, but you can go back and say to the vendor, you said you were gonna give us this performance. In the past, we've just had your theory on that. Here's what we're seeing. Here's the data. Yeah, here is the data. Okay, how about resource management? I'll say that's on everyone's mind. You wanna know what's going on everywhere. We're gonna tie that together. How does that fit? So resource management, you've got very complex pools of people that are working together across different applications in IT. Some people are on specific projects. Some people may be doing firefighting with incidents and problems coming up. What this allows you to do is manage the most expensive resource in the company, which is the people. So you can now look across your workforce and see who's being utilized at what points and what availability you've got. So it means you can plan ahead and you can source out capacity for peak periods and get a feel for what people are doing on a day-to-day basis. And the third one is the password reset. What's new in that area? And why is that in point? Everyone asks, what's the reset the password? We've all been there, done that. But why is it a big deal here?  But on an enterprise level, we've never done that. So enterprise is typically people phoning up to get a password reset. And the analysts say that's 20% of calls. So a fifth of your time, just doing something as dull as that. So what it allows you to do is take that whole social aspect and put it on the enterprise. And as you look at the way, we've moved through the different generations of people using IT. So we've gone down the Gen X, Gen Y digital natives. Now you've got the new layer, the entitlement generation. They're not expecting a system where you phoned up and someone reset your password. They want to do it themselves. I mean, it's a consumer experience. It's the consumerization of IT.  You want it to be like eBay. You don't want it to be like something stuck in the Aces. David Wright, vice president. Great conversations, a cube conversation. I'll give you the final word. You just put a bumper sticker on the car for this ERP for IT. If you're going driving down the freeway, what's the bumper sticker say? What's the big deal in this announcement for the folks to walk away with? So what this enables you to do is manage the whole of IT in a consistent manner. So you're not just looking at your operations or your development phase. You're looking at the entire lifecycle of any IT asset that you have in the company. Thank you very much. This is a cube conversation live in Silicon Valley. We're happy to have service now here. Thanks so much for your time. Dave, appreciate it. Appreciate it.


Hi guys, so welcome back to another video. I am going to do a video today on Dealing with and dealing with not really a good word but dealing with patients who have dementia or confusion and I think I finally decided that I'm going to record my videos in here I really like my headboard backdrop if you see on my vlog channel, which is linked below I made a headboard so and blue is going to accompany me because he's needy and he needs his mom huh, so anyways dementia patients can be really um, I Don't want to say anything that's offended anyone so please don't take these words offensive But they can be very difficult patients to deal with with their confusion and lack of understanding And as a nurse it can be very hard. I like to think of dementia patients as almost Babies and hang hang with me here When you're a little when you're a baby or toddler you have a really hard time communicating your wants in your needs so instead of communicating by using words you communicate by Maybe hitting or biting or yelling or crying and you have a really hard time communicating your feelings So you communicate using your emotions and a lot of dementia patients do that as well It's sometimes they're so confused It's hard for them to communicate their feelings and their wants and their needs and so a lot of times will communicate in other ways other than using their words and as a nurse that can be very hard and confusing and Frustrating at times and yeah, so whenever I have a dementia patient I've tried to make sure all of their basic needs are met As far as you like toileting, you know if they're continent and continent what not making sure they feel clean warmth food and drink making sure that they have those Necessities because if you have made those basic needs then you kind of can take away some of that Need to try and communicate in other ways if that makes sense It's also important to understand that imagine being so confused already in your head and then you're used to your routine in your environment and then you go to a new environment and there's new people new Smells new everything and that has to be even more confusing So a lot of times dementia can be even more exacerbated in a new environment, which rightfully so it's confusing So it's important to try and keep a routine as best as you can in the hospital talk to the dementia patients family members or if they came from like a Nursing home or place like that talking to the staff that works there about the routine and trying to keep that similar We had a patient one time back in my old job who was with us for a long time and she didn't even know She couldn't even tell us her name half the time So she was very very confused and you know, we found out that she liked gospel music And she liked playing the piano, so we ended up getting a piano Electric piano we found one for her and we played gospel music for her and that really helped even though her routine was altered She had something that was familiar and comforting to her So finding something that's comforting for dementia patients is important as well It's also important to really acknowledge their feelings a lot of times I'll say things, you know if I'm asking them those orientation questions like do you know where you're at or can you tell Me your name or birthday and if they can't if they get frustrated about thinking about the answer a lot of times They that's okay. I understand that you're in a new environment right now and that can be really confusing and just acknowledging their confusion acknowledging that their emotions and Instead of getting frustrated about oh my gosh, they're being so difficult. They're climbing on a bed. They're picking at things They're pulling out their lines Understanding that it's got to be really confusing and I understand there's different stages of dementia Not everyone totally doesn't remember Everything when they have dementia But if you think of it as communicating more like to a toddler then that can kind of help you Better process things as a nurse it may be not get so frustrated I know it can be frustrating talking to a toddler when you're trying to communicate But hopefully when you're talking to someone who is young you don't yell at them or get aggressive or things like that And so, you know same thing when you and we get older we forget things we you know lose some of our bodily functions and If you just have more of a sweet Approach to it then that can be helpful. So yeah, that is some of my tips on caring for dementia patients It's very very common when you no matter what hospital setting that you're in unless you're like in Peds or OB or something But you're gonna take care of elderly patients and you know dementia is a very common thing in the elderly So I hope this gives you guys some tips if you have any other tips leave them in the comments below Give this video thumbs up and subscribe to my channel and I'll see you guys next time. Bye


Hey developers, my name is Christy, and I want to invite you to take control of your identity this year. And if you're like, how do I even do that? Well, the good news is, is we have the perfect event coming up for you on May 16th and May 17th. It's called Developer Day 2023, developerday.com is where you can find out more about it. But the theme for this year is take control of your identity and you'll get to hear a bunch of awesome talks, workshops, and have architecture reviews. So go ahead, sign up online to get reminded about the event and I hope to see you there.


The challenge of the Yukon It's Yukon King swiftest and strongest lead dog of the Northwest Placing the trail for Sergeant Preston of the Northwest modern police in his relentless pursuit of lawbreakers Gold gold discovered in the Yukon a stampede to the Klondike in the wild race for riches Back to the days of the gold rush with Sergeant Preston and his wonder dog Yukon King as they meet the challenge of the Yukon Sergeant Preston was sent down to San Francisco to pick up Red Gavin who was wanted for murder in Whitehorse He took King with him The tall broad-shouldered red-coated Maudy attracted great deal of attention in the city But people were even more interested in King a real lead dog a veteran of the fabulous Yukon trail The men at police headquarters were no exception and there was no work done until the sergeant and King were ushered into the superintendent's office Thank you, sir This is lieutenant Captured red Gavin my congratulations lieutenant. Yes That's a great dog you have there. I think so it's a beauty. I've never seen one like him Does he mind if you pet him a bit? Oh, not at all. Yeah, I Brought him along so I wouldn't have to be handcuffed to Gavin all the way back to the Yukon King will guard him for me You'd better tell the sergeant the sad news, sir sad news. Yes It's true that Gavin was captured and we had him in jail waiting for him, but He isn't there now. He broke out the night before last every possible. They've been disciplined But he isn't as bad as it might be We know where Gavin is hiding out at least we're fairly certain and Fair will pick him up again tonight. That's fine. I hope to be picking him up again tonight He's hiding out in the Barbary Coast though the reputation of that district may not have reached the Yukon Indeed it has there and you'll know that making the arrest may turn out to be a little difficult There are tough lot down there in the waterfront. Well, believe me lieutenant I realized perfectly what the San Francisco police are up against and I'd consider it an honor If you'd let me go with you tonight, would you now like to see you in action? Well, I see no reason why the sergeant shouldn't go with you pal. We're begging you pardon sir. I do I have one paint objection Sergeant we're not accustomed to invading the Barbary Coast wearing red coats Did you ever hear what happened to the red coats who marched up Bunker Hill as I remember it they marched down only a few of them Those red coats made wonderful targets. Well, I don't have to be an uniform lieutenant. I could wear plain clothes. That's true And maybe you should come along with me.  I apologize for him sergeant. He evidently hasn't heard of the Northwest Montage record Oh indeed I have sir indeed. I have it's perfect And that's what makes me feel that they can't be up against such spellpins as we have on the Barbary Coast Well, perhaps you're right lieutenant. Has it agreed about tonight? Yes, sir. It is now. I'll pick you up at your hotel at 11 o'clock Lieutenant Farrell drove up to the sergeant's hotel that evening in a shiny black touring car The sergeant and King were waiting at the curb. This is a wonderful car. Lieutenant. Yes, it's a new winter Makes 30 miles an hour. Oh Sergeant Preston meet Muldoon and Jenkins and four. How do you do? Now, where are you going to leave the door? Why can't I take you with me lieutenant? Just what do you think you're going on man a picnic? No, not at all But King might be able to help and how do you figure that? Well, I have a glove here that belonged to Gavin I've let King smell it and he'll trail the man who wore it no matter where we go We know where Gavin is. I thought in case he got away. Oh, he won't and Besides, there is no dog can trail a scent in the Barbary Coast. There's too many of them. You'll have to leave him behind We got a ride with us lieutenant to stay in the car. I promise he won't even bark. He minds perfectly Let him come Mike. We could use a mask. Oh, all right. All right, but mind you keep him quiet We want no barking dogs announcing our arrival.  All right, Jenkins King lay down at his master's feet This was the first time he'd ever ridden in an automobile and his instinct was to jump out and leave the strange coughing monster as far behind as possible He was trembling until the sergeant laid a reassuring hand on his head. It's all right boy A moment later his master held a glove in front of his nose fine King He understood the quiet command and he raised his head a little to search the breeze, but there was no scent for him to follow 15 minutes later when the monster came to a stop and became wonderfully quiet King knew they were near the sea That's the place across the street Jenkins you and Forbes get around to the waterfront end of it. We'll be coming through the cafe in 10 minutes now get going Ten minutes past now sergeant stay here King be quiet boy. Come on Muldoon follow me King waited nervously for his master to return And when he heard a volley of shots inside the cafe across the street He was unable to hold back a growl But he stayed where he was in the backseat of the car watching the door the cafe where his master had disappeared Then he heard another sound a man was climbing out the second story window the cafe He dropped to the ground and started to run King had a bad moment He caught the man sent and knew he was the one the sergeant wanted to find but the sergeant had told him to stay in the car King's intelligence was equal to the problem though to find the man and follow him was the more important command King leap from the car and started out in pursuit Five minutes later the sergeant and the detectives came out of the cafe with two prisoners And crossed the street to the car Pulling that gun on us is going to send you to prison for ten years. I didn't know who you were Oh, yes, you did and you know where Gavin is.  I haven't seen Gavin since he was arrested. I get him The dog's gone. Yes, I see I thought he always obeyed your order. He does He wouldn't have left here without a good reason. Well, we haven't got time to look for it. Listen, I hear him You better call him fast somewhere down on the waterfront I'll go and get him lieutenant, but I think you'd better come with me Why because King would not have left this car except to follow red Gavin. Are you daft man? I mean it and he isn't far away. He'll only take a minute. All right since you're a guest to the force I'll view me your notions. What's he's too Muldoon right lieutenant? Let's go sergeant The sergeant the lieutenant ran down the street toward the sound of King's barking They cut between two buildings to reach the docks. At last they saw King standing at the water's edge There was an old sailor in here King. What's the trouble boy?  What do you do? What's happened? There was a guy getting into a rowboat and his dog come streaking towards him jumped right at his throat Lucky the guy had a belay in pen in his hand.  He wouldn't have got away otherwise What did the man look like? Red headed husky red Gavin which way did he go? Who are you? I'm feral of the homicide Here's my badge that red heads wanted for murder now. Which way did he go out of that schooner the one that's just getting Underway come on sergeant the Harbor Patrol boat is at the next wharf We'll catch the schooner before it clears the gate on King The Skipper of the Harbor Patrol boat knew the lieutenant well and in a matter of minutes He had the boat cutting through the choppy waters of the bay toward the schooner She was under full sail and driving toward the Golden Gate with the help of a brisk offshore breeze But the patrol boat had speed and overhauled it Like that'll do the trick Others some activity on our decks. She knows we mean business. They'll be holding down their sails stand by the board Just gonna The schooner's sails were furl and she wallowed in the choppy sea as the patrol boat came up alongside Had an order from the police skipper a rope ladder was dropped in the deck Feral and half a dozen men from the patrol boat started up it Sergeant lifted King in his arms Taking the dog aboard sergeant. He's mighty big to carry up that ladder. We can make it think and find Gavin no matter where he's hiding Up we go boy When the sergeant King reached the deck of the schooner the crew was lined up You're not taking their word for it Oh, of course not sent three men down to the hold and the others are searching the cabins in between decks But half an hour later when the men reported back to Feral They had been unable to find Gavin the lieutenant lost his temper. What do you mean not on board? He must be we'd have seen him if you'd have gone over the side I'll get down below again and find him a king and I go this time lieutenant. Yes, and I'm going to Yes, sir, you and Adams keep an eye on this game At the foot of the companion way between decks the sergeant spoke to King blinding boy King barked and started forward Feral and the sergeant followed him the dog led them directly to the folks This must be the place lieutenant You can see isn't here those lockers are too small to hide a man all the bunks are empty He's sniffing at the flooring perhaps. He's underneath the deck. There's only the hole below. We've searched that isn't down there Wait a minute wait a minute. I've got it. Why are the dogs smarter than you are? There's probably a trapdoor somewhere around here, but we won't bother looking for it All of you get some tools and rip up this planking go on now get to work What's your idea lieutenant I guess if they've been using this schooner to smuggle aliens in huh? They probably got a snug little compartment built just below this deck. We'll soon find out And when the planking had been ripped away the lieutenant's gas was proved to be correct There was a compartment and red Gavin was hiding in it covered red Now climb up out of there You're going back to jail The evening's adventure was finished red was returned to jail and During the next few days the extradition proceedings were completed The sergeant was scheduled to sail for the Yukon with his prisoner on the following morning When he received a message from the superintendent asking him to stop in at his office He reported at once and found lieutenant Farrow also waiting for his chief. Oh, hello, lieutenant You have any idea why the superintendent wants to see me. Oh, he probably wants to say goodbye I'm glad to the chance myself. I appreciate all you've done for me. Oh not at all not at all Did you enjoy that taste of excitement? I give you the other that's indeed I thought as if I'd stepped right into the middle of Treasure Island Treasure Island. Oh the smuggler schooner. You mean. Oh, that's nothing new around here We have pirates too. There's the Blackhounds gang for instance Sergeant it'll do my heart good if I had you and King working with me when we went after them Look now where would you consider it at all? I'm sorry King and I must go back to the Yukon Nothing ever happened. Oh now.  I tell you I couldn't stand it myself Excitement is meat and drink to me. I couldn't do without it. Oh Here's the old man now How do you do sir? We'll get down to business right away Well You're going to be Yukon with the side. I I'm what here's your ticket Perhaps the sergeant will help you buy whatever you need in the way of closure supplies this afternoon now wait a minute What's this all about? I'm in no need of a rescue if that's what you're planning for me, sir. This is duty fell Link Davies and Ben Walters have been seen in Dawson Their bank robbers sergeant wanted from Burley too I'm sure the north way smothered will help Pharrell track them down. You can depend on it, sir When they're caught it will be up to you to bring them home by Sergeant will it be much of a job picking them up? Well, you're not if they're in Dawson. It's only a small town Once they're caught it'll be a simple matter getting them home Do you hear that sir couldn't one of the newer men take over this assignment? I don't want to leave town and I set my heart on going after the black count. That's out Mike Out. I'm sorry Ivy just been talking to Washington the secret services taking over The case is out of our hands completely You don't say Well, if that's the sad state of affairs, I suppose there's no reason why I shouldn't take a vacation up in the Yukon And that was how Mike Pharrell happened to accompany the sergeant and King on their return trip to the Yukon When they reached the territory winter had already set in Red Gavin was delivered to the authorities in Whitehorse and Mike and the sergeant headed north on the Yukon trail by Dog-sled Mike was completely unimpressed by everything he saw I'm thinking for a grown man to be riding around on a sled I feel like a six-year-old out for a day's tobogganing. We're having good weather Mike Sometimes the trail can be pretty rough. I ought to be back on the job Well, maybe we'll be able to scare up some excitement for you in Dawson. I doubt it So do I especially after what you're used to in San Francisco Oh Events conspired against Mike's desire for excitement The trip to Dawson was made in record time over a hard-packed trail The weather was mild and so was the boomtown when they reached it It was nearly deserted for there had been a new strike north of the bonanza So this is Dawson, hmm and this is the famous Monte Carlo Caffé With a fine funeral parlor it would make nearly everyone who doesn't have some sort of business in town It's going up to nugget Creek. Now. What about the two men I've come after I just had a talk with Tex Rickard the owner of the place And he remembers them but they left when the rush started and I suppose that means another toboggan party for us You can stay here if you want to Mike. I'll go after them. What would I do waiting for you? No, man. It's better to have a little action and none at all. Let's get started at once I'll need the rest of the day to get our supplies together. Oh, what for a few cans of beans is all we need No telling how long will be gone might be a few days might be a couple of months a couple of Will you say that again? We may not find Davis and Walters at nugget Creek And we can't expect any more fine weather we do for a cold snap that happens to be about zero now I'm speaking of 40 or 50 below. Yes. I understand Naturally, no human being could go out in weather like that. Oh, we'll go out It's only a blizzard that might hold a set when the wind gets up to 60 or 70 miles an hour is impossible to fight it 60 or 70 miles an hour Yes, oh, I think you're pulling my leg sergeant. You're just trying to pull an element of danger into this business Well, I'll admit we don't often have girls like that. I'll bet you don't still it's just as well to be prepared Mike We'll start the first thing in the morning There seemed to be no concern no reason for concern about the weather The sun shone the sky was clear during the three days. It took them to reach nugget Creek But after inquiring up and down the creek they learned that Davis and Walters had arrived too late to stake a claim and had gone on The sergeant shook his head when he learned the direction they had taken north by northeast Mike It takes us in the rough country. Can you follow them? Yes, they had to break a fresh trail There hasn't been any snow since they left King won't have any trouble. Well, then let's get going Let's pick up the spelled peen so I can go back home to a place where there's something happening. I'm on board the sled. I'm ready All right Lieutenant Michael Farrell wanted excitement He had his first taste that afternoon The trail led to an Indian village in the wooded country at the foot of the sawtooth range The place seemed deserted as the sergeant drove into it and stopped the team Oh And then a second later the sergeant and Mike was surrounded by a horde of Indians Shouting war cries and brandishing knives and spears We got in to go for your gun.  There are only chances to talk them out of it. Oh, leave it to me. We come in peace Take us to your chief At the sergeants words the Indians rushed forward and taking hold of the sergeant and Mike They pulled them toward the largest hot in the village King would have tried to win a fear, but a word from the sergeant stopped easy boy I know the chief I'm going to find out what this is all about. Well, I can tell you that they're going to kill us Oh wonder the excitement Mike, but I don't want to be murdered. This is all the day's work The chief will listen to reason. Oh, it's a reason. He says he's at him standing in the door of the cabin. Yes Logano I Listen to reason you say Logano Tell your warriors to let us go we're friends. No white man friend of Indians. You know that is not true Like men come here. We give food shoulder them steal nugget.  It looks something like a man Tell us about the nugget God Rogano get charm against winter Now winter bad. No game. No food many Indian died How long ago was it the white men stole it last night Davis and watches must have been Rogano Those men are not only enemies of the Indians. There are enemies We're trailing them and when we catch them, they'll be hanged. You'll not catch them Indian trail into canyon where ghost packs day. No man lives who enter We have strong medicine against the ghost pack Let us go on to the canyon will capture the men and bring your nugget God back What what did they talk about deciding whether to let us go or not and What was that about a canyon where the ghost pack leaves nothing but superstition Mike? They believe there are we're wolves in the canyon What are we many who've taken the form of walls and only pray on human beings now there's a pleasant thought And what's wrong medicine? Do you have against such things is that there are walls in the canyon? Take care of them Indians have made a decision Will you let us go Rogano? But only one way go in the canyon. Are you sure the men are still in there? Why you stand guard at opening this is working out well Mike. I don't like any part of it But don't you see Davis and wallors went into a blind canyon I can't get out because the Indians are guarding the entrance all we have to do is go in after them We're ready Rogano warriors follow you stay at opening you go in You not come out without men and nugget God. We agree to your terms Can we reach the canyon before it gets dark and we'll start right away You wait for warriors to harness team. Whatever you say chief The sergeant and Mike left the village with one Indian sled in front of them and half a dozen behind They were still under guard and it was clear the Indians believe they were being led to their execution To a death as certain as the stake Suddenly the temperature began to drop the wind from the north increased in violence The sky was clear, but the snow whipped from the ground was driven against their faces with stinging violence keep well covered Mike The Indians wouldn't let us for one thing for another this will be a blizzard by morning Let's hope we're out of the canyon by then Sound The entrance of the canyon was narrow and forbidding But as they neared it the sergeant could see that it widened out beyond the opening The floor was strewn with great snow covered boulders and toward the upper end. There was a thick growth of pine See those woods up there Mike that's probably where Davis and Wallace have made their camp None for us on our way to get them. What's the matter with the rocks? We'll leave the team here and just take King with us, you know, I think you're good luck Mike How do you figure that? Everything's working out so well Davis and Walters bottle themselves up in a box canyon the Indians lead us straight to it All we have to do is go in and get them I don't blame you for thinking we have an easy job up here Sometimes we run into a lot of difficulty. Oh, you do. Sorry. I can't seem to provide you with any excitement I give you credit sergeant for doing your best The Indians have been holding a conference and now the chief approach the sergeant We're ready Do you find nugget God don't worry You not come back Suppose if we tried to come back out they'd stick their spears into it. That's the general idea. Let's go one king The sergeant with King by his side led the way into the canyon Mike followed a few reluctant steps behind The steep wall of the canyon broken here and there by caves Towered on either side, but only a sure-footed animal could climb them the sergeant and King Concentrated their attention on the woods ahead They had covered half the distance to the woods when the great dog began to growl uneasily What's the matter with him? You must have caught a sense Follow me Mike work our way forward from one bullet They're hidden in the trees they can see us, but we can't see them I can't see as well our parkers are covered with snow Just then a volley of shots rang out from the woods the sergeant Mike hit the ground can't see us Hmm. Well, they're coming mighty close with those shots. Yes The light's failing though. They can't keep us pinned down for long What's that The werewolves there's somewhere up above us. That's right. They have their dens up in those caves Look look they're coming out. They're starting down the sides of the canyon. Are they coming after us? It looks that way all the saints preservers what a spot to be in Outlaws in front of us Indians behind And the pack of Gorsuch devils descending on us from above if we can only turn them up the canyon They may drive Davis and Wallace in the cover King ready boy Mike Ferrell found himself unable to move a muscle and he could hardly credit the evidence of his eyes during the next few minutes He saw the wolf pack racing toward him and then King leaping forward to meet the leader The sergeant was shooting fast on either side of the great dog his bullets accounted for half a dozen of the grey phantoms and King disposed of the leader the pack began to circle the men and dog then Mike finally sprang into action and began to shoot He saw what the sergeant was trying to do not only to drive off the pack But to force them to retreat up the canyon toward the woods and at last he succeeded Sergeant King broke from cover After them they're starving I'll catch the scent of Davis and Walters dogs. Most crooks will have their hands full now's the time to close in on them Sergeant had called turned correctly The wolf pack was heading for the woods now and the outlaws guns were turned on them Preston Mike and King headed for the trees to the left of the killer's camp and reach cover as the wolves attack the Follow me Mike I'll drive off the wolves and when we get rid of them we'll be able to take our prisoners The sergeant and Mike took cover behind two large trees less than 50 feet in the camp They opened fire on the wolves who were circling the campfire The new attack from the force was too much for the pack. They started to run down the canyon Davis and Walters stared after them unable to realize for a moment that the fight was over Unaware that it was the sergeant and Mike who had turned the tide of the battle But the sergeant's voice soon made them understand how vulnerable their position had become Davis you and Walters stop your guns You're covered Under arrest in the name of the Queen And that was the simple end of the adventure The huge nugget the men had stolen was found and returned to Riganu The sergeant traveled with Mike and his prisoner all the way down to Skagway They were locked in the ship's break for the return trip and as the steamer prepared to sail the sergeant and Mike said goodbye It's been a pleasure working with you sergeant. Thanks Mike and with you to keep You're a fine detective. I'm sorry. We couldn't find any excitement for you Will you say that again? I'm sorry. It's been so dark Just what did you put in your report why the facts Trail Davis and Walters wanted for murder in bank robbery the ghost pack Canyon Arrested them returned large nuggets stolen by Davis and Walters to Riganu. That's all. Oh, is it now? You didn't find it worth mentioning that those Indians were going to burn us alive Well, I know there was no point in making trouble for them. They're simple people. They're well treated. They're harmless Regano understands now that he must call on the force if he has any grievance in the future And couldn't you find a word for those were wolves who were trying to tear us limb from limb They were only timber wolves. Oh What's the use what's the use? I'm beginning to understand now that one has to read between the lines of a Northwest mounted report to find out what really happened Don't you apologize to me for any lack of excitement? I told the superintendent this trip would be a vacation for me Well, I'm going home for my vacation after what I went through in that canyon I'll need a long rest among those babes and arms that inhabit the Barbary Coast. I'll tell you the truth You'll admit it last that this was a tough assignment. It could have been worse true. We might both be dead But I am willing to admit that I'm just as happy as you are that the case is closed Now here's Sergeant Preston with a preview of our next adventure the case of The Phantan gold robbery when I first heard the story of the Phantan gold robbery It appeared that a friend of mine named David Owen was involved in the crime Dave was a family man, and I hated to think he was guilty Luckily King and I managed to turn up some new clues But before we could clear Dave there was some mighty dangerous and exciting moments for both of us Be sure to listen to this exciting adventure Next Monday These radio dramas a feature of the challenge of the Yukon incorporated are created and produced by George W. Trendle directed by Fred flower day and edited by Fran striker the part of Sergeant Preston is played by Paul Sutton The challenge of the Yukon will be brought you every Monday Wednesday and Friday starting next Monday September 12th. This is Jay Michael wishing you goodbye and good luck till next Monday so long Now a listening reminder The exciting adventures of Jack Armstrong the all-american boy will be heard over this ABC station this afternoon You won't want to miss today's thrilling episode. So listen