I'm John Wilson. I'm the CEO at Agenta. We're a technology company that focuses on education and learning. We build, manage and operate platforms for education, for video collaboration. Externally, we prefer to work with what we feel is ethical industries, obviously education, teaching, learning, healthcare. We feel that we can really contribute to these industries by creating exciting platforms, easy to use platforms, secure platforms that people can utilise. What we feel is one of the most important things for Scotland to boost economic growth is investing in rural areas. By investing in broadband in these local areas, we can attract more talent, we can attract more companies and we can drastically improve the delivery of education and learning within these schools, within disparate regions, within Scotland. We'll get started because we don't have a lot of time. Let me introduce myself. I'm Bronwyn Swinarton and this is my colleague, Taryn Coop, and we're from the University of Leeds. Today I'm going to tell you about our ESRC-NRF project that looks at higher education in South Africa and in England. As I said, it's funded by ESRC and NRF. We have a Cape Town team who are funded by the National Research Foundation in Cape Town. You can see the five members of the team there. We also have the Leeds University team which is funded by the ESRC. The project started in October 2016 and ends for the Leeds team in November 2018, so in a couple of months' time. Even though we've put in one application, we have two different funding bodies, and so the South Africa team have an extension on theirs until March, so we are going to carry on working together after we formally end in November. Today we're going to give you a background to the project and a little bit of the context, and then my colleague Taryn is going to show you some of the maps, the interactive maps that we've created. So the project focuses on marketisation, digital technology and on bundling, and we're particularly interested in the role of the market in higher education and how private providers are coming into the market of higher education using digital technology to unbundle some provision. And don't worry about unbundling, I'll explain what we mean by that in a moment. So if we look at marketisation, and the growth in demand for higher education has grown rapidly since the Second World War, and especially so in the last 30 years, and this, along with the financial crisis of 2008, has put a huge financial strain on universities, and so they're looking for different ways to bring in revenue. And one of those ways, obviously, is fees, and fees have increased to over £9,000 in the UK, and in South Africa they've increased 9% per year in the period 2010 to 2017. At the same time, universities are becoming increasingly accountable, and rankings are very important, things like REF and TF are used to judge institutions, and at the same time, private companies are increasingly coming into the higher education landscape. So digital technology, we know that digital technology is ubiquitous in society, and in education, and especially in higher education, and there's a growth of online and blended courses. The ed tech industry is estimated to be worth over $120 billion, and it is expected to double in the next 10 years. And many of the reasons, there are many reasons for bringing technology into education is to widen access, it's to allow people who can't ordinarily come to a campus-based university to be able to do distance courses, people who work, or people who've got caring responsibilities. And then we've seen one of the ways in which digital technology has really increased provision is through MOOCs. MOOCs have rapidly expanded in the last 10 years with global platforms such as Coursera, FutureLearn and edX. But is digital technology really widening access, because when MOOCs first came out, they were free, open access, you didn't really pay for anything except for certification, whereas now they're increasingly monetised, you pay for credit, and now on FutureLearn platforms you pay for access for more than a few weeks after the course has ended. And you're also in order to be able to access which courses you need, a device, you need data or Wi-Fi, you need to have a certain level of digital skills and literacy, and that's not always the case, especially in South Africa. And so the third part of our project is looking at unbundling and how marketisation and digital technology has led to unbundling. And the term unbundling first arose in the computer industry when IBM started to unbundle its software from its hardware, and so instead of buying your computer with all the software already loaded, you then could buy each separately so the software was unbundled from that package. And for us, unbundling is the process of disaggregating educational provision into its component parts, likely for delivery by multiple stakeholders, often using digital approaches and which can result in re-bundling. And an example of unbundled educational provision could be a degree programme offered as individual standalone modules available for credit by an online platform to be studied at the learner's pace in any order on a paper module with academic content, tutoring and support being offered by the awarding university or the universities and a private company. When Taryn shows you the maps in a few minutes, you will see some examples of unbundled provision in the UK. So the key questions for our project have been where does the discourse of unbundling come from and how is it used by the research literature, the policy literature, the media and the interviewees, how do different stakeholders in higher education understand unbundling and re-bundling? How is unbundling happening in practice in South Africa and in England and that's what we'll be focusing on today? How does the intersection of unbundling, marketisation and digital technology change the pedagogies available in the higher education system? Which aspects of pedagogy and provision can be or are being unbundled and are marketised? And what is the nature of educational provision currently available in the UK and the South African higher education at the intersection of those three aspects? So in order to answer these questions, we carried out a range of different data collection with a range of stakeholders. So we carried out interviews with policy makers, ed tech developers, higher education leaders and private company CEOs. We also held focus group with academics, those people who were using those provisions for teaching and learning every day. And then we carried out surveys with students to find out whether students were aware of this provision and what they thought of the potential use of it in the future. We also used desk research to look at the relationship between universities and private companies and that's where the data from that is used to make maps and that is what Teran is now going to talk about. So part of our project involved creating maps to present the partnering of private companies with public universities to offer online education and bundled, re-bundled or online programs. And this just gives a bit about the theoretical underpinning of this. Social cartography specifically involves the mapping of ideologies but it's the premises of social cartography that provide a rationale for visualisation techniques in this case. So for insights into relational trends across the context and for large amounts of data to be explored simultaneously. So what has become recognised as a subset of social cartography is tactical cartography which involves creating interactive maps usually using digital tools or digital software. So our tactic or strategy involved designing the maps using our theoretical framework that Bronwyn briefly explored to inform the scope, visual features and monitoring options in order to reveal patterns for analysis. So for example Bronwyn spoke about marketisation so in a theory around market making in the higher education sector and there's exploration of the dynamics of ranking and brands of institutions and this is visually represented on our maps by including data from the times higher education ranks the historical status of the institutions and university membership groups. So just a little bit of the literature the connections between public higher education and the corporate sector for research, employment, outsourcing, services offered in partnership and various other models of academy industry relationships are well documented and in 2001 Anderson suggested that more expansive views of these interconnections be captured particularly through visualisation techniques and using social network theories. I'm in a number of authors have contributed to the space as well as a number of blog posts that have specifically spoken about online program managers and providers in the landscape how, what business models they use and what type of certificates or degree they're providing. So the maps were made at the beginning of this year and then updated last month. And everything on the maps was sourced from the public domain from distance education websites the university websites, the partner partnering private company websites and pressure leases in the media. The basic elements that you'll see on the maps are circles and squares and arrows and the circles represent the universities squares represent the private companies which I will refer to as OPMs. So this is a map for online program management providers for example companies like Pearson to academic partnerships who provide a variety of services to do with managing online programs such as market research enrolment management, student retention and sometimes technical support and MOOC providers offer a digital platform to host university content but some MOOC providers are also diversifying their services to become more like OPMs. So this discourse is debated but for the purpose of this presentation I'm going to use the umbrella term OPMs to refer to these private companies bearing in mind that the services range from a digital platform to different managerial services. We have used shapes, sizes, colours, borders and clusters to represent different features or characteristics of the institutions and companies in the terrain and we've done it all on an open source platform called COOM which is a data visualisation specifically for relationship data. So I'm going to show you one or two of the South African maps their static images from the interactive platform and from there we will move on to the UK mapping which is more the focus of this session and I'll show you that on the interactive platform and just so you know the maps and the underlying data will be made available at the end of our project. So this is a mapping of the relationship as you can see between the public universities and private companies OPMs in the South African space so the circles represent the universities they are scaled according to the number of students enrolled in each institution the borders around them represent the historical status of each institution and this system of differentiation is quite debated and criticised for embedding historical disparities from the apartheid era in South Africa into today's discourse but it is still used in policy documents especially around funding models so the dark borders represent the historically advantaged institutions that were privileged during the apartheid era and the thin light borders represent the historically advantaged institutions with historically disadvantaged sites that came about as a result of mergers after apartheid and the thicker light grey borders of the historically disadvantaged institutions mainly located in poor rural erics and institutions without borders are new in the terrain so the connections between squares which are the OPMs and circles are also colour coded according to what type of teaching and learning is provided through that partnership so what we see here is the pattern shows that the OPMs work predominantly with the specific type of institution research intensive historically advantaged institutions the two exceptions are comprehensive or more teaching focused institution and a research intensive institution that are both historically advantaged institutions with historically disadvantaged sites this distinction becomes clearer when we look at which research intensive institutions do not have any partnerships and they are all predominantly historically disadvantaged institutions so this is the same elements on the map just clustered in a different way this is clustered according to the times higher education rank and world rankings in the three highest ranked universities in South Africa there are only three in the top 400 in the world are collectively partnered with all nine of the private companies currently active in the terrain so now I'm going to move on to the interactive platform so this is the UK terrain and you can see it's a lot busier a lot more universities in the UK so we have 166 UK public universities represented by the circles color coded according to membership groups Russell group Guild HE million plus the former 94 group which disbanded in 2013 University Alliance and those that are not aligned with any specific membership group they also scaled according to the size according to the number of students enrolled at each institution and there are 21 opm active in the terrain and they are represented by the squares color coded according to which country in which they are established and scaled according to the number of employees as a proxy for size of the company so I'll take you through a few points of interest that's our discussion for our research team so if we use the interactive techniques to highlight the Russell group universities almost all the Russell group universities that's 22 out of 24 of them partner with at least one opm to offer either MOOCs or online programs and that's specifically what we're looking at in these connections MOOCs and online programs we're not looking at short online courses in this map so almost all former 94 group universities 12 of 14 partner with one or more opm and then when we go to those that are not aligned with any brand or membership group less than a third of those universities partner with an opm so we see a pattern around the universities external brands and partnering with opm that are active in the terrain for clustering is the times high education world rank same as in the South African maps and again if we look at those that are not ranked the majority of them also do not partner there's some that do partner but the majority do not and same in the lower ranked universities there are partnerships there but it's not as dense as when we go to those in the top 300 in the world which are dominated by the Russell group and the former 94 and that's where we see most of the partnering happening in the terrain so next we're going to have a look at individual institutions and as examples of how this trend is unfolding and we'll also look at how we can use the interactive tools to sort of unpack the underlying features of each institution first they have King's College and they are typical of the pattern we've seen emerge they are a Russell group university they are ranked in the top 136th in the world they partner with three companies to provide online programs and one to provide MOOCs and we have included the number of MOOCs and online programs that they are currently actively providing and they are typical of what we see the trend we see in the terrain the next example is an institution that is atypical of the trends we're seeing the University of Cambridge doesn't partner with any OPMs it is also a Russell group university it is ranked in the top 100 second in the world and it does not offer any MOOCs or any full online programs and it's getting stuck and this could be for a number of reasons we've discussed perhaps the high status, the high demand is already there they don't want to dilute the brand perhaps their teaching model is more small seminars but maybe that's something some of you all have ideas about afterwards why perhaps this is why the University of Cambridge is not partnering with the trend or the Russell group universities our next example again fits with the trend the University of Lincoln it is not aligned with any membership group it is ranked quite low between 600 and 800 in the world rankings it does not offer any MOOCs it does offer a couple of honours degrees online top of degrees but not through any partnerships with OPMs and our final example is another exception to the trend and that's Coventry University and Coventry is part of University Alliance which is a membership group which represents institutions that specialise in technical and professional education Coventry Coventry is not in the top universities ranked in the world it's between 601 and 800 according to the times higher education rank but we see that it partners with FutureLearn and you can see by the colour of their arrow partners with FutureLearn to provide MOOCs and online programmes full and online degrees and Coventry at the moment is the only UK University providing those full online degrees with FutureLearn at the moment this is a snapshot from last month and the other universities partnering with FutureLearn for that are Australian universities so Coventry has been quite active in terms of teaching and learning unbundling teaching and learning provision in the UK so they offer fully online programmes with FutureLearn that can be taken module by module on a page you go basis and you have the option to take the futureLearn and use those MOOC credits towards a full degree so as we have showed you at the beginning our desk research was only one phase of data collection and we have used the maps to contextualise some of the evidence that we have collected through our other data collection phases through the interviews, focus groups and surveys but what we have hoped to illustrate in this very short time is how we have used the interactive mapping techniques to gain this panoramic view in two places two national spaces and how we have weaved our conceptual framework of unbundling mobilisation and digital technology into visualisation through the visualisation features and interactive clustering tools. Thank you. Thank you so we are ready for we have got a few minutes for questions I have noticed the question of why are they moving around if people want to know why the map moves ok so this also has the social network analysis capabilities and what that does is try to find points of power so it will try and jiggle them around and show you where a point of power is now we are not really looking at those measures in this project but the reason it jiggles like close to futureLearn is because futureLearn partners with almost 40 institutions so that is a point of power so it jiggles around that point of power it is sort of like it is called gravity on the software and you can change the amount of gravity that is what makes it jiggle but it is just also fun Did you find a software after you already knew how you wanted it? We knew we wanted to like I said weave our conceptual framework into into these maps and a lot of social network analysis mapping software is for very big data sets so that you kind of just see these little nodes and lines and that kind of thing and we our data sets were not that big we wanted to see individual things so what was important when choosing the software was the decorative features so that we could use colour and borders so that is why we chose Croomoo because although it does have the social network analysis capabilities you can run the metrics on it it was just it was a good option in terms of for a small data set having those very strong decorative features I would just like to add to that that when we carried out the interviews that was where we came across CEOs of private companies saying we want to partner with the top 200 universities in the world so that is where we started looking at ranking and types of university so it was something that had come out of the earlier data collection that meant that that was the focus of how to put the variables into the mapping interested to get your appreciated is not the kind of focus of your research but to get your insight into whether you think this is a trend in terms of partnering with commercial organisations that is still growing and kind of where it is is it something which you perceive that is mature or is it immature that if you did this in five years time would you expect to see all of those unaligned institutions find a commercial partner I think it's certainly immature and we know that from the earlier maps that when we created the mapping at the beginning of the year it's changed already so for example in the South African mapping the University of Cape Town had no distant students but they were creating provision that you thought might be for distant students and now they've got some distant students but in terms of leaving those non-aligned universities behind I think that's a real risk I think that yes this will mature and in five years time the landscape might look very different but to what extent that will bring in all the universities I don't know this could really leave some universities behind especially with those universities that perhaps didn't fall into the higher rankings whether there was a desire by those institutions to engage in partnering and if they actually saw any particular barriers to that or whether they were comfortable in going to Commons to stay where they were and not partner and just plowler and furrow What we actually found from the interviews was that the more high ranking universities were almost saying with fed-up of people on the door with fed-up of getting emails with fed-up of people ringing this up but then you would get the much lower ranked universities saying nobody wants to partner with us we would love to develop this sort of provision and we were at an event yesterday where somebody said this sort of provision needs a disproportionate amount of funding at the beginning and the lower ranked universities just don't have that amount of money so if a private company doesn't come and help them do this they can't do it and so I'm not saying every university wants to do it but there are certainly universities that are not being approached that would like to start developing this sort of provision If you had to pick just one what is the most important conclusion from this work I think we're still working on that but I think it's that private providers and private providers are really keen to work with universities that enhance their brand that they don't just get the business they get the association with the high ranking universities sorry I saw another question which was easy that's what I'll take the software is called CUMU K-U-M-U I'll just do one more question before we finish another variable which would be interesting is the size of the ed tech team at those institutions to see if that has an impact that's interesting because at another presentation somebody said well why aren't institutions putting ed tech teams together to do this themselves and I think that means you've got to have the resource to do that you've got to have the skills within people who already work at that institution or be able to set up a team and when the digital education service at LEED started they had three or four and now they're into 40, 50, 60 because the university has decided to allocate resources to that and I think it would be really interesting to look at that and we are really interested in other people's views of what variables we haven't captured on these maps so I think yes that and I think you would find that the higher ranked universities with more money would have bigger ed tech teams and that has come up in discussion and I introduced the sort of dilemma of build or buy and whether that is possible or not or to purchase or partner and arrange between that spectrum building or buying Thank you, we'll leave it there and we are around, I'll have to know if anybody wants to come talk to us Thank you This is where I hope that the tech is seamless fantastic so I'm going to talk for about 20, 25 minutes about a project that we've been running at Manchester Metropolitan University in collaboration with the local secondary school which kind of picks up on some of the theoretical things that I've been thinking about recently around failure is that we don't really let our students fail so this came about kind of from three different areas and three different interests of mine so as I said I've been very interested in this kind of theoretical idea of learning through failure and I see this very much in our own students and my own time at university so I do remember when I was a student that actually failing and being kicked out of your year was an absolute option now I've never seen a student with institutions I've worked at fail for plagiarising for anything it's not something that I think we really is really something that students think about I also think that failure is really really important so for me, when I regularly fail to get papers accepted to get funding proposals accepted I see those as learning experiences and I kind of worry that we're not really giving students that kind of opportunity to accept and laud positive failure so all the things that I've been thinking about is in terms of my work around games and play is a construct called the magic circle and the idea of the magic circle is that it's a safe space in which you can fail now video games use this all the time people play video games because failure is intrinsic in them if you could play a video game and we're going to get it right first time it would be really boring it's that cycle of failure reconstruction, learning through failure that's really important in terms of video games so I've been thinking for a while about how you could take this kind of learning through failure model and apply it ethically and safely within higher education a recent other interest of mine is escape rooms so can I just get a sense of how many people have played in escape rooms here so about half and how many people know what an escape room is so who doesn't know what an escape room is okay so the idea is I've seen looks on people's faces of absolute horror it's in an escape room you and a number of other people usually 2, 3 or 4 are locked in a room often not really physically locked in and you have to solve a series of puzzles over the course of an hour or so to get out of the room and I first played this with a bunch of young technologists down in London about four years ago and all four of us came out going wow that's absolutely there's something about the time and the physicality and the being locked in a room and having to work with other people and having to collaborate was really powerful and over the last four years there's some really interesting work that's been done around using escape rooms for education Andrew Walsh at the University of Huddersfield has done some amazing stuff around escape rooms for libraries Helen Whitehead has done some sort of online escape room work Liz Cable at Leeds Trinity is an absolute master of the field in terms of getting she has boxes and boxes of this kind of escape boxes that I know she carries around in her car and the students absolutely engage with that and Samantha Clark particularly at the University of Coventry is doing some really good theoretical thinking around what might escape rooms for learning look like and that was the kind of original space that I was thinking in but we worked from 2012 to 2014 and I was involved in a project called Magical which was looking at how we could get children in years 7, 8 and 9 in high school to engage in collaborative learning through digital game making and this really undermised the kind of power of collaboration and in terms of skills such as problem solving collaborative learning teamwork, creativity and just kind of lateral thinking and thinking outside the box now whilst the outcomes of that project were genuinely positive one of the big issues that we had was around the technical provider, the ability to have game making software the ability then to take that software and use it on different machines it's basically a lot of technical issues and for me that's when it all came together of thinking about what can we do to learn through failure how could I do something cool with escape rooms and how could we build on the principles of learning through game making that we developed in this previous project and it came from a chance conversation with one of our ex-students who happens to be a senior maths teacher at a local school and me saying I'd really like to do something with escape rooms and failure and him saying I've got a bunch of really good sixth formers and they're doing an enrichment project they've got an enrichment week and we don't know what to do with them so I forgot to use my slides excellent, I've just got two inverted in talking that I forgot to actually use my slides so that's my slide that's just the point I've just made about failure so I'll move on from that so this was talking about the magical project so the this is when the edge escapes project was born and we've actually run it for three years in collaboration with a high school in Cedle Hulm which is just about five miles south of Manchester in the second year we got some funding to Morasmus Plus Network funding from a consortium called Learning Games which is really nice because that's given us the opportunity to take some ideas and build on them with international partners the big problem with it has been fitting it into curriculum despite having a senior person in the university who's totally committed to the project because it's with students who are half way their AS level year the curriculum by that point is so full that there's very limited spaces in terms of being able to fit in something that's cross curricular and something that teaches these kind of failure based skills so in the first year we had 12 students who were pretty much cherry picked because it was a bit of an experiment and they developed escape rooms during the two week enrichment period in year two based on feedback from the first time we used a smaller one week enrichment period and it worked both times and we got a lot of very good feedback but the issue for me was that it was very much the best students the brightest and the best we had a small cohort, we had thrown a lot of resource into it and I was very interested to see whether it was something that could be expanded beyond kind of the best 12 in the school so based on our funding for the Learning Games project we decided in the final year that we would go for a different model so instead of having it pushed into an enrichment period we'd run it as an optional work experience type activity, one hour a week over six months and we'd open it up to 40 students so we'd have a much different student demographic and see if we could which was much more representative of the types of ideas we were looking at I mean it's all very well to have a model that works very well but if it's not inclusive there's not a great deal of point and what I wanted to talk to today is kind of the ideas and models that came from that and some of the findings of the research that we've done so this is the sort of model fairly based learning that it was built around with three key areas first being initiation, development and then presentation and the idea of this in the initiation period it's about introducing students to the escape rooms and about giving them the opportunity to build up their teams so we take them to a commercial escape room and play games with them so the teams that are going to work together actually go and work through an escape room we've then developed a kind of training programme which starts with looking at how you design puzzles the different types of puzzles you might use and looking at how you then would go through the steps to put an escape room together in the first two years we ran this as a single day in the last year we ran this over the course of the first month of the project and getting people together and that's very much about taking time to team build to scene set and to make sure that students know what they're aiming at then we have this kind of failure based development phase and the whole point of getting it in an escape room is that you're designing something for other people and you're never going to get it right first time and that's absolutely fine that's part of an accepted part of the process that you will have to test it and you'll have to test it over and over again and that seeing how other people interact with your puzzles and with your theme is part of the learning experience so this cycle of test it, fail it reflect, revise is absolutely integral and one thing we say to students is that you need to come up with phases of it so generally we'll say come up with your theme and then come up with your puzzles test all of your puzzles and test them again watch other people doing it get other people to think aloud while they're working through it because what we've found is that teams will come up with something they think is really obvious and people solve it just don't get it or something that they think is really difficult that people solve straight away so that sort of process is absolutely crucial and then a couple of times during the period we'll send in people from the university so again I've sort of brought in friends who are game developers or professionals and it's that kind of bringing a professional in and giving them the sort of professional feedback that's been quite key to it and then finally there's an event and we run the playful learning conference which runs every year in July and for the last three years we've brought these students in given them conference passes, they've been part of the conference but also they've been running the rooms for the delegates at the end of the conference and this has been the really key moment for them at the end of last year we had students going I didn't realise it was a real thing and suddenly they'd stepped up because they had this new level of professionalism because there was this professional presentation at the end so while it's absolutely key to this that there isn't an assessment this presentation is the point at which actually you've done all your failing and then you've got a big public celebration at the end and it's been very important for the students now one of the reasons for going into the physical is to get away from the problems that we have when we're game building with digital games but actually there are lots and lots of ways that you can build technology into this and it's been quite eye-opening the way that the students have kind of taken this and run with it themselves so one of the things was using digital puzzles so using various types of electronics so there have been puzzles that have been around unlocking iPads, finding codes and one of the groups in the first year built this amazing electronic box with internal circuitry that you had to plug in three various things at different points that would open the box I think it allowed lots of creativity in terms of digital as well as physical and mechanical puzzles I was quite surprised in how many of the groups got really creative in terms of scene settings so they put their theme we had one group that had ambient music at different points along with different puzzles lots of groups that went to a lot of trouble producing videos that would allow people to come to do kind of do the scene setting for them so there was a lot of creativity there a one thing that we asked groups to do when people are in the room, there has to be a way of monitoring them so we've got groups who've set up go-pros who've set up various camera solutions and ways of getting information so some have used different types of computers or way laptops and tablets and digital ways of getting information to participants on the other example we've had some groups that have had a physical person in the room and used that as a way so it allows you to be as digital as you want equally you can go completely analogue and we've also been quite impressed with the kind of project management tools and different digital tools that students have used in terms of being able to design their room being able to test their room being able to monitor and reset their rooms really quickly a lot of them have done that digitally so what I like about this is it doesn't you could do the whole thing without engaging in digital at all you could make it a very very high tech digital room but it allows groups to engage in the digital as much as little as they want to so for this final year we ran some more structured research around the project so this kind of had three questions and three different types of data that we were looking at so we were interested in student learning and we had prophesied that student learning would be around team building that it would be sorry was that a time thing there would be around collaboration then it would be around creativity and problem solving so we used a post and pre evaluation now obviously we didn't have control group so the significance that we could get from this was fairly limited anyway but I was interested in whether things around their confidence in these areas were going to change so what we learnt from this is that students won't fill out a post questionnaire once they've left the project and so we've got very limited data on that but there's some quite interesting qualitative stuff coming out of that too in terms of trying to work out with the student experience did they like it or did they feel that they got out of it we've got quite a lot of rich qualitative data so we've got an RA to come in and interview all of the student groups over the one day of the conference that they were that they came to run their rooms out and that's kind of the analysis I'm going to talk about here we're also interested in the staff experience so we do have some qualitative data from the three staff members that run the project as well and that's backing up some of the data that I'm going to share with you so we're at the point now of having the interview transcripts and having some of the qualitative data from the pre and a few post questionnaires and we plan to do a detailed both thematic and discourse analysis on it at the moment we've done a very early kind of thematic analysis just to see what the main things that come out and I'm going to share that with you now so in terms of the key findings something that came out this year which didn't really happen in previous years and we should think it's to do the size of the cohort is that the whole process of managing failure is not easy so despite the fact that it was always we were very clear with students all the way along that you will not get this right first time you have to accept critical feedback this is part of the process it's about change and iteration and refinement we still on the time the first time that myself and colleagues went in to give students feedback we got quite a lot of feedback from staff that students had found it quite demotivating because we'd been critical but positive and that they hadn't they hadn't been used to receiving that kind of feedback now whether that's something to do with how we weren't horrible to them but we were very constructive in terms of this bit isn't really going to work you're going to need to look at that again so I think they hadn't quite been expecting about this time whereas in previous years they were getting much more close engagement with members of staff because it was over a short period of time so we're really thinking about how we set students up to accept that failure is part of the process and to manage it and to deal with it constructively and that perhaps we need to put more support in around doing that and a more preparation for the students in terms of this is how how you accept how you learn from the feedback that you're being given but the structure is crucial so coming back very strongly that the time spent in scene setting and doing initial training and support and playing was really key equally the students really really value the fact that they could come to a public conference at the end and that they're engaging with the general public and what they said at the end was actually we want to go back and do this again can we go and do it to the the governors and we want our parents to come in and do it this sort of feeling of celebration for them came through very strongly but also the idea of this iterative stage and if anything what we didn't do was perhaps chunk it up more in terms of development because realistically they were having to have it tested in times to get something good and I don't think they had the feel for exactly how much of this iteration was going to have to happen so in terms of future work that something we'll definitely be looking at the middle section needs to be much more structured in terms of the learning outcomes well we didn't gather any evidence on things that we did expect there was some interesting things that came out that we hadn't expected one was very much around the kind of power relationship between the students and their teachers and because they'd got their teachers to play the games this had opened up this new avenue of oh my teacher is doing something that I've made that they can't solve and it really I think in terms of building the students confidence and their ideas of what they could do it was absolutely amazing this came up from lots and lots of the students this feeling of independence as well now this I think we didn't have any evidence of this in the first two years but I think this was in the third year at a time that they liked being trusted that they really felt they'd gone off and done a thing and I think also this idea they really got that other people don't think like them and I'm not sure that there'd ever be a chance to sort of see that as obviously as when you're watching somebody try and solve the thing that you've produced and several of them I was talking to said I just never thought that and it's really opened my eyes to people are different and if I'm producing a thing there's lots of different takes on it so I think in terms of those learning outcomes we're quite interesting that we're coming out from it so words of project going now we've got three universities in the UK who are taking it and going to be using it with their students which is great I'm really interested if anybody else wants to take the model we have a website everything's available online I'd be really interested if anybody else wants to take it and run with it with their students we've received some funding from Erasmus Plus so we're going to be running a two year project which we'll be taking it into schools in six countries to try and sort of embed it my issue with it is I think it's got a lot of potential but it works much better with small groups and high input whereas I think once to make it valuable it needs to be much more accessible to larger groups and how you can then put that into different curricula so we're looking at six different curricula six different countries to see how we can push it forward so thank you very much for listening please contact me if you'd like to know anything about the project my emails there and that's the project website does anybody have any questions are they constructing escape rooms to explore a certain topic when they started in the first year the colleague that I was working with was a maths teacher and I'm a computer scientist so we were interested in maths and computer science but it turned out that that was kind of adding another layer of complexity that the students were finding it quite hard sometimes to come up with puzzles anyway and that making them all maths computer science based wasn't a massive problem but it just ended up that some of them were incredibly complex and required certain prior knowledge which is kind of against the ethos of an escape room so in the previous, in the subsequent years we didn't have any sort of curricula in terms of what the puzzles were about because it was very much about cross-curricular objectives so next one I have a question here too so there's a couple of questions on the screen which have been on the tour I wanted to ask too about being a gamer myself, I get very motivated by the random badges that show up on my PS4 which I wasn't necessarily welcome to show up and have more ability to do new things and find new things in the game and I just wondered given that I see badges everywhere whether you could potentially use badges here not only for motivation but also in terms of showing the things that the kids and people have learned through failure and distance outside of the project yeah I think, that hadn't occurred to me but now you've said it, it's ringing lots of bells I think part of it was why would the students do it and we'd always said well it's something to tell acas about but we'll give you a letter but actually they'd never come back and ask for that I think by the end of it they've retrospectively kind of enjoyed the experience I think this year because it took so long because they were only doing it for an hour a week there were certain points when it kind of lost its impetus and the staff had to get back in there now I can then see actually if we are putting more of a structure in and we did that around badges that each group has to collect certain badges at certain times that would not only allow you to structure it but it would allow you to see where you were in comparison to other groups and I think because there were more groups this time the first two years they were all working in the same spaces and there was a lot of competition between the groups we actually ran at this competition and gave prizes we didn't do for various logistic reasons in the final year but actually I think being able to see that group's done 10 iterations or that group's done failed 20 times could actually be really much I might come and talk to you about that are there any other questions in the room if you have time I'll pick up one more if university education was a board game would it be at £27,000 peace puzzle, trivial pursuit or the beginning of a very long game of Monopoly or a bit of each maybe I can't see the time but I suspect it's moving onwards so I'm going to stop here thank you for listening I'm around for the whole conference I'm very happy to talk to anyone please get in touch I'm John Wilson I'm the CEO at Agenta we're a technology company that focuses on education and learning we build, manage and operate platforms for education for video collaboration externally we prefer to work with what we feel as ethical industries obviously education, teaching, learning healthcare to these industries by creating exciting platforms easy to use platforms secure platforms that people can utilise what we feel is one of the most important things for Scotland to boost economic growth is investing in rural areas by investing in broadband in these local areas we can attract more talent we can attract more companies and we can drastically improve the delivery of education and learning within these schools within Scotland


My name is Vernon Gale, I'm Professor of Sociology and Social Statistics at the University of Edinburgh and I'm part of the National Centre for Research Methods. I hate appearing on video, however in this short presentation I'm going to tell you about Jupiter notebooks and how they can help benefit social science research. If you're a social science researcher and interested in improving how you undertake statistical data analysis then this video is designed to introduce you to Jupiter notebooks. I'm a fairly recent convert to using Jupiter notebooks but in the next 10 minutes I hope to convey some of my enthusiasm and to encourage you to consider using them in your research. But first I'll start by talking about the social science workflow. Having a planned and organised workflow is essential for high quality statistical research using large scale social surveys or administrative social science data sets. The workflow refers to a coordinated framework for conducting social science data analysis. The workflow includes planning, organising, executing and documenting analysis. Jay Scott Long has provided an extensive and almost rabbinical account of good workflow practices. I suggest that any data analyst who has not read Long's book would benefit from doing so whatever their age or their career stage. Central to a successful workflow is the audit trail. The audit trail is nothing more than a chronological account of the activities undertaken in the data analytical process. Alternatively you can think of the audit trail as a line of breadcrumbs helping you navigate through the research process. The audit trail is important because within the statistical analysis of social science data sets a minor decision such as dropping some cases from an analysis or recoding a variable can have major consequences later on. Keeping track of even the most seemingly inconsequential actions within the workflow is important as it facilitates transparency and makes contributions to efficiency and accuracy and ultimately to the overall success of the research project. My acquaintance Philip Stark at UC Berkeley says that not having a planned and organised workflow can be compared to drinking and driving. In both cases it doesn't matter how careful you are it's still highly likely to end in a wreck. In just the same way as I would never advocate drinking and driving I also don't advocate undertaking research without a planned and organised workflow. There's a long history in the natural sciences of researchers continuously making notes that contribute to high quality documentation. The Nobel Prize winner Linus Pauling used bound notebooks to keep track of the details of his research and the 46th notebook spanning a period of 1922 until 1994 are available online. Professor Pauling's notebooks include calculations, experimental data, scientific conclusions, ideas for further research and numerous autobiographical reflections. For example, notebook 24 on page 151 contains an entry detailing his golden wedding anniversary. But the use of notebooks goes back much further. For example, Galileo used notebooks which directly integrated his data. For example, drawings of Jupiter and its moons with key metadata. For example, the timings of each observation, the weather and even telescope properties. The data and the metadata were annotated and the text which included descriptions of methods, analyses and scientific conclusions were woven together. This links us neatly to Jupiter notebooks. The three Galilean moons are visible in the Jupiter logo. Why is it called Jupiter? Well, the computer languages Julia, Python and R almost spell Jupiter. So that's why it's called Jupiter. Jupiter notebooks are currently used in big science. The recent detection of the gravitational waves is heralded as a major scientific discovery. If you follow this YouTube link, you can watch a short video of Fernando Perez who first conceived the Jupiter notebooks, demonstrated a Jupiter notebook that includes data and analyses of the first gravitational waves detected by the LIGO team. What are Jupiter notebooks? They are an open source web application that facilitates the creation and sharing of documents that contain live code and supporting commentary in the form of explanatory text. It's a platform that can be used throughout the research process to organize and articulate elements of the social science workflow. The Jupiter notebook is open source and supports interactive data analysis in over 40 programming languages. What do Jupiter notebooks offer social science data analysts? First, they facilitate easy documentation alongside research code. They have good portability because notebooks are easy to share. They are language agnostic and analyses can be undertaken using many different languages. They can produce rich visual outputs. They can leverage big data research tools, for example using Python. They can be used as integrated tools in teaching, training, knowledge exchange and research capacity building. And finally, they support and facilitate collaborative work. Now, let me quickly show you around a Jupiter notebook. Cells can contain three things. Live research code, for example, STATA or R syntax that can be executed. In this case, it's a STATA command. Cells can contain the results of data analyses. Here we see the output for the STATA command. And finally, cells can contain text comments that form the documentation of the research workflow. Documentation in Jupiter notebooks is relatively easy. You can use Markdown, which is a simple and easy to learn form of plain text. Jupiter notebooks are language agnostic. It's possible to work in many languages within a notebook. I'm going to show you a few screen grabs now of a statistical model estimated within a Jupiter notebook. Within the notebook, I've estimated a logistic regression model first in STATA, then the same model in R and finally the same model again in Python. These models have all been estimated within the same notebook and hopefully illustrates that I can simply and easily move between different data analysis programs within a single Jupiter notebook. The next example involves rich text outputs. I had a great wee group of PhD students a few years ago who absolutely loved XKCD web comics. So just for them and they know who they are, I've produced a plot in the style of an XKCD graphic to show the graphing ability of Jupiter notebooks. Continuing on the theme of rich visual outputs, here's an example that uses an open source street map. I've recently moved to a more commodious office around the corner in Brackeau Place in Edinburgh and here's an example of embedding a map within a Jupiter notebook. The inclusion of maps offers a great deal of potential, especially when working with geocoded data. One of the many exciting features of Jupiter notebooks is their interactive facilities. This next example uses image processing to identify galaxies in an image of the sky provided by the Hubble Space Telescope. This is a live example hosted by the journal Nature and after running the cell we can explore the parameters of the detection algorithm to find galaxies of different sizes and prominences. Just before I finish, I want to direct you towards Professor Lorena Barber's website. Lorena demonstrates the value of using Jupiter notebooks in research-informed teaching by weaving executable code within multimedia contexts. In a nutshell, when you use a Jupiter notebook for social research you end up with an uberstatter.do file or an uberr script that contains your research code and your output woven into a literate research narrative. Jupiter notebooks can be converted into other formats, for example PDF or HTML ready for presentations, for publication, for collaboration and for sharing. Hopefully, this will have helped convince you of the benefits of using Jupiter notebooks and how they have an obvious appeal to undertaking reproducible research. It's very easy to install Jupiter and you can get started relatively quickly. Very soon you'll be able to use a Jupiter notebook standing on your head. Well, almost. But on a more serious note, I'll conclude by saying that overall Jupiter notebooks offer a useful and usable environment in which to plan, organise and execute your data analysis while simultaneously being able to record, document and archive your research results alongside the code that produced them. Jupiter notebooks have the capacity to transform how statistical analyses using large scale social surveys or administrative social science data sets is routinely undertaken. Good luck.


Hi, I'm Matthew Coast head dating coach and founder at commitment connection calm and today We're gonna talk about what to do when he doesn't text back. So here's what you should do So let's first talk about why guys don't text back sometimes and and here's a few of the reasons why a guy might not text you back One is that he's busy, right? so he might have other things going on in his life and he's You know taking a shower or he's at work or he's got something that's important going on that he needs to pay attention to and so he Can't contact you back Another one is that he thinks that you're acting needy So a lot of times when a guy feels like a woman's acting needy He'll just cut off contact and just be like oh god I'll just wait until she's not acting needy and then I'll try to talk to her which can drive you crazy but It's something that you should know is that a lot of times guys won't contact you back if they think that you're acting needy So when he isn't really that into you, he might not contact you back. He might not text you back and and That's also an important thing to know If he's not near his phone, so maybe he left his phone somewhere Maybe he's doing something and he left his he didn't want to keep his phone with him So if he's not near his phone, he's probably not not gonna I would assume that he wouldn't be able to text you back then Or if this is kind of a Rough one, but it could be if he's with another girl So if you're just casually dating each other you first met each other and he's with another girl He might not text you back in that situation So here's what you shouldn't do when a guy Doesn't text you back. What you shouldn't do is you shouldn't shame him or condemn him a lot of women do this because they think that I Don't know that this will work. Has this ever worked. I mean, does it ever work long term? I've never seen shaming or condemning anyone ever work long term But people still do it and it doesn't work But it will probably lead him to never wanting to talk to you again more than Doing what it is that you want him to do and and it what it could also do is make him start hiding Things from you if you condemn guys or shame guys for doing things that you don't want him to do a lot of times Just start hiding that from you Which isn't good. You don't want that So Yeah, I mean treating him Another thing that you shouldn't do is you shouldn't treat him like dirt Because he didn't respond sooner like treating him badly or or just you know being passive-aggressive about the whole thing that's Not a good thing to do it's not attractive and it could make him pull away and disappear all together So here's what you should do Have something else that you can put on to put on your mind, right? Have something else that you can focus on and and You know if If he Isn't texting you, you know and you're starting to freak out about it or get obsessive about it go do something, you know go Do something that you like right and which kind of leads into the next one Which is having passions and hobbies right if you have passions and hobbies that you're interested in you have a life That you live that's really interesting Then you know if he doesn't text you back It's not a big deal because you're doing things and you know, what's the big deal if he's not texting you back?


From theCUBE Studios in Palo Alto in Boston, connecting with thought leaders all around the world, this is a CUBE Conversation. Hello everyone, welcome to this CUBE Conversation. I'm John Furrier, host of theCUBE. We're doing a content series called Leading with Observability, and this segment is Network Observability for Distributed Services, and we have CUBE alumni, Mike Cohen, Head of Product Management for Network Monitoring at Splunk. Mike, great to see you, it's been a while. Going back to the OpenStack days, Red Hat Summit now here, talking about observability with Splunk, great to see you. Thanks a lot for having me. So the world's right now observabilities at the center of all the conversations from monitoring and investing infrastructure on premises, cloud, and also cybersecurity. A lot of conversations, a lot of broad reaching implications of observability. You're at the Head of Product Management for Network Observability at Splunk. This is where the conversation is going, getting down into the network layer, getting down into the packets move around. This is becoming important. Why is this the trend, what's the situation? Yeah, so we're seeing a couple of different trends that are really driving how people think about observability, right? One of them is this huge migration towards public cloud architecture, and you're running on an infrastructure that you don't own yourself. The other one is around how people are rebuilding and refactoring applications around service-based architectures, scale-out models, cloud-native paradigms. And both of these things, they're really introducing a lot of new complexity into the applications and really increasing the service area of where problems can occur. And what this means is when you actually have gaps in visibility or places where you have separate tools, analyzing parts of your system, it really makes it very hard to debug when things go wrong and figure out where problems occur. And really what we've seen is that, people really need an integrated solution to observability and one that can really span from what your user is seeing, but all the way to the deepest back-end services. Where are the problems in some of the core infrastructure that you're operating so that you can really figure out where problems occur? And really never give visibility is playing a critical role in kind of filling in one of those critical gaps. You know, you think about the 10 years past decade we've been on this wave, it feels like now more than ever, it's an inflection point because of how awesome cloud native has become from a value standpoint, value creation, time to market all those things that why people are investing in modern applications. But then as you build out your architecture and your infrastructure to make that happen, there's more things happening. Everything as a service creates new dependencies, new things to document. This is an opportunity certainly on one hand and on the other hand it's a technical challenge. So balancing out technical debt and or deploying new stuff, you got to monitor it all, right? Monitoring has turned into observability, which is just code word for cloud scale monitoring, I guess. I mean, is that how you see it? I mean, how do you talk about this? Because it's certainly a major shift happening right now and this transition is pretty obvious. Yeah, no, absolutely. And we've seen a lot of new interest into the network visibility, network monitoring space. And really again, the drivers of that like, network infrastructure is actually becoming increasingly opaque as you move towards public cloud, kind of public cloud environment. And it's been sort of a fun meme to blame the network and say, look, oh, it's the network, we don't know what's going on, but it's not always the network. Sometimes it is, sometimes it isn't. You actually need to understand where these problems are really occurring to actually have the right level of visibility in your systems. But the other way we started talking to people think about this is the network as an empowering capability, an untapped resource that you can actually get new data about your distributed systems. SREs are struggling to understand these complex environments, but by, with the capabilities we've seen and started taking advantage of with things like EBPF and monitoring from the OS, we can actually get visibility into how processes and containers communicate and that can give us insights into our system. It's a new source of data that actually has not existed in the past that is now available to help us with the broader observability problem. You mentioned SRE site reliable engineers, as it's known, Google kind of pioneered this is become kind of a standard persona in large scale kind of infrastructure, cloud environments and whatnot like massive scale. Are you seeing SREs now that role become more mainstream in enterprises? I mean, because some enterprises might not pull on SRE medical or cloud architecture. I mean, can you just help us, to get that tie that together? Cause it is certainly happening. Is it becoming proliferated? For sure, absolutely. I think SREs are because the title may vary across organizations as you point out and sometimes the exact layout of the organizational breakdown varies, but this role of someone who really cares about keeping the system up and caring for it and scaling it out and thinking about its architecture is now a really critical role and sometimes that role sits alongside developers who are writing the code. And this is really happening in almost every organization that we're dealing with today. And it is becoming a mainstream current. Yeah, it's interesting. I'm going to ask you a question about what businesses are missing when they think about how to think about observability, but since you brought up that piece, it's almost as if Kubernetes created this kind of demarcation between the line between half the stack and the top of the half and the bottom half of the stack where you can do a lot of engineering underneath the second half of the stack or the bottom of the stack up to say Kubernetes. And then above that, you could just be infrastructure as code application developer. So it's almost kind of like leveled out with nice lanes there. I mean, I'm oversimplifying it, but I mean, how do you react to that? Do you see that evolving too? Cause it all seems cleaner now. It's like you're engineering below Kubernetes or above it. Well, absolutely. It's definitely one of the ways you see sort of the deepest engagement and as folks go towards Kubernetes, they start embracing containers. They start building microservices. You'll see development teams really accelerate the pace of innovation that they have in their environment. And that's really the kind of the driver behind this. So we do see that, that sort of rebuilding refactoring as some of the biggest drivers behind these initiatives. What are businesses missing around observability? Cause it seems to be a, first of all, a very overfunded segment, a lot of new startups coming in, a lot of security vendors are in, you're seeing network folks moving in, what's almost becoming a fabric feature piece of things. What is it mean to businesses? What are businesses missing or getting? How are people evaluating observability? How do you see that? Yeah. So I'll, for sure, I'll talk, I'll sort of initially talk generically about it, but then I'll talk a little bit about network area specifically, right? That I think one of the things people are realizing they need in observability is this approach that's an integrated suite. So having a disparate set of tools can make it very hard for SREs to actually, take advantage of all those tools, use the data within them to solve meaningful problems. And I think what we're seeing, and as we've been talking to more people in the industry, they really want something that can bring all that data together and build it into an insight that can help them solve a problem more quickly, right? So that, I think that's the broader context of what's going on. And I think that's driving some of the work we're doing on the network side, because network is this powerful new dataset that we can combine with other aspects of what people have already been doing in observability. What do you think about programmability? That's been a big topic, but you started to get into that kind of mindset.  How does that play in? What's your vision around making the network adaptable, programmable, measurable, fully, fully surveilled? Yeah, so I think what we're focused on is the capabilities you can have in using the network as a means of visibility and observability for systems. Networks are becoming highly flexible. A lot of people, once they get into a cloud environment, they have a very rich set of networking capabilities, but what they want to be able to do is use that as a way of getting visibility into the system. So to talk for a minute or two about some of the capabilities we're exposing, use it in network observability, one of them is just being able to visualize and optimize a service architecture. So really seeing what's connecting to what automatically. So we've been using a technology called EVPF, the Extended Berkeley Packet Filter, part of everyone's Linux operating system, right? You're running Linux, you basically have this already. And it gives you an interesting touch point to observe the behavior of every processing container automatically, and you can actually see with very low overhead what they're doing and correlate that with data from systems like Kubernetes to understand how distributed systems behave, to see how things connect to other things. We can use this to build a complete service map of the system in seconds automatically without developers having to do any additional work without forcing anyone to change their code, they can get visibility across an entire system automatically. That's like the original value proposition of Splunk when it came out, it was just a great tool for Splunking the data from logs. Now, as data becomes more complex, you're still instrumenting. And these are critical services. And they're now microservices. There's the trends at the top of the stack and at the network layer. The network layer has always been a hard nut to crack. I got to ask you, why now? Why do you feel, and you mentioned earlier, everyone used to blame them, oh, it's not my problem. You really can't finger point when you start getting into full instrumentation of the traffic patterns and the underlying processes. So it seems to be good magic going on here. What's the core issue? What's going on here? Why is it now? Why is the time now? Yeah, so, yeah, for, well, so unreliable networks, low network, DNS problems. These have always been present in systems. The problem is they're actually becoming exacerbated because people have less visibility into them. But also as you have these distributed systems the failure modes are getting more complex. So you'll actually have some of the longest, most challenging troubleshooting problems are these network issues which tend to be transient, which tend to bounce around the systems. They tend to cause other unrelated alerts to happen inside your application stack where multiple teams would troubleshooting the wrong problems that don't really exist. So the network is actually caused, some of the most painful outages that teams see. And when these outages happen, what you really need to be able to know is, is it truly a network problem or is it something in another part of my system? If I'm running a distributed service, which services are affected? Because that's the language now my team thinks about. As you mentioned, now they're in Kubernetes. They're trying to think which Kubernetes services that are actually go, affected by a potential network outage that I'm worried about. The other aspect is figuring out the scope of the impact. So are there a couple instances in my cloud provider that aren't doing well, is an entire availability zone having problems? Is there a region of the world that's an issue? Understanding the scope of this problem will actually help me as an SRE decide what the right mitigation is. And by limiting it as much as possible, it can actually help me better hit my SLA because I won't have to hit something with a huge hammer when a really small one might solve the problem. Yeah, this is one of the things that comes up and you almost just hearing you talk, I'm seeing how it could be complex for the customer, just documenting the dependencies. I mean, as services come online, someone's going to be very dynamic, not just at the network, both the application level, we mentioned Kubernetes and you got service measures and microservices. You're going to start to see the need to be tracking all this stuff. And that's a big part of what's going on with your suite right now, the ability to help there. How are you guys helping people do that? Yeah, absolutely. So, just understanding the dependencies is one of the key aspects of these distributed systems. This began as a simple problem. You have a model effect application that kind of runs on one machine, you understand its behavior. Once you start moving towards microservices, it's very easy for that to change from, look, we have a handful of microservices to we have hundreds to be a thousand. And they can be running across thousands or tens of thousands of machines as you get bigger. And understanding that environment can become a major challenge. Teams will fill end up with the handwritten diagram that has the behavior of their services broken out, or they'll find out that there's an interaction that they didn't expect to have happened. And that may be the source of an issue. So, one of the capabilities we have using network monitoring out of the operating system with EVPF is we can actually automatically discover every connection that's made. So if you're able to watch the sockets that are created in Linux, you can actually see how containers interact with each other. And you can use that to build automatic service dependency diagram. So without the user having to change the code to change anything about their system, you can automatically discover those dependencies and you'll find things you didn't expect, you'll find things that change over time that weren't well documented. And these are the critical level of understanding you need to get to in this environment. Yeah, you know, it's interesting, you mentioned that you might have missed them in the past, people have that kind of feeling at the network, either because they weren't tracking it or they use a different network tool. I mean, just packet loss by itself is one service and host health is another. And if you could track everything, then you got to build it in. So I love this direction. My question really is more of, okay, how do you operationalize it? Okay, I'm an operator, am I getting alerts? Do I just auto discover? How does this all work from a usability standpoint? How do I, what are the key features that unlock? What gets unlocked from that kind of instrumentation? Yeah, well, again, when you do this instrumentation correctly, it can be really, it can be automatic, right? You can actually put an agent that might run in one of your instances, collecting data based on the traffic and the interactions that occur without you having to take any action. That's really the holy grail. And that's where some of the best value of these systems emerge is it just works out of the box and then, you know, pull data from other systems like your cloud provider, from your Kubernetes environment and use that to build a picture of what's going on.  So Mike, I got to ask you a final question. Explain the distributed services aspect of observability. What should people walk away with from a main concept standpoint and how does it apply to their environment? What should they be thinking about? What is it and what's the real story there? Yeah, so I think the way we're thinking about this is how can you turn, you know, the network from a liability to a strength in these distributed environments, right? So, you know, what it can, you know, by observing data at the network level and, you know, out of the operating system, you can actually use it to automatically construct service maps to learn about your system, improve the insight and understanding you have of your complex systems. You can identify network problems that are occurring. You can understand how you're utilizing aspects of the network.  So you can actually get better insight and, you know, be able to troubleshoot problems better and handle the blame game of, is the network really the problem, you know, that I'm seeing or is it occurring somewhere else in my application? And that's really critical in these complex distributed environments. And critically, you can do it in a way that doesn't actually add overhead to your development team. You don't have to change the code. You don't have to, you know, take on a complex engineering task. You just, you can actually deploy agents that'll be able to collect this data automatically. Awesome, and take that complexity away and automate how people get the job done. Great, great stuff. Mike, thanks for coming on theCUBE. Leading with Observability, I'm John Furrier with theCUBE. Thanks for watching. Yeah, thanks a lot.