the stakeholders will involve a legal entity and it will involve a decision and revision process for the rules of participation. There's no point in having rules if you cannot have some way of enforcing them or at least monitoring them. However, there's no point in having rules if complying with the rules is so difficult that nobody actually wants to participate in it. So we have to make that balance very carefully. This version of the rules does not address a couple of things. It doesn't address the issues around how the rules themselves will be monitored, enforced, maintained, changed, involved, etc. And it doesn't concern itself with the legal basis where I suppose that work is being done elsewhere. Finally, for this section, a comment about the timing of the rules where because EOSC is because we're working to very tight time scales within EOSC, the rules of participation are being developed concurrently with other working groups which are defining things around fairness of data, defining the architecture for the EOSC, defining how EOSC will be sustainable, etc. And clearly the rules of participation need to incorporate elements from that work. So in the current version of the rules, there are some placeholders which refer to the outputs of that work which will be filled in later as those other groups develop concepts that need to be incorporated into the rules. We use the term rules throughout. It's not a perfect word. It means very many different things. And in fact, there's a broad range of different types of condition that are in the rules. Some of them might be considered terms and conditions of use. Some of them might be considered acceptable use policies, etc. So the rules is an all-encompassing term that has many different levels within it. One thing, as I said, we're not covering our legal regulations which we'll be looking at later in this process. So current status of the rules. Here, I'll go through the timeline that we're working to and then the actual rules. The first version of the rules were developed during last year and were reviewed by the EOS Detective Board and the Government Board in December of last year. And the feedback we got from that was put into the version that's currently open, version 02, which was put up on the EOS Secretariat website with a consultation period that rents between March and April of this year. And from that feedback, version 03 is currently being prepared. Which will then be opened again for further feedback from the community during the summer this year with a further version to be developed by the end of the year with the stakeholders event made this year as a clear milestone for further consultation. But today is also an opportunity for consultation. So when we come on to the third part of the talk, you'll see there are several ways that we're going to gather feedback from you today and the next few days. It's worth noting that the current rules are very deliberately as short as we could possibly make them. And our view here is that the final rules will also be very short and any issues that need discussion or commentary will be put in a separate place. So each rule will have a paragraph or whatever is appropriate discussing the issues that lie behind it. But we want to keep the actual rule text as short as possible. We're well aware that these rules need refinement. I've just gone through the process we'll go through for the rest of this year to do that. And that we're waiting for other working groups also to give their input to these rules. So if you want to have a look at the rules themselves, they're still available from download from the EOS Secretariat website. If you go to the rules participation web page, you can see there the version of two of the rules is downloaded. So briefly to go through the rules themselves, there are four sections in the rules. There's a section on some ground rules, some of the Rashi principles if you like. There's a section on data, there's a section on services, and there's a section on the operators rules for the operators of EOS itself. So firstly looking at the ground rules, and this slide contains the full text of the rules. So I won't go through it all, but you'll see it is reasonably short. The first basic rule is that EOS is open to all the openness principle applies across all the resources across all potential users, etc. However, of course, things can't be completely open all the time. There are issues around privacy, etc. And those issues where we're not going for fully open openness, those issues will be delegated to the specific resources to make their own terms and conditions. And we would, the rules here just say that those terms of conditions must comply with the principles of openness to find a mutual sponsorship. The second ground rule, and this is perhaps a bit more of a definition than a rule. It's saying the resources, the EOS resource, if and only if it's registered in an EOS recognized catalog of resources. So something becomes part of EOS when it registers as EOS, and it's part of EOS only if it's registered in a catalog. Notice we say a catalog, a recognized catalog of resources. There will be many of these. So there's a two tier system where centrally we recognize catalogs and those catalogs recognize resources. And as part of the registration, we will, this will be the point at which resources have to indicate their compliance with the rules of participation, as well as any end-report onboarding requirements for those particular systems. That's covered further later. Finally, we put in this is probably, sorry, finally we put in this is probably for enforcement later, and this isn't the case right now. The use of EOS branding is available to the registered resources. So at some point in the future, we will say that you can only call yourself part of the EOS if you have gone through that registration process. At the moment we're in a sort of very open stage where the use of the term is rather relaxed. Moving on to data. We have six rules of data and these are almost mirrored in the rules for services. There are just slight differences between rules of data and rules of services. The first one is about resources being free of charge for data resources. The policy here is quite clear. If it's public, this public sector funding that has gone, public funding that has gone into the collection of the data, then that data should be made open. And however, you can see, and I won't go into the details, so there are four little caveats on this. As you can imagine, this can be quite complicated and we'll discuss this later perhaps. Data producers will adhere to the principles of proper research conduct and I think that's fairly obvious. As I said, in D3 we capture the idea of subsidiarity where data providers will determine the use, terms of use for those data resources that they provide, but again within some overarching principles that are set out centrally. Data providers will respect the principles of fair data and this is where we defer to the fair working group to give us more details. So you see the first spoiler under D4 says that data providers will implement the fair principles as defined by the fair working group. So here we either put a link to their outputs or bring in some outputs from that group. We also make a placeholder in D5 to say that those who do provide data can impose terms of use on the users of that data. So data users will adhere to the terms of use of the data resources as those are defined by the data providers and they are not willfully violate any of those terms of use. And again another sort of general principle that the data users will reference the source data and there's some caveats there about where possible because as data gets used and reused and reused in a sort of hierarchical way, traceability to the original data might be quite difficult to follow. Hopefully in the form of time we will have automated systems that contract problems of data which will help with this. Moving on to services as you can see these rules sort of mirror the data rules that we structured it in this way because there are some conditions which are slightly different for services. Again services exposed through ESCO free of charge at the points of access very similar to the one but data however the sub points here are slightly different. Service providers adhere to the principles of proper research conduct of course service providers determine and publish the conditions of the use of their services very similar to data but the actual detail might be slightly different. Here whereas the data resources adhere to the principles of fair data set out by the fair working group the service providers will adhere to the ESCO service architecture as defined by the architecture and then S5 and S6 similar to D5 and D6 are there saying that users of the services will comply with the terms of use of the service they consume and they will reference the services they consume. As I said at the detail point in the sub rule as I won't go into but clearly here the way this is done is slightly different than it is with data. The final section of the rules is about the operators so these are those systems that make ESCO itself work and here we have been more stringent than we do for those resources that are offered on a sort of voluntary basis which make the content of ESCO valuable but where if any individual resource disappeared it wouldn't bring down the whole system. So the operators of ESCO sometimes called the core are federating core those are the systems which keep ESCO alive and working and here the rules are more stringent however we imagine that the system that these providers of these resources will be compensated directly for their providing it under some sort of contractual arrangements. Here we say well there will be a registry of data and service catalogs this is so that we can register with a catalog as we put in the overarching global rules that system has to be reliable etc. The operators will deploy processes so that you can register your resource and so that you can onboard it and go through any conditions that might be required in that onboarding process. There'll be monitoring and accounting systems so that we can see what usage is made of ESCO this will be essential to ensure that the emphasis is put in the right places in further development there'll be assistance for authentication and authorization. There'll be a global search function and other functions across the whole of ESCO. My personal view is that the global search function will be just one of many ways of searching ESCO. I think there'll be field specific or domain specific ways in as well where the portal that you go through will be tailored to the work that you're likely to be doing within your free domain. There will also be a global system coming in and the core will also offer APIs so that service providers can build on top of the core resources in EOS and add value and make innovative steps etc by building on top of the things and in order to do that the features of the ESCO to be available not only through you who's are interfaces but also through programmable interfaces. So that's what I wanted to say by way of introduction. I think we have about half an hour left for discussion so here I will hand over to Dale who will take us through the discussion part of the session. Thank you Dale and as I said I can't seem to see the chat at the moment so if there is anything going on in the chat room please let someone please let me know. Over to you Dale. Thank you Juan good morning everybody and it's nice to be with you even if it's virtually rather than face-to-face in Carl's room as we had all been expecting. If you could move on to the next slide please Juan. So our aim in this section of the the session is to both give you some insight into some of the feedback that we've actually received during the consultation on the the draft so version 0.2 which Juan has just run through. That's the version that was put out for public consultation and we've collected and analyzed the feedback that we've received so this next section will will give you some feedback on the feedback. I'll tell you some of the key points that came out that we received from people but it's also intended as an opportunity for you to give us your comments on the information actually so it's a means for us to collect further feedback and input and suggestions of views and so on from you. So some logistics. First thing we're not planning to use Slido in this session so that's one less thing for you to worry about. What we're proposing is that we ask people to raise their hand if you want to speak and we use the zoom chat for posting your comments and questions so please just go ahead and do that. You can see that there is also a Google doc which I'll share the link for in a moment in the chat and the idea is that similar to what was done in Budapest if you attended that session we had a document where we were able to collect comments over several days and that was very successful and quite valuable actually so it gives us a slightly longer lasting record so we were hoping to be able to do the same thing today with that so I'll put the link in just now in the chat but just bear in mind that I probably can't both monitor a Google document and zoom chat comments so for the purposes of the session now please put your comments into the zoom chat and that's what I'll be monitoring to try to see what people say. Okay if you can move to the next slide please. There is one hand up at the moment, Mario David. Ah yes I see that. Mario. So hello. Good morning everybody can you hear me well? Yes thank you. Yes okay I have a couple of questions. The first one is about authentication and authorization saying that you allow basically federated academic users. This is okay. The point is even in the host there will be services and data which will be accessed by non-academic users so the question is about one of the questions is about that. The second question is if you consider for both the services and data repositories any type of quality assurance for those services to be included in the host portal? So I can answer both of those by saying those issues are delegated to other groups so the authentication and authorization is being done by a task force under the architecture working group so we it's not really an issue for the rules it's so much as a technical issue that will be developed by then so we'd wait to see what they do. And the second question was about extending that. Any type of if you will include any type of quality assurance on these services? So this is mostly delegated through the different research infrastructures that will be federated in the OSC so the type of quality assurance and etc will be done by the separate systems compliant with overarching quality principles that we put in these rules so these rules will contain overarching principles about quality and reliability but then when resources get onboarded into the OSC through one of the subsidiary one through one of the federated infrastructures and they will go through an onboarding process where those things are enforced so there are some high-level things in these rules about quality but the detail will be worked out in the individual resources that infrastructures that federate together. Okay I hope that answered your question let's move on. Okay so for the purposes of just going through these issues that came out in the consultation we've grouped them into four different sections status scope substance and structure so if you could move on to the next slide please Juan we'll get straight into the issues around what we call status. So the first point that came out was what should be the overall level so to speak of the rules of participation? Should they be principles or should they be actual rules? The point was made that if they are principles at the relatively high level then actually the consequence of that may be that there are considerable efforts required between communities to jointly interpret what the rules are and how they should be applied. On the other hand there was also feedback saying that the rules really should be applicable at a practical level and there should be a means of validation so in terms of applicability a suggestion was made that the actual rules themselves could be in the form of models or frameworks which would then allow some flexibility for them to be applied to different resource types for example. In terms of validation the point came up about well okay what are the consequences of breaking the rules then and of course that then takes you into the territory of monitoring and penalties and enforcement which the rules currently don't say anything about so I think it would be very interesting to hear some input or questions from you, comments and views in these sorts of areas to try to gather some more input from you to find out what in general the consensus if any might be about these sorts of aspects. Also in this cluster of pieces of feedback what should be the priority in case of divergence? I thought that was a very interesting question that was raised so should the terms and conditions of a particular community take precedence over the principles stated in the rules of participation for example if there was a case where they turn out to be different? Okay so what we'd like to encourage is for you to make any comments or pose any questions you'd like around these that's a summary of the strongest pieces of feedback that I think came back in this section that we've called status about the general sort of level of them and so on. So if there's any questions I'm watching the zoom chat to see if anything comes in and just give you a little moment to say anything if you would like to. You can also raise your hands if you prefer. Just to note that I have managed to get to see the zoom chat and the raise hands will go now so I can raise that for you. Okay whilst you're thinking I will move on to the next slide and just give you a summary of the feedback on that section but please do just keep keep thinking we're happy to take questions and comments from you on any of these aspects. So the next cluster of pieces of feedback we've grouped under the heading of scope of the rules. The rules which one ran through they currently talk about digital resources and about data and services that raises the question about other types of resource such as software and training which you're not specifically addressed in the draft at the moment. So what should the rules be? Are they the same? What are the distinctions? I'll break off there because I see that a question has been put in into the chat and it refers more to the points on the previous slide so rather than keep going with this slide I'll take this question. So Yin Chen has asked what drives the evolution of the change of the rules which I think is a very good point actually that is raised indeed by several of the points of feedback that we received. The draft that we consulted on made the assumption that the rules would be owned essentially they would the responsibility for the rules would be that of the EOSC legal entity and although nothing was explicitly said in this current draft I think it has to be the case that whoever owns the rules will have to in the future be responsible for their evolution and their periodic review based on experience or their updating as the landscape changes over time. So I think the evolution would need to be driven by experience, by feedback, by input and I think the processes around that and the communication to support it would all need to be developed by the entity which becomes responsible for the rules in the future once they're actually adopted and implemented. So thank you for your question. I'll continue going through the scope slide. The second point here was the point was raised about whether the issue of ethics and research integrity should be mentioned in the ROP. So should the ROP actually contain a statement relating to the ambition of the EOSC in the area of ethics and research integrity? Or is this something that really is covered or should be covered in a more general sense somewhere outside of the ROP? So where should the ROP be covered? I think it should be covered somewhere outside of the ROP. So where's the appropriate place to cover that type of thing? Another issue that came up which we had varying feedback about was the section about EOSC operators. So essentially those entities which provide the functions in the EOSC core or what sometimes called the core, do they or don't they belong as a section in the ROP? And there was not a consensus in the feedback that we received about this. Some people felt that yes the rules for operators should belong in the ROP because that way users and providers of the ROP would be able to see what the conditions were that were being required of the operators. On the other hand I think there was an opinion that said that really these these sorts of rules should sit elsewhere and were not necessarily part of the ROP. And if there was a desire as there probably is for people to know what the terms and conditions are that apply to the operators that information can be communicated elsewhere but outside of the ROP. Finally on this slide what research services are included in the EOSC and so subject to the rules. So this this is really about commercial services. So in terms of convenience for researchers it would obviously be convenient if all the possible services that they might wish to discover or be able to use including commercial services could all be findable in one place. But of course commercial services often are or tools are often in response in return for payment. So it raises the question of whether it's appropriate or viable or possible for the ROP to be applied to commercial services. Would they need to have their own separate set of ROP? What's the the viable or appropriate solution for that particular category of services? And I think there's a balance there really between convenience for researchers and the application of the principles of the of the EOSC itself. So quite a big area there I think and I think your input would be very interesting. I'll pause there for breath just looking at the chat. What to do if the rules are broken says Luchano Guido. The ultimate step is to revoke the possibility of using the EOSC branding. But the main issue is how to deal with the process in between in other words how to manage the interaction with the rule breakers in order to keep them on board if possible. Yes and I suppose that gets into both monitoring so you know if the rules are being broken anyway what are the processes around that and then yes as you say perhaps some sort of mediation process or discussion with the rule breakers to try to keep them on board. Yes it's a good point thank you. Just add briefly that one of the sort of approaches we like to take with this is to make the process as transparent as possible so that feedback on the services feedback on on anybody using EOSC can be done in the open and that would be a degree of community monitoring of systems and operators to see what's going on and maybe just the need the fact that comments will be made in the open might be a way of the first stage in trying to encourage people to stick to the rules. But as you're quite right that there will need to be another process behind that to take up if things don't work properly. Okay thank you and I see that Per has raised his hand so Per I've unmuted you if you would like to speak. No hello can you hear me hello hello hello Daylan Juan good to see you. Long time. So now I was thinking this question around the the rules for the providers there. I would assume that they probably have a part of it here but especially for the providers it should be held on a very general level because and keep open because when let's say the providers here including providers into the EOSC when that hits reality one need to be open and to the fact that providers comes from many different and several different conditions here and the actual rules will be set in service level agreements other type of agreements between maybe if the legal entity have a specific agreement with the provider or a service is provided by the change of policy of a government that or a funder that they allow that resources they are funding are actually become available also for general European researchers use and so on instead of only on a national level and I think one have to be put the rules here in the very general level to allow for this very different I would expect very different ways of that services and various type of data resources and other services will come in open be open for that to come in so I for example this rule on monitoring and accounting yes but one can't to whom should monitoring and accounting be provided where when is this important when is it possible and why is it important that is not fully clear until you have a real agreement on how the services provided so I would be I would be careful to be too detailed in these rules but but keep them on a general level what are such a kind of barrier that need to be and then there will be more detail as a license on for various services yes I think I could agree completely and we try to stay fairly abstract in these rules we will have a place to commentary about issues and explanation what goes behind them so for example where things that delegated to the individual resources or infrastructures then most things can be pointed to in that country with respect to monitoring and accounting I think it's clear that the usage will have to be monitored when added up so that we can see what's happening my personal view is that those metrics would be open just because we follow the principle of as open as possible and see why usage metrics should not be open and then that is a baseline for any other agreements elsewhere around compensation I mean one thing we have to recognize is that all the different lines of funding that will go in to support different resources on that are available through EOSC those will all have their own governance their own rules their own ways of monitoring and compensating and their own feedback systems etc and review systems so these rules that we're writing here do not replace those they sit alongside whatever governance is in place for the funding of the resources that are going in to EOSC okay thank you I'm conscious there are some really good questions coming in in the chat but I'm conscious that we're getting quite short of time so I think we may need to move on now and if I just remind you that we've got the google document where we can capture these points and and perhaps have some some more discussion over the next few days on them so if you can move to the next slide Juan please I think we we probably need to keep moving so the third set of feedback points is grouped under the heading of the substance of the rules so several points relating to the the point about the EOSC that it should be the services should be free at the point of use the question was actually raised about whether this principle should belong in the rules at all in fact partly in relation to the fact that actually compensation is provided through many different mechanisms so it's a very complex area and of course ultimately nothing is free it's all got to be paid for somehow somewhere just perhaps not at the point of use so one of the the points that was made was that you may well for example have free discovery of a resource but then the use of the resource um has costs applied to it um as long as those costs are transparently communicated is that acceptable um and a question was also raised actually about um well what about compensation for the cost of data use so not not just focusing only on the cost of service use but what about the cost of data use so there's a whole cluster of of points that came up in this area not surprisingly since it's a very large complicated area um there was also feedback relating to acknowledgement of use of resources which of course there there is some proposed content about in the draft rules um the point was made that users really won't always know which resources particularly which services they've actually made use of so they will not always potentially be in a position to to be able to make acknowledgement um but the question was also raised about whether a standard citation scheme was possible and I think that relates to the fact that um in broader terms incentives and rewards within research organizations will take time to um be be modernized if you like so that the the motivation of users in particular to um move to um citation of use of services for example is is potentially going to be somewhat low for some time so is it possible actually to implement um a standard citation scheme um and also um quality assurance so there was there was a point about this earlier on um this did come up um so there's there's nothing in the draft rules at the moment about this but um in terms of quality assurance of the available resources a suggestion was made that perhaps what would be workable would be to have quality assessment of the fairness of outputs produced through or provided by the EOSC um so um allowing people actually simply to to judge for themselves essentially but based on information provided um and that of course comes back again to the issue of monitoring so what should be monitored um when how and indeed by whom actually as well so there's a whole a whole pot of different questions there um right I've not been looking at the chat during the last moment so are there any questions relating to any of that that we need to pick up I see a comment from Joy Davidson data should be findable but free at point of access is in some cases very costly yes exactly I think that was recognised in some of the feedback we we received um when Appleton fair is by no means the only measure of quality assessment it fits poorly for many services right thank you okay and what I see you've replied to that okay I'm running a bit behind hand with keeping up with the chat okay yeah I'm trying to reply but it's difficult to do everything at once yeah I think you could move on okay if you move to the next slide then please okay so finally um relating to the structure of the rules um the current draft structured um the rules along the lines of resource types so data services there was there was some feedback um to the effect that it might be easier if the rules were structured by the type of actor such as users providers and so on um so it would be interesting to hear some more views on um which you feel might be preferable or indeed actually could each rule stand independently from the others anyway so if each rule um had its own explanatory text then essentially you would release yourself from the ability to actually choose definitively which structure to use um a rule could could either refer to actors or resources or indeed something else um as was necessary and would that actually be a more successful way of structuring the rules so your further feedback on these points I think would be very helpful actually because it's it's quite a thorny issue actually trying to figure out exactly how to structure these rules in the most successful manner and see we've got to vote for type of structuring along type of resources as is currently the case thank you the channel and Owen Appleton says there needs to be a hierarchy of rules starting with general ones then move down to um a structure by type of actor so users providers and so on and then by type of activity so provision of a service provision of data yes thank you I think I think this structured approach is very interesting and probably has some good potential actually um so there's any other comments on that I think that would be very interesting because it's a um I think it's a an approach that has some good potential which we should look at um yes and you've you've got a schema of that sort as a as a proposal that could be looked at thank you I think that's potentially very well yes thanks so and I've looked at that schema and we'll bring it up in a working group meeting in the future um I think it's worth commenting on Sophie's serve and um comment about the process uh going forward from here um yes it's a very good point we got feedback from a good number of um EOS projects I think probably getting close to 20 um but that's not all the projects and it's quite possible that within this process we feedback was only given by a small group of people so it it is difficult to know how whether we have really collected uh feedback widely enough um it's fair to say that even from the feedback we have got um there's also already an awful lot of very interesting points made and some contradictory points made so um we clearly have uh some balance to strike there are checks and balances all over this and there are many places where we have to find the right balance and it's difficult to know where that is so I think you know arguments um anybody's free to put forward comments at any time and in the next few days especially that google documents if you want to make specific points about where the balance should be in some of these issues please do so um of course in the end we have to kind of make decisions um based on uh all the feedback received and we can't necessarily uh satisfy what everybody's suggestions but clearly we haven't received feedback from as wide as possible yet um but we already have uh a lot of feedback with many different views so we're already in a position which is quite important to satisfy everybody but if you do have suggestions about the process going forward please put those in the google document as well yeah okay thank you so I see some there's some very good comments coming and actually there's a point from Renée Bell so about um relating to the fee at the point of access what's the business model yes it indeed does I think that's a very good question that you've made uh Renée um you may have also noticed though that Rob has pointed out that we are very short of time now before we'll all get moved back into the main session so I think we may need to wrap up there um I'd like to say thank you very much for the for the comments for your interaction um please keep it coming using the google document um and obviously all the points that you're made we will we will do our best to take them on board um in drafting the next draft of the ROP can I just check with Rob that the comments from the chat will be preserved for us as well yes that's right we will save the chat okay so thank you maybe you could put those at the end of the google document so we've put everything in one place okay in that case I think we'll have to draw this session to a close so that people can get a coffee before the start of the next session um thank you all very much I hope it was useful hope you understand uh the challenges we have and um and please try and help us by making suggestions um on these rather difficult issues that uh exist in this but thank you very much and I think we can all return to the main session


From New York, extracting the signal from the noise, it's theCUBE, covering Spark Summit East, brought to you by Spark Summit. Now your hosts, Jeff Frick and George Gilbert. Hey, welcome back everybody. Jeff Frick here with theCUBE. We are live in Midtown Manhattan for the Spark Summit East 2016. I think it's the second year that Spark's been in New York, it's a really growing event, really the Hilton Midtown, which has a lot of big data events that were kind of at the birthplace of what was stradded at World and Hadoop World and now really Spark is kind of the next wave of innovation. So we're excited to be here bringing all the action and joined in this segment by Wikibon, Big Data, Analytics Analyst, George Gilbert. Welcome, George. Good to be here, Jeff. Oh, that's my cue. That's your cue. All right, so we're taking it around the table. So we have Anand from Stream Analytics and Anand, I'm not going to butcher your last name, so I'm going to let you introduce that part of, that part and tell us what's new with Stream Analytics. So am I looking there? Yeah. So my name is Anand Venugopal. I'm the head of product for Stream Analytics, which is at Impetus Technologies. The company that brings to you the product called Stream Analytics is Impetus Technologies. So as many of you might know already, Stream Analytics is a real-time streaming analytics platform designed for enterprise usage that's based on open source technologies. And we're very excited to announce that we just announced our 2.0 product that's based on Spark Streaming. And with that announcement, what that makes us is the industry's only multi-engine streaming analytics platform. So you saw what Databricks is doing in terms of Spark. They're strengthening Spark for streaming, increasing the performance, increasing more ease of use, things like that. But as far as enterprise users are concerned, they really need a much more abstract, easy to use type of interface with drag and drop buttons and visual IDE, right? And they need that environment independent of what technology it is. They just cannot keep up with the number of different technologies coming up. They can't keep up with the speed of innovation. So they really need one common abstraction for streaming that enables them to build streaming applications on any chosen engine because there are different use cases, right? There are low-latency use cases, there are use cases that are suitable for batch or micro-batch. So you want to do with storm, great. Use the same interface, build a storm pipeline, deploy it into storm. You want to do a Spark streaming pipeline, build it, same interface, build it, deploy it to Spark streaming. And even better, you can start interconnecting multiple subsystems based on different engines using, you know, queuing and other technologies, again, visually, right? So you build a preprocessing, maybe module with storm, you build a machine learning module with Spark streaming, interconnect those two and build more complex real-time pipelines with multiple technologies, right? The biggest thing here is we want to enable enterprises to focus on the business value and application layer, not for them to be frozen about, oh, which engine do I pick? If I pick this engine, am I going to get stuck with it because it's going to be the wrong engine six months down the line? So we want to really take them up a level in terms of abstraction, right? And say, focus on your application, focus on the business impact, just take it for granted that you have the right technology under this platform. You have, you know, the most favorite technology, whatever it is. Spark streaming, right? It was storm, it's now Spark streaming, we got both, and who knows, two or three quarters down the line, that could be another one, right? One more. Tell us, first, before we get into which engine, you know, or why, tell us the class of applications that you see coming up that's pushing customers towards, essentially, faster and faster reaction or lower latency. Yeah. Why is that happening now? What are the first applications? So, there are two classes of applications. One, I would say, is customer analytics. Anything to do with improving the customer experience, delivering much more context-sensitive real-time service, which literally surprises or even delights the customer, right? Hey, how did you know that? I was calling about this, but you already knew this, it was great, right? Customer analytics is one large category. Operational intelligence in general is another large category, right? Where again, you have security, you have predictive maintenance, you have just maintenance monitoring of different systems and things like that. Yeah, of your systems itself, like... Systems itself, yeah, predictive maintenance, and of course, IT log monitoring, right? All of that. So, there's, from a business value point of view, we classified into three or four layers based on business impact, right? When you're trying to protect your assets and minimize losses, right? That's more preventive and predictive type of use cases where you're doing predictive maintenance and security. Even preventing a customer that's going to churn in some ways is actually protecting an asset or reducing a loss, right? So that's preventing losses, okay? And as you know, if things go bad, these losses can make CIOs lose jobs. People, companies like Target, those is tens of millions of dollars and data breaches and things like that. So, that category of applications is critical. It's basic table stakes, right? The next is routine operations or more profitable operations with much better optimization, right? Let's say you're optimizing network traffic. You're making much more optimal use of your network and you're avoiding like a $50 million capex upgrade just by optimizing your current infrastructure based on real-time analytics. That's routine operations, right? Tracking a fleet and making the best use of all your trucks and all of that. So those are routine operations. The third category which gets into the more sexy domain is getting new, finding new sources of revenue, right? Whenever a customer is ready to buy something and you just are not offering that because you don't have enough information in the moment of truth, you're not able to offer something that the customer would buy. That's the missed opportunity for revenue. Real-time streaming analytics would come in right there at that point and say, context-sensitively, when I'm engaged with Jeff at this point, get all the information about Jeff that I possibly can and put in front of the agent so that I can offer you the right product that you want to buy. We, our CEO, Dave Vellante was talking about that earlier this morning in terms of vendors being, first, consumers got more and more information about vendors and so they had pricing power, but now vendors are collecting more and more information about consumers. Is that process of gathering context about a customer? Does that ever end? Is the vendor always in a better position to offer something, whether it's better experience or better product or service if they have more and more information about that end customer? I guess there is an endpoint where we could say there's no future context enrichment possible, but it's many, many years away, I think, many, many years away. So it is an evergreen process? It is an evergreen process at this time, right? Because life is constantly ongoing, right? Two weeks down the line, my situation changed. Something happened in my life that you as a seller to me ought to know. So it's a constant ongoing process. But the other piece is the nuance, right? I think the nuance can only get better with more sophisticated tools and more better information, better algorithm, because somebody once said, you know, done poorly, it's creepy. You know, done well, it's magic. So, you know, what is the nuance between the creepy and the magic to put the right offer in front of me at the right time? Right.  I went and put information on your website. I called your call center. I gave you my data from a provisioning interface whenever I did. So all the information I voluntarily gave you and you are not able to correlate all of the information I gave you voluntarily, right? And so there's a huge value add that's still yet to be done by companies just in centralizing and converging all that information. That's in the data lake scenario. It's already there in the enterprise, right? Bringing them to the moment of truth. It's not just about doing single source of truth and doing offline analytics, but can you bring it to the moment of truth? The guy calls in to the call center. Does the agent have the required context to provide him that information, right? And it could get as real-time as this. I go on the website, I'm trying to do something. I was unable to do it.   Is that what you're calling about? You want me to complete that for you? I was like, whoa, right? That's an experience we don't have today. And to go from where we are to that, and where that becomes the common experience, it's still many, I think, many quarters, many years away by the time all of these enterprises really get all this capabilities together. Well then, how do you talk to customers about the trade-offs between speed, you know, real-time, close to real-time, not quite real-time, versus complexity to execute, versus really the value. Do you really need, you know, to do that relative to the value to the complexity and the difficulty? How are people going through kind of that analysis to figure out where to tune those levers? I think the low-hanging fruit is what people get to first. In case there are, like I said, asset protection type use cases, right? If you don't have your security apparatus in place, if you don't have streaming analytics to detect your security breaches or things that are absolutely essential to protect the business, that's where you would go first, right? And there it's a no-brainer. Are you waiting to lose that, you know, $100 million dollar situation, or you want to plug that hole right now? It's like insurance, right? So you first go after all the safeguarding use cases, right? And there's not much of an argument for whether it's valuable or not. You really want to do that. The next is people have done Hadoop-related analytics, et cetera, right? Batch analytics, and they've gotten a degree of lift. One large telco said they've got 25 times ROI on their Hadoop investment, right? And now they're looking out for the next big thing. So the CIO and the technology team is now proactively thinking, how do I use technology as the next level of differentiating, right? How do I use, and everybody knows about streaming analytics now because they're all bought into Hadoop. And it's a lot more easy to move into streaming because you already, the Hadoop revolution has already taken place, right? You're just now adding a capability on the same stack. So that makes it much easier. So I want to go back to that comment you made about, there's always more context we can learn about, say, the customer. If that's the case, we really can't build big packaged apps based on predictive analytics, machine learning. Like, it's not repeatable enough. And I guess that takes us back to where you came from, your firm's origin, as in professional services. How do you see the applications or the solutions you build evolving? And what might have to fall into place for you to be able to build repeatable applications as opposed to repeatable tools that you've got right now? I'm not sure I quite fully understood the question. Okay, so ERP came from, we knew how to codify accounting and order taking, inventory control, the sales order process. But with predictive analytics, when it comes to making a next best offer or a recommendation engine, it's like you have to ask for what, and based on what type of information. So it seems to come from professional services firms like yours and the tools. But we don't have ERP-type applications. Or predictive analytics. Yeah, so does that have to change? Or what happens? I see that changing. I see that changing. So for example, all the platform players are going up the stack, as we can see, right? Hortonworks is now going up the stack in terms of adding a security framework, security application, cybersecurity application. And it is the natural progression for all platforms to start getting verticalized to go up the stack. So let's say I could bring in all the best practices of let's say churn reduction, focused on the telco industry and create a churn reduction application, okay? Which is a predictive analytics app. I can package all of that intelligence and with small customization, so it's not coming all, right now, you might see 80% customization, 20% product, but it will grow to 80% product, 20% customization. But you're still talking about a relatively narrow function, churn prediction. You're not talking about the whole OSS or BSS, you know, the whole back office or front office for a telco. You're talking about this one, one app silo, let's say, right? But that would be in the, if you want to make it more abstract and larger than that, you could, I mean, customer 360 is something that most large enterprises are now going after as a use case, okay? It receives integral feeds from both on the batch side as well as the real-time, right? So if you want to go bigger than just churn, it would be the whole view of the customer. And I can create a verticalized app for the retail industry versus the telco industry. There'll be some nuances that are different in each case, but customer 360 would be a more broader space, right? As an app, with some customization involved, I could easily envision that application being a category in itself, right? As in like the next ERP. Right. So there are other, for example, the whole security space, there's this fraud space, risk analytics, all of those, those are all actually doable in terms of being apps in their own self. The one other space that we are interested in is call center analytics, right? We have some very powerful use cases that help call centers visualize the real-time performance of what's happening. And this is a problem we have seen that banks, telcos, insurance companies, across the board, they all have call centers. And they want to look at that customer. And that is the key moment of truth area where people are interacting with the customer, right? And there are two things in a call center. It drives a lot of cost, right? Every call amounts to like tens, or maybe even hundreds of dollars, right? In terms of amount of time it takes. So large companies are all trying to reduce that cost point when you reduce the cost. And when you deliver context-sensitive service at that point, it becomes a double whammy on the positive direction, right? You're reducing cost and increasing customer satisfaction and profitability. So call center analytics is another, I would say, app space that can be verticalized as an app. It can be a significant chunk sitting on a platform that's both using real-time feeds as well as batch information. So now I'm running out of time. I want to give you the last word. What advice would you give to customers based on your experience with all the customers that you're out there, where they should go? What's a little hanging fruit? What's your experiences where people really had some success in getting into this? Yes. Clearly the Spark revolution is here to stay, right? I would say adopt Spark and Spark streaming ASAP because it is definitely going to be the future. We're seeing adoption across the board. Number two, if you think that streaming analytics may or may not be useful for you, be clear. It is going to be useful for you. There's at least like five to 10 use cases that will drive revenues and profits in your enterprise no matter which vertical you are. So go find a platform which can abstract you from the technology, help you to focus on the business and get you to production fast. And that's stream analytics. Good tips. All right, Anand, thanks for stopping by from Stream Analytics from Empathist Technology. George Gilbert, Jeff Frick, we are here in Midtown, Manhattan at Spark Summit East. We'll be back with our next guest after this short segment. Thanks for watching.