and celebrating the first Australians on whose traditional lands we meet and pay my respect to elders past, present and emerging. Welcome everyone, I understand that due to a quirk with Eventbrite some tried registering at 11am rather than 12 noon so I'm sorry that we had that shaky start but with no further ado it's wonderful to have you here today. And I am going to share my screen. Not that this is going to be PowerPoint heavy. There we go. All right, so the purpose of today's update info update is really to ensure that we maintain open lines of communication. I just want us to make sure that we're all on the same or shared understanding of where we are with the house and indigenous research data commons. We're all here because we heard the wonderful news in the 2020 research infrastructure investment plan, the support for the first steps in the house and indigenous research data commons. So I thought it was really important that we didn't leave too long a gap before we brought the community together. So, first of all, we all know that funding was committed by the federal government back in October. And the ARDC will be the lead agent in this space so this morning I'm going to just share a little bit about what ARDC is working on at the moment, where it fits. We are going to provide the broader context what's happening in the national research infrastructure space at the moment, and what are the indications for this work how this work has the opportunity to provide a really strong foundation. And then where we're up to in the specifics of this activity. As I say, it's more of the general information sharing and update that we can obviously have a question and session at the end. There we go. The I might not want to do that. I'm not sure what that's sharing. I'll just stick with this. So the Australian research data commons incorporated as a company limited by guarantee back in May 2019. You'll understand why I'm emphasizing this a little bit further on, but it's a not for profit. And we have very pleased and very pleased to say that this slide is just out of date because we have reached 19 university members plus CSIRO just last week. So we have the membership across the country. And you can see that our staff are hosted by universities across the country as well. And our DC very, very simply has the purpose to provide Australian researchers with competitive advantage through data. We talk about a mission to accelerate research and innovation by driving excellence in the creation analysis and retention of high quality data assets. So that's the mindset that we bring to these activities. So in this establishment, we've worked with five different themes. The commitment of the members is really embracing the shared vision, as we're delivering it through these five different themes so you can see them there. Data and services, storage and compute software and platforms and people and policy. And of course, in writing all of that we have the coordination and coherence. And just a moment talking about where we sit with the four themes and how that will lead into where we have the interconnectedness of the research data comments. So ARDC, it comes from a very, very strong heritage in joining together and NECTA and RDS. So three previous absolutely visionary research infrastructures coming together and providing a portfolio that includes some ongoing services such as the National Information Infrastructure, the NECTA Research Cloud activities in the skills and the policy space that really underpinning contributions that work across every discipline. And then on top of that, we have the platforms and software and the data and services open calls that we ran in 2019 and 2020. Many of you will have engaged in these open calls and we have a series of projects that are currently underway. And I know that some of you on the call today are active participants in those projects. So it's in concert with these in parallel with these that will be working on the past indigenous research data comments. But we're all doing it under the increased larger research infrastructure framework. So the merger of fans NECTA and RDS came following the 2016 National Research Infrastructure Roadmap, which also spoke about the need for investment in the HAAS space as well. And one thing that I think since 2015 is a huge change, but it's incredibly positive that the government have committed to a particular cycle in the research infrastructure roadmaps and that's what I want to highlight today. And we have the 2016 roadmap, the government response to that within 2018 with the research infrastructure investment plan. And that was the, that was the plan that brought the ARDC together. And it was also the 2020 research infrastructure investment plan that created the current funding for the project we're talking about today, the 8.9 million. But that particular cycle of a roadmap followed by two research infrastructure investment plans is a pattern that has been set out by government. The first is the roadmap to investment plans. The first is the government response to the roadmap and the second is a top up or new activities that were identified in the roadmap. And so now this leads us to the 2021 National Research Infrastructure Roadmap Consultation. This is running at the same time as the HAAS and Indigenous Research Data Commons activity is an enormous opportunity for us as a community to demonstrate the foundations for future and increased support in this sector. So it really is providing us with the opportunity to shine a little bit about the timelines. At the end of 2020, the Enquist community were asked to consider some chapter headings effectively for the research infrastructure roadmap coming up. All of the chapter headings or the possible framework so one of them was the same as the 2016 roadmap. Another example was to use mission driven. This was for investment. Another one was based on international models. However, one of the things that unites all of those is a commitment to the research infrastructure. But we're now. This is very, very front of mind for us at ANRDC. Early February, we are in what's known as an ideas jam. The department have recognized much like today's discussion that consultations during 2021 are for the best part going to be online. So they have, they have appointed a consultancy external consultancy to help them with the online delivery of the roadmap in consultation sessions. And the platform is being tested this week, last week and this week. So we're in week two with an ideas jam for the Enquist capabilities where we're really posting some of our thinking for what could come in the future. The expectation is very much that the bugs and the quirks and the learnings for this online platform will be ironed out now. And a broad consultation phase will be taken over the coming months. I mean we've put March to September here. The conclusion and the submission of the next roadmap is expected in October of 2021. That's all we know about the timelines at the moment, but I am stressing it because I do feel we are really focused on laying excellent foundations with the work that we're here together to discuss in the past space. So keep that in mind. Going online, the scope of work that we're dealing with at the moment, it absolutely isn't a blank sheet of paper. We don't have a contract from the Department of Education at this stage, but we do know that these four particular activities of what we will be asked to deliver on so it's, it's really quite specific. The way that we take these forward of course is what we're up to developing now, but we have a very clear, very clear task given to us by the Department of Education to deliver in this space. So you can see the four of them here. I don't suppose there are any surprises because it's exactly the same text as used in the government releases and on our website as well. So we have the high level description of the projects. We don't have the contract yet, and I understand that one of the things that's being tweaked is simply the increased guidelines that will support the contract. So we're just waiting for that. We're not concerned by that. And indeed, the next step is that ARDC will undertake recruitment of a program manager to advance us to the next stage. Now, we have a well developed position description for the program manager and we anticipate that we will be recruiting very, very soon. So very short term, as soon as that is finalised. And then we'll have the program manager on board in advance of contract. The first thing that we will be required to do under the contract for each of these four components of the project is to complete an activity plan. Now, an activity plan. Again, it's not a blank sheet of paper. It's a very specific template with word limits that the Commonwealth will be requiring us to submit. And we will have four of them, one for each of these components. The activity plan sets out what will be achieved in the project, what the milestones will be, what the funding will be. So it really is a short document, but it's an important document. And the role of the program manager will be working with the relevant communities and stakeholders to put those activity plans together. So it is the first step, but it's a big step. Now, with these different activities that we've got here, the four different streams, there are obviously different foundations, different backgrounds, they're each at different parts in their own journey. For example, I spoke about the ARDC open calls, the data assets creation and the platforms for analysis of software, and already the linguistics data commons is being supported by completely separate ARDC projects. But that's fantastic. It allows us to get a very rapid start, use it as a launching pad for going into the next stage. Some of these other areas, there will be the need for more discussion and more consideration to understand what's possible in these first stages. So that will be the work of the program manager with the stakeholders and with the research community. And a couple of other things I want to talk about before we go to any questions is what would be the structure that we envisage taking place here. So I mentioned that we've done the open calls, but at the same time, we have another ARDC program which is called the translational research data challenges that we've run in the past year with a rather different format. So the translational research data challenges instead of working through open calls has had very focused consultations, working groups and discussions to really bring together the identification of the needs and the scope for a particular project. And I do think that we would be striving to follow that consultative pathway to really build consensus in identifying what's required for these projects, rather than any notion that we're going to run open call and people will bid. And these are isolated things that go away into silos. So this is really the time for building very, very strong links in the relevant community. That's the aim. We'll see how much convergence we see in those ideas as we move forward. So that's about how another important consideration to talk about is the co-investment expectations. Now we're in increased capability. The plan will be undertaken as part of the input, the increased guidelines and the increased principles, and they do require us to maximize co-investment as we are able to. We have already in our programs recognized the impact of COVID, the very difficult financial conditions that are facing the sector at the moment and the constraints that people are challenged with. Therefore, in recognition of that, of course, we're very sympathetic and trying to balance the need to secure co-investment, which we still have to strive to do with the pragmatic approach for what's possible. And recognizing that we still do need to secure cash co-investment, but there are other ways of having very, very valuable in kind, very valuable, actually essential in kind contributions for these projects to be successful. So I think those are really some of the key messages we've spoken about the defined areas, the timelines and the next steps and the co-investment. And I think that is actually where I will pause and take questions. Now, we have a very large number of people on the call today, so thank you very much for joining and participating. It's sitting at 100 people at the moment. So if you could drop any questions into the chat. And Ian Duncan, our director of outreach, is going to moderate the questions for us. So Ian. Thanks, Rosie. So we've got a couple of questions from Douglas Roberts. And the first one is, do we have an indicative timeline for the activity plan completion? No, we don't. So the, I think actually we don't have the timeline for when we expect the contract to arrive with us. And this is something that I'm chasing. So that's the, that's the big unknown when the contract comes through. However, the first step is actually the appointment of the program manager, because we can progress and start the discussions around completion of activity plans in advance of that contract signature date. So with the expectation that the recruitment process takes place in the next couple of weeks, or certainly opens in the next couple of weeks, we will be looking to have someone on board, certainly before the Easter break. And their first task is those activity plans. So we will just keep working away at that. The second question and do please tap your questions into chat if you've got any. The second question again from Douglas is, will the department expect co-investment from state governments as this has caused challenges for other increased facilities? The question I can answer by simply considering ARDC's operations already, the nature of research infrastructure has made it much, much harder for state governments to invest. And at this stage we haven't had to, we haven't been held back by the absence of state government funding. It would be very welcome, but it is not a prerequisite for the program to go forward. There's one from Howard. Trove is excellent, but much of the indigenous data is in museum collections in Australia and globally. Will that material data also be included? Howard, that is a specific detail about the particular program that I'm not in a position to answer at this stage. We'll certainly hang on to the question, but I can't give you an answer today. One from Steve McKetkin. Regarding co-investment, there's been a significant level of co-investment into the various ARDC and related programs, so platforms, data partnerships and so on. It may be beneficial to look at how the co-investment in those projects may be considered in combination with the co-investment expectations for the HASI program. Steve, you're trying to count the money twice. Yeah, not getting away with that. I can see you on the screen. We will do what we can and be pragmatic about this. It's definitely worth a try, Steve. You can't count the same dollar twice. Okay, one from Katrina Grant. In terms of the Trove research platform, what other corpora are in scope? Focus on Trove precludes a lot of other collections. I can't comment on the specific details other than saying that we have to follow the identified components as I've set out previously. Can I add one thing to this? It's really useful if people remember that this is a foundation set of investments. You're absolutely right. Trove doesn't cover everything. The social sciences stuff doesn't cover everything, and it's aspects of those particular pieces. And so, as Rosie referred to earlier, the road mapping is the opportunity for showing how these early investments really can make a high impact and fit really nicely together for the community. So you write that there will be a whole bunch of collections and bodies of data and applications and platforms, which would be fantastic to have. And these first ones will provide a basis for some of those and also really, the really important bit is to show that there can be success in developing them. Okay, so another one from Nick T. Berger. Can we envisage that the three name programs excluding Trove could be considered capabilities in the next roadmap? I'm reading the next question as well, which is throwing me off. We can put them forward as ideas for consideration as capabilities in the next roadmap. I think the idea is, are they at the right scale? Is there some way of connecting things together that's required for scaling? To put this in context, there's less than two dozen capabilities at the moment, and I think we do see those numbers decreasing. But I'm not a member of the expert working group this time round, and I'm really reading the tea leaves. Nick, I don't have much more information. But if we think they should be then we should definitely be saying it. From Christopher, had a have philanthropic forms of funding being considered universities are research providers, not funders. Every, every type of contribution is absolutely valid in this space. There are some cases where philanthropic funds are the most appropriate funds, and indeed we're exploring them in the translational research data challenges. The other RDC component that I spoke about earlier, so it's all in scope. Another thing, why not industry? Yeah, all of this is possible. Just before we get on to Kylie's question, there was a comment from Steve, so fair enough. He's not trying to count things twice, but that there has already been quite a lot of requests for co-investment. I think, again, as Rosie referred to earlier, that's why we're really trying to be very accommodating and flexible in terms of what co-investment is and means and can be. And it's cash is king, but it's not the only valuable co-investment. Okay, so question from Kylie. Can you give us an outline of governance structure and advisory processes for the program? Kylie, that's a critical question. And at this stage, this is, it's not something that we have matured, ready for conversation. It will be an important component once we get the program manager on board. Richard's an ARDC question. The Indigenous Data Network has mentioned in the Indigenous task, does IDN have a specific role in this activity? Yes, IDN does, as outlined in the dot points. Thanks, Richard. Another one from Howard. Many of the Indigenous community users are in remote regions with poor digital connection. And in the case of cultural data, including language, how are we going to facilitate access? Howard, I don't know the answer to that. But if that's one of the challenges that we have to consider to be successful in this part of the project, then we'll have to redress it. And I think it's not confined in the current environment, even with people working from home in big cities, I'm seeing some technological challenges. But keeping that in mind, they will need to be other ways of making sure that input is heard. Yes, anybody else got a question? Oh, here we go. Anna Johnston, is there an expectation that all four programs will be consistent in their approach and development? While some of the identified programs seem to be researcher driven, e.g. the Linguistics Data Commons, the Trove program would seem to be led by the National Library. So, very, very clearly, this is being undertaken as part of the National Collaborative Research Infrastructure Strategy. So we're doing this to create the environment and the assets for research on data. Obviously, that data comes from many different places. So I would say that it's a research infrastructure. However, I think we need to be very clear that within that, each of these projects are going to be really quite different. And that's okay. That's okay. There is a set of requirements that we have to meet. We have to meet the increased principles. We're going to need to make sure that we have the activity plans on an annual basis and the progress reports against those activity plans. But beyond that, we have definitely the ability to nurture and support each of these projects in the way that best suits them. And if you think about it, the activity plan, we have to put down exactly what it is we're trying to achieve, and then come back and report against it. And that there isn't any intention to have a cookie cutter approach across each of the four. Notwithstanding that, we do feel that there is definitely the opportunity to enhance connection between these projects across these projects. Or with other projects in a way that shares knowledge, shares best practice, enables people to build on the work of each other and not have to devise each of their solutions from scratch each time. So docking in to even the broader e-research infrastructure space is something that we will be hoping to drive forward with these programs. All right. One from Angeletta. What's the overall timeframe allocated to the HASS investment? My expectation at the moment is that it's to June 2023 as the conclusion of the project. That gets harder and harder when we don't have the start date that since ARDC is able to support the recruitment of the program manager and drive forward those initial discussions on the activity plan. We're really working to make sure that that's not the impediment. Nick Teeberg again. Is it likely that the funds will be split four ways with ARDC keeping some how much Christian Mark for admin? So the first thing ARDC will recruit the program manager and that will be funded from the 8.9 million. We will not apply any kind of top-slicing. So there's not going to be a percentage cut on these things at all. There may be some absolute costs that are incurred to undertake the project. In the digital world there are a lot less. We're not paying for the tea and biscuits for the consultations. We're not flying a program manager around the country in the way that we would have done previously. Obviously, I do think that subject to the constraints around borders, the ability to have this person engage deeply and face to face where possible with people will be very critical. So direct costs that would come from the project. Yes, but there won't be any kind of administration overhead in post. Do I see the funds will be split four ways? Definitely be split into four, but I don't have any preconceptions about whether it will be evenly or anything else. That's something that we will be looking at the activity plans. What are we delivering in the activity plan and how much does it cost to do that? Okay, Sandra Silva asked, following on from the theme of foundational investment, could you clarify what any expectations the department and ARDC has that partner institutions are providing baseline facilities? For example, institutional research data repositories. Keeping in mind the context of HASS being long tail, many researchers and many collections. How does ARDC its role, if any, in ensuring this kind of baseline is in place at the institutions and that those facilities are supportive of overall national research data reuse objectives, both fair and care for Indigenous. That's a really long question for me, Ian. Sandra, maybe I can paraphrase, but you tell me if I get it wrong. But it's basically HASS is, you know, HASS data becomes more valuable as it gets older and there's a lot of it and often small, not big collections, some big, some small. Is the question to what extent is ARDC making sure that the institutions themselves are providing the underpinning infrastructure to make sure that all of those pieces don't get lost. And also that the institutions are more generally supportive of fair and care. So do we have other programs that encourage institutions to be good data custodians and good data stewards? Sandra, you're very happy, you're very welcome to speak here. I can see. Yeah, look, yeah, look, that's a good summation of Ian said there of, I'm just wondering what ARDC's assumptions about what support is out there already for HASS. And I think you vaguely alluded to it in a previous answer, sort of generally about the research space. Okay. I'm wondering, are you with, you know, what do you expect institutions to be providing? Yeah, I think expect is probably the difficult word here. Because I think what you're talking about is a question of whose responsibility is it longer term to deal with the tsunami, as it's been described of data, not just in the HASS space. Some of the collections might be smaller in the HASS space, but you know, they're still acute. So what you're, what you're touching on really is I don't think we have expectations, but we have concerns that somewhere in the system. It has got to happen. So the way that we are addressing this at the moment is we're working to change the conversation from saying to institutions or others, you must provide X amount of storage to a consideration of what is it that's being stored. We ensure that the storing of data, and by that I mean fair data is optimized. And so we have done one pilot program, we're about to launch another very, very soon that works with the institutions and the increased capabilities to identify collections with particular associated metadata and work with them to incentivize the improvement of metadata associated with that collection, and then pay for that to happen. Now what we would like to do is to be able to submit into the next road mapping process. We're creating the framework where you can make a very considered judgment about what it is you're storing. And with that improved knowledge, you can better understand how much it's going to cost for that to happen. Okay, so I don't think we're saying we expect institutions to do X, Y, and Z. We say there is a real need, the need continues to grow, and unless we put in place the framework whereby we can make reasonable decisions about what's stored and what isn't stored, we won't survive this. So by being able to say that data that's being collected from a microscope, you've got 100 image but actually, we know how to recapture that we don't need to keep 100 images, we might keep one image and the protocol for re acquiring that data. In contrast to another collection that we can't re acquire. That was a bit of a long answer but it was a long question and a really big question that's relevant to all Australian research. There are another couple of questions from Marco and Ryan which are similar in the sense of the sustainability of the outputs. If your questions weren't answered, if you could just drop a little note and then we might do a written answer to that, that's something which would be good for us to write down. There was also a comment from Douglas Robertson about that critical nature of governance, which I think we absolutely agree on. Really, that's a key component. Peter Riley has a question regarding the Indigenous data network dot point. Really is the point into the dot point that references the name role of IDN is the IDN therefore the single or primary vehicle pathway or pathway through which Indigenous research capability is expected to be supported. I think, again, we're at the stage of I'm not presupposing what will be in this project but certainly the conversation will be started with the program manager and with the IDN in the formation of the activity plan. But Len Smith makes a comment which I think again refers a little bit to the underpinning infrastructures that it really is key to make sure that those ICT infrastructures also the people infrastructures. There's a sustainability path for those. If I paraphrase that incorrectly, Len, then let me know. Stephanie von Govill has a question. How will ARDC move these foundational or quite limited in scope in some aspects activities to broader scale and application. So I think the point here is some ARDC itself is only funded through to June 2023 at this stage the same as these particular projects. When we're going through the activity plans, there will be things that we're capturing and saying yes this is in the scope of the project at the moment these things aren't in scope and then we would submit those to the road mapping process. But at this stage, I can't say what the federal government investment will be beyond June 2023. I certainly can't say that I will see these activities as an ongoing ARDC stream. I think on their, if you consider a hassle and indigenous research data commons, I could definitely see that being a capability on its own. So a new increase capability running but but I don't know. I don't know. Christian from Katie Wilson will care and fair data principles be managed for accountability or monitored at some level and if so by whom. ARDC is underpinning work is to advance the fair and care agenda. Obviously that the fair data agenda has a longer timeline behind it but all of our workers in that space. The data retention program that I mentioned previously is absolutely at looking at how we improve the metadata associated with a particular collection. But it won't be a case of ARDC or anyone else externally policing something in the activity plan. There has to be some consideration given to what this is what we are building. This is how we will judge it. Interesting question from Peter Riley so extending on from the care and fair are increasingly emerging focus within indigenous data research is indigenous data sovereignty is a complex and systemic question and one that warrants a national consideration including around storage and custodianship of any data that relates to indigenous Australians. Do you see this as a component of this next stage of work. Maybe do you mean this stage of work or the subsequent stages of work. Maybe you could. I'm I guess both this this stage and and then following on into the next stages. Peter, I think that the concept of data sovereignty is a critical consideration, and one that the design of the activities has to consider in the same way we think about sensitive data in the health context. Data has a particular information payload associated with it. And I would expect that this is a key consideration and it's the reason that we don't build a single massive data center for storage and say absolutely everything in Australia has to sit in this space, which is a terrible pity Rosie. Yes, I know I so didn't drop you in there. I have a question from Stephanie. How will our DC connect with all the other government, government agendas with respect to indigenous data and digital platforms so it says, and I delay productivity commission and so on. Who is coordinating here is our DC going to take up this role. Yes, and that will sit within the remit of the program manager. Obviously, a critical part of the communication strategy and so building those linkages and working to ensure the transparency and openness of the next stages of the program are completely critical. But I do see it is key responsibility on ARDC to make sure that we get the right people feeling that they have every opportunity that they want to contribute to the discussion. Good question from Ingrid what are the hallmarks of success through this program for the ARDC and for the four groups outlining. So there's two things. One, we write an activity plan and do what we said we do. That's really simple. And as a result of that we create some fantastic outcomes and the ability for significant support for both those and the broader scope is picked up in the next roadmap. I'm not saying anything particularly surprising there, you know, do a really good job and demonstrate that this broad cohort, I'm not going to lump all the projects in together as a very, very broad cohort, but is recognized as the mature and appropriate vehicle for future investment. I don't have any more questions in the list. Has anybody got anything they'd like to put forward. Okay. So, the purpose for today was obviously not to leave too long a gap, following October's announcement and bring us together and start some of these conversations so thank you for all of your questions. Obviously you'll see that the details in each of the projects I'm unable to answer at the moment, because we do need to appoint that program manager and have the full of discussions. But nevertheless, it's appropriate to capture these questions and take them forward going into the discussions. And Chris testing of the ideas jam is ongoing at the moment, and in the same time, the finalization of that position description is really in the next few days. So I think the important thing from today is for me to say that as soon as we have this job advert published, and out there. The first thing we will do is circulate that link to you and ask you to, again, pass that on to your networks or stick your hand up. There could be someone there today that's just the person to join us as the program manager. I don't know. I can't see all the faces, but you might well be listening so the position will go out very shortly. And it will be open probably for somewhere between two to four weeks and we'll press on with that. So that's the next step. During that time, we are still very, very happy for you to log questions with us. But some of these we really won't be able to advance until we have the program manager, and we can move forward with some constructive and productive stakeholder discussions around writing up those activity plans. Okay. So thank you all very much indeed for joining us this morning. It shows the depth of support and commitment for these very important projects. And I look forward to speaking with you over the coming weeks.


We've been fortunate to have received another one recently, but prior to that we did have one in the office. So what we were able to do is, through King's Daughter Hospital, we acquired another AED through a program that they did and sponsored. So this has really helped us and it allows us to provide a very well needed service to the community, hopefully not needed, but obviously we have it on hand if needed. The training for an AED, it's just, it goes along with the CPR and first aid class that you have now, it's a part of that. You have it for healthcare providers and community CPR, the different levels that you train, but the AED is so important now that you train that at every level, so you don't have to have really even any medical experience to be trained on an AED because they're really simple to work. For the most part they only have two buttons, they have a power button and a defibrillation button. I've been doing this for 30 years in law enforcement, and in the rural parts of the county. It's a long time for an ambulance to respond to you, I mean it's often, sometimes an hour before an ambulance can arrive. And I recall over the years that I've done it, that we did CPR for 30 and 40 minutes of waiting for an ambulance to come, some successful, some not so successful. The quicker that an AED can be applied within certain minutes of a sudden cardiac arrest, AED is very instrumental in bringing back someone and reviving and bringing the heart back.


Good evening, everybody. The next talk is on these R&D APTs you're looking for. We will be hearing about the bad guys on the internet, how they are behaving, and how they are changing their behavior when we look at them to improve the defending against them. And here we have Inbar Ratz and Gadi Evron. I hope I pronounced it correctly. Who will give us an insight there? Stage is yours. Thank you. And play. Guys, we started. Well, we started. Anyway, while they fix that, Gadi, feel free to help them. Welcome to our talk called APT Reports and Obstacle Evolution. Or these are not the APT reports you're looking for. This is actually not the first time that I start talking without my presentation. So this is not at all exciting for me. Basically, what we're going to talk about today, we're going to talk about how APT reports mostly are beneficial, not to the defenders, but actually to the attackers. Now, when we say defenders, we're not talking about fellow malware researchers. These guys, they know their business. They've been doing that for a while. They're very technical. By the way, this presentation, oh, thank you. This presentation is not technical. If you're looking for IDA screenshots and stuff like that, it's not that. Malware researchers. Walk away now if you're looking for kernel shell code. Yes. Malware researchers, you guys can go to the encryption talk. So yeah, I know. I can see that. Thank you. So these are not the APT reports you are looking for. Quick introduction. I'm Inbar. This is Gadi. That's about it. And why are we here? So I used my time without the presentation to say that. We want to simplify the attack process and demonstrate the evolution of various actors over the years and suggest ways to close the gap. We're going to stipulate that there's a gap, an information gap, between the attackers and the defenders. And while I let Gadi start, there's just a little tradition that I have over the years.  It makes me feel more comfortable. And I'd start with David with Kaspersky. He'll be watching that. So David, this is for you. Let's everybody watch Inbar take his shoes off. He's hiding behind his podium. So a little disclaimer. We're Israelis. So last year when I was talking with Tillman, I interrupted him. We talked together. But essentially, I need to do a disclaimer that as Israelis, we interrupt each other. We're not actually fighting. As a disclaimer, just so you're ready for that. Or at least that's the story. That's my claim and I'm sticking to it. So let's tell you a little bit of a story. We're going to get into several stories and several examples about APTs and their evolution, how we can counter that, as we believe we should. But before that, a couple of examples for what we are essentially interested in doing this talk. So we always admire. That's the beginning of cyber. We can agree to that. But then we had APT-1. And on the one hand, as a security guy, I said, what, what, what? I know this is happening. Why is this such a big deal? Why is it in the press everywhere? But APT-1 was cool. It was the first time that an attacker threat actor was fully compromised. They showed everything. APT-1 with their pants down, essentially. They even showed a picture of their offices. That was pretty awesome. And they changed how we see things because now we actually had proof this was going on. And it actually affected the bad guys. But they were not alone. They were not alone in finding out that their entire infrastructure is now gone. And what are we going to do now? Oh, my God, it's going to take us a year and a half to come back and build back our entire infrastructure. And all our new Trojan horses are gone and the vulnerabilities and everything. But then there were also other campaigns. Stuxnet, as compared to Flames. Stuxnet was very tight. Bigger than most malware, I guess. But that's debatable. Well, essentially it was 500K. Modular, built for a specific target, very much about Opset. It was all about these centrifuges in the Iranian facility. And then on the other hand, you have Flame. It's huge, 20 megabytes. Everything you can imagine, all the modules, all the vulnerabilities, everything that can possibly go wrong when the threat actor loses this specific campaign, this specific Trojan horse, is now done. Just try to imagine if APT1 was affected badly, at least according to Mandiant, were they to replace their entire infrastructure, their entire tool set, as far as we know. I wonder, we don't have any information, but how did the Flame guys act? 20 megabytes. It's insane. So how do people evolve? How do they cope with that? How do other threat actors react? So we can see a few examples. For example, Gauss, it was a scaled operation, but the example Kaspersky gave was very target specific. It would only open on a specific machine. They couldn't open the encryption. It was pretty complex, technically. I don't get it. Maybe you would if you read the report or read it. But it was pretty interesting. APT3, Perpy, depending on the name you like, we recently responded to an incident responsible with APT3. And much like many other types of APTs, they first now put a dropper down, do their thing, and then use their heavy tools, not to lose their tools immediately as they enter the network. You've got Rocketkitten. Tillman and I did the talk last year, the incident train micro and checkpoint and other people came in and talked about it as well. And essentially, they used an off-the-shelf tool, core impact, everybody with their own Opsik, with their own calculation. Now, Inbar. So let's cover Opsik in 60 seconds. What is Opsik? Opsik operational security. At first, you want to ask yourself, why do I even need that? Why do I achieve with that? So one, you have to assure success. You're here on a mission, right? You need to do something. You need to steal information. You want to sabotage. You want to do something. So if you came all the way here and went through all the trouble, you want to succeed. And then you want to prevent detection. Detection is not good for you. It's not good for your reputation, for your end of the year bonus. And it might even prevent you from finishing your task if you got detected too early. And last, but not least, is the thing called attribution. Attribution is... Well, it started as a serious thing. Not so much. We're going to come back to that later. But you would like to not be identified if you do get caught, because sometimes you do get caught. And this actually also exists in other processes as well. When you do software development, you're expected your QA process, your design, if you have security by design, and stuff like that. It's intended to basically achieve the same goals, maybe accept attribution because, obviously, they know who you are. But when is it compromised? When is your OPSEC not what you want? Well, first of all, time to market. OPSEC bears costs. It takes time to do. You have to invest resources in maybe developing tricks, or maybe you have to be very careful or do something very slow, okay? For example, I can walk through metal detectors, and obviously, you know, I am half metal, but there are some detectors that if you walk through them slow enough, you don't get caught. And you need to see the faces of the operators when I do that. True story. Scalability.  For example, what happens if one sample gets caught? Well, there are gonna be many others, and they're going to be looking for them, so that's something that goes away. And of course, ease of deployment. Maybe you wanna use the same CNC infrastructure. Maybe you wanna use the same distribution channel. If your distribution channel gets compromised, then now you have a big problem. And what we're basically saying, and this is a generalization, by the way, the entire talk, it's a generalization. So yes, there are always contradicting examples, but we have a storyline which we're trying to follow. So if you wanna tell us that there is a contradicting example, you're probably right, come tell us later. Most of the APT reports represent some sort of an opposite failure. Someone got caught, and someone managed to discover what happened there to a certain degree. So what we're trying to say here is, you need to know the enemy. Now as a defender, you don't always know the enemy, because when APT is being created by nation state actors, they don't really share their failures with you. You end up reading APT reports. So what we're trying to do is trying to figure out what the other guy thinks. Now we do have one good example, the hacking team. I hope there's no one here from that nice company. Well, they got caught, and not only did they get caught, but their emails were leaked as well. And we actually have information. We have their emails, and as the report says, their primary concern seems to have been not getting caught again, which is understandable, because it's kind of bad for your business. But when it comes to nation state actors, we don't have that information, so we're gonna try to figure it out. That's just for the record, we didn't really emphasize it, a citizen lab research blog quote. Yeah, thanks to citizen lab. And we're gonna try to sort of reverse engineer the thought process of an attacker, and here's a problem. Many APT reports suck. Now when I say suck, I am trying to be provocative, because I've learned that that's sometimes a way to achieve things. Inbler, didn't you write an APT report? DVI cover, what? Didn't you write an APT report? I did. It was one, one, I think a co-author is here, I'm not sure. And then I stopped. Didn't you present one here? I may have. You may have. I may have. You may have. Okay, so what's wrong with APT reports? This guy, he's a commentator. He sits up there, and he tells you what's going on in the game, right? And an APT report or a malware research is a lot like you telling me how good the other guy is, right? Look at this most sophisticated attack platform. Look at these amazing deployment technique. Look at this amazing root kit. And it's very nice, but as a defender, if I'm not a malware researcher myself, that's not useful to me. APT reports are commonly very long. Some of them are as long as 60 pages long. And in those 60 pages, there is so much technical information that sometimes you just don't know what to do with that. And many APT reports that we see, the public ones, they're not full. The ones that we see are intended for PR purposes. And the full reports are only shift to some, maybe paying customers, or maybe there's not even a fuller version than the one that we see. And as a result, there is an asymmetry. You were horrible. I see not everyone gets it. Asymmetry. He's calling me fat right now, just so we're clear on what's going on here. Oh, I called you bald. So there is an information gap because the attacker can use all that malware research stuff. So the information gap benefits the attacker, but not just the attacker. Everyone learns because all the other actors are reading the same reports. And even though this talk is about APT and nation state actors, we'd like to remind you guys that the malware writers that work in the cyber crime world, they also read the reports. And actually what we're seeing whenever there's an APT report out, the technology is leaking to the criminal world. And that makes APT reports actually free QA for the attackers. So sometimes you can see lessons learned, right? The APT1 C2 infrastructure was huge. Parts of it, like big parts of it were registered with the same name, same email address. You might remember ugly gorilla at 163.com. But the Terla malware use a very sophisticated satellite downlink hijacking through ISP to inject packets that could be received without actually exposing the location of the destination. And then we had learning in progress. So they're learning, but they're not done yet. So Stuxnet and Dooku and Flame, they all share the same code. All reports clearly show that. And guess what? Dooku too is still using large parts of that framework. Now remember we talked about Opsic. It's a lot of time and money to develop such a thing. So you do try to use whatever you have left. And some things, well, you never know. And attribution is a good case. If you look at Iron Tiger, clearly Chinese. But it was sent to Taiwanese targets with traditional Chinese versus simplified. The attack emails we're talking about, the matters of the straits. Coretto, well, everything fits so well. The language, the identities, everything looks perfectly Spanish. In fact, it looks too perfect. Even if you look at geolocation, their attacks were against some activists nobody would care about except for Spain. But in Dooku too, they were already playing games with the researchers. There are multiple false flags. We know that they put in the Ilegarilla string, which is Chinese. We know that they put in the Romanian anti-hacker, which is Kostin's Twitter alias. So they start playing back with us. So you read an APT report. You take the time. You read 60 pages. What do you get? Well, you get a lot of malware analysis. That's the major part of what you get. After that, you get a little bit of IOC's, indications of compromise. And they will be about the malware. That's actually actionable intelligence. But not all samples go on multiple targets. So you look at the C2 infrastructures. You get the main names. And that's also actionable. But with the development of Opsik, these stopped being shared across campaigns. So the long-term value of each of these IOC's is very small. And at the end, at all, there's very little about the attack vector, how the attack was actually facilitated, how did it all start, and what was the attacker objective? What did they steal? Because you really want to know what they were doing there. It's nice that they hacked this company, but what were they looking for? So I'm a little bit confused at this stage, because we see a little bit of this. We see a little bit of that. There were maybe some false flags. We're trying to make sense of an APT report, perhaps, on research, perhaps, to defend an organization. What is actually going on? Are we getting the correct picture? So what we did so far in the previous slide is try to re-engineer, reverse engineer, what the forensics process essentially reverse engineers what the attacker does. So how about we re-engineer what they do and actually talk about the attack process, about the engagement of the attackers. In a simplified model, we just simply call it an engagement process. So we start with simple intelligence requirements, IRs, the thing we have the least information about. And it's essentially like going shopping. What am I interested in today? Is this this nuclear deal? Is it this interesting product that is developed somewhere around the world? What would you like to know? Now, let's just take an Iraq example, because it's older now, so people won't be as sensitive to it. That's what I'm saying as WMDs. Where are the WMDs? Does he intend to use said WMDs? Who is working on the WMDs? And can we save Matt Damon? Can we get Matt Damon back yet again? Just wondering about that. And then the second part is, let's compile a target list. Where can I actually get this information? So sometimes we're a person who would hold the information I want. So verticals, banking, pharmaceuticals, energy, aerospace, that's interesting enough. Or we can talk about specific targets. We're interested in this target because they hold the information we want. Then again, we said we won't do many counter examples. But one important counter example that Tillman gave us was the Sufacy Group. They're everywhere right now, very high-profile. And they seem to be very opportunistic. They don't seem to be working with any specific IOCs. It's like, they find the information, then try to sell it. So not everybody works according to this model. After the intelligence gathering, and I'm gonna pass it over to Inboard in a second, it starts with reconnaissance. And then we have a target report. Essentially, we're trying to figure out what's going on, what can we find out, how can we get in, and then get all this information in organized fashion. Inbar? So the target operation is basically everything you need so you can do your job, okay? And once you have that, you can start acting. And you do that by attack plan and execution. And this is an iterative step. You start by an attack plan. This is how I intend to plan. Let's say I wanna send an email, or I wanna use force scanning or SQL injection. I choose some technique. And for that, I need to choose my tools, right? Sometimes I will use off-the-shelf tools. Obviously, this is a very large shelf, as we've all learned. Sometimes I will customize. Sometimes I will write something particular for this target. Sometimes I'll just take somebody else's malware and make small adaptations, right? When you do that, you get these examples. Let's look at two, Stuxnet and Gauss. Stuxnet, as Gatti mentioned before, was very targeted. The code that was there was meant to deal with PLCs of specific vendors doing specific things. That thing had absolutely no use anywhere on the planet, anywhere else, right? And Gauss, it was a big multifunctional tool, but there's still one mystery that no one managed to solve. On the USB infection mechanism, they found an encrypted payload. The payload is encrypted by an MD5 hash run 10,000 times on certain parameters of the hard drive. And in fact, till this very day, no one managed to find out those parameters. They don't know which computer was the designated target. Many people tried to enumerate on that. We still don't know. We only know that there is only one computer on this planet that will have the payload decrypt and execute. Or in a Dell Secure Works report of one of the targets that they analyzed, turns out that the attackers took advantage of a platform that already existed in the target. It was an endpoint management program, and they used that to lateral move throughout the organization. By the way, we saw the same with target, right? They used accounts installed by another program to open shares, so we see that all the time. And once you're in, you acted on your plan so the first time you just get into the target. Now you're running code inside the target, but your job is not done. Now you need to move forward. Lateral movement, maybe get to the real place because you usually use the weakest link to get inside an organization. So now you go back to intelligence gathering, and this time it's a little different because now you're no longer outside of the screen. Everybody's staring at the screen. Stop talking. Who got that to their Facebook? Who didn't watch it all the way? Really? OK. So we're going to save you. That's a minute 40. But when you're inside the target, things look different. No, no, just kidding. We wanted to do that, but copyright in two more minutes of your life. Intelligence gathering is different now because now you're inside the target, and the target has all sorts of defenses. So your OPSEC gets revisited, right? You need to map the target's defenses. What are they using? Do they have an AV? Any peripheral devices? Well, what are they using? Any sandboxing? No, the interesting thing about this is when you think about intelligence operations, you think about your target. What am I going to face? Am I going to face an IDS? Am I going to face something else? I need the clicker. You talk the clicker thing. OK. There we go. Didn't say please, though. Please? OK. So essentially, originally, you would say, what am I facing? Then is it a threat to me? So for example, they may have a security control, and you wouldn't care about it because it wouldn't stop you. But then things started to change. You would start saying these antiviruses, for example, may not threaten me, and I can bypass them, but they have an entire home base, the back end, where they can go and later on threaten me. Quiet signatures, whatever it is that is written about right now, they can essentially find me after the threat. So that's a threat. I have to take different antiviruses into the threat now. That changes everything. But it's still not good enough. Look for other players. Think about it. There is another player on the machine, and Reagan is a very good example. When Kaspersky even called the computer, they found it on an APT magnet. So now am I supposed to think about looking at the computer and saying, which other nation state slash criminal organization slash whoever it might be, have some tools installed here already and I need to collect intelligence on that in retrospect or wait and analyze every system I go to? That sounds like a little bit too much work, but it's something that depending on your app stack, you're going to have to face now. Then the last thing is really, but really try to hide your identity, unless you're some of the Chinese group and then you don't care. So we have a few example. The hurricane panda, you can read the report about quiet strike. There was actually a duel there. They got detected, the incident response team came, they started dueling for a while and it took a while of this dueling before the actor decided to give up. This is from Symantec's report, the Stuxnet. Look at the information. Back then no one cared about anything or maybe they were just naive because nothing had been caught before that with maybe one or two exceptions. Here you have the compile compilation times of all the files used inside the target and then you have the infection time. Now aside from the fact that as you can later see in Kaspersky reports, the compilation times are used to determine the attribution. This gives you a lot of information. This tells you how long it takes them from the creation of the file to the deployment. That will tell you about their attack operation. Does it take a minute, a day, a month? You can learn a lot from that. So we have other examples. Actually just one thing is if you look at Ducat 2 you can see they started to randomize that. They started looking at forensic analysis as a threat to their existence which goes back to the previous slide. So obviously the threat actors evolve, right? We have views of previously existing tools or integral tools of the operating system because you can't sign on those, they're gonna be there. So you don't need to deliver anything, you don't need to worry about deployment encryption. You land there and just use like PowerShell or AT or IP config. And Ducat 2, and we keep reminding that sample because it's very impressive, they had this huge leap forward.  They actually, the lateral movement was done in RAM only. They only used several vantage points, computers that they were sure that they would maintain command of and everything else was running code in RAM. So if that machine rebooted then the superior computer could reinfect it from far. But that changes everything about how you act inside your organization. So another aspect which people usually don't talk about when it comes to APTs or other types of operations is their retreat. We often talk about dismantling, but we don't talk about the folding action, the full deck. And we can see some examples over time. Costin Raiou spoke about this a few times. For example, right October, they dismantled the operation after the publication, took them a little bit of time. The mask, Keredo, there was a blog at Kaspersky and four hours into this blog publication, they were gone. And Kaspersky didn't say anything, they gave their own name, the mask. And within four hours, their entire infrastructure was gone as far as I know, we can talk to Costin about that. And Dukut too, they didn't even wait. They hunted the vendor, they went into Kaspersky to try and figure out what's going to happen. Maybe they had other reasons as well, but it shows an interesting story. Of course there are counter examples, again we don't give many of these, but some of these guys just don't care, like APT-12, Gaza Hacker team.  They just don't care. Or they don't know. Or they don't have the operational capability to even know, hey there is a security conference called CCC, let's go watch if we're being compromised. So with that, we would like to take the methodology we built about how the attacker worked, what we re-engineered about their tactics and try to look at the defender side. Now because of the limitations we have of information in forensics and the reports, this is a little bit difficult, this is work in progress. Maybe you can help us out. Maybe we can build it to be better, which is the entire idea of this talk. So we're working by problem, take aways, and action for each one of these issues. So first of all, the intelligence requirements. We do not have enough information about the attacker objectives. If you remember the graph from earlier, that was the least amount of knowledge we had about any attacker, or most attackers. And essentially, the understanding here is they are kind of stalkers. If in their information requirements, they know they're interested in something you have, they're not gonna give it up.  You know, you might wake up and say, squeezy. Squeezy? Actually, there's one example for that. You can see at this report presented RSA conferences, RSA conference by Dell. Silver Debt did that at RSA this year from CrowdStrike. Yeah, from source, yeah. I guess I'm too tired from Dell Secureworks, there we go. So you can see that there was some battling going around there, and then the attacker lost. And then there was a quiet weekend, but when the weekend was over, they came back with new tools. Why? Because they had an objective. They had things that they needed to bring over. And just the fact that they got detected once does not mean that they're gonna say, okay, forget this guy, let's go somewhere else. There's an interesting issue discovered here which Phil usually writes about, which is they will escalate as needed. Meaning they may use pre-lame lateral movement tools and then escalate as they find a position. Right? So the second takeaway we have is that stealing data is just one of the options. And I believe everybody remembers when this happened, and many readers they started saying, what? How did this happen? It is huge. Now again, just like APT-1, coming from a security background, I said, yeah, this is just another hack. Naturally, from their perspective, this was a major issue. Naturally so, if they should take it seriously, and I feel sorry and I would help if I could. But the main point here was there was a risk. And that risk was what might happen once they have a foothold inside my organization? And everybody is used to thinking mostly about data theft. They kicked the body here. And that is something we need to take into consideration. Essentially actions. There is a classic tool in security management called risk assessments. Usually it's a useless tool. It's a huge document, 200, 600, 2000 pages. You write down for regulation, throw it away at some point or you need just to click the box. But risk management is meant to be used and used correctly. Meaning if you know there is a potential risk of an attacker getting in, and then you know the potential risk is them doing, for example, damage check your impact. The impact is important to determine that risk. Make risk assessments make sense for your daily operation as opposed to being some documents that's policy that's never used. The second part is what can we do about the target list? So first of all, our problem is we don't have time sensitive information. We can't really determine a pattern. Now maybe this is available in closed circuits. Maybe not, maybe not always, but our takeaway from this is that we need to be able to get the information which we'll talk about, but more than that, if you have a similar target to you being compromised or you're using similar technologies or platforms, take note. Don't wait to be attacked or for an organization just like yours to be attacked for you to take note and start doing something about it. So if you guys follow Brian Krebs and you know that right after the target breach, every other week there was a new piece about this was breach and that was breach and this was breach. And the thing is, the first moment that there was a target breach and they stole credit card numbers from point of sale devices, everyone who has a point of sale device should have said, oh my God, I could be next. And instead of actually going and making a big, big effort to see if they had already been compromised, everybody was just sitting and hoping that their name doesn't come up on the following week. Which brings us to another tool in classic securities that's been ignored and that's essentially the threat assessment.  So now we know they have a capability but we also know about the intent. Not necessarily against you, but we know this has happened. It should change how we operate. This is based on intelligence. We may not have exact intelligence. This guy is trying to get us but we have intelligence out there now. Somebody is doing this and more than that they may be doing it to people similar to us. Now when it comes to the cyber engagement cycle or so we call it, the cycle of the three repeating steps, we decided that we're gonna change a little bit the format or we're gonna treat all three steps together or we're gonna divide it into two stages, the pre-engagement and the engagement itself. Now, pre-engagement is when everything still happens outside of your organization. The problem is twofold. One, publicly available sensitive data. Anywhere from complete employee lists, network sketches, sketches to say who works in your organization, under who, what's the hierarchy. Anything that you can get from the other. People who will later you will be discovering use security questions whose answers are on their Facebook. Okay, that also happens. And the second problem is lack security awareness which in turn allows probing. Now that probing can happen in two ways. The first one is the one we all know. You just use it automatically. You use tools, you scan the network, you look for open ports or default passwords or bad configurations. Everyone does that. You just take a tool set, you do it. But people always also do that manually. Making phone calls, right? In my previous job I worked at Checkpoint at least twice. I was randomly next to the reception desk while the reception desk was trying to deal with such a decoy call. And on both times then they handed it over to me because I was excited that the opportunity to speak to a scammer. And you start asking them a question and then they hang up, but this actually happens. Next, there we go. One more. The understanding here is that, and this is basic, right? The attacker can gain a lot of information. They can do the full operation sometimes without ever doing anything active against your operation. You need to know this is possible. You need to control what's going on. You need to limit public information as much as possible. Naturally, you won't be able to do everything. You need to act outside your own perimeter, which is a critical thought to even have in this day and age. As well, and this is very important to indoor specifically, awareness refreshments. This is a human problem, not a technical problem. It's whatever awareness can gain, even if it's not much, should be attempted. It helps. You can tell you from my own experience in the Israeli cert, human sensors, so-called, people report to us. We are open to them. We're asking them, what's going on? And sometimes, I don't want to say waste of time. That will be rude. Sometimes the call is negative about nothing. And that's fine. We treat it with all seriousness because many of the best reports we ever got were from people when you were interested, new to watch for stuff and alerted us. Now, the problem with that is that, let's face it, these are quite obvious. But at the same time, they still don't happen. So the attackers still make the same progress because the basic stuff keeps staying under the radar or unattended to. So we come to the engagement scale, so a stage. Now, the attacker is already inside your network. Not a lot of compromised organizations or APT reports, for that matter, share the lateral movement part. Mostly it's about secrecy or privacy or... They may not even have this information sometimes. Now, let's face it. Everyone is being hacked. Everyone will be hacked. Everyone has been hacked. It's not a shame anymore, okay? It happens to everyone. And if you pretend that it hasn't happened to you, then I'm worried because you're probably hiding something else as well. And the takeaway is the engagement is an ongoing process because it's not a hit-and-run thing. They don't not always get directly to the computer that was interesting and the data that was interesting. They will stay around in your network for a while and it gives you many opportunities to get in the way. You have more time, you can think, you can plan, you can influence, and the action is indeed influence. You need to put as many obstacles as possible. Layered security, deception, okay? The attacker needs to spend time and effort and resources in your network because the longer they are in your network, the safer you are. It takes them more time to get to the interesting part and it gives you more opportunities to catch them. We may have separated the pre and the post engagement. Sorry about that. Go on. We may have separated the pre and post engagement but it is an ongoing cycle and that's the important thing to understand. They got in, they used the node, they find. The more obstacles you put in place, the more basic security you put in place, the more time you will have to find them and it will be hard for them to continue operating.  Share your breach data. Yes, someone has to be the first but you don't have to tell everything. Tell about the technique. Tell about the things that other defenders and other CISOs or IT security guys can use and you start sharing, other people start sharing and we're all safer. And the last stage is essentially the fold and retreat. And hey, if you don't wanna share the actual information, the heads up would be nice, just saying. So this is interesting. I never previously had this thought. It's really interesting for me when I think about something that's new for me in security. At least emphasize the security in a different way. Everybody says cyber nowadays. What's different? And one of the realizations is yes, attackers have been deleting logs throughout the lifetime of security but the attacker can destroy forensic evidence. Do we plan for that? A lot of our security today is based on after the fact incident response and forensics. That is a major understanding for us right now. It's not just about endless monitoring and endless alerts. That is effectively where many of us gets our first alert, post the fact. And if an attacker can destroy the forensic evidence, we need to make sure it's there. Snapshots and logs, both, can potentially save the day. So we built up this idea that backup of log files, for example, and snapshots could be, and I'm going to exaggerate, more important than even active monitoring, the backup of the logs. Naturally it shouldn't be that way but nowadays it just might be. And while we were working about this, we decided to, well, INBAR came up with this word to describe this new backup response plan, everything is abbreviated in security, right? Just saying. So with that, let's try and understand what we just went through because some of this was common sense. Some of this was a little bit new but the idea is to be able to make it repeatable. What can we learn from the APT reports and how can we use them on a daily basis? How can we use them whenever a new APT report comes out to better our security against known threats? So we looked at the left here about the information we have in APT reports and the diminishing levels of it and we look at the engagement process as we simplified it and like it. Now, if you get an APT report, first of all, try to understand, not just read the report and look for the IOCs, how much information do you actually get for each of these? Do you have any attacker objectives in there? You know, when you talk about the targets, usually maybe if you're lucky, you have the verticals, pharmaceuticals, whatever, aerospace. Once you're through that, it's easier for you to go to the re-engineering process and say, let's look at how the attacker works, the engagement process, try to put the data in there and to do what we just did. What are the takeaways specific to this report as far as intelligence gathering goes? What are actions? Actions I can actually take based on this report, based on what knowledge I have, the scope of the knowledge, the relevance of the knowledge from the APT report. And the key part of this really is we need to demand better APT reports, APT reports that are actionable. Now, we have one more problem. We don't actually have a solution for that but we thought it was important enough to note it. It is something that we call the decline of shame. In the beginning, like we said, you didn't really want to be exposed. Attribution was a huge risk for you. And it still is for many actors, especially very particular nation-state actors or in the case of other nation-state actors, very particular branches or subgroups of those nation-state actors. But with some of the nation-state actors and with some of the criminal groups, you see that they don't care anymore. They get caught. And you know what happens then? Pretty much this. They continue working and operating while... Leave it on for a while, please. Leave it on for a while. Yeah. Some groups have actually been following the blogs of the AV vendors that were tracking them, adapting in real time. There was a case. I don't want to quote too much about it because I didn't have time to research it, where Trend Micro and Alien Vault were updating their blog live, if I remember correctly, and the attackers were changing their modus operandi just now. They're not much more than that according to network defenses as opposed to something else. Live, which is really interesting. So we were being optimistic. We said, okay, what do we want to see? Because we said in the beginning, this is not a technical presentation. This is a high-level thing and we're presenting sort of a raw thought process that we started... I mean, think about it for a minute. We took the time to study many, many cases of APT reports and to talk to many of our friends just like Thielman and many others we'll give credit to in a minute and to look at our own research and forensic information.  Just think of that concept for a minute. We're thinking, once again, not about the fellow malo researchers but the people that actually have to defend certain organizations. And it's not all corporates with their own IT security teams that are all very skilled and very qualified. Sometimes it's an organization that has an 11-people IT team and then two of them wake up one morning and they are told, you do security now. This is reality. I've talked to more than one customer that has that happening to them. So we would like to see better and more actionable APT reports. We need, and when I say we, it's not just Gadi and myself, it's the community. It's the poor people that need to protect organizations with the knowledge that they are being attacked all the time. We're not saying that all APT reports suck. Some of them are very, very good. We're just saying nowadays most of them are just for PR and that is hurting us. It's helping the attackers. It's the attackers QA. We would like to see better reports for us, for more vendors. They need to be more actionable. If I am a CIS over an organization, there should be something that I can use. Use, and I want to be at a better place after taking the time to read one of those reports. And we need earlier breach reports. Like Gadi said before, give us a heads up. It's important to understand that maybe there's a new trend going on. Maybe if we had the heads up on target and everybody else were after listening to this talk, then all the other point of sale vendors would say, you know what, maybe we should look into our setup. Because even until today, after so many compromises, there are still so many point of sale terminals connected to the internet with default credentials. It's been written in so many places. I was really worried about, well, weary, about the next item on the list. Because I have this thing where I identify, it's not fair given, but I identify people who are new business security by them saying information sharing collaboration. And somebody says, oh, we need to do more information sharing. I say, oh geez, not again. But honestly, actionable information sharing and public information sharing. Information sharing is happening. Some of it is actionable. Much of the public information sharing, which isn't much, isn't actionable. We need to understand that even the heads up we talked about could be critical, really critical. And understanding how this can help us as a common ground. And lastly, and Kudos to Dave Marcosif is watching us, enough with attribution stuff. Yes, we care about attribution. Yes, it helps us. Yes, we can think about the business as opposed to think about who is actually looking at us. Targets, I understand that. But then, what does it actually give us? How much is the attribution work that people spend so much time on just to justify the lizard part of the brain so we feel better about it, better than the other information we could have had to actually protect our organizations? I'm not really sure. But it annoys me in a way. Enough with attribution, or at least enough with the attribution that makes no sense. Attribution is about more than just IP addresses and we've given a few examples during the talk. I mean, yes, if the compilation times, they never work on a Shabbat. It's maybe Israel, I don't know. But then again, nowadays, we've seen a lot of false flags starting to be put in there. It's not easy. There's political information to consider, but we never really know. But whenever we talk about APTs, people go out and say, I was seized on the one hand because it's available. And on the other hand, who did it? Who done it? Tell me. Okay, I'm willing to give it the chance to say, fine, but it's not the most important thing. And if you do it, do it right. So final words, APT reports can be a huge help, but there has to happen some change. The problem with the change is that contradicts certain economic interests of the people that create those APT reports. But we believe that if they start producing more value then that influences the amount of customers. We will be evolving as well as the attackers. Right now, APT reports are so-called bad, which is arguable, because they're the only ones really evolving. If they were made the right way, we're not saying our way is the right way necessarily, but if they were made in a way that would help us more, they could be a huge help. And remember, attackers are not going anywhere. We're not gonna have any less business because of better APT reports. And something that we call, stay on the attacker's six, they need to be worried all the time. They need to be looking behind the shoulder to see if there's anyone on their six. This is a pilot jargon. Think what this evolution means. Gauze, for example, as opposed to Flame. If they have to spend so much time on one target, their cost grows exponentially. If we can do that with APT reports, hey, I like this evolution. It's not necessarily bad that they have to get better. And the last thing, increase the cost of the attacker. Anything you can do to increase the cost of the attacker, do it, whether by installing products, using services, improving the people or the awareness of the people that work for you. You want to make it harder for the attacker. It gives you more time. It's probably not gonna deter them, okay? Let's face it, they're gonna come anyway. They're doing their job just like you're doing yours. There is an asymmetry, and as we said, we are not, the attackers are more powerful than we are. We have to admit that right now. We're trying to change that. But more than that, they're not gonna give up the IRs. They're gonna keep going. That said, we can start making it better for us. Create more symmetry. So this is important, so let's go through this. Yes, we did a lot of research. Yes, we looked at a lot of code. Yes, we looked at a lot of reports. Yes, we have a lot of information that is not public. But trying to construct it in a way that will number one, not be boring, while trying to create this sort of methodology out of it, while trying not to do too technical, while still giving examples that build the methodology of the attackers based on what we know about them wasn't easy, and we stand on the shoulders of giants. People in the community, industry, blogs, reports, a lot of people we wouldn't be here without and they need to deserve their credit. Special thanks and references we took from, Tillman right here, Ned Moran, Phil Burdette, Kostin Rayu, H.D. Moore, Peter Cruz, Chris McConkey, Kevin Mandia, and the Grok. And especially to Nitsan Sedan, who did a lot and provided a lot of significant research support for this presentation, so thank you. And with that, I would like to, before we ask the questions, most therapeutic reports, not all of them, suck. This hurts us. They become better. We can be better than they are. We can use this to keep them on their six, like we have seen with them evolving to a place where they can do less. They can scale less. And that is what we would like to see. Thank you, and we would love to have questions. So thank you, Inbar and Gali. We have a question from the watchers on the internet. Hi, so the question is what is the state of APT response across industries and which industry do you think is most vulnerable now?  Okay, I'll try to respond. I think APT response, there are a few organizations out there that are extremely good at this because they have decent security. They have incident response. They have controls in place. They have monitoring place. They are good at security, which is why they're good at APT response. That said, a lot of the APT response, a lot of the incident response is now outsourced. So they bring companies in to do it for them. And that is why I believe that saving the logs, as I said, as a backup is more important. And essentially the incident response is becoming a way, again, I'm gonna get flints for this, are monitoring. So I'd say some people have really good stuff going on. Most people don't. And those and others bringing the outside help. Another problem with the outsourced security or incident response is that there is a huge difference or there can be a huge change in your ability to do proper response based on what your network looked like before. So if you only call the guys when your house is already on fire, there's not a lot they can do. But if you brought them in before and they help you treat the house, then you're much better off. Another aspect that's interesting, although small, is that forensics and incident response used to be about keeping logs, chain of evidence, all of that stuff. People still do that. But honestly, today with APTs, it's not as important. It's more about finding the actor as fast as possible, a week, two weeks, three weeks, and then moving on to remediation, which will take forever and cost a lot of money. And then it's essentially, in many cases, about installing some agent on all computers and then trying to identify what's going on. So it's not as much about what forensics used to be when you learned the forensics course anymore. But I am not an expert on this as much, and you should ask this question again from other people as well. Thank you. Okay, and microphone two, please. Yes. First, I'm not really sure this is the correct audience for your talk, because at least I'd suspect that most people here, when they build a network and when they, they're already trying to make it as secure as they can, even without considering a special attacker or a special type of attacker or considering what information an attacker might try to get because they'd probably try to make the system as secure as possible as is. And if they do not take a certain choice to use a certain type of security mechanism, it's probably just because their use case simply does the lower it. And even if they had been hacked, they couldn't change that. For example, I work in a public sector and with us the problem is mostly just you have to, you don't have the people to fix stuff. I work at a university and they're basically the situation is that every professor and every institute and every whatnot has its own IT team. And usually the IT team is just the secretary that was put on the list that is saved at the knock where they know which subnet is attributed to which institute, but usually they don't have an IT team at all. So basically what you see is that an institute of let's say 20 or 30 computers runs Windows XP in the year 2015. Most of them not even patched with the patches that are there for Windows XP. And I know that this is not just a single case at my university, but I know that there's plenty of other universities and other stuff where it looks just like this and usually the problem there is they don't even have an IT. So the actual problem is that first you'd probably need to get the, what you'd call the management level above, the lacking IT to actually hire people at all to have an IT team at all. Yeah, well, that was my line. That was my line. So what's the actual question? So let me answer that. First of all, you're correct. And obviously you care about that and you shoot. So two answers. First of all, you talked about the audience. CCC is a huge stage and a lot of people are gonna be watching that, not just the people in the crowd, but people who watched it streamed and people who are gonna watch that later. That's one thing. Second thing is that we're trying to start something here. We could fail. We failed before in certain things, but if we can start changing something then in the long term, we will make a difference. And what you're saying is true. It's easy though to pick that one example where nothing that I've said will help. That does not make what we've said not good because progress takes time. And maybe with time, some of that progress gets into default setups. And when you say that most people here have a secure setup, well, guess what? It's the same opposite that I mentioned before. Sometimes you have time constraints. Sometimes I work at a startup company. You know you rush. Sometimes you get to things later than before. You have a project to make. You have deadlines. You rush. Sometimes you do things later than after. It does not mean you don't know it. But at the end of the day, the reality that the attacker sees is what matters. So maybe not everyone in here was the right audience. By the way, we didn't choose the audience, right? It's not that I have anything against you guys. Thank you. I knew you were coming, though. Yeah, say, say. The thing is, CCC is where the trenches are. CCC is where people who do stuff are. CCC is where the ideas get born. CCC is where people go back to their organizations from where the technology is and grow. This is exactly the right place in my view to do this type of talk. You're asking for a different type of talk. You're asking for how can I do hunting on my own when I don't have a lot of resources? And in our talk, we only gave this a little bit of reference in risk assessment and essentially risk assessment is, connects to how much resources do you have, what you can have, but that's a different talk, how to do that. So I'm sorry we didn't give it much attention to that. I'd also have a second question. You asked that people should release information about bridges earlier. Doesn't that contradict your example where you said that you had cases where the attackers were changing their attack schemes in life while, for example, when Michael was... That's a very, very, very good question. Thank you for bringing that up. It's always about timing. Again, we couldn't bring everything into the talk. That's a comment a friend of ours gave us. If you're still engaging in IR, maybe you can give it heads up to somebody, maybe in a closed circles, maybe in open circles. You don't need to give away the home world. Definitely, it's always about timing, about the right time around the place. We would like, whenever it's possible, to release the information and not necessarily all of it. But you're absolutely right. Okay, thank you. Okay, and we are out of time, but I want to hear the last question from our viewer because these people are not here and they can't be asking you after your talk. Very nice, thanks. So, when you say APT reports suck, do you just mean the publicly released ones or also the ones from security firms to their customers? So, obviously, we talk about the public ones because a lot of work is being done by a lot of vendors, but not everyone is a customer of all the vendors. Usually, you're only a customer of one vendor and some vendors have more luck with APT. Depends on their coverage and some don't. And at the end of the day, this problem is about everyone. So, we don't want to improve it just for people using a certain vendor. We want to improve for everyone, and yeah. Kumbaya. Thank you very much. Thank you. Thank you.