WikiLeaks Vault 7 CIA Hacking Tools Revealed Year Zero Press Release Today, Tuesday, 7 March 2017, WikiLeaks begins its new series of leaks on the CIA's Central Intelligence Agency, codenamed Vault 7 by WikiLeaks. It is the largest ever publication confidential documents on the agency. The first Vault part of the series, Year Zero, comprises 8,761 documents and files from an isolated, highly secure network situated inside the CIA's Center for Cyber Intelligence in Langley, Virginia. It follows an introductory disclosure last month of CIA targeting French political parties and candidates in the lead-up to the 2012 presidential elections. Recently, the CIA lost control of the majority of its hacking arsenal, including malware, viruses, trojans, weaponized zero-day exploits, malware remote control systems and associated documentation. This extraordinary collection, which amounts to more than several hundred million lines of code, gives its possessor the entire hacking capability capacity of the CIA. The archive appears to have been circulated among former U.S. government hackers and contractors in an unauthorized manner. One of whom has provided WikiLeaks with portions of the archive. Year Zero introduces the scope of direction of the CIA's global covert hacking program. Its malware arsenal and dozens of zero-day weaponized exploits against the wide range of U.S. and European company products, including Apple's iPhone, Google's Android, and Microsoft, Windows, and even Samsung TVs, which are turned into covert microphones. Since 2001, the CIA has gained political and budgetary permanence over the U.S. National Security Agency, NSA. The CIA found itself building not just its now infamous drone fleet, but a very different type of covert global spanning force, its own substantial fleet of hackers. The agency's hacking division, freighted from having to disclose its often controversial operations to the NSA, is premier bureaucratic rival in order to draw on the NSA's hacking capacities. In the end of 2016, the CIA's hacking division, which formally falls under the agency's Center for Cyber Intelligence, CCI, had over 5,000 registered users and had produced more than 1,000 hacking systems, Trojans, viruses, and other weaponized malware. Such as the scale of the CIA's undertaking that by 2016, its hackers had utilized more code than that used to run Facebook. The CIA had created, in effect, its own NSA with even less accountability than without publicly answering the question as to whether such a massive budgetary spent on duplicating the capacities of a rival agency could be justified. In a statement to WikiLeaks, the source details policy questions that they say urgently need to be debated in public, including whether the CIA's hacking capabilities exceed its mandated powers and the problem of public oversight of the agency. The source wishes to meet about the security, creation, use, proliferation, and democratic control of cyber weapons. Once a single cyber weapon is loose, it can spread around the world in seconds to be used by rival states, cyber mafia, and teenage hackers alike. Julian Assange, WikiLeaks director, stated that there's extreme proliferation risk in the development of cyber weapons. Comparisons can be drawn between uncontrolled proliferation of such weapons, which results from the combined with their high market value and the global arms trade, but the significance of Year Zero goes well beyond the choice between cyber war and cyber peace. The disclosure is also exceptional from a political, legal, and forensic perspective. WikiLeaks has carefully reviewed the Year Zero disclosure and published substantive CIA documentation while avoiding the distribution of armed cyber weapons until the consensus emerges on the technical and political nature of the CIA's program and how such weapons should analyze, disarm, and publish. WikiLeaks has also decided to redact and animise some identifying information in Year Zero for in-depth analysis. These redactions include tens of thousands of CIA targets and attack machines throughout Latin America, Europe, and the United States. While we are aware of the imperfect results of any approach chosen, we remain committed to our publishing model, and note that the quantity of published pages involved 7 part 1, Year Zero already encompasses the total number of pages published over the first three years of Edward Stone's NSA leak analysis. CIA malware targets iPhone, Android, Smart TVs. CIA malware and hacking tools are built by even CCI Center for Cyber Intelligence at the department belonging to the CIA's DD5 major directors of the CIA. See this organization chart of the CIA for more details. The EDG is responsible for the development, testing, and operational stupus by the CIA in its covert operations worldwide. The increasing sophistication of surveillance techniques has drawn comparison with George Orwell's 1984, developed by the CIA's embedded devices, Perange EDB, which invests Smart TVs, transforming them into covert microphones is surely its most emblematic realization. The attack against Samsung's Smart TVs was developed in cooperation with the United Kingdom's MI5 BTSS. After infestation, Weeping Angel places the target TV in a fake off mode so that the owner falsely believes the TV is off when it is on, as a bug recording conversations in the room, and sending them CIA server. As of October 2014, the CIA was also looking at infecting systems used by modern cars and trucks. The purpose of such controls is not specific, but it would permit the CIA to engage in nearly undetectable assassinations. The CIA's mobile device, Perange MDB, developed numerous attacks to remotely hack and control popular smartphones. Infected phones can be instructed to send the CIA the user's geolocation, audio and text communication, as well as covertly activate the phone's camera and microphone. Despite iPhone's minority share, 14.5% of global smart phone market in 2016, a specialized unit in the CIA's mobile development branch produced malware to invest, control and exfiltrate data from iPhones and other Apple products running iOS, such as iPads. CIA's arsenal includes numerous local and remote zero-day develop developed by CIA or obtained from GCHQ, NSA, FBI, or purchased for cyber arms contractors such as Pate Shop. The disproportionate focus on iOS may be explained by the popularity of iPhone among social, political, diplomatic, and business elites. A similar unit targets Google's Android, which is used to run the majority of the world's smartphones 85%, including Samsung, HTC, and Sony. 1.15 billion Android-powered phones were sold last year. Year zero shows that as of 2016, the CIA had 24 weaponized Android zero-days, which has developed itself and obtained from NSA and cyber arms contractors. These techniques permit the CIA to bypass the encryption of WhatsApp, Signal, Telegram, Weibo, Confide, and Cloakman by hacking the smartphones that they run on and collecting audio and message trans traffic before encryption is applied, targets Windows, OSX, Linux, and routers. The CIA also runs a very substantial effort to infect and control Microsoft window users with its malware. This includes multiple local and remote-weaponized zero-day air-cap jumping viruses such as Hammer Drill, which can infect software distributed on CDDDS such as USBs, systems to hide data in images or in covert disc areas. Actions, efforts, or develop several attack systems for automated investigation control of CIA malware such as Assassin and Medusa against internet infrastructures and web servers are developed by the CIA's network devices branch, NDP. The CIA has developed automated multi-platform malware attack and control systems covering Windows, Mac, OSX, Solaris, Linux, and more such as EDBs hide below CIA-ordered vulnerabilities, zero days. The NSA, the U.S. technology industry, secured a commitment from the Obama administration that the executive would disclose on any ongoing basis that the hard, serious vulnerabilities exploits bugs for zero days to Apple, Google, Microsoft, and other U.S.-based manufacturers. Serious vulnerabilities not disclosed to the manufacturers places huge swaths of the population critical infrastructure at risk to foreign intelligence or cyber criminals who independently discover or hear rumors of the vulnerability. If the CIA can discover such vulnerabilities, so can others. The U.S. government's commitment to the vulnerabilities equities process came after significant lobbying by U.S. technology companies who risk losing the share of the global market over real and perceived vulnerabilities. The government stated that it would disclose all pervasive vulnerabilities discovered after 2010 on an ongoing basis. Year zero documents show that the CIA breached the Obama administration's commitments. Many of the vulnerabilities used in the CIA's cyber arsenal are pervasive and some may already have been found by rival intelligence agencies or cyber criminals. As an example, a specific CIA malware revealed in year zero is able to penetrate, invest, and control both the Android phone and iPhone software that runs or has run presidential Twitter accounts. The CIA attacks the software by using undisclosed security vulnerabilities zero days possessed by the CIA, but if the CIA can hack these phones, then so can everyone else who has obtained or discovered the vulnerability. As long as the CIA keeps these vulnerabilities concealed from Apple and Google to make the phones, they will not be fixed and the phones will remain hackable. The same vulnerabilities exist for the population at large, including the U.S. cabinet, Congress, top CEOs, system administrators, security officials, and engineers. By hiding the security flaws from manufacturers like Apple and Google, the CIA ensures that it can hack everyone at the expense of leaving everyone hackable. Cyber warfare programs are a serious proliferation risk. Cyber weapons are not possible to keep under enormous cost and visible infrastructure. This old material to produce a critical nuclear mass, cyber weapons, once developed, are very hard to retain. Cyber weapons are, in fact, just computer programs which can be pirated like any other. Since they are entirely comprised of information, they can be copied. Carrying such weapons is particularly difficult since the same people who developed and used them have the skills to exfiltrate copies without leaving traces, sometimes by using the very same weapons against organizations that contain them. There are substantial price incentives for government hackers and consultants. A piece says there is a global vulnerability market that will pay hundreds of thousands to millions of dollars for copies of such weapons. Similarly, contractors and companies who obtain such weapons sometimes use them for their own purposes, obtaining advantage over their competitors in selling hacking services. Over the last three years, the United States intelligence sector, which consists of government agencies such as the CIA and NSA, and their contractors such as Booth Allen Hamilton, has been subject to unprecedented series of data exfiltrations by its own workers in the number of intelligence communities arrested or subject to federal crime investigations in separate incidents. Most visibly, on February 8, 2017, a U.S. federal grand jury in the counts of mishandling classified information, the Department of Justice alleged that it sees some 50,000 gigabytes of information from classified programs at NSA and CIA, including the search code for numerous hacking tools. Once a single cyber weapon is loose, it can spread around the world in seconds to be used by peer states, cyber mafia, and teenage hackers alike. U.S. consulate in Frankfurt is a covert CIA hacker base. In addition to its operations in Langley, Virginia, the CIA also uses the U.S. consulate in Frankfurt as a covert base for its hackers covering CIA hackers operating out of Frankfurt consulate sent for Cyber Intelligence Europe or CCIE are given diplomatic blank passports and state the instructions for incoming CIA hackers make Germany's counterintelligence efforts appear inconsequential. Breeze through German customs because you have your cover for action story down pat and all they did was stamp your passport your cover story for this trip answer support technical consultation at the consulate your WikiLeaks publications give further details on CIA approaches to customs and secondary screening procedures. Once in Frankfurt, CIA hackers can travel without further border checks to the 25 European countries that are part of the France, Italy, and Switzerland. A number of the CIA's electronic attack methods are designed for physical proximity.  In these cases, a CIA officer, agent, or allied intelligence officer acting under instruction physically infiltrates the targeted workplace. The attacker is provided with a USB containing malware developed for the CIA for this purpose, which is inserted into the targeted computer. The attacker then invests in exfiltrates. For example, the CIA attack center system fine dining provides 24 decoy applications for CIA spies to use. To witnesses, the spy appears to be running a program showing videos, example, VLC, presenting slides, peresy, playing a computer green, breakout two, 2048, or even running a fake virus scanner, Casper ski, McAfee, Sophos. But while the decoy application is on the screen, the underlying system is automatically infected and ransacked. How the CIA dramatically increases proliferation risks. In what is surely one of the most astounding intelligence on goals in living memory, the CIA structured its classification regimes such that the most market value part of Vault 7, the CIA's weaponized malware in Blast Blunt Zero Days, listening post LP drill C2 systems. The agency has little, the CIA made the systems unclassified. Why the CIA chose to make a cyber arsenal unclassified reveals how concepts developed for military use do not easily cross over to the battlefield of cyber war. To attack its target, the CIA usually requires that its implants communicate. If CIA implants command and control and listening post software were classified, then CIA officers could be prosecuted or dismissed for violating rules that prevent placing classified information onto the internet. Consequently, the CIA has security made, secretly made most of its cyber spying war code unclassified. The US government is not able to assert copyright either due to restrictions in the US Constitution. This means that cyber arms manufacturers and computer hackers can freely pirate these weapons if they are obtained. The CIA has primary had to rely on obfuscation to protect its malware secrets. Conventional weapons such as missiles may be fired at the enemy into an unsecured area. Proximity 2 ordinates the ordinance including its classified parts, hence military personnel do not violate classification ordinance within classified parts. Ordinance, if it does not, that is not operators intent. The last decade US hacking operation, NSA jargon, are being called fires as if a weapon was being allegedly questionable. Most CIA malware is designed to live for days or even years after it has reached its target. CIA malware does not explode an impact. But rat targets device. Copies of the malware must be placed on the target's devices, giving physical possession of the malware to the target. To the CIA actions, the malware must communicate with CIA command and control C2 systems placed on internet connected servers. But such servers are typically not approved whole classified information. So CIA command control systems are also made unclassified. A successful attack is more like a series of com perhaps akin to the whole series of military maneuver against the dark forensic and antivirus. A series of standards lay out CIA malware investigation patterns which are likely to assist forensic crime scene investigations as well as Apple, Microsoft, Google, Samsung, Nokia, BlackBerry, Siemens, and antivirus companies attribute an offense against attacks. CIA rules on the CIA US government forensic review. Similar secret standards cover the use of encryption to hide CIA hackers describing target next-filtrated data as well as executing payload and persisting in the target's machines over time. CIA hackers develop successful attacks against most well-known antivirus programs. These are documented in AV defeats, personal security products, the PSPs, and PSP debugger RE avoidance. For example, Commando was defeated by CIA malware placing itself in the windows recycle bin while Commando 6.x has a capping of CIA hackers discuss what the NSA's equation group hackers did wrong and how the CIA's malware makers could avoid similar exposure. Examples, the CIA's entering development group EDG management system contains around 500 different projects, only some of which are documented by year zero, each with their own sub-project, malware and hacker tools, projects related to tools that are used. Another branch of development focuses on the development and operation of listening post-LP and command control C2 systems used to communicate with and control CIA implants. Special projects are used to target specific hardware from router to smart TVs. Some example projects are described below but see the table of contents where the full list of projects described by WikiLeaks year zero. Umbridge, the CIA's handcrafted hacking techniques pose a problem for the agency. Each technique it has created forms of fingerprint that can be used by forensic investigators to tribute multiple different attacks to the same entity. This is analogous to finding the same distinctive phone on multiple separate murder victims. The unique wounding style creates suspicions that a single murderer is responsible. As soon as one murder is set in the set is solved then the other murders also find likely attribution. The CIA's remote devices branch collects and maintains a substantial library of attack techniques stolen from malware produced in other states including the Russian Federation. With Umbridge and related projects the CIA cannot only increase its total number of attack types but also misdirect attribution groups that the attack techniques were stolen from. Umbridge, keylogger, password persistence, privilege escalation, stealth, antivirus, PSP, avoidance and survey techniques comes with a standardized questionnaire that CIA case officers fill out. The questionnaire is used by the agency's OSP operational support branch to transform the requests of case officers into technique requirements for hacking attacks. Typically exfiltristums for specific operator allows the OSP to hit this to CIA malware of configuration staff. The OSP functions as the interface between CIA operational technical support staff. Among the less than possible targets of the collection are asset liaison asset system administrator, foreign information operations, foreign intelligence agencies and foreign government entities. Notably absent is any reference to extremist or transnational criminals. The case officer is also asked to specify the environment of the target like type of computer operating system used internet connectivity and installed antivirus utilities, PSPs, as well as a list of file types to be exfiltrated like office documents, audio, video, images or custom file types. Ask for information if recurring access to the target is possible unobserved access to the computer. This information is used by the CIA's JQ Jim provides software C below to configure a set of CIA malware suited to the specific needs of an operation improvise. GQ Jim provides is a tool set set up an execution vector selection for surveyed supporting all major operating system like customized tools before CIA malware suit its associated control software. The project provides custom customizable implants for Windows Solaris, micro tick used in internet routers and listening post LP command and control C2. Inflats are configured to communicate with this web server of utilizing these implants as a separate cover domain and the infrastructure resolves to an IP address that is located at a commercial VPS virtual private server provider. The server forwards a VPN to a pot server that handles actual requests from clients. It is set up for optional SSL client authentication. If a client sends a valid client certificate only implants can do this. The connection is forwarded to the honeycomb tool server that communicates with the implant. If a valid certificate is missing which is the case of someone if someone tries to open the cover domain website by accident the traffic is forwarded to cover server that delivers an unsuspicious looking website. The honeycomb tool server receives exfiltrated information from the implant. An operator can also task the implant to execute jobs on the target computer so the tool server acts as a C2 command and control server for the implant. Similar functionality though limited to Windows is provided by the Rick Bobby project. See the classified user developer guide for Hive. Only ask questions. Why now? WikiLeaks published as soon as its verification analysis were ready. In February the Trump administration has issued an executive order calling for cyber war reviewed. While the review increases the timelines and relevant setting the publication, the addresses and external IP addresses have been redacted in the releases. 70,875 redactions in total until further analysis is complete. Some items may have been redacted by these contractors, targets or otherwise related to the agency but are for example authors of documents for otherwise public projects that are used by the agency. Identity versus person. The redacted names are replaced by user IDs, numbers to allow readers to assign multiple pages to a single author. Given the redaction process used a single person may be represented by more than one assign and identify refers to more than one real person. Three archive attachments zip tar dot gz etc all the file names in the archive. As the archive content has assessed thousands of routable IP addresses references exclusive investigation of CIA malware affected binaries. Organizational chart. The organizational chart corresponds to the material published by WikiLeaks so far. Since the organizational structure of the CIA below the level of directors is non-public the placement of the agency is reached 2013 to 2006 and was each part of Vault 7 of 2016. He leaks obtained each part of Vault 7 sources trust WikiLeaks to not reveal positions to write


If you've driven through Sublet County recently, you likely drove through one of the two wildlife overpasses that frame the area landscape. The overpasses are the first of their type in Wyoming and are believed to be the first designed specifically for pronghorn. The overpasses are strategically located near Trappers Point, west of Pine Dell. The location of the overpasses provides connectivity for pronghorn as they make their historic and epic migration to and from their seasonal ranges. The ranges and distance from 72 to 160 miles each way, which is the longest pronghorn migration recorded. It is estimated that 1,500 to 2,000 pronghorn make the journey each spring and fall. The overpasses were designed to replicate the surroundings. On top of the structures is soil that has been planted with native grasses and shrubs. Additionally, earthen berms line both sides to block the view of the highway. An 8-foot tall fence was constructed on top of the berms to prevent animals from accidentally jumping off. The Wyoming Game and Fish and the Wyoming Department of Transportation asked the public to respect the space of the wildlife and to not recreate on or near the overpasses. From the University of Wyoming Extension, I'm Wendy Kelly, Exploring the Nature of Wyoming.


Definitely the Pokemon manga, or anime, or Monster Musume. Monster Musume, you say, huh? Or I gotta do this, I gotta do my ghost impersonation. Oh, yeah! Oh! Nah, I don't kink same. I got my own kinks, but yeah, I just had to just beg a little nod at that.  I could be like, yeah man, I gotta put the giant girl manga in here now, I gotta be the sidelines on this one, or other kind of stuff like that. Oh yeah, gotta put that naga girl in there, how about that cow girl, slime girls, all that.


If you can see my screen today. Yep. Yep. So today I'll be talking about Merlin framework. Like as I mentioned in my introduction, I'm working with salsa since six. I got introduced to this framework for migration, which is called Merlin. So today I'm going to explain, I'm going to give you pointers about what exactly this tool does and how you can utilize this during your migrations. So what exactly is Merlin? So Merlin is a tool that helps us migrate content from a website into a structured data. So basically any source of any source website you can migrate and what you get as an output is structured data, data. This data then can be migrated into like Drupal or any other system. So overall it simplifies the process of migration and yeah. Hello. Can you hear me? Yes, yes. So irrespective of the source of the migration, we can extract data from the source and we get structured data as an output that is the JSON files. We extract structured data into JSON files, which can then be used by other systems during migration. Let me talk about the details, what exactly it does and how it how it works. So Merlin does three things. First thing is it helps us identify all the public URLs that need to be migrated. So for example, if you have a website that needs to be migrated, first step would be identifying all the public URLs and which are the URLs that you want to migrate to the target system. Second thing it does is it loops through each of the URLs to extract or scrape content from the source. And then finally, it gives you the extracted data into JSON format. This JSONs can then be used during your import, like you can import those JSON into Drupal or any other system. So first step, let's see how what what is the first step. So you are Merlin comes with a URL crawler that helps us like generate an extensive list of all the public URLs. Once you have the public URLs, you also have an option to group those URLs based on multiple types that is either you can group those URLs based on type. You can group based on existing DOM elements, or you can based on certain value of a group of an element and I'm going to talk about this in detail. So, just to put it in context, imagine you have a website that needs to be extracted, you will be able to extract all the list of public URLs and segregate them based on the target system like target content types maybe. So, yeah, starting point you have all the URL list segregated based on the content types. Let's see how exactly we do that. The Merlin framework uses config.yml files. Here you can define which is your source domain. You can also define there are plenty of options to provide delay to provide what is the concurrency maximum number of requests that you can you want to crawl. You can also provide and start URL and you can also provide a delay so that you won't have like you won't burden your source system. We have group by which which can be used to segregate the URLs. So this is our typical config.yml file. Let us see group by condition like how we can use grouping. So, while extracting the URLs, you can, as I mentioned, we can group those URLs together based on the target system. So, for example, what we can do is we can have a group by condition based on URL patterns. So that say some pattern needs to be grouped together so you can do that. Secondly, you can use, you can check whether and if an element is present group those URLs together. Third one is if a particular element, if you want to match a property that also you can do and based on this URLs will be grouped together. So these are the three types of groupings that are currently present. So, this is how we run a basic crawl command. We provide config.yml to the crawl script and you define where exactly the output is going to be. Once you run this command, you will get, you will get list of all the URLs and you'll also get the files will be generated based on those. So this is about crawl. This is the first step of migration that we typically follow. Once we have, once we have a list of extensive list of all the URLs in the system that we need to migrate. And once we have finalized what all URLs we need to migrate, we start with generate. Generate command basically what it does is it scrapes the pages that the URLs that we have provided it scrapes all the content based on the mappings that we have provided in the config file. So, so if you have a page if you want to extract some data out of it, we can do that using the config and defining that in the config and and and using the generate command I'm going to explain how this is done. Well, it provides a lot of inbuilt types, which we can use. There's a dog over here. Merlin provides Merlin has a documentation which provides all the predefined predefined types. For example, if we have an accordion on the source page, which we need to extract this is how you do it. If we have some media. And if you want to extract file name, alt, etc. So we can do that. There are all the predefined types which help us during the migration, and there are also pre processors. For example, white space removal. If this is our basically to do some operations on the extracted content. For example, string replace, while extracting only you can do replacement and few other things. So, this is about generate I'm going to. Yeah. So, so for example, let's see a mapping. If we have a, if we have a H1 tag, and we want to extract this H1 tag, basically, from all the pages. How do we do that we add a mapping field title this type is defined here type is text. Define the field name and selector based on the H1 tag. And this will extract the data. Let me show you an example of how, how this can be done. Let me give you a demo. So, I have, I have a generator comma script. I have, I'll extract some data from salsa digital website. Let me show you this website first and I'm going to target only to do elements at this moment. First is the title tag, and let me extract this description in order to do that. So, I have created a Yaman file, defined a domain as salsa digital. I have provided list of URLs in this at this part here. That is, I'm going to extract only five content at the moment. Few other configurations, which include what is going to be conquer concurrency delay. I've kept it like as standard as of now. The two main things that we need to look at is mapping. So, the mapping contains two fields. First one is title and second one is summary. So, first one, I have used a selector here that way that is going to extract H1 text H1 tag and secondly, I've used a long text which will extract the hero banner description. So, let me run this and show you I've provided the config file as an input and I've provided reports folder. So this is how I run it. It would let me it would tell me what are the, what are the URLs it has passed and it will tell me where the file has been generated. So it has generated a JSON file for me. If I go back to the reports folder, I have the article or JSON file. It has extracted title and also the summary for me. Similarly, you can do it for a lot of content types which are defined, including complex content types like, say, a media or an accordion or a paragraph, ordered list and menu as well. You can also extract menu here. So, yeah, so once, once we have this extracted JSON, so this is the JSON that I have, I can then proceed with the migration, I created a migration already, a Drupal migration for a Drupal JSON migration, which has article.json as my input. I have title and the summary as the input and I have processed, I have process plugin for to import it into title and body. So, yes. So let's migrate. So I have created this importer already. Let's import the content and yes, we have this content imported. So, mostly about Merlin framework. Benefits of using this framework is you don't have to get access to the source system irrespective of the source system. It might be any source you can you can migrate and yeah, it simplifies the migration overall. So this is this was my presentation for Merlin framework. Any questions? I don't see any questions. One question from my side about the like, this is the one use case that I can see it is applicable on a public website only. Is there any way that we can use for internal sites or internal application? So when you say internal, do you mean, which are hidden by some login, etc. Yeah, something like that. If it is hidden by just basic shield, then yes, of course we can provide the URL within that. But yeah, that is that that I can understand that the basic authentication is a different thing but in terms of like internet side that is only accessible internally or maybe it's like how exactly that might be the other use case that we can do with the Merlin. Not yet, but it is an excellent feature that can be added. Merlin is open source. I will put in the links to Merlin documentation as well as the Merlin GitHub repository for anybody to have a look at it and have a play with it if they want to. Hi, I see that you have your hand raised. Yeah, thank you. I do. I have a very selfish question. And I'm sorry, I came in late to the session. So I may have missed the answer. Sorry, hold on. Sorry, Max, can Max go first. I can. I was wondering if it's also able to bring across assets that are attached to the pages being scraped as in like PDF images that type of stuff. It can bring in the URLs of the assets, not really. So the aim of Merlin is to get the content in a JSON, you know, structured format. So when we are doing media migration, for example, it can bring in the URLs and then Drupal Migrate API can be used to actually bring on, bring those URLs into Drupal. Cool. Are you there now? Can you talk now? It's okay. It's answered my question. Thank you. All right, cool. I put in the links for the Merlin documentation as well as GitHub for Merlin. Merlin is an open source tool. Please feel free to create issues, etc. If you want to, if you have, if you see something that you want to get at it, please go ahead and create a full request. We can go ahead and look at that. Thanks a lot for your presentation of it. I'll just stop recording now.


Okay guys, so I am back with another Destiny 2 video now, all over social media, all over Twitter, Facebook, YouTube, Instagram. I'm seeing posts from people talking about Destiny 2's frames per second on console. Now me being at the event, me seeing the gameplay right in front of my eyes, me playing it, I'm going to give you guys my honest opinion on what I think is happening, speaking to devs, getting information from other people, seeing what people are tweeting from the events itself and so forth. But before we get into the video, if you guys do enjoy it, leaving a like is much appreciated on my behalf, it really is. And also guys, remember drop a comment down below in that comments section to beam with a chance of winning my weekly giveaway, either a Destiny 2 beta code or a resurrectionist emblem winner will be picked and the choice will be yours and what you want. Okay so getting into it, 30 frames per second or 60 frames per second. Now we would all expect a new game 2017, late into 2017, running on next gen only being Xbox 1 or PS4, Pro PS4 and PC. We would expect this game to be running on at least 60 frames per second. Now I can tell you guys that me playing the game on PS4 Pro on 4K monitors, in my honest opinion what I saw there, it looked to be 30 frames per second, unconsol anyway. Now I saw people tweeting like SniperWolf saying how she spoke to three devs who confirmed it was running at 60 frames per second. I saw one of the Planet Destiny dudes tweet out saying it was confirmed it was locked at 30 frames per second. Me personally looking at it, I would say to you guys that it was definitely running at 30 frames per second and everybody in the group we were in playing the game all said the same thing. There's no way in my opinion it looked like it was running at 60 frames per second, it just didn't look that. But still the game did look absolutely incredible. I mean it looked incredible and this is on the PS4 Pro. Now to talk about the PC version of the game that was at the event at the capture event, I mean it was outstanding. What I saw was running 4K 60 frames per second. Now I know in the capture event for PC anyway, it was locked at 60 frames per second 4K 4 PC. A few people were told when the game is released on PC which is actually at a later date and when it comes out on console is that there will be no FPS cap. So the game will be capable of running 200 plus frames per second.  I ain't no PC buff but all I can tell you guys is what I saw at the event and what I saw was it looked absolutely incredible on PC and on console it looked great don't get me wrong but it did look like it was running at 30 frames per second. Now the devs I spoke to and I asked about this they said that this was just a build of the game and improvements may be made but they can't comment on what frame rate the game is running at. I asked blatantly is this game running at 30 frames per second and a couple of them said we can't answer that right now. I asked a couple is it running at 60 which you think they would be quite proud of and they said they can't answer that right now so they were literally holding back they knew they blatantly knew what was going down but they was holding back but yes as you guys probably would have seen on Twitter people are tweeting that it's confirmed to be running at 60 frames per second on console people are then tweeting that it's locked at 30 frames per second on console in my opinion I will go with the second one right there. This definitely looked to me to be running at 30 frames per second which is kind of disappointing I mean it's 2017 now you would at least expect 60 frames I mean you just really would. If not on Xbox One definitely on the PS4 Pro but taking nothing away from the game it looked great on PS4 it really did and it looked absolutely outstanding on PC. Now I actually sat there and watched someone play on PC for about 15 minutes and I was gobsmacked at how great it looked going back from a PC screen to look at a PS4 screen and I'm like my life differences is unbelievable well it is what it is us console gamers know what PC is capable of it's as simple as that and we expect it to perform better on PCs that's the bottom line but yes if I had to give you guys my honest opinion especially after what I've learned and saw from being at the capture event I would say that the game is running and probably will run when it comes out at 30 frames per second on console which is like I said disappointing in a way but it still looks incredible the graphics have definitely improved over Destiny 1 and the game is definitely more fluid it really is but yes guys that is my opinion on Destiny 2 and how it runs I mean we won't know for sure until Bungie actually leave a statement on the matter I mean content creators can have their own ideas they can say what they think they've heard or say what they've actually heard but until Bungie actually comment on this we won't know for sure but that was my opinion guys hope you enjoyed it and that's going to be it for another Destiny 2 video if you enjoyed daily Destiny content make sure you subscribe if you're new around here press that bell button next to that subscribe button if you want to receive email notifications of when I upload and I will as always guys see you on that next one


Hello, and as a native French speaker, I don't practice English very often so I hope you will be able to understand what I say. If you don't, well you can have an app or you can try to use the slide to follow me. So the idea is that today we can use NFS, which is old or we can look for something better. And what something better could do that NFS doesn't do today. For example, it could have a high availability, which means that if you've got a server that goes down, the clients can still access the data. And you want this without a single point of failure. You want storage clustering, where you can add storage servers to increase your volume storage. You want elasticity, which means doing this without any downtime. You want, of course, position semantics. You want file locking, you want performance, and you want secure communications, which means cryptography. And what do we have to do this? Well, for a long time, your file system used to be implemented in Kernel, which means that we are not portable at all. And development cycles were quite long because developing in Kernel is not easy. And recently we have seen a file system in user space appearing, where the file system is a unique process. And the Kernel is a client for this unique process. We've got message exchanges, sorry, between the Kernel and the file system, which is a bit like a micro Kernel architecture. So what file system do we actually have? Well, we've got Luster, we've got Hadoop, we've got GlosterFS, we've got ExtremeFS, we've got lots of file systems. And if we want to move to one of these file systems, we need either a Kernel module for it or a few file systems. Some file systems have both the Kernel module and the Fuse interface, but more and more we see a few file systems. And after looking at a few file systems, it seems that GlosterFS was an interesting target. And it requires a Fuse interface. So the goal of my project was to run GlosterFS and obviously it was to be through the Fuse interface. So what is Fuse? Initially it was Linux only thing and it became a de facto standard. Now you have Fuse on 3BSD, you've got Fuse on macOS 10. I heard that you even have Fuse on Windows, which is a bit strange. So in a Fuse file system, you've got the Kernel, which is the client of the Fuse process. It sends requests to the Fuse file system and the Fuse file system reply. You've got a message passage interface for a character device called DevFuse. You've got a user library, which is there to upload the basic course so that the file system is not too complicated to write. And you've got free API for the Fuse file system, which seems a bit insane, but it's the way it is. In this diagram, you see how the thing works. So everything starts by a process, for example, here at LS, which issues a system call. For example, here, read, it does this through the libc. And once the system call is executed inside the kernel, it reads a layer which is called the VFS. I'll talk about it a bit more in the next slide. Basically, it's a switch that let the kernel use a different file system. It can be the X3FS file system or it can be a Fuse file system. If it's a Fuse file system, we've got a Fuse tab inside the kernel, which will immediately send a request through DevFuse to use around here. You've got three options. You can have a file system that is directly connected to DevFuse. It uses the kernel API. You can have a file system connected through libfuse using either the low-level API or the high-level API. So you've got three APIs. The VFS, so it was introduced by Sun a long time ago. At that time, the idea was to have the ability to use both UFS and NFS. So you can think about it like a standard interface inside the kernel to reach any file system. And below that interface, each file system implements some methods. So for the VFS, you've got some object like remounts and vnodes, vnodes represent files or directories. And you've got methods. Lookup to find a vnode, open to open it, read, write, which are self-explaining. What about NetBSD? NetBSD didn't have a Fuse when I started to look at GloucesterFS. It had Poof. Poof, which is very hard to pronounce, is similar to Fuse, but not compatible. It started when Fuse was not objective standard. It was not objective that everyone would be wanted to use Fuse. And it seems that Poof is there to stay today because it still has merit as a native interface because the native interface will always fit better your own VFS than a foreign one. And if you want to add a new feature, you don't have to talk to many people. You can do it within the project. How does it work? Exactly like Fuse. So you've got your process that does a system call. You go to the VFS, which is that we are talking to Poof's file system and the request with which you use our land for dev Poof's. And here you've got only one API, which seems a bit more reasonable. So it looks exactly like Fuse. The name of the device is not the same. It's dev Poof instead of dev Fuse. The name of the library is Poof instead of dev Fuse. But despite the similarity, it is still very different. The message spacing in protocol is different and the API is different. At least for Poof, we've got one API. Only one. So Poof is there to stay because it has merits. Fuse is highly desirable because today people write file systems using Fuse. So we need to use Fuse on NetBSD. And a first attempt was done, which was called with Fuse, which is a nice plan, which is an implementation of Fuse over Poof's. How does it work? Here, on the top of LibPoof, you can have LibFuse and LibRefuse, sorry. And on the top of LibRefuse, you have a Fuse file system using the high-level Fuse API. But it's only the high-level Fuse API, which means that we like two APIs. So we don't support file systems that use a low-level API. We don't support file systems that use a kernel API and GloucesterFS, which was my own target. It does exactly that. It opens the Fuse and talk to the kernel directly without using LibFuse. So Refuse is not enough and we need something better. So a new project is called Perfuse. The idea is to implement the Fuse kernel API. And once we have it, we can just use the stock LibFuse on top of it and we support everything. How does it work? Here, I omitted the first part where you've got LS doing a read system call. I think you understand the concept. And now, you see that your request that goes from the kernel to userline through DevPuse and through LibPoof. Now goes through LibPerfuse. LibPerfuse is used by a demand, which goes PerfuseD. Perfuse is to translate the Pufs request into a Fuse request, sorry, and to send it to a DevFuse socket so that the Fuse file system can use it directly through the kernel API or through LibFuse using the high-level or low-level API. But there is one small problem because I said it was a socket and a socket doesn't have exactly the same semantics as a character device. For example, you can open and mount a character device where it's impossible for a socket. You have to use the system call socket bind and connect to use a socket. So we need to cheat a bit and use some defines to wrap Perfuse open which will do the socket bind and connect stuff. And we define it as open in the Perfuse header if you include the Perfuse header the things just work. LibFuse was modified so that the change is hard to support NetBSD so today we can just use the standard LibFuse and it works. And later I realized that after all I didn't really need the DevFuse socket and it was replaced by Anonymous socket using socket pair. Here I talk a bit more about VFS because I need it in the next slides for anyone that didn't heard about it in depth. So the idea is that when we do file system and we translate the VFS operation into request for Perfuse. So the VFS operation start by obtaining the root venode and it's obtained at the same time. And then we use the lookup method to find another venode. You give lookup the name of a node and you get a reference on a new venode. You've got a method that get attribute to set attribute to get the metadata for a file, for example the mod, the owner that will open with or write on the file directory. Here is some pseudo codes that show how it works. So your first month of file system you get a node. Then you use the lookup method on the root node to lookup for something that is called foo and you get inoen because it doesn't exist. Then you try to lookup on the new node to get the inoen number you can open it and you can read from it. Sometimes VFS operation are not really obvious, for example in 8BSD we've got the release operation which is called at close time we've got the inactive operation which is called when the last reference on a file is dropped and we've got the reclaim operation that the kernel will send when he will not use the node anymore and that memory should be freed. Linux has only release and forget. It's a bit tricky because as you can see in this example after you get release method invoked you can still have some read codes that happen. In fact you close your file and it still has something to read so the release method which is called for close doesn't mean that you will not have read afterwards. So that just means that we've got to wait for inactive to close files. Now I will make a nice list about funny things that I encountered during this project, bugs and traps. So here is the list which is them one by one. So first another problem about socket semantics when I use a socket it can be of type stream where it's non atomic and reliable or of type datagram where it's atomic but unreliable and my problem is that a character device is able to provide a semantics message or send atomically and it's reliable. So I couldn't use a stream socket I tried because the kernel can split the request into several packets. I couldn't use the datagram socket type because it's not reliable I'm not sure it has to be unreliable for local communication because the sender could just sleep waiting for the receiver to get the data but changing that would have been quite intrusive and with the risk of breaking things. So we prefer to implement the second packet socket type which is an atomic and reliable socket type which is exactly what we want and fortunately someone did it and it was sitting in a bug report for I think 5 years and it wasn't being committed so I just had to make the code build again and we had some things that worked. Another problem which is specific to GlusterFS is that it makes heavy use of extended attributes on the server side and NITBSD has no support for extended attribute well it had some support which came from UFS1 code which was very broken and didn't build so I had to make it work again I choose the sparse file to store attribute and we have one file for each attribute so if you want to use another attribute you have to create a back end file so I had to add code so that the back end file can be auto created every minute I added support for copying extended attribute and preserving them in the CP and MV commands and then I had the question because the code coming from 3BSD used the 3BSD API and there is also a Linux API for extended attributes and we had to choose between the two and finally I didn't choose the limited buff API which turned to be a good idea because no porting software is much more easier because we support both there are still work to be done for extended attributes we need it in a few of our utilities for example backup utilities because for non-NITBSD it's not able to back up the extended attribute which is a big problem extended attribute support impact, dump and restore it would be nice to be able to copy extended attribute over the network using ACP and AirSync AirSync has support for it using the Linux API so I can do it easily it's just a build option to set up ACP is a bit more problem I know that macOS 10 implemented it upstream someone will have to work on it and there are some commands that could benefit of extended attribute but I'm not sure that it's ways to do so it's TAR and CPIO because if you add extended attributes to the format you somewhat break the standard that doesn't expect them I'm not sure what will happen with the binaries that will extract the archive but I don't know about standard attributes some other improvements that have to be done adding extended attribute in UFS2 someone has to steal from them better storage for extended attribute instead of sparse files we can store them directly into the file system like we did for Quota here again someone has to do the work another funny bug it's an un autoradlist this time it's about raising a get attribute a method as you can see in this diagram when you send a get attribute then you write some data at the end of the file you end up in the situation where the kernel has an idea of the file length because the length is free and that get attribute replies come with another value which dates from before the write and the kernel will tronquette the file because it discovers that the file is lower than what it expected and when the write request replies comes the file size is extended to 3 but we've just zeroed between 0 and 3 we've lost some data so the solution of this race is simple, we need a mutex on the side another problem which is probably well known is dianame thread safety or I should say thread unsafety the problem is that the standard doesn't tell us if dianame should alter the buffer it is given or if it should return static memory both behavior are standard compliant so some software will assume some behavior and Gloucester FS assumes the linux behavior which is to modify the buffer while NetBSD use a static buffer unfortunately there is no consensus on thread safe dianame underscore error so the solution was to just add the dianame code from GNU inside the country directory of Gloucester FS they want the linux version just give them the linux version don't try to be compatible in NetBSD another funny problem what happen if you call the link system call which is used to do a hard link on a sim link will you link to the sim link or to the sim link target here again it's not specified the standard will not tell you what is the good behavior unfortunately linux will link to the target while all the BSD will link to the sim link itself Gloucester FS of course relies on the linux behavior and it's a very important feature for migrating data from when storage break to another when you want to replace a server so we really needed it so the solution is to use linkat which is part of POSIX Extended IP set 2 which is a big family of new system calls which improve a lot of things and this new linkat is just like link but it's with a flag and the flag can tell if we've got to follow the sim link or not in the NetBSD 6 branch we only implemented a feature linkat just to support Gloucester FS and finally in NetBSD 7 we've got the whole extended IP set 2 except FXXVA that was not implemented because some people always cared about it seems it's unreasonable and security wise another problem and it's not the last but the previous with the Pajaman action app so Pajaman is a kernel thread that is responsible for freeing memory when memory is scarce and one of the job is to take vnodes that have memory that are using memory and put them to disk freeing memory the problem is that when you want to do this with a pure file system you will need to allocate the message so you will need to allocate memory when memory is scarce and the memory allocator can sleep waiting for memory to become available or not sleep which means it will fail if there is no memory and we had some situation where the Pajaman was able to allocate a proof message and sleep for memory we end up in the situation where the kernel thread that is responsible for freeing memory is sleeping waiting for memory and the problem will look obviously Pajaman must never sleep so the solution is inside the proofs kernel code to check what is the calling thread and if it's Pajaman then we prefer to fail of memory allocation and return to the calling code that it doesn't work instead of sleeping so what happens Pajaman will try to put a vnode to disk which fully will not be a proof vnode and sometime it will have enough memory to start freeing a proof vnode last funny problem it was not a complete list I've got a lot of funny bugs but the idea was to give you an insight of the various direction where we went GlosterFS use swap context which is a system code which let you change the CPU register and posix thread it's a bit odd because usually you use swap context to implement threads but here they needed a stranger concept of task, the idea is that when GlosterFS will send a request to the network we've got a context which has to wait for the network reply and it's a bit a bad idea to send the thread to sleep and to wait for the reply because you need to have as many threads as you have a request and you've got a lot of sleeping threads so what they do is using swap context they are able to load the context to restore the situation where they were when the request was sent and the big question is that threads use a CPU register called TLS to store the address of the private data for the thread and should swap context touch this register or not on Linux the TLS is preserved TLS is thread safe on NetBSD it was not very clear in fact we had a machine dependent mess where each port was doing a different thing so what happened you've got two threads one thread that prepare context the second thread that swap context and then you discover that both threads suddenly return the same value which means that you are going to crash because mutex logs will not work anymore and we've got to fix that so what should we do with the TLS register obviously the behavior should be different depending if the problem is linked with libpeshred or not if it's not linked with libpeshred then we do not preserve it we change it using swap context and if we link with libpeshred then we preserve it we had for some ports an option to make context to decide if the TLS register was to be restored or not so what I did was adding this option to all ports so that the various NetBSD ports no share the same behavior that's all for my list I could present bugs for a long time but I got to finish my presentation so to do list so as I told you we need to add support for extended attributes in dump and restore we need a better storage for extended attribute because the current situation it's just scary because we don't have any tool to check the integrity of the extended attribute storage if the file is corrupted and we cannot save them in backup so it's a real problem there are some few features that should be implemented few negative catching which let the kernel know that a file doesn't exist and don't ask every second there is a mechanism for the kernel to send notifications to the file system the idea is that kernel will be able to tell the system that a given file has been removed for example and perhaps there is queues which is a character device in userland I'm not sure about this one because it could be interesting if there were some queues driver a lot of queues driver to use but I've not seen many so I'm not sure that it's worth working on it yet at least here I am it might be a naive question but I don't quite understand why you had to go around for userland again to me it would seem simpler to say ok I've got the PuffFS implementation I just clone it and then I turn it into a fuse implementation considering I have some code in FreeBSD or something like that what could be simpler about going again for userland implementing a new socket making sure it works etc etc well the answer is that it's already faster to develop in userland than in kernel in kernel you crash or reboot you crash or reboot it's very long and here in fact there are not many drawbacks to do it in userland because we are working with a network file system so performances are most about the network and not about going from userland to kernel to userland to kernel I have not measured it but when you have to wait for a network when more system code is not really a problem more question roughly how much code did you have to write for the perfuse implementation so how much code did you have to write how big is the perfuse library how much code I have to run WUC on it I don't know exactly I'd say that it's a a few thousand line of C but it's a very simple code in fact you get a few request you create sorry you get a few request you create a few request you translate arguments so it's not complicated well it's time consuming when you have to spot a bug in fact because the bug list is not the most trickiest but it's the one that took me the most time to understand the thing with the pagiment that is sleeping waiting for memory has been very difficult to find from the outside you just see that the system looks I have a question myself have you measured the overhead of going back and forth in userland not yet but I know that it's nothing compared to going back and forth any more question but by the way there have been a heated debate between the people that did the cluster FS and Lennox Torval Lennox Torval told them that the thing was a toy because file system should be in kernel and the people for cluster FS had just this kind of argument that if you have to wait for the network it's not a problem to do more system codes there are a lot of questions about the differences you find with GNU, BSD, other Lennox specific stuff have you talked to upstream about it because on free BSD upstream cluster FS is themself working on porting it through free BSD so they might be interested in those feedbacks to be able to fix and have pretty more portable I'm not sure I understood the question you asked me if I send my change to cluster FS or just talk with them about those differences I do talk with them and I've been contributed a lot of portability fixes to cluster FS and today, NetBSD just works out of the box for cluster FS you just have to configure it just works and now I'm working to pass all the regression tests for cluster FS another example of communication with the cluster FS team is that today there is a NetBSD smoke test when you do any commit in cluster FS so we know that it will not break the NetBSD port and soon we know that regression test will pass so yes I'm talking with them a lot in fact I'm a bit cluster FS contributor thank you for your attention no more question ok thank you all and next talk is in 25 minutes