Namaste to all of you. This is our second session on financial accounting. I am sure you would have enjoyed the first session. In the first session, it was just an introductory session. So, we had just started with understanding what is accounting, then what is the distinction between various streams of accounting like financial cost, management accounting. After that, we have moved to financial statements. So, we have seen how are financial statements emerging. So, basically, they emerge from money cycle. So, you can have a view at the slide where we have just seen what is a money cycle. I am trying to go from very much of a layman who does not know accounting. Some of you might be well versed with accounting terms. So, please bear with me, we will go in details in the coming sessions. But to begin with, this is what is a money cycle. It starts with money, procurement and so on goes back to money. So, at the stage of procurement and consumption, the costs are incurred, then delivery collection is a time you get the revenue. Now, from this money cycle, the basic financial statements emerge. So, on one hand, we have got a PNL account. It records your revenues, it records your costs and it compares the revenues and the costs, the net result is given to you in the form of profit or loss. On the other side, we have got balance sheet. Now, what does the balance sheet do? It lists out the resources for the business. Those resources for the business are known as the assets. It also lists out the providers for resources which are known as the liabilities. In the last session, we had also seen what are the important assets. So, you can again look at those slides. I think you know it all now, the first one is fixed assets, which is nothing but an infrastructure needed for the business. Can you think of any example? So, it can be land, building, it can be vehicle, it can be software, it can be patents, these are all your fixed assets. What are the current assets? All those assets which are getting converted or which are moving in money cycle. So, it includes your stocks, it includes your receivables, it includes your cash balances, it includes any monies which you have to receive from, let us say from electricity authority as a deposit. All these are included in current assets. Human assets are important, but not part of balance sheet. Then on liability side, we have got three important items, we have got capital. This is the money which owners have put in, then we have got borrowings. This is the money which lenders have put in as various types of loans. And then we have got current liabilities. Current liabilities again are coming from money cycle. For example, the payables to suppliers, so if we purchase some goods, do not pay that money immediately, then that much balance will be your payable. Or if we use, let us say electricity, but have not paid the electricity bill. Then the outstanding electricity bill will become your current liability. So, this was a brief of what we did last time. If you look at the bottom part of the slide, it is sort of trying to tell the purpose of these two statements. So, balance sheet is one which shows you the financial position of the organization. At P&L account, it tells you about the profitability of the organization. Now, moving ahead, again it tells you what does the balance sheet gives. So, it shows the cumulative position of resources, that is your assets and the sources of the funds, that is your liabilities at the end of the year. Of course, it can be at the end of the month or a quarter also, but keep in mind it is a cumulative position. So, if you purchase some land 10 years before, it will continue to appear in balance sheet today unless you have sold it. So, all your assets continue, your liabilities will also continue. So, if you have obtained a loan 4 years back, it will appear in the balance sheet at that time. It will even appear in the balance sheet today as long as you have not yet repaired it. So, you cannot say, okay, this loan was 4 years ago, let us forget it now, that is not possible. Loan tabbita, if there is an outstanding balance, it will be shown in the balance sheet of today also. That is why it has been called as a cumulative position as on a particular date. The profit and loss statement essentially gives you revenues and cost performance, but for a particular quarter or a year. So, if you have made any sales in this month and if you make profit and loss for this month, it will come as a profit and loss of this month, but it cannot be shown in the next year. It can only be shown in the current year. So, it is not a cumulative performance, it is a performance for that period that is shown in the PNL account. Now, very important statement is as we have seen in the beginning, your resources and sources match. So, your balance sheet exactly matches, but after doing business for some time, your business cycle is continuously going on, will the balance sheet still match? Is it possible?  Now, I am just going back to the slide so that you can have a view of the balance sheet. So, you can just look at the asset side.   So, your liabilities are falling, assets are rising, then will the balance sheet still tally? Is it possible that the total of assets and liabilities will match? What do you think? Just have a thought on this, is any item missed out in the given assets or liabilities which should be there? Actually very important item is known as reserves, which was not shown earlier, but I am just showing it now to highlight it or bring your attention to it. So, what will happen is after say one year, you have made some profit in profit and loss account, that profit either is taken away by the owners, it is their profit, they can take it away, but mostly owners do not take away the whole of profit, they may take some money, remaining money is accumulated in the business in the form of reserves. So, that reserve goes on increasing, see we have already seen that assets are rising, your external liabilities are not rising, in fact they are falling, so there is a difference, that difference will be given in the form of reserves. So, what happens is for a healthy business, profit goes on building up and that accumulates in the balance sheet in the form of reserve. Now, suppose you have no time to study the balance sheet, the whole balance sheet of a company, just have a look at balance sheet and look at the reserves position. If the reserves money is high, it immediately tells you that company is able to generate profit on a consistent basis, because one year profit, but it is only for that year, but that whole profit is getting added up in reserves, if the total profit for so many years minus what profit they have taken away is what is shown in the reserves and if that amount is very high, it will mean that company is having a good financial position for a pretty long time. You would have heard of blue chip companies, most of the top rated companies have a high level of reserves. Now what happens is if you have good amount of reserves, then company can easily expand, they do not have to go to bank to borrow new money, they can spend that money on anything which they want and the company will grow in a very healthy manner. Now, once again I have summarized the financial positions, so balance sheet, it may be prepared at any time, not necessarily at the end of the year, but it gives you position as on that day and profit and loss account is normally prepared and presented at the end of a particular period. This is once again summarizing the items in the balance sheet for you, so we have got liabilities. Now the first item, I have put it as owner's fund. Now what do you understand by owner's fund? If we just go back, we had earlier put capital and we had the second item was reserve. So these two items taken together capital plus reserves are known as owner's fund. Now how do you define capital? If we remember we have seen that money which owner's put in is a capital, here business is generating profits and giving it to owners in the form of reserves. So capital plus reserves together is the first item in the liability side of the balance sheet that is known as owner's fund. The second item is non-current liabilities. In the earlier balance sheet we have called it borrowings as an example, but any liability which is of long term in nature is called as a non-current liability, mainly it will include borrowings or loans. It can also include any other item which you have not paid in one year's time. The third item is current liability. So current liability we had seen that from the money cycle certain liabilities arise. As far as the definition is concerned if that liability is intended to be paid in one year's time then it is called as a current liability. So we have seen 2-3 examples, can you think of any other example? For example if your employee is with you, you should pay salary at the end of the month but if you do not pay that salary then the amount of salary which is still unpaid it will be called as a current liability. If suppose there are some other expenses let us say office expenses you have not yet paid them. Then that is also a current liability. So all those liabilities which are due and which are payable within one year's time they are in current liabilities. All those liabilities which are not intended to be paid in one year are non-current liabilities. So both these non-current plus current these are external liabilities and the top item or the first item was owner's funds. This is also called as an internal liability. Now go to asset side, we have already discussed what is a fixed asset. So this is a infrastructure or these are the assets which are long term with the business to be used for the business then we have added one more item here that is known as non-current investments and the last item the bottom most item is a current asset which I think we have already discussed. So what we mean by current assets is these are those items which are to be liquidated or which are to be received within one year's time or and they are normally coming from the money cycle. So all those items which are likely to last for more than one year are called as non-current. So assets broadly are non-current current, current is the last item. Within non-current we have got fixed assets then next is non-current investment. Now what is this non-current investment? First of all what is meant by investment? If you see I will just take you back in the earlier balance sheet we had not mentioned of any investment we had just seen fixed assets and current assets because that was a balance sheet which emerges from money cycle but in case you have invested some money outside your business then that money is called as an investment. So here any money which you have put outside your business and which is to last for more than one year then it is called as a non-current investment. So can you think of any examples that which item will come in non-current investment? As we have put some money in the bank as a fixed deposit for more than one year not in your bank account because bank account is a current asset but if you keep the money in fixed deposit with a bank for two years then that will be an example of non-current investment. Similarly if you invest in shares of some other company then that is non-current investment. Are you getting me? So you can have some examples of those money put outside your business that is a non-current investment apart from this if you have any non-current assets then they are categorized as other non-current assets. So are you getting? So these are the important items in balance sheet under liabilities and assets. Now we will come to the next item we are going to discuss balance sheet and P&L in the next sessions but right now it is very important for you to know where from you get all this information. The best source of information for you is a annual report because every company comes out with an annual report and that annual report has financial statements. We have discussed financial statements like a balance sheet P&L, cash flow they will all be available in the annual reports along with notes to accounts and some more disclosures. Then there are some other things in annual report also there will be chairmans statements there will be board of directors analysis there will be auditors report there will be report on corporate governance there will be consolidated financial statements then there is a management discussion and analysis. There is also a list of employees receiving remuneration more than 60 lakhs maybe some of you are aspiring for that they are earning more than 60 lakhs per year. So there is a list given out there that who are these people what is their qualifications and experience and so on. So this annual report is a authenticated document and a very useful document for any learner you will get financial statements here I am telling you this right away because I am expecting each one of you to download the annual report of the company this is freely available. So please choose your company download your annual report read it go through it because in our course in coming lectures we are going to discuss about balance sheet P&L, cash flow some more requirements of preparation of financial statements all these should not be theoretical go through your annual report look at the balance sheet and P&L of your own company your own means you do not may not be owning it. But consider it as your company and start studying it from now that is why in this session I have told you what is annual report in detail we are going to look at various financial statements in the coming sessions. I hope you have liked the second session also this was bit shorter one but the third session onward now we will go into technical details about each of the financial statements thank you so much and namaste.


about six months now so this is all like it's kind of what i'm seeing here you know just got this big green open expansive lot with a little bit of solar power going on in the background because this journey of delos for me this is branding on kind of that dinosaur you know that in this case that gargoyle that archaic image you know from times past i was a windows server administrator i have a windows problem now part of breaking out just my windows server shell it's been a lot of retrospection a lot of diagramming out a lot of figuring out what it is i know you see i've gotten the dev ops because of my love for power shell i love being able to automate um i love this idea of having a plan of having things mapped out and i got really tired of spending my sunday mornings click drag click drag i wanted something more what something better actually what it was i was studying for the s7410 microsoft server exam and i got so you know you know swirl with the power shell stuff i just absolutely fell in love with it or this case bird because just like all the other birds like server administrators everybody's going this way i'm like no wait a minute i like this automation thing i like this idea of being able to tell the computers to do it for me and i'm seeing a house here so you know this is kind of the houses i wanted to build you know that was operating system agnostic i wanted to be able to do more so in the last six months i have learned so much and shout to my colleagues back there um tom sherry erin lindsey thank you guys sherry has really taken me under her wing uh kind of like the cat down here that's going to keep an eye on everybody to stay back and she is just really kind of giving us that development mean cat right there actually and this is my manager right here i tell you he is just uncharged but the captain's hat on he is steering the ship which has been a wonderful opportunity to be a part of this dev doll shop and this is erin right here you know top of the line first class vehicle i mean it's a key of you know this is one of the nicest keys that they got that you gotta understand who's been with the bank 15 years and he just transferred into the dev ops team well at the same time i did and you know we both kind of learned things as we go lindsey there you are right there you know lindsey and i and erin all came onto the dev ops team around the same time so all ahead me sora and i don't recognize that character that is not kairi but again this idea was a brand new adventure none of us really knew anything about what dev ops was again we kind of had this old-school technology get the way things work uh lindsey the developer erin was then we have uh access management and myself and the server admin so we kind of came in with this different mindset and we didn't know how to get there but we kind of got behind some of the you know i'll call them wise old sages you know the head goat if you will and they just directed us and really what happened is um the bank is getting so much new work taken on well they had to expand they had to build this is a good thing for anything to have one of looking at here about publisher worth glasses all right so we got a train come on here you know what maybe there's a bigger train because that was for us there's a bigger train coming there was more work coming and they needed to expand the dev ops team so it was time for that team huddle it's time to get fresh bases out there time for us to get trained up on brand new skills skills that we didn't have before so here i am i'm learning linux i'm learning docker i'm learning the jukes and giles of docker not knowing one thing what i'm doing seriously i asked a friend of mine cj another one of my co-workers wanted the guys in charge here um this is yeah i like this one so i'm learning how to store fight with docker i'm learning my doctor's here and like okay i've got my docker stood up it's on this vm what do i do with it it's just it's there it's a thing i do i jump over it do i seriously i'm sitting at this terminal docker's installed docker version i have no idea what i'm supposed to do with this thing i think my computer crashes a week later and i lost it however i had also been studying um technology called aws and aws was the long lost time i absolutely fell in love with aws and really what it was is i was learning okay how can i make these vms on the cloud and then i kind of got distracted again swirl or bird or in this case castle i read an article on somebody how they made a really kick butt gaming system out of their uh a uh ec2 uh box on the cloud and they run all their games on the clouds i was like forget work i'm gonna do that i want my castle to move forget castle let's just make it a mountain let's make this really big gaming ring and then i'll just go on my video games up on it and you know i've had to continue to study and i've had to continue to develop learning ec2 and learning the s3 everything along with um aws and i know i'm not being very positive here and honestly i don't know enough about this technology you know the competitors for aws well in here i am the microsoft guy right um docker you guys have said lxc lse i see i don't even know yet i'm so new to the to the thing well guys thank you very much


Welcome back on the chaos zone stage. I hope you had a great day so far. And after the Algorave talk, we're happy to host a talk by Andi Müller-Magun. He is a long time member of the CCC. Now he is at Wauer Holland Stiftung and he's also a data journalist. And today he will tell us a bit about things between WikiLeaks and the CIA. And this talk is some kind of successor for talks he gave previously. But for all the details, he will tell them by himself. And yes, welcome Andi. And we're happy to see what you can tell us and all the interesting details that are in your talk. Thank you. Okay. Good evening. So I named this talk when WikiLeaks bumped into the CIA, Operation Kudo Exposed. So explain a bit later what that is. Just as a reminder, the hacker community and the CCC, even in its bylaws, one of the core things has always been information wants to be free for the sense of the hacker ethics. I brought a small snippet from Vau himself where you will not hear the sound at this moment due to technical reasons. But where he talked about the hacking of society through freedom of information. My talk will have two parts, what happened so far and what should be done now. In the first part, I just want to refer a little bit on the context of what I'm talking about. So this is about what happened surrounding WikiLeaks in the context of the CIA and the United States government getting on them. I had two talks about similar topics already in 2019 at the, unfortunately also, no, that was still the last real Congress. I talked about the technical aspects of this violence. And you will see one image that I needed to copy from that again. Then last year, I talked a little bit about the CIA versus WikiLeaks interpretation tactics. That was more what happened to me and other surrounding WikiLeaks. Now, in the meantime, this year and of September came a very important article in this context on Yahoo News. It seems to have been doing that. Some guys have been hired there who previously worked for Newsweeks and others. The article from 26th of September is called Get Dapping Assassination at the London shootout inside the CIA secret war plans against WikiLeaks. And it did reveal quite some things. It finally referred to my talk. It links even to the video of my talk. It takes some quotes from it. It confirms a lot of it and adds a lot, but it also frames. And with framing, I mean, there is some disinformation.  What I'm trying to do today is to reconstruct the whole thing a little bit to reframe it and help everybody to understand a little bit what happened here. The Yahoo article rightfully distinguishes the timeframe of the interaction, so to say, between the United States government and WikiLeaks into four to five timeframes. One of them at the beginning of the WikiLeaks project or let's say before Snowden, so before mid 2013, the Obama administration had, although the diplomatic cables have been published by WikiLeaks, Afghan, Iraq warlocks and so on were out. They had the view that as long as some entity or some people are publishing are engaging in publicistic in journalistic activity. There's nothing they can do because first amendment of the United States Constitution talks about the freedom of publishing the freedom of speech and the freedom that does include journalistic activity of all kinds. After the Snowden, not revelations, but the fact that Edward Snowden was getting from Hong Kong on the way to somewhere else, but he got to Moscow with the help of WikiLeaks editorial member, therefore in acting in what you could call journalistic source protection. However, that brought the US government to a slightly different view of WikiLeaks. They didn't really like it. So Obama allowed the intelligence community prioritized collection, WikiLeaks, search warrants, opponents, national security letters. So here we're not talking about, as far as the article mentions about the legal investigation, yet this is more intelligence work, like they allowed them to get on them. They also, in the context of the Snowden revelations now, where it wasn't WikiLeaks, it was Glenn Greenwald and Laura Poetras, who had been given by Edward Snowden the material, and they published the material together with Guardian, the Spiegel, others. I was personally also involved with the Spiegel, I should disclose. However, they tried to relabel not only WikiLeaks, but also Glenn, Laura and others from journals away to like information brokers. They tried all kinds of definitions to circumvent the protection of the United States Constitution, you could say. That went not that far. At least I have now actively knowledge of a criminal prosecution running against Laura and Glenn. However, there were for sure intelligence activities that they also reported on and everybody who was involved in the publications. As you might know from history, the Guardian was later forced to even destroy the computers where they had possessed the Snowden material and so on. So there was quite some things going on. In 2016, the next milestone in the change of the relations between the United States government WikiLeaks, to say it nicely, was the publication of the DNC emails that by the definition of the National Security Agency, like they said, this was goods effort to zero was the Russian military intelligence, the GRU, and that the whole publication was with the intention to hurt the interests of the United States. This now is a first point where we could sit back from our European perspective a little bit and say, wait a moment. This was about leaking. I mean, this was leaked emails or however, let's say it was emails that somehow got leaked, obtained or otherwise, but in any way WikiLeaks published them. What the discussion was about was how Hillary Clinton had treated Bernie Sanders as the other candidate of the Democratic Party. She obviously did not made it. She made it. So this we could call this exposing of facts in the public interest. But as I said, the United States at least National Security Agency and others seem to have agreed that this was, you know, intended to harm the United States, not what Hillary Clinton did, but what WikiLeaks did in this publication. I think it's important that we distinguish between how we evaluate these things and how the US government puts these into different baskets or categories. However, then it got much more wild when WikiLeaks started at the beginning of 2017 to publish with a so-called fold seven series. Some documents from the Central Intelligence agency from the CIA, Mike Pompeo was in charge of it. I did talk about this at length and I want to repeat this last year. So he got very upset personally because he was also potentially personal responsible for it. So it was under his watch, so to say. However, the framing aspects of the article are worth having a brief look. The, what happened this year was also that the key witness of the prosecution Icelandic guy called Sigurdur Tardazan made it public that actually he lied to the FBI and that they fabricated part of the evidence based on his lies. Also, they could have verified things. He later even was imprisoned for his multiple illegal acts and the Icelandic government saw it as reasonable enough to declare him a danger to society and therefore lock him up. And that's not happening that easily in a country like Iceland. Normally people are very calm and down to earth. However, the article came just after a few weeks after the publications on this fabricated evidence. And it's fair to say that the gravity of the Yahoo article was a lot higher and a lot more was discussed than about the fake evidence of the key witness and so on. However, one other aspect that was in the Yahoo article was a thing that is from my reading and I've talked to many people. There was no evidence for this whatsoever. The Yahoo article claimed that there was the Russian government also having like kind of officers in front of the Ecuadorian Embassy or in the immediate surrounding preparing to help Julian to ever create him, so to say, from England to sneak him out. As the article says, Russian Interpreting to Sneak Assange out of the UK. This is a little bit wild and it's double wild when one looks at how the involvement of the Russian government, how that upsets American people, American media and so on. This is such a polarized environment where the moment the Russian government is declared to be involved, it changes everything. What happened really here was something different and that is that Julian had in cooperation and in coordination with the Ecuadorian government found a way to legally leave the embassy and the United Kingdom. By becoming first an Ecuadorian citizen than an Ecuadorian diplomat and then in theory he would have been able to leave the UK because a diplomat on the way to a different working place has under Vienna diplomatic assurances is immune from any kind of interference. However, the article does reveal some aspects of what happened. For example, the kidnapping plans, the assassination plans that the US government considered this year a play through ways to kill him in the embassy to poison him, to kidnap him from there. These kind of extreme acts did not happen and the article claims that justice prevailed, the White House lawyers had doubts the National Security Council and the heads of the Senate and House intelligence committees ensured that these wild ideas because they were not compatible with the legal framework, not even with that of the United States, that that did not happen. So the article gives you kind of this American song melody of, yeah, we had some wild things that this year are going on, but you know, we are a democracy and we stopped it. However, there were some actions that were according to the article and the witnesses and lawyers had talked to well court out, extensive spying on Wicked X SSCAs, dealing with electronic devices. Then there were things there we could talk about like the article claims that to what was also carried out so in discord among the groups members. So now if any one of you is longer than a few weeks in a in a CCC like hacker club or working for a journalist organization or working in any other group.  And I'm personally, I wouldn't always be able to distinguish between is this now a CIA operated, you know, group fight or is this normal normal group dynamics. People don't like each other, people having disputes, people having different ideas and how to do things and so on. So I would suggest to take this kind of claim with a with a grain of salt, not every dispute among a group has been created by the CIA. Also, I'm very generous on bashing them. However, they also talked that at some point, they changed the whole context of Julian and Wicked X from a target of collection to target of disruption. Well, for sure, some things happened there. But this is not what I can go into detail as if I know no detailed report on it. And however, the project I talked about that Julian would get legally out of the embassy as a member of Ecuadorian diplomatic staff is coming together in a very is like the most critical time frame also according to the article. And that we were able with the we were going through with the lawyers, the lock files of the embassy security service, the videos and so on. So we have been able to identify the time frame. And this the time frame is the 16th of December 2017 until the 26th. This is the most critical time frame because around the 16th he was officially not only declared a diplomat. There was a publication in the Ecuadorian like legal got set or what it's called so like the legal publication in Ecuador to have him declared. He had around the 21st the head of the Ecuadorian intelligence visiting him. So that means he also had the diplomatic passport. It was fully formally done. There was a discussion of the process and this meeting on the 21st. I had mentioned it in my talk last year was the most high priority conversation that ever happened in the embassy. At least as far as we know from the witnesses of the security service who later revealed to the court that they had been instructed on behalf of the CIA to do other things than to protect the embassy but to spy on Julian. So this meeting on the 21st was extremely important to the Americans. And we do know roughly that the whole story ended through various means but mainly to pressure on the Ecuadorian government in Quito in Ecuador around the 26th when they actually called the plan off. Because the Americans knew about every detail including how he would get out of the embassy in what type of car and so on. And they also then at some point denounced his diplomatic status after pressure from the United States government. In this timeframe I made here a little bit of an event matrix which is completely incomplete. I have to say there's many things missing for legal for other reasons. You know some things are just too wild. The US government for example would never break into European law office.  That's conspiracy stuff. They don't do these things. They of course comply with the law. However we have some events that are funny and fit well into our picture. For example that after on the Saturday the lawyers from Spain and England were sitting together with Julian. Like two days later in preparation of that meeting on the 21st came the fire protection service into the embassy. And those who have seen my talk last year know that one of the fire extinguishers played in the meeting room the main role for holding a buck. However I'm coming to that. Then we have this observation that every day in this timeframe there was a silver gray Ford car with sometimes two sometimes three sometimes more people sitting outside the embassy. Seeming obviously to wait for instructions something to happen. I'm coming to that. And we have other things going on at that time frame that kind of fit in the end of frame. So on the I selected three events to talk about them a few minutes. The first is this fire extinguisher. Here you see it. And in the on the right picture you see this black bottom of the fire extinguisher. That's where they had a magnetic little box with an audio microphone. I mean audio buck in it that seemed to have not only recorded but also transmit the conversations in life to the American intelligence outside. Finally this on the 18th comes a company not even from London the Iceland fire protection limited a guy and goes into all the rooms in the embassy to check the fire extinguishers. Now according to the lawyers there had been intensive discussions with the employees and David Morales the owner of UC global the company that was originally hired to protect the embassy is known to have talked to his people and emailed them. Mentioning that the Americans want all the other rooms at some point to be booked and want access to the fire extinguishers. We don't know exactly what happened in that discussion to the last detail but we know that on the 18th came this British company. And this is a little bit cross and I think there will be many other embassies of other countries who will be interested to check if they don't are maybe serviced by the same company. Now the other nice event that I selected is the night from the 23rd to the 24th so the very morning early morning hours on the 24th so December morning Christmas morning so to say. Where you have the three guys sitting in the car and on the backseat on the right side someone reads the briefing notes. I will show you the hopes don't tell me this hopefully works. Okay the video doesn't work. I'm sorry I can't show you the video today maybe courtesy of the CIA. However so the guy in the backseat browsers through the briefing notes and we have been able to at least read part of what they have been what this briefing notes say. It says this page that we have been able to read mostly was in the event of loss of camera coverage. So there was a process to be established when the surveillance cameras and the embassy wouldn't deliver pictures anymore and the guys outside sitting partly according to the article the British police guys was guns. And this year eight people maybe without guns would be ready to jump into the scene a crash diplomatic car shoot into tires of cars that would try to bring Julian away and so on. In the case he would he would walk out and so there's a few keywords here that I just want to emphasize in the event of loss of camera coverage then there's talking about and something called GS seven that might be code word for CIA or something different. I mean he is clearly the Metropolitan Police. That's a normal acronym in England, and they talk about the context of the operation kudo. So we looked up the word kudo kudo is something saying roughly like friendship. So we have to assume this was a joint British American operation and that's exactly what the Yahoo article describes. However, what it does not describe is the legal implication because this could well be one of the most or best well documented breaches of the Vienna Convention is basically saying that the premises of the mission shall be in in voile inviolable, which is normally means that you shall not bug you shall not, you know, put surveillance devices, cameras in cameras or whatever you shall not hack into the camera surveillance system often embassy as to host state and so on. So that intelligence to it and that the CIA was doing it in the case of the equivalent embassy is already part of a Spanish lawsuit. However, the dimension is a little bit different as the British police seems to have access have had access to that video surveillance and that is potentially legally a different thing that will be subject to some legal steps going on in the next weeks and months. The third event I selected more for relaxation for relaxation issues is on the last day. You see here to police officers carrying an astonishing amount of eight cups of coffee coffee for relatively small police car. That gives you an idea what was going on there the British police being prepared to see at the site. The conference room is about in the area where the where the trash bin on the left side is. So giving you an idea of how intense the British police was also in the scene outside. So what is currently happening with this and a lot of other material is checking the violations of the Vienna Convention. Then pausing together many of the events and observing patterns and trying to see those patterns at other places as we of course still do not know the full scope of the operations of CIA and other intelligence agencies against WikiLeaks. This is just the tip of the iceberg what happened in London. But also to see where other journals where other citizens where other governments organizations whatever where maybe targeted with same or similar ways and methods. So with this brings me to the second part of my little talk. The question what needs to be done. So and I try to first invite you to a little reflection because as some of you might know during a science presented the WikiLeaks project in the CCC Congress end of 2009 if I recall correct he made another talk in 2010. This was very much a project of the hacker community and it was highly welcomed at the time. Because it was like combining the idea of freedom of information which had always been in sharing information which had always been the spirit of the hacker scene with those of journalists and democratic. Yeah think tanks to ensure that we would have actually informed society not this this very weird concept of an information society which which does not really say anything between the relationship between information and society. But an informed society is a clear picture I think. And therefore the better wording. So the other question is of course is what what what does this whole thing this what we have been reading in the article and what we're now a step by step here revealing and starting to understand. What does it tell us about the United States governments. Yeah prosecution at the U.J. Pompeo the CIA all these people. How compatible are they really to decide to society that is based on an informed electorate like the people making decisions based on knowledge. And voting based on knowing what is going on and that's slightly disturbing I think what we what this thought brings us to. So here's my little ideas and then I will just come with some questions to the audience. So yeah what can we do and what what maybe should we do this is here just some ideas of mine. Well we could of course hope that the United States the people of the United States the government of the United States. Would understand the core democratic well it attacks here when going against assigned WikiLeaks and so on. So in theory we should we could hope that the self healing or the self understanding mechanisms of the United States society will stop this madness. Because they will see hey wait a moment this is our constitutional first amendment that we are taking here indirectly. And if we don't have like the publishers right the journalists and publishers right to inform the public then we have nothing. Well the second obviously a level would be to dissolve the CIA. I mean Kennedy had this idea before Shadrow it in the wind and so on. But I don't know how at least this show will continue with that budget with the information operations with the influence operations. Where actually where the dog is just a tiny little aspect of it because the question is how should a democratic government work as long as there is an intelligence agency that has all the knowledge about every person involved in all the little compromise boxes and the aspects of how to nudge and how to influence and how to manipulate and so on. Well and then the third aspect outside the United States here in Europe is of course the question of how can we immunize those people entities government organizations and so on where it still seems possible to understand that this is the journalism and the right to inform the public by making also information and material public that governments corporations or whoever would like to keep secret but if that documents are playing a role in informing the public in the public interest then it must be allowed to make it public and that was what's called the fourth stage or the right of the press to inform the public. Yeah how can we do that that's of course more a question. Here's my list of questions that I will want to address to the audience we should have 20 minutes and maybe a few seconds for discussion of this. So guys how do we get your message out of jail ladies and gentlemen how do we do it. How do we stop the communalization of journalism and those who ensure access to information in the public is just in order to achieve an informed society. That's our duty I fear. How do we ensure a value driven community. So as everybody knows the CCC had always different fractions of political and the technical fractions. Then came at some point the party the event and the hedonism aspects all together and we had a great fun time but I'm not sure that we also took care of ensuring that we are value driven community all the way. I mean when we look at the CIA and the NSA methods there's obviously some kind of atmosphere between those who work in the IT security industry and those who maybe then take offers from the intelligence community. And that's not the spirit of the hacker ethics and that's not the spirit of the CCC and that's not the spirit of an informed society that's people with money who instrumentalize technology people and you don't have to like look at the CIA as the most crassist maybe entity. It starts with the so called open technology fund. I mean we had various years the ability to observe how the Tor project had its issues between the two worlds of the US government having this and that ideas and our community having other ideas of how anonymization works. And I'm not sure we can say that our values have been preserved and we have ensured that the OTF finance projects do not serve just some funny governmental interest and when it was relabeled partly from internet freedom to circumvention measures that I think gave already some ideas on what could go wrong if yeah governments start to fund projects of the so called hackers. Yeah, so this is my question to you guys. How do we get him out? How do we ensure our society stays intact and democratic and how do we as a scene avoid to be corrupted by governmental money and funny interests. And I hope the moderation cannot take over and provide some answers from the audience. All right. Thank you very much Andy for your talk. Let's see how this will work. Thank you also for your questions to the audience. I mean time to fix this video and make it this one minute 23 seconds video so that I can show it but maybe you can start to take the questions. Sure. Yeah. And yes. So let's let's say to the audience please put your possible answers to these questions in the chat. I will I will follow them as good as I can and so that we can have a lively discussion. I know it might be a bit limited because in the Presence Congress it would be easier to to interact with with each other. And but yeah let's let's let's see that. And but first of all maybe Andy if if you have the capacity for a question from the interwebs. Then the question would be how did you obtain the pictures and camera footage from the embassy. Well this has to do with the legal analysis of this material. I'm myself here by the way you could switch on the video if you wanted. Well I'm myself accusing the Spanish company to have spied on me and other colleagues. And so I'm part of that legal proceedings as such. I'm also helping the lawyers to obtain the technical evidence. There was a shitload of digital evidence confiscated that needed forensic examination and so on. So this is material accessible to those who have been affected by the illegal activities performed by UC Global and others. All right. There is also the question of are there pictures of the Ford or the people inside it. But I think that's pretty much part of the video you have just shown. Is it. I'm sorry. I don't see what is being broadcasted. Do you have access to my sliding to the streaming laptop. Okay. Yeah. Yes. So this is the full video where you when you see the guys reading the briefing notes on the backseat. We have been able to zoom in a little bit and so on. And the question was where did you get that from. But I think you already answered that in the previous question. Because it's no answer to my question. What should we shall do guys. Yeah. So we have one line of feedback for example that how to get Julian Assange out of jail. What one proposal is ask our foreign minister give Julian German citizenship. Make it a chef's. So part of the part of the chancellor. That's what it means in every in every German activities. Question mark. Would that work. It's been worked on. I mean the new we have a new foreign minister who's a woman from the Green Party. And she seems to be very much a fan of United States German relationships. I'm not sure how much she sees about lack of values that the US government represents watching the history of the US Constitution and so on. But I'm sure there's a lot of work to be done there and the Green Party used to be also interested in the society and stand for human rights and so on. So I would say yes. It's definitely there's a path to go. All right. There's also a question. Are you so you personally still under surveillance. Do you know. Well I have taken some legal and technical measures and the German authorities have some evidence I provided to them still in their analytical labs and so on. It's a little bit unrealistic to assume that the Americans would not continue watching those who surrounded a certain big leagues and as a member of the Valhalla Foundation and we finance the we finance many of the publications and things or aspects of the publication. So it would be unlikely that you lost interest. At least for the moment they seem to behave a little bit more especially after the Yahoo article. I think it became very obvious also to the German authorities what was going on. So the article was helpful.  All right. May I ask you to maybe just also bring up again the slides with your questions. So we will have a second. I think this will help to spark a bit of discussion also. Good point. All right. Thank you. And another answer how to how to get them out of jail is keep talking about Julian Assange and the public attend vigils. I don't know what that means actually. Organ or articles write comments called the Department of Justice talk to politicians communicate. So this is this is one answer like like keep keep keep keep the word out. Yeah. I mean let me let me briefly try to interact with whoever gave that suggestion. I think it's well known that in in Germany in France and some countries there was quite some campaigns going on in the last months quite some people on the street acting for Julian and series of events and so on. Also a little bit in England but England seems to be a very tough under two aspects. The one is that they don't have that of a self understanding of a country with a constitution guaranteeing freedom of rights. The United Kingdom does not have a constitution and it doesn't have what's called constitutional rights. The United States have similar statements but they are not as clearly defined as a value system of a democratic society. So most British people if you ask them to do something for freedom of press they're like the press these assholes why should I do something for them. It's it's it's all very complicated and a bit polarized over there. And the other aspect is that the UK government to say bluntly there's quite some people who say that the UK government does what the US government says. And in this case there is no way according to that interpretation that you can avoid the UK government handing Julian over to the Americans. So the problem needs to be addressed in the US and Germany and other European countries have a different history obviously and I'm at least sure that if Julian would be in Germany I'm not sure he would be not having any issues but there would be a different discussion. However the the question how the so-called old Europe or the continental Europe that is now even more ignored after beer exit from the Brits can have any influence here in England I would say forget it. On the grass it's more complicated but for the moment it seems that similar to what happened to Julian and Wikileaks in our own community that there was quite a time frame when the reputation the character assassination had taken so much that actually he was seen as a persona non grata more or less the United States political atmosphere is even more complicated and more polarized between left right and nuts and whatever that it seems a very tricky task to bring some sense into that discussion as long as you have the military intelligence apparatus and Hillary Clinton saying like hang him on the highest trees. So there seem to be quite and that's also mentioned in the Yahoo article a revenge aspect of the United States legal system here not only Pompeo that want to yeah basically kill Julian as a symbol that no one should ever try to reveal the dirty laundry of the United States. So yeah that's a bit tricky and we will need more ideas and how to also initiate a better discussion in the United States maybe. Related to that another answer we got was for example how to how to stop the criminalization of journalism and maybe also other question of these questions is vote for the right people and well it's probably can help for some things and what comes to my mind is I mean indeed in this and also other prosecutions and trials very often there's some ancient laws involved on those grounds people get prosecuted right isn't that for Julian there is there is the intelligence act or what's the name of the S. Bill Nash act so basically what the US prosecution does is this is a secret grand jury that might have even more investigations running against Julian and we can extend that what has been put into the expedition inquiry to the UK at this point however that one already accuses him to violating the espionage act not declaring him having spied for another country but finally having revealed secrets to the American public and to the of course public of other countries that's what they call espionage that's a little bit ridiculous and it is however even more of a concern watching the fact that a US journalist would be able to claim the protection of the First Amendment the right of freedom of speech and the right of publishers and journalists to and so on however they deny that because he's not a US citizen so the US partially exports their laws and says well he violated against this American law called the espionage act but they do not grant him the protection of the US legal system and that is to call it hypocrisy is I'm sorry is too nice this is just really fucked up okay shouldn't we shouldn't we try to get rid of maybe like the espionage act or at least I'm all for it dissolve the CIA get rid of the espionage act I'm all for it I just fear that at least part of our community will have to become I don't know lawyers lobbyist maybe we need to look for better communications with the US hacker scene and see if they can kindly get into political consciousness mode and get for a moment distracted from technology developments into society developments and see what can we done to ensure that also in the future we have the right as a citizen to know what's happening in our name by governments and so on mm-hmm all right yeah because for example I I I I remember a couple of years ago I don't know whether it was in the 2013 the year of Snowden or later where we also had a talk at Congress about the German post surveillance for example where back in these I think it was the 70s where where we had the NATO troop statute got got into play but there was a verbal note from the from the from the German government who told the allies well we will be part of the NATO troop statute now but don't don't be afraid you will be able to have the powers as before under allied law you could say and only after this the information the investigative journalism of I think it was a historic historicion exactly mr. Fosher pot only after that came out the government had to say okay well we we want to stop this and now this at least officially is over well I mean it's not really over Germany is still member of the NATO and this regulation is still in place and just to have it said I mean the four seven revelations if you look at the publications with leaks you will see the modules the CIA had developed to make a software atroian a malware whatever kind of manipulation so fair look like it was coming from a specific country or time zone so to make malware or attacks on it is to make them look like they come from Russia China Iran you name it most Korea issued list as well and this is the scenario we're looking at already if you if you look at the news what happened the last year is we had all these attacks it was Russia it was China it was Iran it was North Korea most probably have forgotten some other people who it was blamed on but the discussion that the CIA would be having the tools to make attribution misleading to a country so what's called a false flag operation in military terms is creating a scenario where exactly we as a NATO member are now looking into military like conflicts again because the media environment has been so poisoned with guys and those guys hacking our IT our parliament our you name it this worries me it worries me that we as a technical community have not spent more attention to avoid that the media environment was able to like create again just paintings of enemies and create an atmosphere where war between countries seems possible again and that's something that's deeply disturbing to me and I think this is something we have to work on more as a community also to ensure that technical knowledge is not abused for like yeah political games by withholding information what I should mention is as we are only having a two minutes left here something I did agree to be available for a little discussion in the whistleblower tent that somewhere in that virtual world and the audience will hopefully find it all right so whatever it's called so the once again what's the name of the of the of the and the whistleblower tent of whistleblower village okay all right so go out to do whistleblower tent so after after this talk and so maybe one last question is it possible to sue the UK government for their treatment of Assange before the European Court of Human Rights it's a little complicated what's happening right now is I might I think other talks have covering it is that Julian tries to avoid his expedition and there is specific aspects of this which he might at some point be able to address at the European Court of Human Rights that in theory could stop his extradition but only if a specific criteria are met and so on how much now the UK government will listen to it after the beer exit and so on is due to political atmospheric reasons that's all a little tricky the European Court of Human Rights is not part of the EU agreement so it doesn't matter that the UK stepped out of the EU but it is still an instrument of Europe and not of the friendship between the United States and Great Britain so the atmosphere of the British government does not suggest at this moment to be overly sensitive to anything coming from continental Europe to say it carefully and that's pretty bad all right so yeah thank you Andy for your for your talk for everyone who's interested into for a discussion with you please go over to the whistleblower talk on this channel at Kauston stage the next talk will be reproducible building network infrastructure by Astro which will start at 9 30 p.m. so tune in for the next Kauston talk as well and that's it for now thank you


We are walking in Memphis but we're broadcasting in Lifford today and we're live from Lifford old courthouse and we're being royally looked after here. It has to be said with the food and refreshments so thanks everybody here for that.  There's spot prices to be won and it really is a family day because there's children's characters and storytellers and face painting and you can explore the cells as well and go down and get that whole experience down a couple of levels and bring the kids and scare the living daylights out of them if you like down below but it is a family friendly experience and if you're here on a tour you'll know that some great stories attached to the history of the place as well. Now I'm joined by a man who the last time I was here actually I was locked up with because it was my cellmate we were here raising money for a charity it was a few weeks ago and a whole group of us came together and spent a night down in the cells raising money for a charity and Kenny Crawford who got to the semi-final of Lord Sheer earlier this year. The final, sorry, the final, the final was was my cellmate that night so Kenny the first big question is do I snore? No you told me I snore and I was blissfully unaware of that. Well you didn't snore too much because I remember I had a good enough sleep that night it was no it was noise from elsewhere that kept us awake. What was funny was when you get up to go to the bathroom I switch the light on on my phone until you come back because I was kind of half scared. Any of the characters that were down there whatever about whatever they were like they weren't scary no that night. And most of the noise wasn't coming from the ghosts. No no no this is true this is true. Anyhow how are you doing? Oh it's good to see you again.  I am I'm heading to Spain now in July to play but I'm not sure the length of time because the way the bar is yet it's not open yet so hopefully if things go great with the crowds I'll stay longer out there you know so hopefully. Oh so they're at a different stage of reopening to us here. Yeah the bar I'm going to play in hasn't actually opened yet okay because it hasn't opened since obviously. So hopefully the crowds come back now in the summer and go out to October so hopefully the things will well. Not your first time playing out in Spain because roll the clock back a number of years yourself and a couple of guys a couple of guys yeah and mickey joe mickey joe funny and drum yeah funny and drum uh yeah played with all those guys yeah uh telerife as well with mickey joe doing this now 20 years this is the 20th year I think this year we're going oh wow yeah every year well apart from this most COVID years yeah every year we always do we stunt in Spain you know right but it's great that you can just dip into it and do it for a while and during during the really busy season yeah and then come back and gig around yeah it's really good because the plisters will be jammed you know it's a good learning experience as well they handle a big crowd you know it's great I love it plenty of Irish obviously Irish and Scotch it's just fold to the doors yes great yeah I love it your exposure and glory chair and and do you remind me there they actually went all the way to the finals so the exposure on that must have been great it was a good experience oh a lot of doubt you know um I can't speak hell enough off the the whole experience the whole glory chair production company from everybody from the stage manager to the runners to everybody they treat it me so well it was a lovely experience great and you know the phone hasn't stopped really since it came off the show you know so it was great and your mentor was Johnny Brady it was yeah and between the two years you went all the way to the final we did yeah so I say well you thanks to Johnny as well for taking me on the show as well he's a great guy really genuine person down the earth you know he is he is genuine a really nice man you know Johnny yeah so I say thanks to Johnny Brady for taking me on right right yeah okay then so the the diary book is it's pretty full by the sound of things for the for the coming months and and you're going to get off to some guaranteed sunshine and hopefully very shortly and try to send some of it back to us here you're going to do a song for us now well I'll do a Bruce Springsteen song very time is on the news as well yeah yeah absolutely yeah this song called I'm on fire oh great hey little girl is your daddy home did he go and leave you all alone now got a bad desire oh tell me now baby is he good to you to to you the things that I do I can take you higher it's like someone took a knife baby edgy and dog drove a six inch valley through the middle of my chest hey and I don't wake up with the she's so good wait free train running through the middle of my head on you kill my desire got a bad desire stuff can I cross with you did you get tickets no because apparently it's cheaper to go to Rome the one that's the word that is a word oh it's okay I have I have seen him five times yeah I've seen a number of times as well but three years shows all the time I couldn't resist it I know it's expensive are you going as well yeah yeah but I just I could I'm not going to Rome no might not stay in Dublin yeah no no stay up in Dublin anyway listen Kenny thanks somebody for coming in new watch all thanks for a slide best look with the the rest of the summer yeah see yourself Joe and we'll chat to you again thank you very much thank you


Okay, so I'm here today to talk to you about some work I've been doing in Linguistic mediation of visual saliency and a unique analysis that is able to incorporate large data sets and infer some differences in different language contexts. So before I begin, I like to acknowledge my collaborators, Jeffrey Viat, who graduated with his master's in experimental psychology and my lab, which is called the Lab Lab or Language and Behavior Lab. I have several undergraduate assistants that have worked on this project and thank you again for inviting me. So we use language in a lot of different contexts. We use it in various social situations, like when we're talking to undergraduates in the lab and telling them how to analyze data, as this picture clearly shows. We also use it in very complex dynamic visual environments that change as we move around the world and move our eyes and explore our environments. And we even have various different auditory contexts, whether that's a noisy room that you're speaking in or you have to compensate for an impoverished speech signal or perhaps in auditory contexts like hearing loss or other speech processing disorders. So my work mainly focuses on this question, what are the principles and mechanisms of comprehending language in realistic contexts? So there's a lot of different sources of information that we use in language processing. It's very context sensitive to a lot of different things. So we don't have a very good understanding of how exactly we know to draw information from these different sources and incorporate them during language processing. If we did, we would be able to have conversations with computers that included things like sarcasm and discussion, having jokes and things that we naturally do with language every day. So one way to think about context is where the source of information is coming from. Where it comes from is going to affect how it's incorporated and processed during speech comprehension. So there is visual information that we use to incorporate into the context of sentences. So for example, if I were to just wander up to you and say, the eagle is not in the sky and just walk away, that makes no sense. It does no context for it, right? By itself, it's underspecified. If we were out bird-watching and I point to a tree and I go, the eagle's not in the sky, suddenly it's a perfectly reasonable sentence to say. So your visual context continually shapes whether or not you'll be able to comprehend a sentence. I think this is most obvious when you're trying to debug a friend's computer over the phone and it takes two hours when, if you were right there with them, you share the same visual environment, it would take 20 minutes or so. So visual information, it's very easy to use. Having a shared visual environment is very powerful during language processing. The other thing about language is it serves as a context for itself. So this classic sentence, it's a garden path sentence where you initially have some ambiguity and then resolve it by the end of the sentence. So the sentence, the horse race past the barn fell. How many of you are familiar with the sentence, first of all? Okay, just a couple of you. How many of you understand the sentence? Okay, only one person. So now let me give you a paragraph and see if you can parse the sentence correctly. So if I say earlier this month the horses were in training for an important race to test out different training techniques. One of the horses was raced past a barn in a quiet corner of a farm. While the other was raced past a noisy schoolyard, they did this every day for a week. In the end, when it came to the big race, the horse race past the barn fell. Does it make sense now? Yeah, so I've added linguistic context. Now you know that there's two horses. I'm clarifying which horse is being raced and that that one fell over. So it's a fell barn. What was that? Yeah, so this is one of the most difficult garden path sentences and I hate it in a lot of ways because people almost never understand this even by the end of the sentence. A better one is something like the man who whistles tunes pianos. That one has a temporary ambiguity, but you resolve it by the end. But this one I think it's well illustrated by when you add more linguistic context then it becomes unambiguous. Okay, so we also have social context. For example, if I'm talking to a guy with a really nice beard and I say what a lovely beard you have, that is a perfectly acceptable thing to say. However, if I am talking to a woman and I say what a lovely beard you have, suddenly the meaning is very different. So your social context also continually plays a role and shapes what meaning is. The last type of thing I'm going to talk about here is long-term memories. This one is very difficult. It's more effortful to incorporate while you're having a conversation with someone. So if I ask you what kind of eagle did we see in the sky last week, you have to search your prior memories. You have to maybe visualize what eagle you saw and then turn that into linguistic information. It's very hard for people sometimes to access prior memories. We don't like to do it unless we absolutely have to. So in more recent research I've looked at when you give people information, even if it's completely false, they will rely on that as opposed to their prior memories. So they rely on the more recent information. So you can tell people the layer of fat around a whale is called flubber. And even if they know it's called blubber, they will reply, I believe it is called flubber. So they don't take that current information they're getting and try and reconcile it with prior memories. They just kind of go with the most recent information they've encountered. Okay, so in my lab it's all about context. The goal is to begin describing at a more computational and algorithmic level how we do this kind of language processing. It's very fluid, it's dynamic, it incorporates all these different sources of information that we have to switch between and eventually lead to systems that can use things like sarcasm and empathy and use behavioral and visual cues that lead to richer understanding. So our current understanding of speech perception at very low levels is quite good. We have very good models of language processing up until about sentence processing. But when you get to this pragmatic level, it starts to be very complex to describe how exactly you do this because it's so interactive. So some examples of other work I've done. I've looked at how we process language when you have two visual references and you have to build meaning in the moment. It's a different kind of mechanism than when you're doing something like recognizing a familiar object. So if I say the towel is not on the rack and they have to look at a towel on a rack and a towel on the floor, they have to build that meaning in the moment. They have to accumulate the information in front of them and then form a response. So they don't come into the lab with representations in their mind that they're accessing and fighting between, which would be a more competition mechanism. They have to build it in the moment. So there's a lot of different characteristics of language processing at this level than there is at the lower levels of language processing is what we've been finding. I've also looked at different work looking at the linguistic context, different behavioral contexts, where you take out the task and just have people listen to different kinds of language and just look at a blank screen to see how it affects their behavior. And as I mentioned, the research on prior knowledge, looking at how people process information when it conflicts with their prior memories. So largely what I do in my lab is track behavior in real time. We use a Tobi eye tracker, which just sits at the bottom of a computer screen and it picks up on people's pupils and how far away from the computer screen they are. It records their eye location at 60 hertz, about every 15 milliseconds or so. And typically in these language experiments, they're viewing images and listening to speech over headphones and clicking on a picture on the screen. So it's a very specific constrained kind of task. And why this works is because eye movements are very closely time locked to speech. So within 100 milliseconds of the onset of a word from the very start of it, people will start to look at a picture that sounds like that word. So from the moment you start to say the word bear, anything that starts with a B, people will start to look at on the screen. So it works very, very well for a window into processing and seeing what's going on in the mind as it's happening. So you might see something that looks kind of like this where you start out, you look at the center of the screen and you hear click on the pair, or I'm sorry, click on the peach. And you have the peach and you have the beach, which is just one phoneme different from peach. So you might look at the peach, look at the beach, and then click on the peach. It's a very simple task. There's two unrelated items on the screen. You might not look at them much. But what's important here is that anything that sounds similar, anything that is similar semantically or visually will compete for your attention on the screen. And you can see how much it competes for your attention with eye tracking. So one of the classic studies in this area shows how the visual environment shapes your sentence processing. So in the picture on the left you have the apple on the towel, you have the pencil, an empty towel in the box. And when you say put the apple on the towel in the box, people might look at that apple and then look at the towel and then look at the box. Because when they hear put the apple on the towel, that could be the end of the sentence right there. So they end up looking at the incorrect destination before they look at the correct destination. Now with the one on the right, you have two reference. You have two apples. So people here put the apple on the towel in the box. They know that put the apple on the towel is disambiguating information. It tells you which apple you're supposed to be drawn to, you're supposed to put into the box. So you look at the apple, you might look at the other apple, might not. But typically you do not look at that incorrect destination at that towel. So in this case, this study has led to about 20, 30 years of research of this sort of style of changing the visual reference and changing the speech to see how those interact and really dig into minute details of language processing. But I've always been curious about how well this generalizes. So typically we're not sitting around on a beach with a peach and just asking someone to click on the peach. We're having real conversations and real situations and using humor and we use it for communicating with one another. So in a very recent study, they asked what happens when you're processing language when the visual reference go away. So instead of simply looking around at the screen, use your memory to search and see if the image is there. So what they did is they have three pictures on the screen and it's either there when you hear banana and you have to say whether the banana is there or not. It's all you say is presence or absence or the image of the banana would be there in the target present trials. I'm only going to talk about these target absent ones though. So with banana, you have something that is visually similar, the canoe, you have something semantically similar, the monkey, and then you have an unrelated item like a hat. Now when the pictures are right there in front of you, what people tend to look at most is the visual competitor, the canoe. They look a little bit less but still look at the semantically similar picture, the monkey, and not quite so much at the hat.  Well previous research has shown is that if you say banana and the banana was on the screen, you'll look to the region where the banana was. The question is do they also look at the related items when they've disappeared? And they don't. So when those visual references go away, even if they were just there on the screen, suddenly there's no competition. So just that tiny change in those visual reference being there or not being there, suddenly this mechanism that we thought was very robust that was always there is not there anymore. So this might not be as generalizable as we thought it would be. So what I wanted to do with the research I'm going to talk about primarily today is see what happens when you don't have a specific task, just like when you're sitting around having a conversation with someone, you don't have a very specific goal like clicking on a peach. And you have speech that's very loosely coupled to the visual environment. So every once in a while I'm using these slides as reference for my speech but it's up saying things that also are not on the screen. So regular conversation has this loose coupling to the visual environment. So in order to do this, I'm going to look at visual saliency. Visual saliency is a quantitative measure of where approximately you will look at a particular static image. So I have this picture here of the different colored houses. A visual salience algorithm will take things like orientation, contrast, color, and predict the areas that people are likely to look at. You can then collect data from people with eye tracking and measure those regions and see where they look and see how it aligns with your predictions. So in this particular one they tend to look at this green door, they look a little bit over here and a little bit at the silhouette of the woman. When you look at the visual salience of this image, it looks something like this.  And you notice there's always this center bias. People tend to look at the center of images. And you can almost see the image there. It's very, there's a lot of contrast, there's a lot of lines, and so there's many different regions of salience. So in this experiment we had images like this where there's lots of different things to look at, like these bond icons. These are icons for all of the different bond movies over the years. We have Moonraker right here. And so there's a lot of stuff to look at. There's a lot of things to think about. And in the salience map of this, there is a lot of contrast and you can almost see the image. So again, there's this very high resolution, lots of different alternatives on the screen to look at. Other images we used, it's very what I would call low entropy. These images you have regions that are predicted to be salience and very large regions that are predicted to not be salient. So you have things like this. This is the salience map of the parents. So what we did is not just have people view these images but introduce language. So there are two conditions, one in which we have the language presented before the image and one during. Do you have a question? I will talk about the particular algorithm that I used in just a minute. So the language that I used, we had little paragraphs that somewhat related to these images. And some of them were in an affirmative form and the other half were in a negated form. Now I'm going to spoil it right now and tell you the negation has no effect on anything. What's going to be interesting here is whether the speech is presented before the image or concurrent with the image. So this is an example of one of the vignettes that was played. Will decided not to paint his house a bright color. He does not prefer a building that is painted vibrantly. Will did not often visit brightly colored homes. Sometimes Will would not want to stay at home. So it's not very interesting but it's language and they were all sentences that we could negate in kind of the same way. So in order to analyze this data, what we needed to come up with was something that could compare how predictive a model of visual saliency is for the language during and the language before. And what we found is an analysis called receiver operating characteristics. How many of you are familiar with a receiver operating characteristic analysis? Excellent, a few of you. So for those of you who are not familiar with this, this is an analysis that was developed during World War II to detect enemy objects in a battlefield. They take radar data and in radar data you want to be able to tell if a bomb is coming at you or it's a cloud. And so being able to detect a particular pattern in very noisy data on a single trial is incredibly important. In this case might be life or death. And so they developed this analysis that parses out the signal from the noise. So all of the graphs that you'll see are these graphs where you have a what's called a line of discrimination which means that it's all in the ways. This line of discrimination means that it's not predictive at all. If the curve goes above this line it means that the algorithm is predicting the data. And if it goes below that means it's not predictive at all. So this was used in the 1950s in psychophysics to measure detection of very weak signals, psychophysical experiments. This has also been used more recently with multi-dimensional data like eye movements. There's only a couple studies that have used this with eye movements. It's been used with fMRI data to look at individual subjects. So if any of you are familiar with fMRI data it is very noisy. And you have to aggregate over multiple subjects or else it's just all noise. This analysis you can look at individuals and get reliable data. It's also used in EEG pretty much any large data set. It's often used in the evaluation of visual salience algorithms. So there are several different variations of visual salience, how to parse that picture to determine which areas are salient versus non-salient. And the higher the curve is the more predictive the better your algorithm is. So it's very common in that area. So the only limitation here is that you need some kind of baseline or a prediction of what your data is going to look like. With that you then put your data on top of it and see how much of it matches essentially. So the area under the curve is how much of your data matches that baseline. If you have a very low area under the curve less than 0.5 it means you either don't have a good baseline or the data that you've collected is not aligned with that baseline. But if you have it higher than that 0.5 then it means that you have both a good baseline and the data is matching it. Okay so again the fixation data that I used participants either had the audio first or the audio concurrent. Participants that had missed lots of missing samples were discarded. There's no task here. They were just asked to sit and listen to these stories and they viewed each image for 15 seconds. So that way we have approximately the same amount of data for each trial and each image. So the question is how well does purely visual salience just the visual characteristics predict viewing behavior in these different language conditions. It could be that people completely ignore the language altogether and just move their eyes based on the visual characteristics or these different language conditions might be mediating people's eye movements, changing the behavior a little bit and suddenly that visual salience becomes less predictive. Okay so a little bit of details of the visual salience algorithm I used. It's called graph-based visual salience. It's a somewhat more recent model of visual saliency. It extracts feature vectors from the image, various features like I mentioned the color orientation. Then it forms an activation map that is continuous between one and zero over those feature vectors and then normalizes it. So it's non-binary values between zero and one but for the ROC analysis it needs to be binary. It has to be either the eyes are there or the eyes are not there. Either it's salient or it's not salient. So what we did is applied a binary mask and it's sampled at different rates in increments of five percent. So what we get is an image that has lots of regions predicted to be salient or very very few regions to be salient. I'll show you that in a second. So what we do is classify each fixation as either a hit. It hits one of those salient regions, a miss, it hits a non-salient region or it was completely you know they were looking at the wall because sometimes they do that. These are undergraduates so they look at all kinds of things and we do this for each individual trial. So this is kind of amazing to me that this works because eye tracking data is very messy and on each individual trial you can actually see the ROC curves. People are looking at salient regions. Okay so this is what the graph-based visual saliency might look like originally before the binary thresholding. It's a continuous value between zero and one where the white regions again are the more salient and the black is predicted to be non-salient and that is the salient's map of this image. Now when you apply the binary mask it turns into this and this is at the 85 percent sampling rate. So not all of the regions are that could be salient are represented here. It's a little bit less. So back to this image this one that's a little more high entropy there's lots of contrast. When you start to down sample it starts to lose some of those points and it is biased again toward the center so you start to lose a lot of those edges. So of course when it's at zero percent saliency none of the fixations are going to hit salient regions because none of them are predicted to be salient. As it grows as it gets larger more and more are going to be salient regions so that's where we get that ROC curve. Okay so qualitatively in this study when we first looked at the data we went this is amazing. It looks like the affirmative language and the negated language are doing different things. When they hear the negated language what our predictions were is that people would look for alternatives. They would look around and try and see what else is around that you could be referring to since you're saying not the green door they might look at the blue door the yellow door and that's what it looks like they're doing. But when you do the ROC analysis the lines are right on top of one another so this is actually two lines for the affirmative and the negated and they're right on top of each other so it was not at all a difference in how it drove eye movements to salient regions or not. Now this is the language at the same time the audio concurrent so we're looking at the image they're hearing the speech the overall model is significant it does fairly well at predicting people's eye movements but again no difference between that affirmative and negated. Now when you play the audio first and then they look at the image it changes to this so again they're right on top of one another the negation makes no difference in this analysis but it's a lot more predictive so the visual salience is driving people's eye movements eye movement behavior more than when it's the language at the same time so the language is exerting an influence it is changing and mediating people's eye movements at the same time and doing something that just purely visual salience cannot predict so a comparison between these two experiments they are significantly different from one another the audio first is better at classification than the audio concurrent. So one of the things I like to play with data and one of the things I was curious about is how robust this analysis is in face of individual differences if any of you have worked with large data sets from people people tend to be very different especially eye movements eye movement behaviors are completely different from person to person so we might expect that this would be all over the place for different subjects so this is what the image variability looks like this is each individual image in the audio concurrent only so there's a good amount of variability it's it's uh you know there's there were a huge range of images so some of them the algorithm was very good some of them that was a little less good this is the subject variability this is almost unheard of I don't know what other analysis is so robust in face of individual differences so no matter how different your individual subjects are this seems to be able to pick up on trends that are driven more by the image characteristics rather than the individual in the audio first of course this algorithm overall was better at predicting eye movements when the speech was before the picture so there's not quite as much because we're kind of pushed toward the ceiling here and the subject variability again is very small so there's not a lot of differences between participants when you look at the data this way so this is a really sensitive analysis it's really robust in face of subject variability which is typically quite large in eye tracking or any kind of noisy data set and what's really cool about this is you can actually test significance on individual trials I did not do that I collapsed over many many trials of different people but you can do that so it's a very sensitive measurement and if you wanted you could test a very much smaller data set um so what appear to be differences in looking behavior what looked like there might be differences in the affirmative versus negated it didn't seem to make a difference in this analysis it was still guided by the salience of the image so those regions that they looked at even if they were looking at more diverse regions on the screen what they were looking at was still salient so it seems to be driven by that characteristic so one last thing I'd like to tell you about one last analysis we did is look at these very high entropy images versus the ones with lower entropy where there's these large areas of salience so if you hear something like she did not see the 007 logo you've got a nice 007 logo right here you have a bunch of other things to look at if I say um he did not look at the parents what else do they have to look at not a whole lot so there's not alternatives available when they hear that negation so what I looked at is whether the visual image had alternative reference available or not and there is a difference so in the audio concurrent only when they have the language and the image at the same time there is a small difference between these images that have alternatives available and ones that do not have alternatives available it's more driven by the salience when you have those alternatives so but in a very small way so this analysis it does an excellent job with multi-dimensional data sets that have some kind of baseline that is all the qualification you need for this you can do this with n-dimensional data not just two-dimensional eye tracking any number of dimensions you want uh GBVS it it's a pretty good model of visual salience it's fairly good at predicting eye movement behavior under different conditions um this could be done of course with any model of visual salience or any kind of predictive algorithm you might have and people might be looking around a little bit more with negated language you might be looking around for different reference but what you're looking at what you're going to be driven to are still regions that are salient so there are visual characteristics that are still guiding that behavior so when is language what's language really doing here when is it changing eye movements what I think is going on is that it's mainly the task it's when the only task is to listen behavior is a lot different than when you have to click on the peach or when you have to do a very constrained task and in fact the very first eye tracking study showed this they showed people images and they asked them questions like how affluent do you think this family is versus um what kind of event are these people preparing for people view images differently depending on the question that you give them um this is the title of a paper that I thought described this very well saliency does not account for eye movements during visual search visual search is a very particular task you're asked to look for something amongst lots of distractors and so the way you do that is very different from other kinds of looking behavior the salience does not seem to drive in that case so semantically and visually based competitors they have these fixations during word recognition in these visual world paradigms click on the peach but looks to blank regions where those competitors where are no different than those unrelated regions so back to the study I talked about with the banana and the boat and the canoe if they aren't there there might not be that competition so potential applications I know a lot of you do lots of different kinds of research but I feel that the ROC analysis is a nice sensitive analysis that could be applied in lots of different areas and I don't think it's being used as much as it could be so of course in psycho linguistics things like eye tracking um even motion tracking and models uh that predict body movements could be used there in movement of robotics uh it could be used for there's plenty of predictive models and economics and politics machine learning all of that can it could be used you could use it to compare different analysis techniques and fmri and eeg data and even looking at something like experts versus novices um expert dancers versus novice dancers and comparing their performance so those are some of my ideas and I do have code available for this if this is something you're interested in uh this is something that I programmed in matlab uh if you are a matlab user uh just send me an email and I can send you the code and thank you for letting me mediate your eyes. Any questions?


In a sense, I'm grateful I had this experience, but it's totally terrifying. You inhale the substance, and then within five to ten seconds everything becomes black. The visual field is fractured in hexagonals, and then those hexagonal sort of themselves disappear, and you go down into this tunnel, this black tunnel. And the last thing I remember is that holy sh... What have I got myself into here?