Hi, can you hear me? Nice, that's what it's all about. Yeah, welcome to my talk. I will talk about hearing aids and what the state of the art is. There will be a little hacking with not my own. Yeah, since I know that there are some people interested in my talk that cannot hear very well or not at all, so I will publish slides that have very detailed speaker notes so that you can read it afterwards if you missed anything. I hope there will be a recording available and if it's possible to add subtitles, I will do those. Okay, first a few words about me. I'm a software engineer. I'm based in Munich. Some people might know me from my time in Cologne as well. I'm a software geek than a hardware hacker, so all this is also new to me. From university, I have a background in data mining and yeah, signal processing. I worked in the medical industry for a while, but that had nothing to do with hearing aids. Also my current job has nothing to do with hearing aids, so this is really just my personal project. I'm hearing impaired for about three and a half years, so this is when I started to dig into the topic and well yeah, that's just what I will talk about. Since I have since I haven't seen many talks about audiology here at the Hacking Congress, I will start with a short introduction and the process of how you get hearing aids, actually. Then what are the current hearing aid models and what can they actually do? Some words about the peripheral hardware. There's quite a lot of it outside and yeah, some hacking and another point is self-tuning. That are people that tune their own hearing aids, although they are not audiologists. Yeah, so this is an audiogram. This is a result of a hearing test that you usually do at ENT doctors. The x-axis shows the frequency in kilohertz and the y-axis, the loudness level of volume. The silence is at the top, the really loud sounds at the bottom and the green line you see here is that result of a normal hearing person and this is obtained by the audiologist or the doctor, plays sounds in the different frequencies, starts at a very low volume and as soon as you hear it, you hit a buzzer or say yes and then they create this curve. So the blue curve is a typical curve of a hearing-appared person. So what you can see here that at the low frequencies, the hearing is quite low. That's very typical. The hearing starts to get worse and the high frequencies and bear in mind that the scale of the decibel scale is actually logarithmic. So if you have a hearing loss of 60 decibel, it's 1 million more than 10 decibel. So it's not linear. Another thing that gets measured at the audiogram is actually the maximum that you can hear or that you can stand hearing. So the audiologist raises the volume more and more and you have to say yes until it hurts. So what you see here is the red curve, it's a level of discomfort and also a typical thing is that it raises exactly at the areas where the hearing goes bad. This is a very complicated thing for tuning hearing aids because they cannot just amplify everything. Because you would like hurt people very soon as soon as you get below the red line. Another thing is the area where speech takes place. This is called speech banana. Actually, that's a technical term for it. It varies of course according to the language and the speaker, for example, female speakers have a little slightly different curves than male speakers and this is the area where hearing aids are targeted because they are used to make you understand speech again, so they focus on this area actually. You can see the blue line here, so half of the banana is actually cut and this is the high frequencies which in speech are the consonants like S and F, for example. The vowels are usually understood quite well. To give you an impression how I hear, I made a sample. So this is a song called Sad Robot from Pono Phunik. It's a nice band which makes music with Game Boy and the guitar. And the original has really nice high frequencies at the beginning. This is why I use it for testing. And later there's also some singing. So this is the original. And now I'm going to play my version, so with less high frequencies and the tinnitus as well. There's actually a website where you can download tinnitus sounds. And it says that you should actually turn down the volume when you start broadcasting that because it can hurt audio equipment, so I hope I don't destroy anything, so I will turn it down first and slightly increase it. So this is not a feedback loop. It's the tinnitus and without the high frequencies, this is how it's so it's really hard to actually hear the high frequencies. Yeah, so this is just an impression of what hearing aids have to work with. Getting hearing aids is mostly, I like to compare to getting glasses. So one day you wake up and everything is really blurry and you decide, well, this should probably not so good.  Getting hearing aids unfortunately is not that easy, so one day you will realize you cannot hear very well anymore.  You have to go through all the difficult hearing situations to test if it works with that tuned hearing aid.  So this whole process, this iteration, it takes weeks, a month until you have something that is actually fitting to your ears and after that you're sort of happy, actually I haven't met a person that has hearing aids that actually compensate for the hearing loss completely, so whenever you are done with that, you're usually just stopping because you don't want to spend any more time on that. And it works well enough. Compared to glasses, this is actually a lot more effort and a lot more frustrating. Another thing is that hearing aids are really expensive. A good hearing aid starts at like 1500 up to 3000 and I only have the numbers for the German health system. The normal German insurance pays 500, so there's a lot of money you have to pay for yourself. Yeah, hearing aid models and their features. There are roughly three types. In ear hearing aids that go completely into the ear canal, a more common one is behind ear hearing aids. They are for mediocre to severe hearing losses. The main part is behind the ear and another special thing are cochlear implants where parts of it are implanted into the head and some is attached from the outside. I will mostly talk about the behind ear hearing aids because that's what I have and yeah, there's a lot of variety on the market. Yeah, hearing aids got pretty invisible. These are pictures of me wearing my hearing aid and not wearing it, so except for this little wire on the right side, you cannot really see it. Most people that don't have hearing aids find this an advantage. People who have a hearing aid actually are not that sure about it because sometimes when you have to ask someone to repeat a sentence, if they know you're wearing hearing aid, they think okay, she didn't get it because yeah, because of the acoustics and if they don't see it, they think yeah, she didn't get it because she was stupid. So it's really not that obvious and sometimes it just helps that people see that you have a handicap. Yeah, they also got pretty small. This is an example of my hearing aids with a 50 cent piece to have an impression of the size and you can see that actually half of it is used by the battery compartment. Out of curiosity, I took my hearing aids apart and of course. You can see they have like shells which you can take off which is like like for mobile phones you can switch the color and choose a different one and the body still, I know this picture is not really good. There's still a lot of plastic around it and the white part there's two microphones, the signal processor and antenna that's used for for hardware, I would come to that and the speaker is actually at the part that goes into the ear and what you can also see here that the part that goes into the ear has also like holds where the natural sound can still come to the ear. So if you still hear like low frequencies then they can you can perceive them naturally and only the hearing aids only adds what you cannot hear. This is called open hearing aid and it has the advantage that you still hear a natural sound which is really nice if you like if you're really sort of an audiophile person that likes to listen to music and so the first thing you do is actually choose that one. Yes, in the last, I think in the last two centuries hearing aids got digital and with that they come with a lot of new features that you couldn't do with analog hearing aids and right now they're standard in most first world countries and yeah, the most important thing is that they can analyze the situation and react to it instantly. Since I mean they have signal processing in it much more sophisticated than analog hearing aids did. Coming back to the to the audiogram, this is actually a feature that can also be done by analog hearing aids. I don't know in what extent actually. So if you have a look at this audiogram and only consider one frequency band like for four kilohertz, here the hearing loss is between 60 and 90 decibels. Over 90 decibels is gets too loud and the input of the environment still has the whole range. So the hearing aid has to map zero to one out of 30 decibels to this small area between 60 and 90. It cannot just amplify everything because that will hurt and this is called compression. This is should not be mistaken with compression in audio files like MP3 or something. It's a little bit different and if you have a look at the software that is used to adjust hearing aids, it looks like that. You, I feel mouse here. This is the this controls here at the amplification and this here reduces the maximum level. So my hearing aids can at maximum do 108 decibel and if you have like minus 12, you just subtract it from that. This has a problem that if you compress it, the volume gets like increased and decreased all the time and this can actually make it harder to understand speech because the hearing aid is adjusting all the time and so to avoid that they don't compress every time. So the first area here is linearly amplified and only after a certain level they start compressing and this is called knee point. So that they usually try to avoid compressing the speech signal and only something above that. And the hearing is adjusting software, it looks like that. This is actually an example only from Siemens. Every brand has their own adjusting software. So or tuning software. So this is just an example. So here you have the knee point in the first row and the factor where what it is compressed and this is actually the third row is factor regarding how fast it should react. So within one syllable, it's adjusted to the right volume. The problem is what do you do when your hearing loss is so bad that the red and the blue curve actually meet each other or the blue curve goes all the way down. And this is actually a problem because then you cannot do compression anymore, at least not in the original sense. So you have this whole, this is a simplified audiogram. This is the area that is that. So instead of compressing in one frequency band, you compress the frequencies. So you reserve a part of the still alive hearing frequencies and map it there. This is called frequency compression. And it works only if you have closed hearing aids, meaning those where you cannot hear natural sound anymore, where the ear canal is really blocked by the hearing aid because otherwise it would be really confusing. And this is actually offered by only one brand right now by a company called Phonak. And yeah, it's actually quite hard to get used to that. The brain has to adjust to that very, very for a very long time. What I heard, I mean, I don't have this. But still, I mean, it's interesting that they try to do it like that. A very common problem with hearing aids is feedback loops, especially if you have open hearing aids, then it can happen that the hearing aid captures its own sound and amplifies it a lot. This is the squeaking what you get when you get too close to them. And this is really annoying. It happens every time something gets close to your ear. So it's just like, it can simply be hair wearing open or you put on a hat or you hold your telephone handle next to your ear or you just want to lie down on the chauffeur.  Yeah, this is really, really annoying.  So they look for clear signal signals. And when they detect one, they send an unhearable flag. So, oh, I detected one. And then the affected frequencies get damped so that until it doesn't squeak anymore. This can, they can adapt in real time. So it actually goes, works really fast, but it's still hearable. So the problem with that is that music contains clear signal signals. And those get damped, which like make your music experience, yeah, a lot worse. And also the damped frequencies can be in the speech banana. And then that means whenever you put on a hat, then you get a feedback loop, then the frequencies get damped and then you cannot understand anyone anymore. Analog hearing aids did not have a measure against that. So this is something that is clearly new with the digital hearing aids. The screenshot here is also taken from the tuning software. When you have tuned your hearing aid, you can make a feedback loop test, where it plays a lot of different sounds and tries to detect the hearing, the feedback loop. And then it reduces the maximum power of the output of your hearing aid.  A very common problem for people with hearing impairment is the cocktail party problem. This is when you're in an acoustic setting where a lot of people are talking and you have some background noise and then someone is talking to you and you have really problems to figure out the version that is talking to you. And there are several factors in that and hearing aids react to that in several ways. So first of all, directional hearing is impaired when you have an hearing impairment. The human ear uses the two ears and the brain to locate the sound. And we use the pinna that is actually the outer part of the ear or muscle in German. If you have behind ear hearing aids, of course, most of the microphones and everything else is behind the ear so you cannot use the pinna. You have to simulate that in a different way. And this is why both hearing aids have two microphones. So you have four microphones in total when you have hearing impairment on both ears. And this way they can detect if the sound comes from front or back. And they talk to each other so they can also detect if the source comes, the source of the signal is right or left to you. Additionally, they try to recognize the situation and automatically focus on the person that is talking to you. And also to reduce the background noise in general. In ear hearing aids I showed before, they of course still can use the features of the pinna. This is also a screenshot from the tuning software. You can actually test your directional hearing in real time so you can wear your hearing aids connected to the software. And then you can do something like that and see if it's recognized correctly. It works more or less in a silent room, but not in a cocktail party. Generally, it's really hard to extract foreground from background noise because foreground noises has all the high frequencies and background noise doesn't. And if you don't hear high frequencies at all, everything is like one blob of sound for you. Hearing aids help with that because they are mostly focused, they mostly focus on high frequencies and they have filters to filter out the background signal. But actually that doesn't really help so much because those situations, recognitions, they tend to fail as well. So sometimes the person that is talking to you gets faded out because it's considered as noise. But sometimes they also work pretty well. I mean, it's also coincidence. Sometimes you're in a setting which has exactly the situation that the hearing aid can work well with. So sometimes you're here with a hearing person actually and you hear him quite well and then you start talking in a normal voice because you can actually understand everything and then the hearing person is asking you to repeat a sentence. This is really weird sometimes. The tuning software for hearing aids also has a real-time monitor where you can see some parameters. So you wear hearing aids and then, for example, listen to music and then you can see here the dark areas are where the hearing aid actually started to work. Below that it doesn't need to amplify and the gray thing here is the speech banana. So you see that it's optimized in a way that it starts amplifying in the speech banana. It also has this situation, recognition in different settings and one of this is music actually. I tried with different types of music. Actually, if you like heavy metal, you will never see music here. It's usually noise. A problem with hearing aids is humility actually. So most hearing aids are not waterproof. That doesn't sound so bad but actually a lot of things are related to that. So that means no swimming with friends, no pool parties, no water sports where you have to talk to someone. Sweat is a problem, especially for people who do a lot of sports. No audio books in the bathtub, no heavy rain. So if you go to an open air concert and it starts raining, you will really have to take care of that.  A very recent development is that FONAC also offers hearing aids that are waterproof or water resistant. They claim that you can hold it under water for 30 minutes and then there will be no irreparable damage. I'm not really sure what that means so I guess you probably have to dry it. Or if you have to give it into repair for weeks until you get it back.  And also audiologists have opening hours for the elderly so if you're hearing it's break on Friday night you have to wait until Monday until you actually can hear again. Yeah, those were the important features of hearing aids but there is some peripheral hardware that I would like to present. So there are different interfaces for which you can use to flag into your hearing aids so the oldest one is the direct audio input which is just a cable. So it looks like that and it's usually connected to the hearing aids with some sort of shoe which looks like that. And yeah it has the person kinds of cables of course you feel like being on the leash but you also have no interference with other wireless stuff. It's usually used to plug something else in for example FM or Bluetooth adapters and yeah I mean there's pretty old technology but it's still around. Although for really small ones the plug is actually too big so they leave it out there. Another common technology is FM systems. You can buy those from several vendors. You have receivers and transmitters. There are different setups like for a meeting that you can put a microphone in the meeting table and hear all the participants talk or for lecture halls like that that you can connect the microphone to the transmitter and listen to it. There are some standards but receivers and transmitters don't work with others from other companies but at least if you plug them into with these or direct audio input you can choose a receiver FM system from a different vendor than from your hearing aids. The sound quality is said to be quite good. I actually could never try it but I heard that actually in schools hearing impaired students listen to music while actually they should listen to the teacher and teachers call the audiologist and ask like could you make this stop. I mean they don't listen to what I'm saying. A really common problem for hearing impairment is calling on the phone. The problem is first of all most people do lip reading so they don't rely only on the audio input. We use the visual as well and of course you don't have that on regular telephone and also for technical reasons the frequency range of the phone line is reduced and in Germany it's like 300 hertz to 3.4 kilohertz. That's the blue box in this audiogram. Also the background noise that you have in the room where you're talking on the phone does not have it does have the full range so you get actually background noise and a better quality than the signal from your person that you're talking to. Then often the signal is altered and unnatural. It's sometimes you have bad reception and also you hear it only in one ear if you just use a usual telephone and if you hold it to your ear you get a feedback loop. So a lot of things make it really annoying to talk on the phone and there are some technical solutions for that. The most old one is the telecoil or g-coil and it's connected and the source is connected to an induction loop and you take off the electromagnetic and this is from the telecoil. It's there on the picture really small antenna and there are different ways, different setups for the induction loop so they are adapters that have the induction loop actually used to hang it around your neck or there are induction loops installed in lecture halls like this. I don't know if there's one here. It's rightly used in Europe especially in Scandinavia they even have laws where like every public building has to have one or at least public lecture halls. You have some pros and cons of course you have interference. When you move inside the induction loop the level of volume changes so it's nothing where you should like dance or something but if you sit still in the theater or something it works. Installing an induction loop in a lecture hall is quite expensive but there are actually DIY kits available. It's quite common that people build their own ones. Telephones also have an induction loop even very new ones so all telephones which are called hearing aid compatible they have an induction loop that can be used with the telecoil even like the new iPhone for example. And then there's Bluetooth. They are right now no hearing aids that can do Bluetooth directly because mostly of the batteries usually hearing aid batteries last like one week to 10 days but with Bluetooth I think they will only last a couple of hours although there are no Bluetooth enabled hearing aids on the market right now. I've heard that they are working on that. Siemens is located in Erlangen which is not that far from Munich so I have heard about people who like test prototypes. Right now if you have to use a Bluetooth adapter to use your hearing aids it has a set and there's a different version of it. So this is an example of Phonak. They built a really nice thing. This piece you hang it around your neck and the ribbon is actually the induction loop that is used with the telecoil and to this gadget you can connect several things. It has direct audio input which you can use to plug in an FM system. It also has aux in that you can like directly plug in your MP3 player and it has Bluetooth. It actually also has a warning to combine this with pacemakers and I have a friend who has a pacemaker and this thing. He just ignored the warning but I think this is luckily he's still alive. I think this is a good example for what we are heading in the future. I mean we will get more and more cyborgs and I doubt that every hearing aid vendor is trying their adapters with every pacemaker there. So we will get a lot of compatibility problems in the future and I mean if this really works it might be an option to kill people really silently remotely. I mean you have to think of that. Yeah, the solution was of course not to use any of the existing standards. Just build something new. So they built something called Siemens Tech and it talks Bluetooth to the phone or to whatever you connected to and some wireless near field protocol to the hearing aids. So the hearing aids also have a small antenna in it which looks kind of like a telecoil but it is not and it is not compatible to anything the telecoil is compatible to. So you can only use it with this. It has a signal around 3.3 megahertz. You can see it on the picture. I tried that. And it's compatible with every Bluetooth speaking device in theory and practically you have to check it with everything. So whenever I get a phone at work or whatever I have to check if it actually work. So of course it works best with Siemens mobile phones but you can imagine how old they are. Yeah, they also are supposed to work with landline phones but on the websites they say only two, I think it was only two Siemens mobile landline phones of course and other than that they don't guarantee that it works. It comes with an additional transmitter that is connected that you can connect to a source that is far more far away. The tech itself has a range of like one meter. So if you want to have an inflight screen TV like I don't know five meters away from you, you use the transmitter. This thing costs about 400 euros for just turning one wireless protocol into the other and no insurance is going to pay for that. So you have to pay for this on your own. If you have like a generous employer you might get some for it but yeah. They also released a new version of it. This is on the right side here. It's called Minitech and it has actually less features than the old one because they reduced the they removed the display and they still want 400 euros for that and you don't get a discount if you bought the old one. Well, but guess that's marketing. This is the set up with a transmitter so you connected to the computer and you wear the tech around your head. It also has some buttons for the different programs. So here is a different programs that you can change manually so that you have one for like listening to music or one for your living room and one for outside or whatever. Of course I took that apart as well. You have to couple this tech with the tuning software of hearing aids. So there is some kind of authentication via a 7k repair serial number. I doubt that there's actually a lot of encryption in there because the latency is crucial. Bluetooth already has a latency and you don't want to add that much to it. But I also try to use a different tech with my hearing aids and it actually doesn't work. So some kind of authentication must be in there.  There are like lots of talks about that here. And of course the pin is 000. And something that is not directly about the tech but the hearing aids also communicate with each other.  If I switch between the programs, on one ear it also tests that to the other ear. And that one doesn't use authentication. I have heard that when people have the same model of hearing aids and are close to each other, for example, as happens in couples by the same hearing aids, then you switch your program and your spouse also gets the program switched. And this can actually only be changed by the audiologist by changing the channel. So they have like, like in superviolets, different channels and you set it to a different one. That's the security about that. Yeah. Hacking. When I started to dig into that topic, I was really disappointed that there is not very much hacking. So there's one forum called hearingaidhacks.livejournal.com. This is the biggest one I found. And better if you go through the entries, it's mostly people asking for technical advice. So I bought this and this hearing aid watch peripheral hardware. Can I use with it?   So but there is a little hacking on the peripheral hardware. I will show some two examples for that. There's one guy called Gertlex who built his own Bluetooth adapter. He posted this on Flickr, quite detailed. And what you can see here, he used a Sony wireless Bluetooth headset and hacked it in a way that you can connect the direct audio input cables from the hearing aids. The picture here in the upper right corner shows the setup when he tested it. So he actually didn't test it with his original hearing aids. He used an old one that he had and he even used an old MP3 player because he was afraid of frying that as well. So this is sort of the precaution that you have to do when you start like frying your hearing aids.  And there's another guy who also made a Bluetooth adapter and he also took the DIY cables here, those shoes that you used to connect them and the Bluetooth mono thing here. And this is actually the result that you connected directly to the hearing aids. He also provided some nice diagrams for that. The slightly bigger scene is actually the self-tuning scenes for hearing aid. Because as I said, it's kind of frustrating to get hearing aids. You have to go to the audiologist a lot of times and he asks you, yeah, what's wrong and then you have to describe the situation. But you're sitting in this silent cabin at the audiologist and so the adjustment is not really done in realistic circumstances. And a lot of people get frustrated about that. So they spent like weeks tuning the hearing aids at the audiologist and still they're not happy with it. So they try to get the hardware and software that is necessary for it. And those are actually only sold to doctors and acousticians or audiologists. And they're not sold on eBay because it's medical equipment and this is not supposed to be sold on eBay. So you have to use other channels. And there's like a black market for it. It's kind of hard to put a price on that. But I've seen offers for the hardware which is called PyPro and there's different versions for serial USB or Bluetooth. And it starts with a couple of hundred euros. So you can imagine the pain that people have when they already spent like 5,000 euros for two hearing aids and then they spent even more money because they want to tune themselves. And there is this self-tune scene and people really hack the system.  So they register a business for an audiologist to buy this hardware and then stop the business again. So there are actually people who are doing a lot of effort to get this hardware and software.  There's one exception in America. There's a hearing aid manufacturer called America Hears. They sell quite low-budget hearing aids up to $1,000 if I'm right. And you send an audiogram of yours and they tune it at their place for the first time and then you can download the software and tune it a little bit more at home. Unfortunately, I've never seen that software and you can only order it in the US, but I would be interested to have a look at that as well if someone has channels for that. Yeah, of course some of these hardware ended up in my hands. This is a serial hypro. So you have this looks like really fancy like a modem from the 80s. It's connected via serial and you connect the hearing aids to it. I have some closer pictures for that. So you take off all the battery and put on a small cable which has a contact of the battery, size of the battery, very flat cable and this is connected to a bigger cable and that is connected to the hypro. The hypro is the same for nearly all hearing aid brands so you can use it for Siemens and Funak and whatever, but these small flat cables, they are different for nearly every hearing aid. So if you try to buy this hypro on the black market, you also have to buy those cables. There's also a Bluetooth version that also ended up in my hands. It looks like that and has the advantage that you're not connected to it by a cable so you don't feel like on a leash and I haven't really used that much, but this way you could actually go outside and tune it like in the subway or at your offers because you just need a laptop and that and it works without your power for a while. Yeah, the tuning software, I showed you some excerpts from it, there's just another screenshot. On the right side here you see the different programs, so the universal one is one for music and one for the tech, if you work on that. When I was playing with the tuning software, I found something very interesting that is actually spying on me, so it locks some stuff and, for example, how much I wear it, although I find 40 hours a day a little bit, I think I'm awake actually more longer than 40 hours, but they use it actually because people come and complain, yeah, this doesn't really work much and then they see they only wear it like half an hour a day and so, of course, you cannot get used to it and adapt to it. You can also see what different programs I use, so mostly I use the universal program and sometimes I have another one called universal, it's actually tuned to have less feedback loops, that's the one I use when I put on a hat, so that I still hear a little bit, but don't have too many feedback loops. It also tracks how often I was in a noisy environment or was listening to music and since I like to listen to heavy metal, this is actually not correct. Yeah, I found it really interesting what you can see here and, yeah, I hope that it doesn't record anything, what I talk about or what I listen to. We have a little bit more time, so I will talk about cochlear implants as well. I mentioned that those are the ones that I have implemented in the hat and also have an external device. This part is implanted and they have a wire that is drawn into the cochlear and it's connected to the nerve, so actually the whole ear is circumvented, only the wire goes directly to the nerve and then through the brain and cochlear implants are what I find really fascinating, I mean, they really make deaf people hear. It's only applied to people who have a really severe hearing loss, like less than 20% or something, so they hear only less than 20%. Of course, it's a surgery to insert that.  It can also affect other nerves, so I have a friend who had this surgery and they touched some taste nerves as well, so everything tastes as metallic. It was kind of weird, but it actually went away after a while. The signal is really different. The brain has to adjust to that very long. They're actually hearing courses after you get the surgery and the device started.  And like for the other hearing aids, there are not many standards, no interoperability between the brands, so if you decide to take a cochlear implant of one brand, you can never switch to another one. So, yeah, you have to think that through. I have an example of how it sounds with a cochlear implant. So it sounds kind of spooky. It starts with a normal sample that like everyone can hear it and then they have like different channels and reduce the number of channels and then it gets less and less hearable. A boy fell from the window. A boy fell from the window. A boy fell from the window. A boy fell from the window. A boy fell from the window. Yeah, so you can imagine that it takes a while until you actually can understand speech with that. Yeah, I'm coming to an end. So there are a lot of things that I want from the industry. First of all, better service and that goes into the consideration of young people's needs because you can feel that every time that it's everything is designed for the elderly, they don't consider that people actually have to work and have a life and actually want to go out and talk to people and not only like in a silent room one on one. And there are a lot of things where you miss the support when you have hearing aids and are not 60 or something. Generally, I'd like to have better signal processing. Of course, regarding the size, they already did a really good job if you consider what they do already. But actually the cocktail party problem is not solved. So a lot of people who have hearing aids and who also have really good hearing aids, they just avoid social situations. So they don't go out. They don't go to Congress. They don't go to parties. So whenever you would ask them to go for dinner, they'd really carefully choose the restaurant if it's like a more less crowded one so you can understand people actually. What I'm really missing is some standards. I mean, it would be even cool if it was open standards because this way you feel like really trapped as a patient. I mean, there's this saying like if you don't if you can't open it, you don't own this. And I really miss that when having hearing aids. So I have a lot of ideas how to improve that. But I don't see most of it coming in the next 500 years because of the companies are not very open source friendly. So what would be really cool to have something like a hearing aid app market that you can download the newest feature for like background noise removal or something and that you can write your own filters and share those and especially like exchange those between different brands. And the funny thing is when I was browsing through the websites of the vendors, the marketing of some hearing aid companies actually got this idea already. They just call their features apps. So I mean, it's just an enumeration of what the hearing aid can do, but they just call it app. And I mean, it would be really cool if there was something like that you can exchange those. And what I also like, I mean, hearing aids don't not use like reoccurring situations. I mean, most people have a rather steady lifestyle. You live in the same apartment. Most of the time you work in the same office. You take the same subway to work every day. And hearing aids are only tuned for like a general situation. And but I think signal processing works well. The more you have, the more you know about this rounding. So it would be really cool if you have like one program for the office and one program for at home and one for the journey to the office. And I mean, we carry something around that knows all this. I mean, we have smartphones and they have a calendar and it shows where you are and it knows the people that you're talking to. And it's even like if you talk to them on the phone. So there could be like parameters for each person that you talk to and that it could be saved on the smartphone if there wasn't enough space in the memory in the in the hearing aids themselves. So all the information is actually there. But I don't see any of the hearing adventurers adapting to that. So and what what I also think that there are people like building 3D models of houses and you could take this information into consideration as well. So if I have never been to the BCC, but someone has made a 3D model, you could also like calculate the characteristics of the acoustics here. And then you can before you go to the Congress, you could download the acoustic settings for your hearing aids. That would be really cool. But well, this is just ideas. Yeah, and also regarding the the hardware would be nice that there were some open standards. I mean, it would be cool if you could like print your own hearing aids with a 3D printer, at least the part that goes into your ear so that it fits really well. And there there are a lot of possibilities, but the market is really, really slow and they still like try to figure out how do we connect Bluetooth. And yeah, I think for me, it's just way too slow. I'd like to see like more progress and that. Yes, actually, with that, I would like to conclude. And I have to thank some people who helped me with this with this work with the talk itself and all the stuff I talked about. And yeah, I think we have some minutes for questions. Yeah, and that's wonderful. So before we come to the questions, I need to say three things, which means please stay seated while the question and answers are going on.  And then please leave through the front door here, while the last door over there is the entrance for the new people. So now we can go over to the question and answers. We have a signal angel again in the IRC sitting and watching Twitter. And we have an audio angel in the back running around with a microphone. So please questions now. Hi, I have a question about the cochlear implantat. I'm a neuropsychologist and I know that with eyes there's an approach to kind of re-engineer the signal processing of the retina. And with this kind of knowledge, you can make better retina implantats to enhance the possibilities of seeing with this kind of aid. And is there a similar approach for hearing aids and the cochlear implantat? I mean, there are different types of cochlear implants. And I mean, this is really not much of my field of expertise. I know there are like those who still use the membrane in the ear or some directly the ones that I showed that directly insert the wire into the nerve. But I am not that familiar with the field, so I don't know what's coming there. OK, there's a question from the front row. I wonder, is there any connection of research in companies like Siemens or whoever builds these parts? Is that in any way connected with other consumer good research? Like I'm not hearing disabled, but if somebody would come in come up with a decent set of in ear headphones that would fit and would be able to do some noise cancellation or so. I'd be quite interested in spending money on that. So is but this seems to be totally uncorrelated altogether. Yeah, that's true. I mean, it's pretty close to in your ear, their cat phones. But it's from what I've seen, a different market. I mean, you still can buy your new hearing aids if you don't really use them. I mean, no one can prevent you from that. But I don't see any trend in like merging that very much. It's no more questions. Hello, I would like to add two things to your wish list. One thing I would like the hacker community to find out what the real differences in devices from different price ranges are. What is done in hardware and what is actually only firmware. My audiologist reported devices coming back from repairs of hate programming to report as more expensive devices, for instance, or it's just a firmware thing. And the other thing is I see an interesting hacking or attack angle of thing for hacking in the protocol that the devices use to communicate with each other with the devices which transfer Bluetooth into the body area network that really speaks to devices. So do you know any off the shelf components that can speak to body area network protocols? For the Siemens thing I showed, I haven't found anything. So this is just that gadget that I have. But I think people are experimenting with the audio induction loops and the FM systems. But yeah, it's not that much that you can see documentation of that. So I'd like to see more here. I mean, I made this talk because I would tell you hackers what's on the menu and it would be really nice if there was more activity in the scene. And it's kind of hard to start that if you're all alone. So I hope I've risen some interest. And if you have any pointers for me or anything I didn't mention here, so I'm also happy if you send me an email or I will be around in the next couple of days. Before we finish, there are some questions from the internet. The first one is Gilligan who asked, does the fact that you have tinny toes makes a process of tuning a hearing aid for your harder? Does a hearing aid have a negative or positive effect on your tinny toes when you wear it? Of my what? Sorry, a positive or negative? Does a hearing aid have a negative or a positive effect on your tinny toes when you wear it? Actually, yeah, there are hearing aids that claim to counter-fight the tinny toes that they like regenerate the counter frequencies against that. I also asked my acousticians or my audiologists about that. They actually don't work. I mean, they usually are not really offered. And even if you ask for it, they tell you, yeah, usually they don't really make a difference. My choice was actually just to make a normal hearing aid. And because the frequencies that you don't hear are amplified, the tinny toes in relation gets, yeah, less loud and you learn to ignore it. But that's, yeah, that's not really magic, but that's the only thing that seems to work. Okay, the next one is from Speckmarde. Are there hearing aids with documented RP for the tuning official dog reverse engineer? Can you get the official tuning software to tune yourself? Did you write to any people who factor them? I haven't found anything about that. That the only thing you can do is get the software on the black market and disassemble it, if possible, but there are no APIs or something. That's really, yeah, what I'm missing as well. Okay, the last one is from Lucy, I guess. What frequency range can you usually hear using a cochlear implant? How fine can you reserve frequencies? There are about the range that you have in the speech banana, but actually, I'm not that familiar with cochlear implants, so I cannot give the details here. Okay, hi, you showed us the programming tool for changing the settings on the hearing aids. I forgot the name, but the 1980s modem. Have there been any attempts to clone that hardware or reverse engineer the spec? I'm not sure if I understood correctly. You mean like how you get the hardware for this? Yeah, so at the moment you have to buy the programming tool as I understand to update the software. Since it's only delivered and sold by two audiologists and doctors, you have to be friends with audiologists or doctors and somehow they can give you that, but there's no official channel. So you have to find the black market. Yeah, sorry, my question was, has anyone tried to create an open hardware variant? Has anyone copied? No, I haven't found anything like that either. So that was actually also on my wish list that makes the tuning hardware open source as well. That would be really nice. But I mean, it's a serial device, so you could actually do some sniffing there, but I haven't seen any approaches there. Yeah, because it looks pretty easy, I would say. So we are running out of time. Is there any really, really important question left? You mean, because she's around, so you can meet her in the next days on the conference again, on the congress again and ask her more question. And thank you very much for the talk. It was very interesting.


Thanks, hello everyone, I'm from review 4, Han Sheng the Hansen, if you want to call me Hansen, I don't mind. So tonight I'm going to share with you how we are using deep learning to change the advertising industry and some of the issues and solutions that we have encountered and that we have found when using deep learning to solve our problems. And these issues you may have when you are using deep learning or other machine learning techniques as well. So I hope you can... Can you speak louder? Oh, like this? Yeah. Oh, hello. Oh cool. Yeah. And yep. So first of all about myself and the team. I'm the head of the data, but anyway. And my background is actually from electronics engineering and computational intelligence. But unfortunately unlike other speakers, I'm not from Google and I'm not from Facebook as well. I'm just from review 4. And our team we have an analyst with my left arm. And another infrastructure guy who is creating a scalable infrastructure. And my heart has a research scientist based in London. But my two legs are missing. So actually we are hiring. So we are looking to double our head count by the end of this year. And next year we will double our head count again. So please submit your application. After today's talk if you find like what we're doing is cool and you are interested in joining us, please submit. So to start the talk, since we are an attack company, right? So I got a...  Sorry, I need to connect to internet. Am I going to do that? Yeah. Okay so the password is umballfunan. So I'm going to attach that over there. Which one? Funan SG? Oh yeah. Yeah. I wonder use. Probably like their Facebook page or something. Because I'm using Windows. So I don't want to connect to internet. So if you have the passwords, don't pass it to me. Digital ads are great. But eventually their performance starts to deteriorate. And they need to be redesigned or refreshed. So time consuming. But with Refill 4, your ads are automatically designed and revised to keep campaign performance high. To get started. Let's go down as well. Okay. All right. Let's see. Started. Just connected ads are great. But eventually their performance starts to deteriorate. And they need to be redesigned or refreshed. So time consuming. But with Refill 4, your ads are automatically designed and revised to keep campaign performance high. To get started. Just connect your ad account to our platform. And create your brief. Which is sent to over 10,000 designers in more than 100 countries. You'll receive your creatives in 48 to 72 hours. And you can set them live. Request revisions. Yeah.  It's crazy. Yeah. I tell y'all to drag the coach in. Yeah. So I get presented. Okay. I have another one. It's here. Enjoy this one. Yeah. All right. Yeah. Yeah. But eventually their performance starts to deteriorate. And they need to be redesigned or redesigned. Your ads are automatically designed and revised to keep campaign performance high. To get started. Just connect your ad account to our platform. To create your brief. To send to over 10,000 designers. Ah. Okay. I think. So you guys, we are not doing good enough. So that's why we need people. It's okay. Just never mind. Breaking them in the order of printing and performance. Your ad is selected. Your ads. Yeah. All right. So I guess you guys are still not knowing what we are doing. So I'll explain. Yes. Okay. Basically we are quite a mature star. Actually we are profitable already. And for a star, this is actually quite like difficult to be profitable within like three years. And we have offices in London, Tokyo, Singapore. And San Francisco, Austin. But we are headquartered in Singapore. And currently we are like one of the major marketing partner with Facebook and Instagram and Snapchat. We also won the last year's Facebook innovation award spotlight. But this year we are submitting to the Kent Slion. So we haven't received a result yet. So in advertising industry today. Now they are like the global media inventory is like 20 billion. So estimated 50% of the media cost should be spent on your creative production. You should have 3 billion globally in creative production. However, nowadays there's only like 2.5 billion. We are having more demand than supply. That's why our business are booming. And this problem actually affecting the entire digital industry as well. Because more and more advertisers are coming to digital advertising instead of doing the traditional TV advertising. Or even newspaper. But somehow they are suffering several issues. For example, the performance. You create a creative, right? You want to launch an air. So you got to have your creative. Like be an image or a video. But you don't know its performance. So the only way you can do is to have more and more creative and try to test each of them. And the other thing is the relevancy. In digital advertising, those advertising platforms such as Google and Facebook. They don't show like irrelevant ads to users. They only want to show relevant ads to their users. Hence you can have a lower bidding cost. So if your creative is not really relevant. If the algorithm decides that your creative is not relevant to your target audience, right? They are giving you lower priority.  They will not show your ads to the target audience. Unless you are willing to bid higher. So this is the effect of relevancy. And the velocity with the more and more advertisers coming to this industry, right? Now people are pretty quite aware that like a targeted advertising is the key to the next generation of advertising. But more and more people are targeting the same audience or different audience, right? So the result is that you will have to target more detail and detail the audience. Otherwise if you have a very wide target audience, your relevancy will not be high. Hence you will not be able to spend your money and show your ads to your clients or to your target users. And what we are doing is actually that with us you don't need to imagine what kind of creative will perform. We have a fleet of designers that can help you create any sort of, given our domain knowledge and our understanding in different industries. We are actually creating creative in a data-driven manner. And we also have the machine learning algorithm to help you to predict the performance of your ad before you put your ads online. So that you can actually save your money and save your time if you work with us. And the way our process is like this, you want to create, you want to have some creative for your campaign. So this is called creative production. And you submit your brief to our platform. And we have our designers globally to create creative for you in a matter of few hours. Then you can select the creative that you want and put it online. After that if your system detects that these ads are not performing well, we will just swap it for you. And our designers will submit more and more creatives. So this direction keeps going on and making sure that you are refreshing your creative faster enough.  Once, let's say if a user sees your ad, you are a user, right? You see an ad, then you don't respond.  The next day, you will never respond. That's called exhausting. So like what I say, the performance relevancy and the refresh frequency are now the three key elements in digital advertising. And what we have for tackling the problem is this engine called Shaka. I think he's a guy. And we use this machine learning model, given the data that you have. Be it like social media data or your targeting data or your creative. And we feed it to our Shaka engine. Then we tell you whether your ad will perform or not. If not, we'll ask our designer to submit a new creative again.  And coming up, we will have sort of auto targeting for your audience expansion.  You know some like a future, some profile that you should target, but you don't know which one is the optimal one. We will be able to tell you that. And you don't have a creative. You don't know what kind of creative that you should produce, so as to achieve maximum performance. We will have a tool to tell you that, even before you produce your creative. Let's say we all know that in the, I think the last two, three years, video ads are getting more and more popular. Before, people were mostly using image creatives only. And later we have like a carousel, which is a set of static images. But now we are seeing more and more video ads on Facebook and Google YouTube, because people realize that video ads are actually a lot better than static images. But however, producing a video, the cost of producing videos is a lot higher than producing images. And once you, if you don't do it right, you have to do it again. It takes time. And I'll show you some of our clients. We are able to increase our scale and reduce the cost of our client by like 70%. In industries such as e-commerce, online gaming, travel and entertainment, such as casino. Paypal, they are one of our clients. High profile one. And we also have Spotify, Spotify, Tiger, and Cooking Dash, Tiger Air. These companies, they have their own in-house designers. So actually, but the reason why they want to use us, because our creative is better than theirs. And if we don't charge them by the amount of creative that we provide them, instead we charge them by the performance of our creative. So if our creative don't perform, they don't have to pay us. So this is our like a value proposition. Okay, next I'm going to talk about the deep learning part. Any questions so far? No.  Okay, right. So I mentioned the Shakarai. Actually it's not really a magic, but it's a multi-class, multi-target, air performance ranking classifier. Same like you can come up with a classifier with many other techniques such as SVM or simple regression, the sort of thing. But the reason why we want to use deep learning, because we are trying to predict the human's behavior. Unlike predicting the, unlike a classifying image, you have an object in your image, then if it's a chair, it's a chair. You don't have doubt. But we are human, we see a creative. How do we decide we want to click on this creative or not? We don't know. That's why if we use a simple method, simple machine learning method, I don't think it will work well, given the capacity of other algorithms. So these are our input features. We have properties from advertisers, be it their industry kind of information. And they sell out a campaign on Facebook, so we have the properties of their campaign. Maybe they are, for example, if it's a cooking dash, they have mobile apps, so they want to launch a mobile app instead of a campaign. All other companies might be retailers. They want to have a conversion, so they will launch a conversion campaign. And targeting is the most important part. They always have a target, i.e. teenagers, so these are their asset properties. And the other key element is the creative itself. Either they are maybe the image or video, and we have metadata from the creative as well. So all together, we have 48, like a fixed lens or variable lens features. And give it to our chakra engine. So what is our output? We actually output like seven target values for them. These target values are actually the metrics that those advertisers concern. Maybe the click-through rate, or conversion rate, or light rate, or video viewing rate, or video play rate. If you run a conversion campaign, you will be looking at conversion rate. If you run a brand awareness campaign, then you might just want to look at click-through rate only. So these are some per-sample input features. These are actually from Facebook targeting. Everyone uses Facebook, right? So we see Facebook ads every day. I don't want to see them, but they still show me that. And actually, we are all being targeted based on the websites that you have visited or the keywords, the things that you gave Facebook.  And they also know your interests, like vocation or others. But some of these features are input by users. So you don't have a fixed set of possible inputs. For example, the AAH goes to Singapore here. This one looks a bit weird, but it's there, defined by lots of users. Others might define vocation, but sometimes they can be typos. For some vocation, vocation, or Star Wars with space or Star Wars without space. They mean the same thing, but the terms are quite ambiguous. These are country codes. And colors are the data that we have from our creative metadata. These are like a sequence of colors.  So we analyze the video and general metadata are telling us the salient colors existing in these friends, key friends in the video. So basically these features, they are largely recurrent. By recurrent, I mean, if you look at the, for some family status, you have a sequence of words here. Parents back a war or parents with early school age children. So this is actually a sentence, so a sentence sequence of words. And furthermore, if I have, for example, the color here, one image contains several colors. Then I have a video, so I have a sequence of sequence of sequence. So they are sort of in nested recurrent structure.  Three of them, one is the CTR. This CTR is the click through rate and the CVR is the conversion rate and video viewing rate, video viewing rate is for video campaign. So, and why are there like several courses here? It's because for a certain kind of campaign, you only have few metrics available. For example, if you are running a brand awareness campaign, then you don't have conversion, so you don't have conversion rate at all. And if you are running a video campaign, so you have a video viewing rate, but if you are running a campaign, you add with a static image creative, then you don't have video viewing rate. So this is how our data look like. And so now here's this thing. My data actually, my features contains the industry or the campaign type. So all the samples are from different industries or four different kinds of campaigns. And there are also three kinds of creative. So if they are actually lots of distribution, the combination is over 100. So traditionally you should separate the distributions by creating independent models for them. But I have over 100 distributions. I don't want to create 100 models. And from marketer's experience, they know that if I launch a campaign with a video, I know it's most likely 90% of chance, it's going to be a lot better than having a campaign with just image creatives. Why? Because people are more responsive to video. So my model, if I create a model, then my model for sure tell me that, okay, now you have a video creative, right? Then my model will tell me, okay, this one, you should get higher because it's a video creative. So you should use a video creative. But this makes sense, but is it useful? Because a lot of time, for example, you have a video creative, but you just cannot... Oh, I'm sorry. If you have only image creative, right? The model tells you one company tells you, okay, you should use video creative. But you just can't because you don't have time and money for producing a video. Or you are in the, let's say, game-boarding industry, right? They tell you if you were in the, like, travel industry, the purpose of your app will be a lot higher. But you can't, you can't switch your industry, right? But because of the way you mix the distribution, right, your model will only tell you if the campaign is from the travel industry, it's not going to have higher purpose. Whatever it's not from the industry, you have lower performance, right? So what we did is we still had, we still tried to maintain one dataset, okay? But for each distribution, right, we transformed those numerical targets, i.e. our CTR, CVR values, right? The shape of them is looked like this, largely skewed, always like that. But we wanted to be more of a normalizer, Gaussian distribution, right? So we transformed those targets with SparkTux. So after transformation, the distribution of... One distribution only, sorry. Let's say conversion here, I'm using maybe conversion campaign plus, like, gaming industry and video creative as an example, right? So this is one distribution. For each distribution, we do this so that we have, like, a proper Gaussian distribution in target. Then the question is how do we decide that what kind of... what level of CTR is good, or what level of CVR is good, right? You can look at all the values of the CVR of the CTR and decide, okay, you're going to have a threshold. Anything above this one, you're going to label it good. Anything below this, you're going to label it bad, right? But we don't know. And I can't do it for, like, a hundred distributions, seriously, yeah. So we further apply the, so-called the inverse of the Gaussian CDF to figure out all those separation points, the threshold here. The yellow lines here showing four thresholds. So I have actually five classes, right? So for each of them, I converted those numerical values into scores, five, with the five levels, okay? Then I put them in one model, in one dataset. So the dataset becomes this, okay? I have now see categorical values, but for, like, invalid, for invalid targets, I level it as negative one. So this is my, this is my, like, training data, okay? And here you may have a question. So if you have created, for example, a softmax classifier, right, you need to specify a non-negative class, but I am having negative class here, right? On the other hand, right, sample one doesn't have CVR target, but they are, normally people do batch processing, then they are always, if they are included in the batch, right? How can I predict it's CVR? Does it make sense or should I need it? Do I need it? Right, okay. So this is the model itself, right? Or simplified. Let me try it soon. Oh, great. Okay. I'll start from this part. Okay. Basically, this is a, what's your code? An R, there's an RN here. Record new network, right? So it takes a sequence of words as input. Then I stack two layers of it. So I have two state factors, right? And each of them feedback to its corresponding input state, right? Then the state factor of the last step, right? I use it as an input for next level, right? Which is this one. Again, this is RN as well. But here we are using words as input. And actually there's a word embedding in between, converting them into the factors. And here is the output vector already. So the input for these are actually a sequence of factors. And after that, we attach two dense layers, right? Okay. And similar for this one, right? So remember we have 40 something, 40 features, right? So actually we have seven sets of RN dense structure like this, right? And each of them taking different sequence of input, right? And the right-hand side, this is similar. Except that we have one more layer. And here we have dense layer in between, right? The reason why we want to have dense layer is because having dense layer in this way, right? You are creating like a hierarchy of dense layers instead of just having very large dense layers here. So your capacity of your model actually performs better in larger capacity. Although you can argue that, okay, you might want to have a very large dense layer here. But this way you have here, so like a sub-window of your input feature. Like a convolution neural networks. Why convolution works? Because you only look at the local features. Right. Oh, yeah. Sorry. So actually here you can, the input here is a sequence of factor. So this actually becomes like an embedding factor. So usually people do what embedding? Or they do sentence embedding or document embedding. But here I have an embedding factor for my input sequence of factors. Right. Right. And same. And same. Sentence. Sentence embedding. So sequence of sending embedding factor. So it becomes a article. Right. Okay. Right. So together the model has, if you, the model has over a thousand TensorFlow variables. If you have used the TensorFlow, then you will know that each TensorFlow variable actually is of a multi-dimension. So the total number of three parameters is like 34 million. Right. Which is not very big. Comparing to those AlexNet or Inception model, they have more than billions of parameters. Right. But this one so far is big enough for us already. Okay. Next we come to training. Right. Right. So because of limited capacity, so we got to have a strategy. I realized that I don't want to train the whole entire model at once. Why? Because it's going to take a lot longer and takes a lot of time. Right. If I can do the training layer-wise or module by module. Someday, if I want to update a sub-module, I don't have to retrain the whole thing. I need to retrain a few sub-modules and then train the entire module again. Right. So turns out this method is called greedy layer-wise training. People have been using, for example, auto encoder to achieve greedy layer-wise training as well. Right. But here what I did is that I trained this R1 first. I tried to come up with an embedding factor. Then I removed the decoder. Then attached it to the second layer. And then I trained this module again. Then I attached another two layers. Then I trained this module again. Then this one. Then finally joined training. Right. If without this layer-wise or module-wise retraining, right, my number of epochs that I can converge my model will be around like 50% more. Right. I tried it because initially I thought I didn't need to do a pre-training. And the other thing is people, you guys all know RELU, right? Rectified linear unit. So basically it's a formula called max zero energy input. Okay. This is pretty common and proven that are very effective in deep learning. Before that, people were only using sit-moil or logistic functions. And they realized that there's this called vanishing gradient issues. So RELU, rectified linear unit, actually tackled this problem very good. But somehow you would solve a problem that is upon your initialization, right? You might see some big units. Right. Anyone know this issue? Okay. Because every time I ask in my interviews, he seems no one knows. Okay. So that unit is actually that if you pick all of your training samples to this unit, right? This unit never generates positive output. I is always below the negative range. So no matter how you train it, it doesn't generate output. So your gradient never flow back. So this unit actually use less. And during the training, during the training, if you attach it, let's say... For some here, I always... If I had RELU here, right? Then because the activation function in this RELU layer actually hyperbolic tangent. So the output range of hyperbolic tangent from negative one to positive one. So if you have too many negative one, right? Then most likely the RELU unit here will be there. Because once the input is negative, then your output is zero. Or your connecting weights are too much negative. Okay. Right. So my experience that after training, you may still have at least 20% of dead units. Of course, you can argue that or I just increase the number of units that I have. Yeah, that's correct. But you will have more dead units as well. And that's the waste of your commutation capacity. Right. Right. And we train the model on the DeoTitanX machine. Right. The joint training part took like 8 hours for 45 epochs. Actually, I set it to run for 60 or 80 epochs. But because I have validation data. So if the accuracy doesn't improve for a certain period of time, I stop training. So I have to not waste my time. Right. Okay. Make sense? I hope so. Okay. Then I'm going to talk about how I came up with the embedding factor for those iron units. Iron layers. Right. So. Okay. So I think some of you may know the skip thought model. Right. It's actually from SkidGridModel. SkidGridModel says that if you have a word right next to another word. Then these two will tend to predict each other. Right. Let's say like I love you. Right. But actually I don't love you. But I love you. So love actually predicts I and you better. So if you see a love, right. Then you have a higher chance of seeing you after that or I before that. So this is the idea. Then SkidGridModel took this idea and expanded it to sentences. But of course they create, they use like a recurrent structure for it. Otherwise they would have been able to do it. Right. So we implement this model in this model intensive flow. But furthermore we expanded it to our so-called article embedding or document embedding. Right. Skip thought model only does sentence embedding. But we have a sequence of sequence of sequence of words. So we have article. And we need to do embedding at article level. So we do this way in this. We have an encoder here. Right. Which takes another encoder as input. And they are like several steps of it. And there's a decoder taking the output factor. The output stack factor of your encoder. Right. And try to predict your input here. Okay. Right. So this way we are able to verify and come up with sensible embedding factors. I'm showing you some examples here. I think let's look at the download shop. Okay. This is a sequence of, a sequence of words. Okay. Right. So this is one of our inputs. And the minimum input like granularity is what? We don't do it at character level. Okay. So turns out once we done the, after we done the training of the embedding factor, we found that by checking the cosine similarity, you can actually see which input is the closest one to your, to the factor that you want to see. So we realize that the another embedding factor representing Donald Trump is the closest one to here. And save for this one. But how about this one? Queen Ben and the Feddy Immigrant. I don't know who they are, but later I Google it. And I realized that Queen, Queen and this guy, they actually had some dispute with Donald Trump. So turns out obviously there were some advertisers targeting Donald Trump and they were also targeting Queen and the other guy. Right. Maybe they try to convert that, convince them that, okay, we are showing you Donald Trump's air and you should support him. Right. And the other one, the Fed 365, right. It has very, the distance, the distance between Fed 365 and the Ghana Football Association is very close within top three. I didn't know why as well. Then I asked my British colleague, right. Because they, I think he's more familiar with football than me. So he told me Fed 365 actually is a sponsor to some of the football teams there. Although they are gambling, gambling, online gambling website. Right. But they do sponsor football events a lot. And some other teams are as well. So if you are doing like online gambling and you are targeting Fed 365, you might as well want to target football association. Because people who like football association are most likely football players or football lovers. They are Fed 365 clients. Right. So this is how they do targeting and this is how we verify that. After we train the embedding, once we have done the embedding factory training, our embedding factory actually works. Right. Okay. Right. And there are some other issues as well. Right. For some of the memory issue. Okay. Although my Titanic's machine has 12 gigabytes of memory. Right. But it's still ran out of memory. Yeah. Because I have some videos now. So the number of key friends in those videos can be over 30 friends. Right. Which is quite a lot. So I couldn't handle all of them. Then my machine just told me out of memory. Right. So I'm showing you some here. Right. These are again a sequence of sequence of words. Right. Okay. But you can relate to video. Video is a sequence of friends of some metadata. Right. And the length of sequence can be as high as 680. Right. So, but I don't want to truncate them. Because truncating them, I'm not taking the rest of the sequence into account. Right. So I don't want to truncate. And one way to handle it is to have a fixed lens of. It's for your eyes to have a fixed lens. But once you have complete this iteration. Then you take the output factor. And you feed it as an initial factor into the same iron again. Right. But this is time consuming. And I need to do something for you. I didn't want to do.  I'm sorry I said that. So instead of truncating, right. I did reduce the lens of sequence. But I didn't truncate them. Instead, I randomly sampled some elements in the sequence. But I still maintain the order. So example here, I have for some of the second elements in the sequence called call of duty. Right. In one batch, right. This call of duty may be taken out. Right. But in the next epoch, it may be there. So it's sort of random. Then after enough epochs, right. I at least I still know that each elements in the sequence still got trained. Right. And turns out this method works well as well. And so this is how I tackle the long lens sequence. Okay. And coming to the question that I asked before. My dataset contains like a weird target class. I did negative one. Right. I didn't want to include them in my loss function. Right. So how can I do it? So this is what I was inspired by. By the way, we do dynamic RN. In dynamic recurrent neural network, you have you got to supply a mask telling the dynamic RN that where your sequence is going to terminate. Right. So same idea. I apply it to my submax loss function as well. Right. I mask those negative one classes. So for CTR, right. The submax loss includes all samples. But for CVR, it includes only one sample. Right. And for video viewing rate, it includes only one sample as well. Okay. Then you do averaging, then you're all fine. Right. So given what I have said, I transformed those distribution into sort of normalizer and comparable scores and put it in one dataset. And my dataset, my model is able to handle this kind of situation as well. Right. So I don't need to create like four or five models like what I did before. Right. Okay. And okay, so the performance of the model is like this one. Remember I mentioned we are trying to, what we are trying to predict is human behavior. Right. So the accuracy or error rate of the model is currently at 47% for five classes exam match. I say five is five like that. And this actually, this is based on very little data, not training data. Okay. And my average score is 0.427. Okay. And okay. But we have one, the previous generation in production. Right. And here you see the relationship between our prediction and the actual CTR of those ads. Right. So. Oops. Okay. Here we are seeing positive correlation although not all of them. But at least the creative that is, that was scored five performed at least two times better than the worst ones. And I believe that the newer version should be better. Okay. So we also have a marketing teaser for you guys if you want to try and view our marketer. Okay. You can go to the analyze my ads with your world.com. Right. Then you see the interface. Follow the step. One, two, three. Upload your creative and tell us your targeting spec. And we will tell you is your creative good or not. But bear in mind, this one, this marketing teaser is a simplified version. So he only had three classes. Right. Because it's free. The payable one is in our platform. Right. Right. And so, questions? Okay. So it's the recovery bureau that works for you. You mentioned that there shouldn't be any dead units. But actually they are. Sorry. But actually they are. Yeah, yeah. It's the TensorFlow decay function. No, no. To get rid of new dead units, right? Just don't use RELU. You can use either RELU or what I'm using parametric RELU. Yeah. Okay. So I'm going to show you some of the other things that we are going to do in the coming months or the beginning of next year. Right. So we, oh, damn. Okay. The title is wrong. Sorry. So this is what we imagine the workflow of how advertisers should be creating creatives in the coming future. Right. What is it? Oh, yeah.  You have a list of targeting specs. But you don't know whether they are optimal or not. Right. Then you use our design recommendation AI API model. We tell you what kind of creativity you should produce. So that you can ask your designer, you can ask our designer to produce those creatives or video for you. Then you further use our performance prediction AI. Right. To see if these creatives are going to be good or not. Right. By taking more, like, factors into account. And from the base creative that your designer or our designer has created, we are further generating some variance for you by using some moving variation or expansion techniques. Right. And with all those, all those creatives in hand. Right. We will use our automation AI to help you manage your ads online, dynamically. Right. This is going to change the, how advertisers do their marketing or creative production. Right. And one thing that we will have to create is this one. Right.  And you know your target audience. They are teenagers. They are involved. They like gaming.  But you don't know how to design a creative. Right.  Or the content should be more about sports media or computer gadgets. Right. Yeah. Then we will also provide you with an example creative. Right. Right. And how do we do it? Right. Now, so we have had the black box, the X here. Right. So the X can tell you the optimal, the estimated performance of your creative. So if your input can maximize the target value, then your input value is actually the best one for your desired target value. Let's say I want to maximize my CTR. So what kind of input should I have? No. So I have the function already. So what I need to do is to use a solver and solve this optimization problem. Right. But the difficulties are like my input contains integers or words. How can you optimize input sentences words? Or how can you optimize integers? Right. Normally those solvers can solve floating points only. Okay. And next one, we will have semantic integers. Right. Our research scientist has created this prototype. So given an image and a sentence. Right. Yeah. Okay. Taking the blur image as an example. Right. And then I tell the model. All right. I want this blur to have a blue cone and wings. Right. And white breast. So this model, this thing that is model should generate, generate corresponding images for me. Given my input sentences. Right. So with this we will be able to do like a variation and the creative expansion. Right. And simple flow as well. As long as you have the base creative. Right. And the way we do it is this. This is actually a generative adversarial network model. Right. So our generator is encoder decoder structure. So our encoder takes the image as input as well as the embedding factor for your text. Right. Together. And encoder. So from the feature factor that this encoder has encoded, then you can do things like a deconvolution to generate the image. Right. First in encoder you do convolution and you also take the text embedding factor into account. Then decoder you decode the picture factor into an image. Right. So given the image. Right. You feed this image to your discriminator. Right. Your discriminator again use convolutional architecture to encode your synthesize the image and try to discriminate by taking also the text embedding factor into account. Right. See if your discriminator can discriminate this artificial image or not. Right. If you can, you have just created a good image that the computer cannot really discriminate. Right. Okay. Question? No. All right. I think that's it from me today. I hope you have enjoyed and we are indeed hiring. So I hope anyone who feel like you are interested or you feel like you want to contribute or you want to learn. Yeah. And if you feel like you know more than me. Actually a lot of people know me. I'm not the best one. Actually I want people who know more than me as well. So if you know more than me, please join. All right. Thank you. Mr. Xu once again. Yeah. And then this Friday, you'll see the half-time attendance. See how it's going to be here.