So, talk today controlling IoT devices with crafted radio signals. So my name is Caleb Madrigal. By day, I work for FireEye slash Mandiant doing some awesome IR software. My public handle is Metim. You can't know my private handle. Website there. I'm a ham radio operator, WuHack. Bit about me. I've been programming for around 18 years. Mostly for fun originally and then, you know, for pay later. I really enjoy like mathy and hacky kind of stuff. So like signal processing, machine learning, AI, you know, that kind of stuff. I find it really interesting being able to like hear the unhearable or see the unseeable. So you can do stuff like that with math and I'm not like a big math background, but I like it when it helps me do awesome stuff. Ontologically, I guess the best way to describe myself is as like a Christian mystic. And then I like, you know, arts and music and that kind of stuff. This talk was kind of hard to come up with a title for. So, you know, I played around with a few titles like intercepting, modifying and generating wireless signals with SCR. How digital data is transmitted wirelessly. Controlling wireless IoT devices with crafted radio signals which is what I went with. But another way, maybe a good way to describe it is how the OSI physical layer works and how to attack it. So today we're going to be exploring a few different attacks on wireless systems, wireless digital systems which we're going to look at a few concrete examples but these attacks should apply to many, many different areas as you'll see. Bit of background on kind of what led me to doing this talk. So I kind of came in from like three different directions. So the first one, well, so IoT music theory and wireless hacking. So with IoT, I kind of got into IoT before I knew it was called IoT stuff. Obviously buzzword bingo. But so I ended up making like a little home security system out of like a raspberry pi and like this little wireless control system. So like I bought this little $20 pack of wirelessly controlled outlets at Home Depot. You know, it's 20 bucks. You have this remote which you can turn the switches on and off with. So, you know, I was like hardware hacker time.  Basically I snipped off these little tactile switches and soldered in some transistors and voila. You know, you have a IoT device. A little more programming. But the second thing that brought me to this kind of stuff is music theory which would sound probably ironic or not fitting. But like a few years ago I was just like looking at a keyboard. And I was like why are the black keys not evenly spaced?   And so, you know, but I kind of learned some of the math of like Fourier transform stuff and all that kind of stuff which really interestingly applies very, very well for radio hacking. And this is like a spectrogram of me singing like do, re, mi, fa, so, la, ti, do. Something like that. The last, here all day folks. The last, and the last thing was just wireless hacking. So for a while I've been into wireless hacking. I think it's really fun. One thing I like about it is that often when you find like a wireless protocol attack, it's not something that you can just fix with a binary patch. It's often, often you're dealing with systemic problems when you're doing wireless hacking, which is really cool because, you know, when you find them, they don't really get patched, which is kind of cool. So for example, I did a talk earlier this year at Cyphercon in Milwaukee on basically like using 802.11 monitor mode to figure out like if a wireless security camera has detected motion even without being connected. So, you know, put out software for that. So anyway, wireless hacking, music theory, and then all the other stuff. That's kind of what brought me to doing this kind of stuff. You know, if you see me in a coffee shop, you might find me looking like this. Fran took this photo a few months ago. People like don't like it when you have huge antennas for some reason at coffee shops. I don't know why, but. And anyway, so, you know, doing the wireless hacking of other types, you know, art poisoning and all that kind of stuff. I always had, you know, looked at the OSI layer and like way down here, the physical layer, it's always been kind of this like black box. And so this talk, the goal is really to demystify that black box for you and show the kinds of attacks that you can do at the physical layer. So this type of stuff is super exciting to me. I feel like kind of like it's like magic or something.  It feels like the Star Wars or something. There's this electromagnetic radiation all around us, right? It's here in this room. It penetrates us. If you know how to make secret, these secret codes, you can like manipulate it to make things happen in the physical world.   And so this kind of stuff, it kind of feels like that.  So, and it's a pretty cool one. Also, another big thing, obviously you guys know the whole IoT stuff happening. And it's like, it's almost like I got a like a charm or something. It's like all IoT or all wireless attacks now have like way more damage power potentially, because of the IoT stuff, right?  But it's also something where it's really valuable to know, even if you're not wanting to attack people, right? So you know what to look out for. You can analyze various wireless devices if you really know how the physical layer works. And so we know, you know, like almost everything operates through radio. You know, obviously we know standard FM, AM radio, we call that radio, but really TV is radio, cell phones are radio, Wi-Fi, Bluetooth, GPS, wireless security systems. I mean almost any kind of wireless stuff is radio.  I've not looked at those like for vulnerabilities, but there's a lot of stuff. There's a lot of stuff that you can control if you have radio. So the main tool, like if we're doing magic and that kind of stuff, the software to find radio, it's kind of like, you know, it'd be kind of like your wand or whatever. So I have a hack RF, that's what I'm using for these presentation, this presentation. There's also things like the LIME SDR and all that, but RTL SDR for a cheaper version. But it's really cool because like, you know, you got this antenna and like, you're actually transmitting power through it. You know, it feels just like, it's just crazy. I love the metaphor. Anyway. So let's go ahead and do some demos and stop just talking about stuff, right? So the first one, I actually, I've got a video for it. It's actually attacking my Jeep Patriot. And this is like the first thing I did when I got my hack RF. And so let me just try to play a brief video briefly. This is me unlocking my Jeep with the hack RF. And let me try to just see if the sound works here. And if it doesn't work, it's not a big deal. But basically I have a new radio notebook and I'm using it to turn on my Jeep. And you should see the lights blink in the video. I don't know if that's showing up. So I'm about to execute this script to turn on the lights of the Jeep. Talking more in this video than I thought. Come on, Kayla, what's up? Why are you taking so long? Okay, there we go. All right. So, so it's kind of a, and I lock it back or whatever. Thank you.  I'm like, wow, this is awesome. And it's of course, if you're familiar with SDR, it's a simple replay attack. This won't work on every car because a lot of them have rolling codes. There are other ways to defeat that. In my case, the Jeep happens to not have a rolling code. So it was very simple. The next kind of vulnerability I just want to demo, and this is kind of just to give you guys a bit of the, you know, a bit of an idea of what kind of attacks you can do with software-defined radios. So I want to demo basically a jamming attack, but, I mean, jamming is heavily illegal. So like, I'm not actually going to be jamming. Well, it depends on your definition. Okay, so I've got, I am a radio operator, so I can operate these little radios. So I'm going to just, what I'm going to try to do is I'm going to try to play some music from this radio to this one. And then out of curiosity, I want to see what a, just a really loud sine wave signal sounds like, transmitted coincidentally at the same frequency as this other one. So like, you know, it's not jamming because we're actually wanting to see what this sounds like, right? So let's see if this works. That's another demo one second. Okay, so let's go ahead and there you have it. So the music just cuts out and I'll stop it.  So anyway, jamming, obviously be very careful doing that. It's highly illegal to actually jam something. Anyway, but it's a good example. I mean, physical layer, it's one of those things where it's like, you think about all these other layers up here and then sometimes we might forget about the very lowest layer and how vulnerable it can be.  So I don't want to have to like record the signal or just jam it. What if I want to generate it from scratch, basically? Like, how could you generate a signal that controls something? Like in theory, it's just doing these waveforms. So I thought, well, if I can mathematically, if I can just look at what it looks like and then just use math to like, kind of simulate the signal, basically do like the modulation side of a modem, right? Modem, modulate, demodulator. So that was what I was going for with doing this. So I have a script and we're going to go into in a bit how this actually works. But I want to do the demo first just so you can see it. Now, this is really small, but basically it's just this one line script or one line to call the script and I have a configuration file to turn the outlet on. I'll just show you what you guys what that looks like. It's just, you know, you have the bit string, you have the frequency, the baud rate, sample rate, et cetera. So I generate that signal and the outlet comes on. And then I can generate the off signal using the off config file. And we'll see if that works. Someone jamming me. I'll see if I can get a little bit closer. Yeah, it's not going to work. But kudos by the way. This is a hacker conference, obviously expected. But so anyway, you know, you can do this by doing a replay attack, but this signal that I used to turn that on, I never recorded. I looked at another outlet and found the key pattern and I was able to generate the signal from scratch just using math. So I want to go into kind of how that looks, how to do that. Real quick before we do that, though, I want to show you what this looks like. So I want to show you what the signal actually looks like. So this one right here on the bottom, this is the actual signal, this is one of the signals I captured by listening on the SDR. So I'll let you see what it sounds like. And you get the idea, right? It just keeps repeating. But if you zoom in, and this is just an audacity, which by the way, like I mentioned, I did some audio analysis stuff, a lot of the tools actually for audio can be reused for radio hacking. So these are radio waves and we're using a sound editing program to actually see these. You can even edit them in here.  So you have the short ones which represent zeros, the long bursts represent ones. So the way to read this would be zero, one, one, zero, one, zero, et cetera. And that's the signal. Up here is actually my generated signal. So you can actually hear them a little different. This is the original. And this is I think the generated one here. So it's cool because it actually is a little different. So you don't have to exactly replicate for it to actually be accepted by the outlet. Anyway, you can, I have a few notes on how to open up wave files in audacity. I'm going to skip over that, but those are going to be in the notes in case you guys want to be able to do that. So, alright, so understanding waves. So I want to look at a little bit of the details of how radio waves work, mathematically why some of this stuff works, but I want to emphasize something. You guys do not need to know or do this math to actually do radio hacking. There's programs like a new radio that you can use and you don't actually have to go and like write all this math out. But I personally found it very insightful to see how these radio waves work and how waves work to help me understand how to do this stuff. So I want to like look at some of that stuff, should give you guys a really good foundation and understanding of what's actually happening. And then you have a better idea when you use other tools, what they're actually doing. I mean it's, and a lot of times I know hackers aren't really into math a ton and I'm not really traditionally either, but you know like I don't want to be a script kid, I don't want to be a script kiddie. I want to understand what's really happening. And so that's kind of the purpose of this. So understanding waves, what do we mean when we talk about waves? I mean usually just sine waves, right? Sinusoidal wave. In computers we represent them as simple arrays. So if you look at this graph, this is simply an array where the values are going greater than zero and then they're less than zero. It might be like zero, point one, point two, point three all the way up to one, back down to like negative one, and it oscillates. That's what a wave is. Sine wave, it's related to sine on the circle. So waves going up and down like this, if you just follow one dimension in a circle, it's going to make that shape. There are some really, really interesting things about sine waves, about waves in general. Let's look at these real quick. So first they're everywhere. They are the epitome of change. There's this thing called the superposition principle which is really, really important to understand. We're going to look at convolution, deconvolution, the relation to E which believe it or not is actually useful for programming this. So first waves are everywhere. Like any time you have circular motion or vibrations, that's sine waves somewhere in there. All wireless communication which we already mentioned, motion of pendulums or springs, et cetera. You see it in the temperature of earth. You have a natural sine wave created, patterns of breathing, in and out, yin and yang stuff. It's all over modern physics. So it's really interesting to understand waves. Quantum field theory says that even particles are just simply excitations in fields or waves in various types of physical fields. Which is pretty cool. String theory would say that maybe vibrating strings are the basis of all matter, which is cool. I'm sure you guys have heard of the uncertainty principle, like the Heisenberg uncertainty principle. It was kind of cool. Like when I was doing audio analysis, I stumbled upon it like accidentally. Like basically if you look at a spectrogram like this, you have basically the time dimension in the x-axis and then the y-axis is the frequencies and the intensity of the color is the intensity of the amplitude basically. But basically like if you have a sine wave, if you only listen for like a millisecond, like how do you know exactly what the frequency is? It could be, it's going to be hard to tell. The more data points you have, the more sure you can be. The higher probability you have of knowing exactly what that frequency is. But the more samples you have to gather to get that, the less sure you are in time, right? If you're chunking by one second chunks, you're only going to know within one second where this spot was. So what happens when we up the window size for the spectrogram? You don't have to understand all this, but just look at, watch what it does. You're going to see in the time domain, we're going to be less and less sure it's going to stretch out horizontally. But in the frequency domain, which is the y-axis, it's going to get more and more tight, right? So it's kind of cool. You can visually see the uncertainty principle. So it's cool waves help us understand the universe. Another cool thing I found, waves are kind of the epitome of change. Of any shape, you know, like because the derivative of a sine wave is itself another sine wave. So it's kind of cool. It's like the epitome of change. Not important for radio stuff necessarily, but kind of cool. There's also this really, really cool concept. Now this is true of both sound and radio waves. It's called the superposition principle. So let's try this. So I'm going to knock. Can you hear me and the knocking at the same time? And can you differentiate them? Well, you can. But what's actually happening is in the sound waves. My, the pressure from my vocal cords is getting added to the sound of me like knocking on the desk. And they're being added together into one, into basically pressure that's reaching your ears. Your ears only can determine pressure. Oh. There we go. So the idea is there can be multiple simple waves all added together and stacked together to form a more complex one. So what this looks like visually, you can see these four sine waves. One is flat. You add those together, they look like this, right? This is like, and this is what radio waves look like as well. Look at all, think about all the radio devices all around us that are transmitting. It's making, they're all getting added together and it's making this really big complex wave. But we are able to tune in to one frequency with radios. That is only possible because it's able to deconvolute this really complicated wave of electromagnetic radiation. And so this is important because we have to do that. We have to be able to tune in. We don't want to have the noise from the radio station nearby affecting our cell signal, right? This kind of thing is done with a 48 transform. Not going to get into a lot. You know, it goes from, you know, the time domain to the frequency domain. But, yeah, deconvolution is necessary to actually tune in to a signal. One other little side note that's kind of interesting. This kind of blew my mind, okay? It's really kind of cool if you get it. So if you have two different sine waves, let's say you have an 8-hertz sine wave, so it's cycling eight times a second, and you have a 12-hertz sine wave. And you multiply or take the dot product of those together. They will always, basically any two frequencies, any two different frequencies are going to cancel each other out when multiplied. But to a wave that has the same frequency in it, if you multiply it, take the dot product, you'll actually have a big value. That's actually how you can tune in. That's actually how you can deconvolute a signal. So for example, you know, here I have an 8-hertz wave and a 12-hertz wave. Multiply those and you get something close to zero.  If you're curious, like, does it drop off right away? The graph looks kind of like this. It's kind of crazy. And this is kind of esoteric, but you could imagine every single simple sine wave as one dimension in this really complex Hobart space. And you can actually, like, add vectors together. But anyway, it's kind of interesting stuff, but I'm going to skip over that for time's sake. The last thing, and this is actually going to be, this is actually useful to me for generating the script. If you have a single dimensional sine wave just going up and down, what radios actually take is a complex sine wave. So it actually is oscillating not just up and down, but left and right. So it actually looks like a circle like this going forward. And raising E to the I times some power actually generates a two-dimensional sine wave, which is pretty cool. So let me show what that looks like. So basically like this. So instead of having just one up and down sine wave, you actually have this circle, the spiral shape. And that's actually what radios use. So then how does actual digital communication happen? Well, we know modems, of course. We're doing the mod side of a modem. You know, just to go through it quickly, there's a couple of types of modulation. So there's amplitude shift keying, frequency, phase shift keying, et cetera. Amplitude shift keying, you're just making the amplitude of the wave go up to be louder or down. So this would be like zero one. That's what it would actually look like. The sample, the demo that I did with the outlet, that's an example of ASK, right? You can see where it's really small and then it gets really loud, et cetera. Frequency shift keying, you're actually changing the frequency. There's lots of stuff that uses that as well. Actually FM radio would be frequency shift keying. This is what that would look like. So this would be like zero one, zero one, one, zero in FSK. There's also phase shift keying and other crazy stuff like quadrature amplitude modulation. That's actually what Wi-Fi uses. It's kind of a combination of ASK and PSK. Anyway, just to show you kind of what my process was like for actually getting this script developed. So first, this script, it's all online but it's about 120 lines of Python right here. So not really, not bad. But I want to go through kind of what it was like. So basically, I started off with this ASK sample. I kind of just eyed it to try to figure out what is the baud rate, all that kind of stuff. And I said, okay, I need to do basic amplitude modulation. You can ignore all this code. This is a Jupyter notebook, by the way, which basically it's an executable visual thing. So like all the code necessary to do this is in the single document. And it's at the end of the, it's on the last slide. Anyway, so I have this wave and I need to amplitude modulate it. So I wrote some code to do that. And then I basically wrote some code to say, well, I need to make it, I want to make zeros and ones, transmit into low and high. I'm not going to go into all the details there. There's also bit spacing. You notice that if you look at the signal, there's some bit of space in between each bit. So basically, you know, find all those variables and then I have an algorithm to generate a signal from that. I tried this and I was like, hey, sweet, that's like really close to the original. And I tried it and it failed. And actually the reason I failed was I learned the hard way, well, yeah, it's actually got to be a complex sine wave signal. So I basically was able to substitute out raising sine to the X to raising E to the IX and it just worked. Or actually, well, there's one other step that I had to do. And it was interesting, but there's basically when I looked at the waves up close, doing a simple wave, it was almost, you almost couldn't hear it. Basically I had to do another sine wave inside of, so I was transmitting at 315 mega hertz. I had to do another like 1000 hertz wave or something like that. And that's the one I was actually making go loud and quiet. So a few little details if you actually want to do it. Otherwise you can just use the script. It's on GitHub. And there is definitely some interesting obstacles that I had to pass to get that to work. This script, this whole presentation, including these links I'm going to have at the end. So anyway, conclusion. It's really cool because like all wireless communication, like virtually every wireless communication that I could think of happens over electromagnetic radiation. All digital communication rather. The only exception I could think of is sound waves. That's a form of wireless communication that is not radio wave based, right? It's sound waves. Having looked at, you know, the various ways you can attack the physical layer. I mean, hopefully you can be aware of various attacks that are possible. You know, we went through jamming attacks, replay. This stuff I'm doing, generating these signals from scratch. You could use that potentially to brute force, right? Or you may see a sample like in my case, I actually saw a sample of another outlet, not this one, but another one. From that I was able to derive the binary code for this one. Just, you know, a few bits difference. There's also potential for doing things like mixed attacks. So you might do like, there might be a one time rolling code and you want to brute force that, but you replay the other part, right? You see what the key space looks like and then you generate all the possibilities. That's another possibility. And, you know, thinking about this kind of stuff. So like I have a home security system. I have a wireless home security system. So like the jamming attack, you know, you could theoretically jam, okay, so like I have a door sensor and then it has the hub, right? Well, you could jam at the frequency that those communicate with, even if it's armed, open the door, right? And so, well, how could you fix that, right? I mean, you could just do a non-wireless system. That would be one way. You could also, if you're looking at wireless systems, wireless security systems, you might say, well, I'm going to look for one that has maybe active low. So if the power gets cut, we know something's up, right? Or there could be ones that keep track of their state. They actually look for an act coming back from the hub. That kind of stuff would be harder to attack in this way. So hopefully this will give you guys a good basis for how radio works, right? How all the stuff works. You can use various tools. I recommend a new radio, a new radio companion, hack RF. And there's a lot of new cool tools coming out right now.  So, yeah, and then that's all I have for today. I think we may have a few minutes for questions in case anyone has any. The link right here is to this presentation. And the link below is to my GitHub which has the Jupyter Notebook, all the scripts, all that kind of stuff. So let's see, do we have any questions? Yes. I can't hear you. Can you come a little closer?


My name is Don Nicholson, along with my wife Trista and three boys, Aiden, Gavin and Ian. We farm and ranch in Frederick, South Dakota. And one of the things we're trying to do is incorporate our livestock more into our crop ground. Where we live in Frederick here, the climate and the rainfall is very east and west. Across 281 it just starts drying up more, so we have kind of farming in two different areas where the stuff east of 281 we have generally more rainfall. As we go further west it starts falling off and it lends itself more to the livestock side of the operation. Initially I started into no-till. After college I came back to the farm. It was just me and my stepfather farming so labor was, it seemed like an easier avenue to go. Less tillage, less work. So I started out as a labor-saving and then I started seeing, you know, benefits to the soil and doing less work and still keeping up yield-wise with neighbors. You know, we've always had livestock and we've always had crop ground. Focusing on the cattle in the fields longer. Start seeing the benefits of the soil health. The crops we grow here are corn, soybeans, oats, triticali, alfalfa and starting to do a little bit more in the cover crops. The alfalfa I think has been key for me to start identifying areas of the field that are not as productive and allows me a different time of the season to be able to get into those areas if it's fixing saline area spots. In our area also there are a lot of potholes so with excessive water, ponding and ditch effect saline areas start popping up and expanding, you know, if they aren't cared for or treated differently than the rest of your crops. I use a lot of precision stuff in my cash crops so I started using that to identify the poor producing areas and then zeroing in on what those spots need and how to do it. The best way to do that in my operation is through the cattle and bale grazing. Some of the effects that I've seen in the areas where I have been bale grazing after doing it the first year, you know, you get the cover which I believe the cover helps eliminate some of that evaporation. It keeps the soil moisture rather than drying out and bringing all those salts to the top and then wicking moisture from other areas and pulling up more salts. It's part of the healing process. It might not look the prettiest at times but it's a root in the ground, it's something that's growing. Also on our pastures, I have been rotational grazing for quite some time. Golfers say play the course, don't let the course play you. You've got to look at how the ground is and what kind of grasses are there and what you can do with it. Each area is managed differently, just like in my crop ground. Each zone is managed differently. There's challenges every year and every year is a different challenge and what I've found in my soils, you know, building the organic matter, having better soil health allows me to weather through whichever weather storm I face with. I wasn't born into farming but from a very early age, since I was probably nine, ten years old, I was been exposed to farming and then got a great opportunity and a lot of freedom to try different things. My stepfather gave me a lot of freedom that way and that's helped me going forward. I remember him always telling me, you know, do what you think's best and that's what I hope to pass along to my sons, you know, whatever they're doing is do what they think's best.


Communion, like the cross, should be a critique of unequal power structures in our homes, churches, communities, and societies. What if Communion were me sitting on my Rastafarian neighbor's back porch, listening to Reggae and hearing about his week? Do this in remembrance of me. What if Communion were showing up at a protest and taking on issues that are not at all my own? Do this in remembrance of me.


Patrol Squadron 45 flew one of their P-8 Poseidon Patrol aircraft north to Denmark to participate in the multinational maritime exercise, Northern Coast 16. The P-8 brings improved speed, sonar buoy capacity and software for conducting anti-submarine operations to the Europe Africa Theater. It helps us to get to know the accents and the techniques and tactics of the NATO forces. I don't get to train with NATO very often back home, so being out here and you calm and flying with those guys and working right alongside them is actually, I mean that's great. It helps us, helps them. It's good all around. With 14 countries involved, Northern Coast 16 provides the perfect opportunity for the P-8 crews to showcase their capabilities and strengthen interoperability between land, air and sea forces from regional partner nations. This is the first time a P-8 in any squadron has ever got to come up here and do this. It's definitely a lot different environment as far as shallow water, there's a lot of surface traffic, working cross queuing with other nations, using multiple assets to prosecute a sub. It's definitely a learning experience which is pretty good for us. This exercise has been conducted every year since 2007. From the skies above the Baltic Sea, I'm Petty Officer Kelby Sanders.


Hi friends, I am Purva and today we will discuss the following question. Triangle ABC is isosceles with AB is equal to AC is equal to 7.5 centimeter and BC is equal to 9 centimeter. The height AD from A to BC is 6 centimeter. Find the area of triangle ABC what will be the height from C to AB that is CE and we are given this figure. Let us now begin with the solution. Now we are given AB is equal to 6 centimeter and BC is equal to 9 centimeter and we have to find the area of this triangle ABC. So we have area of triangle ABC is equal to half into base into height. Now here we have BC as base and AD as height. So we have that is area of triangle ABC is equal to half into BC into AD. Now this is equal to half into we are given BC is equal to 9 centimeter. So we have 9 into and we are given AD is equal to 6 centimeter. So we have 6. Canceling 2 in denominator and numerator we get 3 in numerator. So we have this is equal to 9 into 3 is 27. So we get 27 centimeter square. So we get area of triangle ABC is 27 centimeter square. Now we have to find the length of CE. Now again in this triangle ABC if we take AB as base then we have CE as height. So we have also area of triangle ABC is equal to half into base that is AB into height that is CE that is we have now area of triangle ABC is equal to 27 centimeter. So we have 27 is equal to half into now we are also given that AB is equal to 7.5 centimeter. So we have 7.5 into CE and this implies 27 into 2 upon 7.5 is equal to CE. Now solving this we get this implies CE is equal to 7.2 centimeter. So we have found the length of CE to be 7.2 centimeter. Thus we write our answer as area of triangle ABC is equal to 27 centimeter square and length of CE is equal to 7.2 centimeter. Hope you have understood the solution. Bye and take care.


Hello, everyone. It's my pleasure to introduce Marley Gatti and Jake Aiden. Marley is a senior data scientist at Biogen and Jake is a senior principal biostatistician. They're going to talk to you about a shiny-based ANOVA and power analysis tool. Please welcome them. This is joint work between myself and Marley, a senior data scientist at Biogen. The status quo at Biogen currently is that the statisticians often receive many requests from the research scientists to perform ANOVA or sample size calculations of three clinical studies. Unfortunately, the scientists often use pre-existing software packages such as Prism, which do not perform ANOVA in a rigorous manner and does not perform sample size or power calculations. In addition, these software packages do not include QQplot, Residual Tynastic Plot or other optional diagnostics, and they don't suggest an appropriate model for the scientists to use. Sometimes the scientists may perform these ANOVA analysis incorrectly, which may lead to incorrect interpretations for their internal biogen experiments. They may use incorrect models or analysis submitted to journals, and they also have to be careful about ANOVA assumptions. If the ANOVA assumptions are violated, this may impact the p-values or significance of findings. Now, for the sample size calculations for pre-clinical studies, sometimes the scientists may just pick a number for the sample size without doing a rigorous calculation. If they pick a number that's too small and the study is underpowered, if they pick a number too large, that's a waste of time and resources. Now, ANOVA app offers a lot of benefits. So, the app we're going to be using to train the scientists to check the assumptions of ANOVA and how to make their analysis more rigorous. So, we're hoping to create an app that is a little bit more rigorous than pre-existing tools such as Graphcat Prism, and we're striving to increase the reproducibility of pre-clinical studies. We're also hoping to increase the quality of analysis that are bound in for pre-clinical studies and streamlining of paper acceptance. Some other benefits of ANOVA app is that it relates to automation and standardization. So, by streamlining our calculation for ANOVA-empowered calculations, we are now obtaining results in less than 30 minutes rather than waiting weeks for the statistician to do ANOVA calculations for the scientists. So, this leads to both increase efficiency for both of the scientists and statisticians. We're also hoping to empower scientists to perform ANOVA-empowered calculations themselves after we properly train them on how to use the app. Another advantage of our ANOVA app is we're presenting one consensus analysis protocol to the scientists for specific scenarios such as comparing means across independent groups. And lastly, the ANOVA app is standardizing both the data input and data output. Now, we have to be mindful of the scope and limits of our app. So, within scope of the app is ANOVA and ANOVA-related sample-heist calculations for pre-clinical studies, and this app is only for exploratory analysis, not for clinical trial analysis. In terms of models, we are allowing one-way and two-way ANOVA models as well as interaction effects or no interaction effects. The app also supports well just ANOVA and non-permeasure ANOVA. And the app is only handling unpaired design and no repeated measures. Out of scope of the app is sample-heist calculations that don't relate to ANOVA, such as non-tuner survival analysis of sample-heist calculations. Clinical trial analysis is also out of scope, and the app does not support repeated measures at ANOVA. I want to discuss some of the features of our app. So, the ANOVA section of our app allows the user to import in vivo data and visualize data. The user can then assess the normality assumption, equal variance assumption of ANOVA, and then the user can decide whether or not a log transformation is necessary. The app also checks for outliers, and the user can remove outliers if they desire. Then the user can set up their comparison of interest, and then the app will provide a results table with adjusted P-values. For the power and sample size part of our app, the app does perform both parametric and non- parametric power calculations. The parametric power is based on a formula approach based on the details, and then parametric power analysis is based on simulation. One main feature of our app is the app automatically suggests a model for the user to use. So, after the user selects their interpretation of each of the diagnostic plots, the app will automatically suggest a model. So, for example, if the user selects that the diagnostic plot is showing a normal distribution and an equal variance, then the app automatically suggests an ANOVA model. If the diagnostic plot suggests a normal data and an equal variance, the app will suggest a well-tos ANOVA. I also want to give an overview of the power analysis tab of our ANOVA app. So, within the power analysis tab, the user can choose their primary and secondary objectives for the next experiment, and then the app will calculate the sample size per group required to observe at least one of their primary hypotheses in the next experiment at the desired effect size for 80% power and a 5% of type 1 family-wise error rate. The app for robustness will plot the sample size requirements per group or a range of effect size scenarios, and the app will also plot the statistical power for a range of sample size per group scenarios. And then finally, the user can output a report of their ANOVA power analysis findings. Some of the parameters the user can select during the power analysis tab is the desired effect size. The app will automatically calculate the observed effect sizes based on the input of pilot data. The user can specify their family-wise error rates and their expected mortality rate in a very pre-penical study. The app will automatically calculate the variances based on the pilot data. And the app will perform a different calculation based on the user's review of the equal variance diagnostic plots of the app supports both equal and unequal variance power calculations. In addition, as I mentioned, the app supports both parametric and non-parametric calculations for power. And now I want to hand over to the presentation to my colleague, Marley, to perform a demo of our ANOVA app. Thank you. For our demo, we're going to use a data set of infar volumes of mice after cerebral artery occlusion. And this data set has three main groups, vehicle, compound X and compound Y, and the vehicle is going to be the control group. So let's take a look at the application. So when you load the application, you have three tabs enabled. The first one is the introduction tab. And in this tab, you can see information about the application, like the type of data sets, as well as the type of analysis included. This tab is just informative. The setup tab contains information about the type of ANOVA. You can upload a data set here. And you have a violent plot, a dot plot, and a review of the data set selected. By default, the application has three these two data sets, iris and empty parts. So you can play around with ANOVA application. For purposes, we're going to use the infar data set. So we're going to perform in one way ANOVA, and our grouping variable is going to be group, 10 points going to be the volume, and the unit identity part is going to be 18. Now, by looking at the data, we're going to be able to create the next tab that the variance is equal. And again, this dropdown is just optional. It's just a way for you to tell the application that you expect that to be the variance. Now, for the covariance tab, we can see that the variance is actually a new code. And so once you click on the covariance, a new tab will load. And in this tab, you'll be able to select if the data is normal on the Rina scale, normal after log transformation, or not normal. Now, since the data is small, I'm going to reduce the number of things so that three percent change is what I create. So this does look like a bell curve shape. And so if we go to the QQ plus, we can actually see that the first QQ plus has all the points on the diagonal line. So we can safely say that is normal on the Rina scale.  And this tab allows you to remove all liars from the data set as well as identify them. Now, reviewing this chart, we can see that there are no points greater than three, nor less than minus three. So there are no outliers.  In the final check tab, the application is going to recommend you a test based on the previous selected choices. So in this case, he suggests a Welsh test. So this bar is only implemented for ANOVA. We can take a look at the ANOVA data here. So we're going to skip to the multiple comparison blocks since that is relevant to what we're doing. And so let's select the pairs for multiple comparison, have the X and the Y compounds. Now, by looking at the comparison results, we can see that the X vehicle is not statistically significant, since the adjusted P value is greater than 0.5. And the Y vehicle is statistically significant since the adjusted P value is less than 0.5. And by reviewing the tables below, the chart below, you can see how similar our vehicle on X groups and how the similar is that from the Y grouping variable. Now, the application throughout has a lot of choices, like increasing the dot size of the multiple comparison plots, the height of the plot, the title. So the application has embedded a lot of extensions that you can take advantage of in case that you have a grouping variable, for example. Okay, let's go to the sample size calculation tab. In the sample size calculation tab, let us choose very close to control group, the primary hypothesis as the Y compound vehicle, and as our secondary hypothesis, the X compound vehicle. Now we'll assume a certain percentage of mortality rate and a 50% of desired effect size. Now below, you can see that based on the previous elections, this table of effect size on the X axis versus sample size per group on the Y axis displays actually that the observed effect for the primary hypothesis is negative. So actually, we expect that the duration of effect is decreased in response. So if we go over the effect size of minus 50, we'll see that the sample sign is 25, which means that we need 25 animals per group to detect at least one of the primary hypothesis in the next experiment. We can also take a look at the power versus sample size. And here power means the probability of detecting an effect size in the next experiment. And when we hover over the default 80% statistical power, we see that we need 24, 25 animals per group to detect at least one of our primary hypotheses in the next experiment. Now application after you have reviewed all the tabs, you can generate reports. And for the sake of time, I've pre-generated for you. And you can take a look at the different sections. So it actually records each step of the application, the analysis set up, the initial diagnosis, the final diagnosis, their respective plots, the comparison table, and all, and the two plots on the sample size calculation for a one-way and all. We would like to end the presentation thanking the rest of the team. I went in one Taylor Reynolds, Michael Pearson, and Bob Inko. And thank you. So let's see, we have a question that's been upgraded. Does the app generate root reducible code for a specific analysis that's been run? It's like that one. So that's a very good question. And so far the application does not. But once application is open source and default, we are open for anyone making requests and working on them. Have you had a chance to look at any other ANOVA R packages? For instance, there's one called DOEX, D-O-E-X. Any thoughts on those? No, I'm not familiar with that one. Yeah, neither. We'll be interesting to look into.  Well, I don't see any more questions from the audience, but thank you. And you know, I really think that you've got a winner there with that and I hope that you get lots of users. Thank you. Thank you very much for joining us. Great. Thanks.


Hello, my name is Duncan. I'm a lecturer here in the Department of Computer Science at the University of Manchester. And in this short video, I'm going to tell you about our industrial experience program, which is a key part of our undergraduate degree. The industrial experience program is a year-long program that students can optionally take, enables them to get some work experience before they graduate. So they graduate with a broader and deeper set of skills than might otherwise have been the case without. When you study at university, you're making a big investment in your future. Yes, you're studying a subject that you find interesting, but perhaps part of the reason that you're doing that is because you're also trying to improve your career prospects. Now, it used to be the case, perhaps in your parents' generation, that the way to get a good job was to go to a good university and get a good degree, and that was all you needed. Now, sadly, that's no longer the case. In order to get a good job, it's not enough just to have a degree. You also need to have some kind of experience and demonstrate something above and beyond what a degree provides on its own. So this is where our industrial experience program comes in. As I said, this is a year-long program. It's optional. It's something that students can opt onto and off of any time up until the end of their second year. Our students work for a wide range of companies from very small start-up companies right through to some of the largest and biggest tech companies out there in the UK and outside the UK as well. In the last academic year, we had over 100 students going out on placement, and they were doing stuff that was relevant to their degree program. So they were all doing a wide range of jobs, but they're all relevant and related to computer science or software engineering or whatever degree program they were doing. Now, in order to find a placement, we have a team of people, including myself, who help students find placements and encourage students to do so during their first and second years of study. And the reason we do that, as I've already said, is it strengthens students' employability because students who've done a placement year typically have a broader and deeper set of skills than a graduate without. During your placement year, you have a reduced tuition fees and you're paid a full salary. So you're not paying the full tuition fees for the year, but you are paid a full salary by your employer. As it stands at the moment, 96% of our graduates who are on the Industrial Experience program are in a professional or a managerial job six months after they graduate. So we think this is a good program. It's beneficial for students, it's beneficial for us because we graduate better students, and it's beneficial for employers because employers find that placement years are a very cost-effective way to hire new employees. Now as I've said, there's a wide range of places that students go on placement. Here's one example. This is Juliana, who is working as a software developer at CERN on her placement year. Now CERN is an interesting employer because they're perhaps more associated with physics, but it turns out that for every physicist working at CERN, there are 10 engineers, and many of those engineers are software and hardware engineers. Because in order for the physics to happen, in order for the physicists to do their experiments, they're aligned on a huge network of very complex machines and software that enable them to do their experiments and interpret their data. So Juliana was working at CERN, and we have students going to do placement years at CERN every year. But as I've said, there are lots of other places that students go. So this is Ben. Ben was working for NCC Group, which is a global cybersecurity company, and he actually went back to work for them when he graduated. Sarah here was working for Nant Health, a company working in the health information industry, so providing systems that enable doctors and clinicians and medics to do things with electronic health records and the like. And we've also got Kieran there at the bottom who is working for PricewaterhouseCoopers. Now some of the companies that students work for are perhaps the obvious ones, the big tech companies that you've heard of. So this is Todd, for example, who went to work at, who is now working at Google. We've had students go on to work for all the big technology companies, Google, Amazon, Facebook, Microsoft. But there are also lots of other jobs as well. It's not just the big tech companies, but if big tech is what you want, that's certainly an option. Another example is Laura. Laura is now working at the BBC, having done a placement year at another employer, and she's worked on a range of different products and services provided by the BBC, including websites for the FIFA World Cup and the general election here in the UK. So where else do our students go? Well, you can see on this map here that's published at that link there is where students are going for their placements, but also where they go after they graduate. You can see that obviously the UK dominates, but because our students come from all over the world, the students are going on to graduate positions at a wide range of places all over the UK and all over Europe, all over the world. And one other thing to say about Manchester is that there's been significant growth in the last few years and the number of employers that are in Manchester looking for computer science graduates. So companies like ARM, for example, who designed chips that you'll find inside your mobile phone. Companies like Jaguar Land Rover, who are designing and building autonomous vehicles. There's GCHQ, recently moved up here to Manchester. There's booking.com, who work in the online retail space, but also lots of other companies like Disney Plus, which launched this year. A key part of their infrastructure was built and continues to be built here in Manchester. There's companies like Couchbase, Wakelit, Cubic Motion and BioRelate. So besides the big technology companies you might have heard of, there are lots of startup companies as well. And many of those are in Manchester. So if you'd like to stay in Manchester for your placement year, there are lots of options for you to do so. And many of the best relationships we have with employers aren't just with employees around the world, but they're with employers that are on our doorstep right here in Manchester. You can find out more about those employers if you follow that link there. Another thing to say about the University of Manchester, just to finish off, there's two things. The first is that the University of Manchester is the most targeted university by the UK's top 100 graduate employers according to the Times Survey and according to High Flyers, their Graduate Market Report published in 2020. What this means is that there are lots of top employers who visit the campus and are engaged with the university and students in various ways. So this gives you lots of opportunities to meet employers and find out what they have on offer and whether what they've got on offer is right for you or not. Another thing to say that is because the Industrial Experience Programme is a significant part of our undergraduate degree, we have lots of students here doing it and it's those students alongside staff like myself and other people in my team who can help you find a placement. Often the best person to talk to about a placement is somebody who's just come back from that company. So that network of contacts is something that you can take advantage of, whether it's the professional network or the student network, it's all part of something you can take advantage of while you're here at the University of Manchester. So I hope that gives you a flavor of some of the ways that our students make their way into employment after studying.  There are dedicated staff like myself who work in computer science, who work alongside the careers service, who are there to help you at every step of your way along your job search and your route into employment. There's also a significant community of students. So I mentioned at the beginning there that we have over 100 students on placement. And what we found is that having a large group of students on placements means they can help each other and advise each other on what the good and bad points about specific employers are or what specific interview questions are. So it's a classic case of it's not what you know, it's who you know. And those students are part of a larger alumni group. There are more than 10,000 people who graduated from the University of Manchester with some kind of computer science degree, whether it be bachelor's, master's or PhD. An alumni network is something again that you can plug into as a student here in Manchester. So I hope that gives you an idea of the stuff that we do here. If you have any questions as a Q&A session or you can post comments below. Thank you for listening.


Today I'm going to talk about a downtown activation and placemaking plan that we recently completed with Graf consultants out of Green Bay. So the introduction and purpose. So this is in 2014, the Harbor Center Master Plan was completed in partnership with the Downtown Business Improvement District in the city of Sheboygan. And that was really the honest for getting a lot of what you're seeing in the vibrancy of our downtown today. Housing, cross connection and marketing, and then placemaking. So I'll talk about placemaking in a little bit more depth as we move forward, but this is really kind of taking this new version of this plan is taking the successes of that 2014 plan, of which about 90% of the work in that plan has been completed and kind of evolving into some additional districts and some additional activities for us to continue to bring some vibrancy to the downtown. So in the 2014 plan, there was a number of goals and actions, install streetscape amenities to reinforce the arts, culture and food district and encourage pedestrian activity throughout the downtown. There was a number of projects that were completed with the largest one being the revitalization of the library plaza that was completed last year. And then some streetscape improvements along A Street and the Niagara Avenue corridor. Another goal was to increase awareness and understanding of the Shabuigan market opportunities among property owners, investors and developers. And you've seen that success with the additional housing that has happened, although this plan added, asked for another 50 additional downtown housing units. And we've actually are closer to 350, but yearly we hold a regional developer summit where we invite people from the outside to come in and see the opportunities that we have. And that was really the success for moving these developments forward. Other goals were to establish the arts and culture district. And we did that with the John Michael Kohler Art Center, expanding public art and introducing family friendly activities in partnership with the Mead Public Library. And then support the coordination among local and regional tourism efforts, events and consumer marketing to leverage investment. And there's a number of events that have been held night market, being one of them in partnership with the Shabuigan County Interfaith Organization. Another goal was to improve the mobility within the bid by promoting alternative modes of transportation. And this is kind of where the downtown trolley, the seasonal trolley service that Shoreline Metro operates kind of came out of. And then shared bike systems, which we continue to discuss today on where we should go with that. Another improve wayfinding and navigation within the bid to encourage visitors to explore additional businesses and amenities. Although we didn't create an app as what was planned in there, we did reestablish street connections by opening up Wisconsin and New York avenues and then installing pedestrian wayfinding signage throughout the downtown. And lastly, improve economic returns and reduce vacancy for property owners in the downtown. So we developed design standards that our staff follows today as somebody's coming forward and wants to make renovations to their storefront that it kind of follows a plan for how we do so and then preserve downtown property values by increasing the aesthetics and encouraging quality renovation. So the new plan, the 2019 plan that we're talking about here today is to continue to create placemaking enhancements at key locations to reinforce identity, celebrate strength and adjust challenges, improve them, continued improving the mobility about within the downtown districts with alternate modes of transportation, continue to support local and regional tourism and then preserve property values. So what is placemaking? Placemaking is a collaborative process for community members, business partners and property owners and municipal governments work to reimagine public spaces. The goal of placemaking projects is to help people feel more connected to places, more excited to walk to lunch and shop locally and play with their kids in parks and other green spaces. So on the screen is some examples of placemaking that you see today that was success of the 2014 plan, sidewalk seating, planters and trees, markets and festivals, downtown district signage, seating areas and community events with the city green. This plan also, this plan looks at four key areas. So the Michigan Avenue and the Indiana Avenue are two districts that haven't been focused on in the past and that are included in the plan as well as what the paradigm folks would call uptown which is primarily Erie Avenue to Michigan Avenue, that area and then the downtown basically from Erie Avenue down to the river. So each one of these districts is broken out with a number of priority recommendations. Michigan Avenue, the three or four priority recommendations are the right size of the roadway. So there's a lot of road on Michigan Avenue, a wide road with very wide parking lanes. So the idea is how do you try to bring people together with bump outs and those types of things we'll talk further about. Strategic nodes of art lighting, district gateway and entrance markers and then facade improvement projects. In the uptown, it's very similar. Priority recommendations are strategic nodes of lighting, sidewalk seating and facade improvement projects. Downtown is the same as uptown and then Indiana Avenue is really kind of going off of the whole innovation district and working with district specific street furniture, gateway signage, facade improvements and bike path signage and landscaping. So in the recommendations that gateway signage and street prints you can see on the screen there's a number of ideas that came forward on different signage to identify these corridors as well as street prints along the key corridors to try to slow traffic. So there's some plans, there could be some plans to paint some areas and try to slow traffic with traffic calming with painting and those types of things on crosswalks and streets. We've seen some murals already on buildings, particularly on the outside of the above and beyond children's museum, but the idea of trying to use more district entrance mural signage on key buildings within those districts to identify those districts is another thing that's planned. Parklets, so parklets are taking parking stalls and putting barriers around them and having seating areas out in the street to try to green up areas. So in this picture it shows the US Bank building and what it would be if some of those parking stalls were created into parklets to bring some extra greenage to the green and stuff along the street and then there's some pictures on the bottom of how parklets and you see these a lot of times in larger communities as a way to try to encourage people out on the street to dine in and socialize. Strategic nodes of art lighting, so there's different examples here. Some of that lighting has already happened in the alleys downtown in partnership with the Business Improvement District. These are just some additional examples of what that might be. Activating vacant storefront, so the idea is to take storefronts and put artwork and or dining to try to activate it so when you're walking down the street you feel a sense of connection with those types of businesses. And then sidewalk and alleys seating for restaurants, so some of that has happened already with the Black Pig Alley, but considering more of that alley seating for restaurants and people to kind of come out of the building and into these public spaces. And then right-sizing the roadway, particularly as it relates to Michigan Avenue, the idea of some type of bump outs and or landscaped barriers with seating and stuff in to try to connect people closer to the street and not feel like it's an airport runway and trying to green up and make it a place, a third place where people wanna be. And then lastly, facade improvements and projects that support the rehabilitation. So we've successfully worked with the cautious company on the property where they are today with a paradigm. And then the other photo is the Mavericks Barbershop which were historic preservation renovation projects funded by the city. Under the innovation district specific street furniture, this would be related to Indiana Avenue and trying to give a sense of place with improvements on the street and kind of tied it into the whole innovation district and how that all plays together as a tech corridor and kind of including the streetscaping into those areas. Wayfinding, strategy to encourage walking and biking and telling people how far it is on distances and those types of things so they know where they're going and what to expect. So the next step, so we see this plan is being implemented with a variety of grants and CDBG community development block grant funds that we have. So the idea would be to start in a document later in the agenda, there's allocations for the 2020 block grant that has streetscaping funding in there. Some of this stuff will be funded with that. And then the plan is to work with representatives of the districts in having, developing meetings and partnerships to advance these placemaking in the four districts. And then I will leave you with the project for public spaces and that placemaking shows people just how powerful their collective vision can be. It helps to reimagine everyday spaces and to see a new, the potential of parks, downtown, waterfronts, plazas, neighborhood streets, markets, campuses and public buildings. So that's what I have. If there's any questions, I'm happy to answer them but the planning commission has unanimously approved this and the document is before you tonight for adoption and then we will start implementation. So thank you.


We call the tooth a helper and that is very true for me. When I see a problem, my first question is how can I help? How can I help? As a justice worker, I need to be very careful that I am helping in a way that actually helps and not in a way that just makes me feel like I helped. Often the thing I need to do first is listen and that is difficult for the helper.


Every week Dr. Jekyll hosts a dinner party for several friends including Mr. Utterson. One night Mr. Utterson stayed after the party to ask Dr. Jekyll questions specifically about his will that he didn't like. Now Mr. Utterson told Dr. Jekyll that there is something strange about Mr. Hyde. Dr. Jekyll said, you just don't understand and I ask that you help take care of him if something were to happen. Even though Mr. Utterson did not like that, he agreed to help.


Hi, everyone. Welcome to today's product school webinar on breaking into product marketing. I'm Annie and I'm a senior product marketing manager at Pinterest on the brand safety team, working on tools and features to help keep our platform more safe and positive for advertisers. And today I'm joined by a wonderful panel of product marketing leaders, and I can't wait to hear and learn from all of their experiences and insights. To kick things off, let's start with some brief introductions on our panel today. Mohit, why don't you kick us off? Sure, Annie. Hi, everybody. So I'm doing currently product marketing for Revolute Business, the B2B side of Revolute. Yeah, and I'm based in Paris, so happy to be sharing and discussing with you all today. Great. What about you, Miguel? Hello, everyone. I'm Miguel and I work as a product marketing manager doing brand reputation and consumer apps for Google in Spain and Portugal. I've been working with marketing for more than 10 years, and yeah, I'm happy to be here, thank you for inviting me. Happy to have you here as well. And last but certainly not least, Lori, would you like to introduce yourself? Thanks, Annie. Hi, everyone. I'm Lori, and I'm currently based in San Francisco as the lead product marketing manager for WhatsApp's business app products at Meta, and I'm responsible for managing the monetization and growth of WhatsApp business. I've been at Meta for close to four years now, and I've been in marketing in general for over 10 years now, and specifically in product marketing for over seven years. Happy to be here. Great. Thanks, everyone. Well, we have a few questions prepared today for our panel, but please feel free to input any questions into our chat, and as a disclaimer, we want to remind everyone that each panelist represents their own views and not those of their companies. To start, I'd love to know how each of you got into product marketing. I remember when I first started in my career, I hadn't really ever heard of product marketing before. It was a fairly new space, and it took me some exploring to finally land into an APMM role. I'm curious what all of your paths were like to get into product marketing. Why don't we start with you, Lori? Sure. So, as you mentioned, Annie, I think it's safe to say that none of us really grew up aspiring to become product marketers specifically, just because the field is still relatively new compared to other roles in tech. So I personally first learned about the product marketing role when I was job hunting for internships during business school here in the U.S. Before business school, I was working in Japan, and I was in consulting, focusing on advising clients on their marketing strategy, as well as, like, target segmentation based on data analysis. And I would say, you know, when people often pursue an MBA to change their career trajectory in at least one of three ways, that would be geographic location, role, or industry. And so when I was searching for internships and full-time opportunities out of business school, I knew that I wanted to change all three of those. And I was laser focused on pursuing a marketing role in tech in the heart of Silicon Valley. And so I was naturally drawn to the product marketing role as most openings that I was looking at really required or strongly preferred an MBA or a consulting background. And I ended up landing a product marketing internship at PayPal. I really enjoyed it there and was successful to get a return offer. And so I joined after business school, and I've been in product marketing ever since. Thanks, Lori. What about you, Miguel? What was your path like? So I commented on the comment we received. I studied public relations, and I started my career as a planner at an advertising agency because I wanted to work with advertising. And then I spent around two years working with clients doing like general planning for clients in Brazil. That's where I started my career. Then I spent some more time working as a business development for a tech company. And at the time, as well as Lori mentioned, I didn't know what I would end up working with as a PMM. I knew what was advertising. I knew what was marketing in general. So I decided to leave the agency side and started working in clients and companies. And also I wanted to work in the tech industry. So I looked for marketing roles in general. And I noticed there were a few transferable skills I could use to learn this role. And that's when I applied to a marketing role at Pugone in Brazil. I was accepted. And at the beginning, it was a pretty generalistic marketing role working with brand and other topics. But then kind of it started as product got prioritized, projects got prioritized. I was kind of being pushed to product marketing. And that's where I'm now. Great. Thanks for sharing. What about you, Mohi? What was your experience like? Yeah, I think as everybody else mentioned, product marketing is fairly new. And when we started, it was not really a space which we aspired to be in. So my experience is a bit different than you all here. I'm an engineer, a turn market here, essentially. So I started off as an engineer. Then I moved into business as I gained interest into actually understanding the business, how it works and marketing side of things. I actually started as a consultant for SAP. So doing tech consulting for a couple of years before doing another kind of marketing degree to really move into the marketing space. I think this blend of interest and tech and marketing together coupled with education, both sides tech as well as marketing. When I was looking out for my first job after my business school, I was like, this is the space which could fit the best in which I could offer the most in terms of to a company. So the blend of marketing and tech led me to my first internship in a company called Amadeus. So it's a B2B tech company based in France as well. And from there, slowly you progress into the role and I've been doing it for the last eight years now, many in Amadeus and then at Revolute. So yeah, I think I would say that this blend of interest, education along with some experience from different roles as well is very much applicable to you actually being a product marketing manager and especially the side of customers in the city which comes in being a consultant. I've seen plenty of other PMMs also who have drawn from their experience in consulting in various ways into actually what we do today. So I think that's a bit of my story. Great, thanks for sharing. I think as we all heard from the panel here, we've all had different experiences getting into product marketing. I know for myself, I started off in marketing for a telecommunications company and I slowly start to see that there is a lot of transferable skills that would apply to the PMM world. And so I think my next question is what are some of the skills or experience that you think was very key in helping you land your product marketing role? Let's start back with Mohit. Yeah, thanks, Annie. I think as I was mentioning, there are a couple of skills which are extremely important for a PMM role. I think the biggest one of them would be customer centricity because everything we do here in this role is about the customer and actually how we can solve their issues, understand them and eventually make the product adapt in a way which answers those needs long term. So that's one of the main skills which is important and a lot of jobs out there offer this skill like sales, like consulting, which you have to be in this position where you actually work with customers very often and understand really their need at the core. Two other things which I'll mention is collaboration. This role sits at the heart of a lot of other functions, other teams. So it's very rarely that a PMM would drive everything by themselves, but rather working with a blend of other teams which are cross functional. So you can be ranging from owners, product managers, to the sales department sometimes, to the social media team, to the PR team. So really at the heart of the thing and you really kind of orchestrate the way there's things with other people. So building relationships with them, connections with them and really managing this collaboration is super important. And the third one is problem solving. Because in this role, I think in my day-to-year job, I'm mostly solving problems for customers or internally how to operate as well in the team in the company as well. So I think the problem-solving part of it is also super important to really be able to solve challenging problems for your customers and build ways in which your product also answers those questions for customers. So it would be my top three skills which would be important in a PMM role. For sure, I think all of those were really important transferable skills. Miguel, did you have anything to add in terms of any skills or experience that you think would be key? Yeah, I agree with Morit. I would also add thinking about in a scenario of people that are trying to start off a career already knowing that this is a role that you can do. I would say that having a general understanding of marketing is something that you should have like marketing strategy in general communication strategy what is a research I don't to start off the career I wouldn't say it's necessary to know how to conduct a research and to end but at least have a general understanding of analyzing data how it plays into my product into my company and as Morit said like a basic understanding as well of I would say product and sales of the industry you are applying to so how does this industry make money how is a general product in this company in this industry structure what are their most important features how can we help users to learn about these features so thinking about I don't think you need to have a specific previous experience in the area to start of course to start a career for example I have many colleagues that came from all the areas the transfers from for example the health industry so perhaps being if you are a red in the health industry you might look for roles in your industry for PMMs and then you already will know a lot about the business a lot about how the industry is a structure so you have a lot to contribute and then you have less to learn in that role so I would say those would be where I would focus if I was trying to start a career as a PMM now and try to look for as many transferable skills as possible because this is not a career at least when we started that you start developing the skills for that career specifically you build from other roles and then you start in that role certainly last but not least what about you Lori what are some key skills that you thought was really helpful to help you get into product marketing yeah I definitely echo what Mohit and Miguel said already but I think for me specifically the role, the skills that really helped me to break into product marketing was really a culmination of my prior work experience so there really isn't I think the beauty of product marketing is that there's really no one way to get into product marketing which because there's so many different sets of skills that really come into play as a product marketer so for example my first job answering customer support phone calls really heightened my ability to empathize with customers and understand their pain points the skill needed when you're trying to understand how to improve products and really deliver customer centric products as Mohit was saying as well and then my consulting role really honed my competitive analysis skills strategy setting, critical thinking skills and then business school prepared me to conduct both qualitative and quantitative customer insights research which is a lot of what we do and also gave me the foundation for understanding marketing frameworks and setting go-to-market strategy and I'll also say the skills that you really need to break into product marketing is not necessarily the only ones that you'll need to succeed in the role so you really need to be able to balance these skills that helped you get into product marketing to execute on like inbound and outbound initiatives but also you really need the ability to drive influence and deliver impact at the end of the day as we are all here to deliver outcomes for our products so if you think about in the 3P framework like product, process and people you really need to understand the product and its customers really well it includes understanding and setting the product vision strategy and being able to influence the product roadmap based on your inbound work and understanding of customer needs and on the process and people side as Mohit mentioned as a PMM you're really collaborating with so many cross functional stakeholders and you act as the quarterback between these teams and so you should be able to identify gaps in the current processes optimize them for more efficiency and help people help different teams speak to each other to deliver these outcomes I think that's a really great point you made about not only focusing on your transferable skills that you had before but also looking to how you continue to develop and grow as you get into further product marketing experiences and with that I know you've talked about our experiences in the past our different paths to get into product marketing my next question is what ultimately had you decide to stay in product marketing for now and what do you like or not like about your roles within product marketing why don't we start with Mohit I think why I got into it and why I really like doing what I do there are two principal reasons one is as I mentioned I'm really customer obsessed so I really love this challenge of understanding customers and really doing the process to understand and build things which can really help them at the same time also deliver messaging which is really impactful for those kind of customers at the end aiming for their success so that is something which drives me and along with this what happens is that I really like the impact you can drive in the organization as a PMM so whatever I do is very tangible so you are having metrics which are associated with whatever you do and those things are very tangible so the overall job remains dynamic very dynamic and ever changing which is a bit of the challenge as well to manage I would not say I would hate that that is the challenge which you manage on a daily basis but at the end when you get results out of it which is in terms of customer growth in terms of revenue growth those are the targets which in the end when you see impact which is tangible and really affecting the quality of the company that is where I think you keep getting the motivation and drive to do that better and better over time so that's the idea so I think for me those are two things which really I enjoy doing this customer side of things success of the company and customers together with very tangible outcomes thanks for sharing Mohit what about you Miguel for me it's really like to understand users behavior and consumers behavior in general so that's and understand how we can use what we learn from from our users and then do something with that that is actionable similar to what Mohit was commenting and that possibility of being a role that you are kind of in the middle of many different areas and you can you have in your role the ability and the skills to bring users inside and understand behavior from your users and get to understand what other areas are talking about what are their pain points and activate that with what you learn from the users you want to achieve that's what I like the most about the role so it's like kind of a process you don't do this every day so sometimes you are having having conversations for a month with the product team or the sales team and then when you do some explorations with users you understand something that you can do so that piece of being in the middle and having to understand from every part of the business and activating this in a way that is as Mohit said, tangible also with kind of a marketing magic to the users that translate everything that it's kind of translating everything that comes in a business language that you are getting inside in a way that the user will see and think about oh okay that's the brand I know the brand I believe and look how cool it is that what they are doing for me that job, that task, that process is what I enjoy the most thanks Miguel and what about you Lori why did you decide to stay in product marketing and what do you like and not like about the role yeah I think very similar to what Miguel and Mohit said but what really keeps me in this role is being able to speak directly with customers to understand their pain points and needs and really seeing the joy on their faces when we launch valuable products and features that actually improve their daily lives so for example at my current role I have the opportunity to visit multiple different countries where we are where we have the most presence in our product and literally going to these people's homes or their businesses and seeing how they use our products and understanding how we can improve their products and just hearing their stories about how they use our tools on a day to day basis is really what drives me at the end of the day the other thing is that with the product marketing role you are kind of like the jack of all trades you can wear multiple hats or you need to wear multiple hats and you just gain so much different kinds of experience and skills that you never thought you would be doing for example I had no background in pricing but as my role in figuring out how to monetize the product I've had to understand how pricing strategy works what are some ways that pricing models that we can explore for our product for example and actually setting the price do a lot of content writing for comms or conference type PR work, do a lot of branding strategy, product naming design, you name it literally PMM can delve into any of these areas so you can really lean into your interests and make the role what you want it to be I think I saw a lot of questions also around what is the difference with product manager or product management I think the difference is that well first of all I think product marketing and product management is very closely tied together I would say product marketing in my role at least we act as the right hand man to product management and we really worked hand in hand to figure out what is ultimately best for the product but I think the difference is what is each group or role responsible at the end of the day the product marketing manager you would obviously be influencing the product roadmap with all of the customer insights that you gain through your inbound work but ultimately the PMM is responsible for finalizing that roadmap for example or product marketing we have a lot of cross functional stakeholders who help with the go to market land, the product launch moment but ultimately product marketing is the one who is responsible for making sure we get that word out there setting the value propositions and product positioning and really delivering that narrative out in the market Thanks Laurie certainly I think having a strong relationship between PMM and PMM is very key when it comes to launching features in the market I did see another question in the comment section how is product marketing different from marketing itself Miguel did you or Mohi did you want to jump in on this one Yeah I can go if that's fine I think that there's one key difference between the two which is that on the product marketing side as Laurie was mentioning you're really accompanying the product very closely so you are able to in a way influence where it goes long term in terms of strategy in terms of actually what is in the product whereas on the marketing side it can be a lot of marketing you could be doing brand marketing you could be doing comms you could be doing other kinds of marketing but not really influencing where exactly a particular product goes and why it should go there so this is something which is I think a core difference however the skill set you would need to execute both of them is pretty much also marketing you would need to still be able to drive you know create narratives create value propositions create messaging which is I think across the marketing domain but the key difference is that you are really accompanying the product through its development also it differs between different companies how it actually would end up being but it can be the case where you drive product creation from the beginning from the business case of why we build this up to the end taking you to market whereas the other things that could be you know just taking you to market and not really delving type into the product Miguel if you have something to add maybe you can add as well yeah no I agree the only the only thing I would add is that I think about people looking for a new role what I would do is not trust as much in the job title because you can see sometimes a role that says product marketing manager and then when you read the description it is like brand marketing or it is as comes marketing so it depends on what you want to what you want to do so the same for the opposite right if you find a role that you should want to do product marketing and you find a role that says brand marketing looking to it because sometimes as it happens in any industry in any role the person writing down the description sometimes doesn't have the full visibility of each of the the role so just wouldn't wouldn't try to I would try to check the job description and not focus only on the on the title whether you are looking for product marketing manager role or marketing role that is not product marketing manager and that's a great point just to add on that what I've seen also is that in European markets the PMM role is not that evident at the moment so in US you would see a lot of roles which are just say PMM in the heading but in European side of things you might come across roles which are called product line manager or only managing for the product which is only marketing manager but inside if you go in the details you would find there are roles which are actually PMM just that the heading is not the right one because it's not that popular at least at the moment I definitely want to echo what you guys both said there's definitely different flavors of product marketing and it differs by company in my past work experience I had roles that were more focused on the go to market side more heavily on the marketing and launching the product whereas right now my role is more focused on influencing the product being closer to the product manager and understanding customer pain points and so when you are looking at roles to apply to I would definitely ask the hiring manager what is the balance of inbound versus outbound kind of responsibilities some roles are very hybrid where they have a 50-50 kind of outlook whereas others are much more leaning heavily on one side or the other Thanks everyone I think it really speaks to the flexibility of being a product marketer and how you can really flex and adapt your skills to what you're more interested in as you pursue roles within product marketing I know we only have two minutes left why don't we do a quick wrap up perhaps one final question I'll measure success as a product marketer I know we are in touch with so many different teams that we have so many different objectives curious how we measure success for each of you why don't we start with you Laurie? I think at the end of the day because we are such a customer centric role we want to understand we want to make sure that the end user is experiencing the product as you intended to be you want to delight your customers you want to make sure that you're delivering a valuable product so that's how I would set my goals as a product marketing manager but also on more of the B2B side it's often tied more to revenue goals so even as a PMM you know if you're more in a hybrid role as a product being close to product as well as on the sales side I think your efforts are measured with how much revenue you were able to drive at the end of the day through your go-to-market activities Great, Rohit what about you? Yeah I think it really depends on what the company objectives are from my experience I have seen two very different kinds of operations here happening in my company there's a high chance that there's less space for growth and the objective is really adoption and really churning most money from these in customers which is what I've seen in my old company the objectives are really based upon adoption of the product so users who are using it and how they are adopting what you're releasing and how you can make the most revenue as Laurie said from this customer base on the other side of things in a more hyper growth environment what we really focused on today is really growth so we are looking at customer numbers lead numbers, how many leads we are generating which are the indicators of success of the activities we are doing because today our most activities focus really on the outbound side of things to really get people to enter the funnel in various ways so it's about customer growth and number of leads so that's essentially what we do Great, thanks Rohit last input for the day Miguel what about you, how do you measure success as a PMM Yeah, I would say that agreeing with Laurie and Mohit it depends on the product we are working on and what's the goal for the product so I would focus on the key metric of whatever I'm trying to achieve being revenue, being reducing churn anything that we are trying to achieve with our strategy but then as a PMM as well one thing that I measure personally is the quality of my relationships in the company I'm working on because as we discuss here we are in a central role interacting with many different areas so if the relationships are not in a good place doing your job is going to be pretty difficult so as a PMM in the role I also have this regardless of the product of the project I'm on how I'm handling the relationships I need to have for what I'm trying to achieve Thanks Miguel Well I know we're at time but I want to thank everyone and all our panelists for sharing their valuable insights and thoughts and for everyone for tuning in today Thanks everyone Thank you


Hi, I'm Olga Blavich. I'm the president of West Pomeranian region in Poland. When I was young, I lived behind the Iron Curtain and common Europe used to represent for me a dream of freedom, peace and prosperous society. Now I'm happy that we are the part of this reality. For me, European Union is a project built on our roots, outstanding European heritage, strengthened by wise conclusion because we finally understood that living in peaceful Europe led us to develop and provide our citizens with prosperity, solidarity and equality. Of course, nothing is perfect. We have to still improve our union, but I have no doubts that it is the best idea has ever been created for Europe. In my region, there are plenty of examples of benefits from Europe, from beautiful field harmony in Szczecin to good support project, the best European social project awarded with registrar's prize. Although we celebrate Europe Day separately, we, cities and regions, stand together for common Europe.