Hello and welcome to this introduction to microphones. In this video I'm going to cover the details of the PDM to PCM conversion. PDM to PCM conversion is the set of processes and methods involved in the conversion of a single bit PDM output to a standard digital signal in PCM format. Several steps are involved in the process. We have a low pass filter, a decimation phase, a high pass filter for DC offset removal, and finally a gain stage. We will go in deep in each one of the involved steps to see why it's required and will give a general idea on how it can be implemented. A low pass filter is a filter that passes signals with a frequency lower than a selected cutoff frequency and attenuates signals with frequencies higher than the cutoff frequency. The low pass filter is required in PDM to PCM conversion to remove the high frequency noise, which characterize the PDM output and it's used also to avoid aliasing and noise in the resulting PCM. Since we are dealing with digital signals, we are going to use a digital filter for this stage and specifically a finite impulse response filter or FIR. You can think at the operation of FIR filtering as a moving average. Each output sample is computed as an average of the last and input samples. So this is the first sample, the second sample, the third sample and so on and so forth. The main difference with averaging is that the input samples are not weighted equally as it would be with a simple average. But specific coefficients are used to weight each input sample used. The FIR filter then can be thought as the set of specific coefficients to be used to weight the input samples in order to obtain a specific effect on the output. Let's see this filter in action by analyzing how it works on a hypothetical signal. The first PCM output sample will be the sum of the product of the first and input samples with the filter coefficients. The second PCM sample will be the result of applying the same operation on the next and samples of the input. The third output sample will be obtained by applying the filter on the next and samples and so on. So at this point the output signal is filtered and the high frequency noise we saw in the original PDM stream will be mostly attenuated. One important thing to be pointed out, which is common to all FIR filters, is that each output sample depends only on some of the input samples. There is no dependency on previous output. The second operation to be performed is the decimation. Decimation is the operation of discarding sum of the samples of the signal in order to reduce the sampling frequency. As we saw, the signal must be filtered properly in order to avoid any issue of alias. More in detail, a decimation ratio equal to M means to keep one sample every M samples discarding the others. For example, given this hypothetical signal and using a decimation ratio equal to 3, we are going to keep those samples and eliminate the others. The resulting signal will have a sampling frequency equal to the original sampling frequency divided by the decimation ratio. As we saw in the operations of filtering and decimation, we compute some PCM samples that are going to be discarded just after in the decimation stage. In order to optimize greatly the process and avoid unnecessary processing, we can simply avoid to compute the samples that will be later discarded in the decimation phase. This is possible because we are using a FIR filter and each output sample depends only on the input samples. There is no dependency on previous outputs. In this way, the process we went through previously assuming a decimation ratio equal to 3 will become compute the first sample, do not compute the second sample and do not compute the third sample because those are going to be discarded in any case, compute the fourth sample and so on and so forth. In this way, we avoid wasting processing power in computing samples which are going to be discarded. The next step is a high pass filter. A high pass filter is a filter that passes a signal with a frequency higher than a certain cutoff frequency and attenuates signals with frequencies lower than the cutoff frequency. In the PDM to PCM scenario, this high pass filter is used to remove the DC offset which can be present in the output signal. When a DC offset is present, the average of the signal is not zero as it should be, but rather it is centered on a positive or negative value. A simple single pole recursive filter is enough to cancel this effect by filtering the frequency below a specific cutoff. Usually this cutoff is below 20 Hz to not impact the signal inside the human listener range. Gain is the process of adjusting the level of the audio by increasing or decreasing the amplitude of the signal. This can be simply done by multiplying each sample by a constant value. Gain is used to focus on softer sound which could be too low in the overall PCM range. It can be described as the volume of the recording and it must be added properly in order not to reduce the PCM resolution. Let's now briefly focus on how to tune the parameters in order to obtain a PCM stream at a desired sampling frequency. So we have three parameters involved. The first one is the PDM sampling frequency. The PDM sampling frequency is equal to the microphone clock. Please note that there are always limits to the clock frequency that a microphone can accept. The second one is the PCM sampling frequency, which is the desired frequency of the output PCM stream. Typical PCM frequencies are 16 kHz for voice acquisition, 32 or 48 kHz when a wider spectrum is needed. The last parameter is the decimation ratio, which is the ratio between PDM and PCM sampling frequencies. Depending on the way decimation is performed, for example with a software library or with an hardware peripheral, not all the values for the decimation ratio may be available. Typical values for the STM32 software library for example are 64, 80 or 128. The PCM sampling frequency is the ratio between the PDM frequency or the microphone clock and the decimation ratio. So here are some examples. If you want to obtain a PCM stream at 16 kHz, you can use a clock equal to 1.280 MHz and the decimation ratio equal to 80. You can obtain a PCM stream at 32 kHz using a clock equal to 2.048 MHz and the decimation factor equal to 64. You can obtain a PCM stream at 48 kHz using a clock equal to 3.072 MHz and the decimation factor equal to 64. In general, microphone clock and decimation ratio can be tuned to obtain the desired PCM sampling frequency. More information are available in these documents. Look for the next videos in this series on the ST YouTube channel. Thank you.


Remember when I used to have to set that up? We're very mustard today. I like a couple of Dwights. You consider this to be mustard? Yeah, together. Baby poop.  I'm Rick, he follows on Instagram. And Twitter for more juicy content. It's so juicy. They both just hate it. I'm a fellow official of Twitter account. Ring the bell to be part of the notification squad. Bang! Ring it. Ring it. No, no, no, no, no, no, no, no, no, no, no, no, no. You can do it. And today we are doing a movie review. Rick, did you watch a movie? You were supposed to. You know I did. Anyways, we watched CyRot. It's right. Is that how you do it? I know it's a CyRot. I think that's right. CyRot, okay. Which we've actually kind of already seen the movie based on how many of the songs we have reacted to from the film. Yeah, and this is our second Marofi film. Yes. It was a long time coming, we know. I think the other film, there's several films we've done this with. Well, it was the Hydroaniac DiDewaniac. That's the one where we'd been reacting to the songs for like a year and I think we'd seen every musical number and then we saw the film. I think the same thing was true with this. I think we've seen every musical number from the film and just hadn't seen the film. Yeah, and so sorry about that. Yeah, so now we've seen the film. But if we had not reacted to those songs, you wouldn't be able to see and they are beautiful songs. And we had wonderful reactions to the movie, we may say so ourselves. And so we reacted to them doing the orchestrations in Los Angeles, which was the first time that that had been done for an Indian film. I think I got blocked. Oh, then it was just Patreon. Sorry, I can't remember if it was just a song. But yeah, that's fine. It is a, read that. The synopsis is in Interior Maharashtra, a fisherman's son and a local politician's daughter fall in love against the restrictions of caste hierarchy. Directed by? Waiting for it to pop up. So I can read it properly. Nagraj Manjul, who did Fandrey as well. And then starring, I'm just gonna say the main two. Yeah, Rinku Rajguru and Akash Fosar. And I don't know if you wanna say the other because the friends and then the... Yeah, there's like three other friends as well as her dad. That's really... There's the two main leads and then an ensemble of friends and family. So finally watched it. Initial thoughts for them. I wanna know your initial thoughts because I have a feeling I know what Corbin... Okay, by the way, we've gotta just be spoiling this right off the bat, just letting you know. If you haven't seen it and you don't want the movie to be spoiled for you, go away. I... The minute the ending came, I said out loud, Corbin will love this film. Am I right? Yeah. But not just because it was a really good film. Like, we could talk about the ending in a minute, but... You're gonna have liked it more than I did. Really? Yeah. The whole why? Why didn't you, what was it? I didn't dislike the ending. I knew it was coming. Did you know it was coming?  Yeah, the minute the dudes showed up, I'm like, bat, these tigers are showing their stripes. This is not gonna end well. No, not at all. So when their little boy walks in, I'm like, yep. Saw that coming. Yeah, I did love what the director did with it though. I did too. The ending. You hated the ending, as far as just...  That's just me. I know, but... The realism. Yeah. But what I particularly liked in the reveal was the way it was done from the POV of the son.   That was awesome. Yeah. It was a great decision. You didn't need it at all. And I will say, before I had a son, I wouldn't have felt as emotional during that. I didn't cry. Oh, I was gonna say, my wife, she bawled. Okay. She was bawling. But I think before I had a son, I would be like, oh, this is awesome. I was more like, oh, this is cool, but mm, that would be awful. That's it. That's it. Orban Miles kind of ended. So, yeah. Let me tell you everything I liked about the film. Okay. All right. And then I'll tell you the couple things I didn't like. Okay. The things I liked about the film. First of all, the thing that towers and soars above everything is the score in the music. Oh, the score, yeah. It's just... And we did that in the fact that they did it in Los Angeles. And the strings of the entire thing was so unique to anything we've ever seen from India so far. And it was so, so... So I love the score. Love the score. It's just quite probably, for me, the most memorable in terms of its overall impact for me in listening to this score. And a big reason for that is because we reacted to so many of the songs on their own, including the one just purely orchestral piece, which is for the beginning of the film. And ding, ding, ding, ding, ding, ding. Yeah, yeah. I just think it's so just freaking gorgeous. It's one of the most beautiful scores I've ever heard. Yeah. Absolutely beautiful. I thought there were also moments where the cinematography matched the score. I thought there were moments where the cinematography was really pretty. There were never moments in the cinematography where I didn't like what I was seeing or felt that there were mistakes. For the most part, I felt everybody was believable most of the time. I'd say 90% of the time, I believed everybody. I found myself caring about the guy and the girl. I didn't find myself. I felt the direction was solid with the writing. I felt all of that was very believable. There were things about the story we'll talk about that really bother me, that I'm sure bother you as well. The things that didn't make it a film that I love were I felt the first hour could have, as beautiful as the score was, it didn't need to be three hours for me. It felt longer than it needed to be, which is just one of the things that contributed to me not loving it. The biggest contributor for me were moments where I didn't believe the emotional complexity that I think the actors needed to portray when they were crying. There was one moment the girl had that I was like, thank you, I believe you there. And the other moments, whenever he or she had to cry, I didn't believe they were crying. I saw them indicating and that always really bothers me and pulls me out of a story and it's hard for me to get back in and it happened like multiple times. So that for me is the biggest reason it wasn't a film that I loved because I found myself on like four occasions going, I don't believe you're crying right now. And I wish I was on set and could say, okay, stop. Let's take time to work on this and see what wall is causing you to do this or why you haven't learned how to cry on cue the right way and the indicating of wiping tears away that aren't there is just stuff that totally pulls me out. But she had a beautiful moment on the phone with her mom. Well, I thought she had, I really actually enjoyed her performance and what the writers and the director did with her entire character. I thought most of the time they were believable. Really, really brilliant. Most of the time they were believable except the crying points for me, but those are like our like critical moments. Yeah, but they're critical. Yeah, it's three hours long. I know, but those are the critical moments. It's three hours long. But anyways, I loved what they did with her character especially. She in his too, because I think the reason I was, it was long for sure, but I think the reason I was okay with it was because it was basically a lifetime that they were showing us. And I loved the arc especially that she had because she came in and I loved her character. And I really liked the way her character was written. She was just a little badass heck.  I was so rooting for her to shoot somebody. Oh yeah. And so I loved that whole dynamic of, because just off the bat, you normally think that the guy is the dominant one and that's just normally how it goes, but it's refreshing to see. At the beginning at least, it's that way. Agreed. And then I loved the way they wrote her character into this arc of she's really strong-willed and she knows what she wants at the beginning, when she's in a very direct, and then when she gets out of it and she's like, okay, let's run away, she thinks it's gonna be like the movies and she's gonna be, because she loves film and so I'm sure she's seen people elope and run off into the sunset. And I liked the writing of her becoming disillusioned and suddenly wishing she was back home. Yeah, I liked that. I liked that a lot where it went there and then she was regretting it when she didn't have that money anymore and so she wasn't in her comfort zone and she almost became a totally different person. And then, well they both did. Yeah, yeah, that whole scene where they're both being just terrible to each other. I have one scene, he's doing it and then the next scene, she's like, okay, I'll make it better and then he doesn't wanna do it. Right.  Right.  That she returned and then it moved into them as older and they've had a kid now.  And also the last thing we watched through Dan, this father makes that father look like the father of the year. Yeah, I don't think that dad would have killed this kid. Now here's something about the film I don't like, but let me clarify what I mean by that. What I'm about to say is not, I'm not saying I don't like the film and think the film is less because of this. It's a subject I don't like. It's something that exists in the world that was portrayed well in the film that I don't like. Does that make sense? It, so because I've said some things about like the biggest thing that caused me to not love the film were those moments where I saw the indicating rather than believability with the crying. And the subject matter, I get a visceral response when I see, I don't call them men, males, slap women. Whether it's a dad slapping his daughter, a boyfriend slapping his girlfriend, a brother slapping his sister, or a husband slapping his wife, it is one of the most repugnant things in the human race that I can see. And what, I know it exists. I know there's a culture that lives the way it's portrayed in this film as our honor killings. There are cultures and they're in all over the world where dads and brothers think and uncles, it is a actual like pleasing to God to murder their daughter for disobeying them in the way that she did. And I find that so transcendently repugnant. It's not one of my reasons for not loving the film. Does that make sense? No, but okay. I'm probably not doing a good job explaining it. What I'm trying to do is talk about this topic without it making seem like I didn't like the movie because of it. I don't want it to equate to that. I'm just pointing out- I don't think anybody likes those things. No, I don't, it's true. Well, I think it's like you're explaining like I don't like murder. No, there are men who, I guarantee whether they're Indian or American or German or wherever, who will watch this film and they'll be like, yeah, put her in her place.  I hate that what's depicted in the film in the same way, for example, this is the grand example. I really, what I was gonna use is Schindler's List. I hate what happened in the Holocaust, but I love Schindler's List. I hate that that actually is going on in the world in places, but it didn't change my opinion of the film itself. That's what I'm trying to articulate. Gotcha. Does that make sense now? It still doesn't make sense? Okay. It's like, to me, it sounds like you're explaining, I don't like murder, that's what I'm saying about this film.  Yeah, but that's a different thing. I don't like murder, but I don't have a visceral response to it the way I do slapping and abusing women. Like the moment he slapped her in the restaurant. Oh yeah, bad. I wanted him to have, when he's on the street with her and he's grabbing her wrist, I'm wanting guys to show up and break his legs. I don't like him anymore. I'm not rooting for him anymore. And when she goes back with him, okay. Now guys can make a mistake, not that mistake. Sorry. Sorry. Oh no, I'm fine, but you also liked Kabir Singh. He made that same mistake. Did he? Yeah. He slapped her? Oh yeah, he did. So that's what I'm saying. I don't, I don't, I- Wait, when did he slap her? I don't know, he's- In the street. In the street. He slapped her in the street? He did. And then he goes to her house. Oh, I think I know why. Yeah, I think I know why. Yeah. Why what? And I had the same response. I had a visceral response to that. No, no, I agree. I'm slapping, it's terrible.  I don't know. But it doesn't impact the way I feel about, I don't know if I'm doing a good job explaining it. No, you're not, but we can move on from that. Okay. My main thing about the film that pulled me out of it multiple times was my wishing the two leads who were believable most of the time and I enjoyed the majority of the time didn't have the capacity and whether it's their fault or not as a material to me. I really wished I had seen actors that could have gone into the depths of the emotional complexity that needed to be portrayed in those moments that their default, which is natural for somebody who can't go there was indicating because it just, I get completely pulled out of the believability of the film for me. That's my biggest gripe. But I did really enjoy the directing part of it as well because even though it was three hours, it was almost three different films essentially in those three hours. The first one was the first hour was like the love story, them falling in love. And then it's them in that relationship and then trying to run away. And then the last little bit last 30 minutes is them in their older life with their and kind of living their life now. And so I enjoyed that it helped the film move along and not feel as long. It is a long film. It's three hours, of course it's long. But then that's the only gripe because I actually really enjoyed this film. I thought it was great. Even without that ending, I would have enjoyed it. That just kind of was a little icing on the cake. I love it. So sincerely, you felt the moments that they had where they had to break down and cry. You were never, you were fully absorbed believing and drawn in with it and found it to be believable. No. Okay. But that doesn't ruin the film. It doesn't bother you. There's lots of films we've watched that has bits and pieces of bad acting, especially in the crying department.  So, well, if I could do that, it has over emphasize but it doesn't have indicating in the crying. I did not have those same overall issues that didn't bother me. That's fine, like it bothered you. I would definitely watch this one over Notts and Rott again. So the two of the two Marathi films we've seen, this is my favorite. And obviously that's not a lot, that's two. No, I would agree. I would agree with you as far as the enjoyability of the film. Especially so much was so enjoyable. And I thought the director did really well, especially certain shots he did were so brilliant.  But then also there's certain shots when they were passing notes or he was trying to get her to open the note. He kind of just panned over. Yeah, those were really nice shots. And it was all, it looked like one shot but it kept just going to different scene and it cut in between the walls. Yeah, it just kept panning. So a lot of shots in the cinematography I really enjoyed in this film. Another thing I really enjoyed was I felt that the supporting cast of his buddies were all 100% believable. There's been times where what I'm about to say has happened and I've had a really big ripe about it where there's multiple scenes in this where he in particular is getting a snot kicked out of him by the brothers and the family members of her.  They did a very good job of staging all of those scenes. I thought every single one of them was believable. The dad, he's such a terrible, terrible person. Oh man. Just such a bad person. Well, and that too, we didn't discuss this in addition to the slap bank. It was a great scene when she called her mom. Yes it was. That was a really good scene. I was actually hoping they would cut to the mom more because I wanted to see what she was going through.   Or it just happened without her knowing. It may have happened without her knowing. She may have given them a guess. She clearly told the dad about it. She may have been in on it.  True. And so I knew that he was done with it. True. And so I was like, why are you calling your family? And so I was mad at the girl at the end. Me too. Because you were so smart for most of the entire thing.  Yeah. But the two other things that we didn't really talk about that are portrayed in our poignant, primary point of the story are the caste system and the differentiation, which is a universal thing because whether it's caste, it'll be social strata. It'll be economic strata, where people are prejudicial toward other people. And I correct us, if we're wrong, I think it does a pretty good job of portraying what it really is like in parts of India where there is still prejudices based on caste as well as corruption that can take place in any particular place where you have power, but particularly anybody who's in politics. Like that scene where his son slaps the teacher and then is like proud of his son for doing it and lets them know he's gonna get away with that because I'm a guy of stature and power in the village. Yeah. Can't relate to that here in America. Yeah. So, yeah, all in all, I thought there was more to like about this film than not and some things about the film to love, some of the best overall score and songs of anything. It's especially like, there's songs like Yajwani or the one we just watched, Yajwani, Higuani, that's what it was called, right? Yeah. Those are just fun, big, huge Bollywood numbers. These are some artistically just beautiful overall scores and just songs. Agreed. And they were already on my playlist. Yeah, I agree. I think this is a better film than Not Some Rod. If I was to pick between the two, I kind of would, there's no comparison for me. I just don't love it because the primary thing is the indicating moments that pulled me away, which I'm hypercritical than other people. So I get it. And if it doesn't bother me, it doesn't bother you. That makes sense. As much as I, again, I won't communicate this right.  But again, we need to have films that talk about it. So it's that love-hate relationship. The runtime was a little bit long for me. It could have, for me, helped if it wasn't as dragged out. And I think part of the reason it felt dragged out was I was dreading any other moments that were gonna come up of emotional import where they were indicating rather than really giving me something I was believing.  Ha, ha, ha, ha. I just hate that ending. Oh, no, it's good. It ended and the little kid's there, and I'm like, oh, and he was good too. Yeah, he did. He did do a good little kid actor right there. I liked him. I like, I think he did very well. And I don't know if the director helped him out by turning off the audio or what, but I thought it was brilliant to turn off the audio. And it was a great choice. I don't think you need any music behind that. No, I don't think you need sound to understand what that kid and your sympathy to that kid. And for a film that has been using score and music so powerfully to convey who these two are to silence the music, that's probably the best directorial choice in the film. And in a theater, it must have been shockingly stark to go from, especially for people who didn't see it coming. Anybody who didn't see it coming, this would have been a punch to the gut. And I appreciate their artistic license. I appreciate movies that end this way. I know you love it, but I like movies that have the endings that say dreams do come true. Realism. Realism. So very excited to watch another Marathi film. So please let us know down below what we should watch and review next in terms of, in general, and also what should be the next Marathi film we explore down below. Yeah.


Richard has a question about user adoption. So buckle up everybody. It's going to be right here. What are the biggest challenges your end users face when starting with Microsoft 365? So Richard likes the bold big topics, apparently. We're starting at a 10,000 foot level, not at a 1,000 foot level. Right. Anybody want to begin? Oh, I could begin, but I don't know if I'd ever stop. So you all just tell me when to stop. One is proper training. That's the thing that everybody thinks, yes, most of the applications and things you do are very intuitive. Or they've done it before. They've always used mail or they've always used this. Things are different. They work differently. There's new functionality, things that are built in that weren't there before. So training is a must. Also communication to them. So there's a lot of... I'm just going to use the word attitude maybe for users. The more they feel communicated with on what's going on, when it's going on, they embrace it a lot more. When things are shoved on them and they're told, they tend to not be as receptive to the change, right? And so making sure that you're communicating and getting feedback from them on things. It is always very key to one, ask for feedback, but then two, let them know you're actually listening. Two different things, right? So go solicit that feedback, let them know you've listened, prove it that you're listening to them. Hey Stacy, I'd like to say that one of the things kind of ties into both of these is... So years ago, one of the reasons why I started doing, initially with SharePoint, I had a session, some folks watching might be aware of this. It used to be like 10 features that you've never used and it started in SharePoint that you didn't know about, or features you've never used but should. And I haven't looked in a couple of years now, but at one point over a half a million views of just that slide deck, which is very popular there. But I got into the productivity tips because you have so many people that are familiar with, of course, the Office Suite and they're somewhat familiar with Teams and maybe SharePoint and OneDrive and the other tools, the other solutions that are out there. But they're not familiar with the latest. They're not familiar with all the integrations around that. So with the productivity tips, that's why I love that category of content in general, like from other MVPs and experts that are out there is because nine times out of 10, I've seen it before, I've used it before, but then there's that 10th one where like, wow, I didn't know that it did that. I didn't know it worked like that. That's how often the end user training is. People are bored to tears, they're going through the basics, things that like, I know this stuff, I know this stuff. Whoa, wait a second. I had no idea that I could do that, that it worked like that. And that happens to everybody in those sessions. So this is exactly why my presentation QuickWinds came about, right? QuickWinds, because people like, but they didn't know how to use it, what they could do with it, or these little caveats, right? I've had people in like my one hour QuickWinds session say that five minutes that you just showed me that, paid for my entire trip to this event, right? Because it made them more efficient, they were able to get more done. So and the cool thing is, is that presentation, I've given it so many times, I bet you over 200 times, right? Because it's constantly changing, because the technology keeps changing and new things keep coming out, right? You know, I've been working on a, literally a workshop, it's Microsoft 365 A to Z, which goes through the different apps and how you could use them within your organization and stuff like that. Well, what did you put down for Q? Actually for the Q slide is literally some like QuickWinds type of stuff, littler things. So yeah, but thank you, because it took me a while to figure out what the heck I was going to do for Q, but thank you. So what I want to say about this is looking at it from an aspect of not only educating the users, which I took a little different approach when I was doing some 365 migrations. I did a, I went and talked to people and found out what the problems that they have currently and tried to match them up with the solutions that 365 gave. And then I would show them that, hey, the problem that you have over here doesn't exist over here or the problem you had over here, you can work around it with this in 365. And that they became more receptive at that point of that transition. But I will say that there are so many variables in this question because number one, it says starting with Microsoft 365, where are they coming from? What environment are they actually coming from? Are they coming from an environment that's all Google apps? Are they coming from an environment that is like prehistoric and they're still using Lotus Notes and things like that? Which is the file share, right? Yeah, or they're just using a file share and they're using basic spreadsheets for everything. It all depends on the situation. And each one is unique. So when you say the biggest challenge is really the biggest challenge is identifying what they currently have and use and how they use it and matching it to what they can be doing, could be doing and should be doing in 365. That to me was the hardest part because it's not a one for one. Usually it's not. Usually it's not as simple as, oh, you're going from Google Docs to Microsoft Word. It's not that simple, okay? The two don't look alike. They don't work alike. They don't anything alike. That's like the features. That's like the user experience, the interface itself. Then there's, and Mike, before we restart recording, you mentioned this too is the migration of the data. Being able to, your adoption could be tied to whether you can find your content. So it's not just moving from that other location, those other environments, all of your data over there, but it's also then how you reorganize that content within the new system. So if it is different. Is it going from Google Drive into OneDrive? Is it going into, or are you going to introduce your users to SharePoint? We're going to move everything from Google Drive into SharePoint and going from Google Drive to SharePoint is a huge jump, huge jump. Going from Google Drive to OneDrive, not so much because it's just file shares, right? Not a big deal. But the thing about it is, is that there are nuances between all of them. I'm just using the Google Apps as an example, but realistically, I had a company that not only a year ago was still using Lotus Notes. That's why I brought that up. They're still using Lotus applications because they're strictly an IBM shop. That's it. Everything was IBM. So to try and match that up was like matching up a dinosaur kind of application to something that is off the bottom. A fighter jet. Yeah, it's crazy. So I was actually going to bring up that exact same example. So I've worked with dozens of companies that were Lotus Notes companies and moving over to the Microsoft stack. And one of the clients in very large financial services company, it was being driven kind of top down. The new CIO said we're going to Microsoft technology forcing the org and they had a lot of problems because people hadn't bought into that. So big part of it was they didn't take the time to really get the buy-in from the organization of the value of why we're moving, what it'll bring us. So they hadn't sold people on that. Like you need to have some degree of buy-in from the organization. Do that feature mapping. Show how things move across, how you move the data. Now, I was going to bring up another point that from the migration standpoint of all that content. Lotus Notes works very different. It's different functionality. This is just one example for a user adoption there. But what it was, the learning from a user adoption standpoint was that there's only so much that you can automate with tools to move on the back end that there has to be a strategy that is user driven to train people on the new features. They know where their old content is and it's almost, you can spend all the time trying to move it on the back end and make it all line up and map it the exact same way and tag things properly. Or you can train users on, hey, here's the new system. Give them time to do it, say within the next 60 days and then they go move their content. They do that. And they know where it is. They know where it is. So funny how when you ask the people who own the content to manage their content, how things get put in the right places or they recognize that I no longer need this content. So that's where training becomes important, right?  And they don't have views and all this kind of stuff. You train them and then you just show them examples, right? Examples of how their content can look in SharePoint so they can find it easier, all that kind of stuff. And they're sold, right? But you got to show them first. The other thing that I do is when I talk about the app, the mapping that Mike was referring to, is I find the things that they should start with because that's what they're using right now. And I turn all the other apps off so that they can focus on their current needs. And once that's then, as the company's deciding, okay, now we want to introduce these apps and then these and then do the training as you go on because you give them too many apps all at once, content is going to be everywhere and everybody's going to be doing it differently. Yeah. Funny how that happens. I know. You got anything, Hal? From my end of it, it's not so much Lotus Notes and things like that.  And in those cases, if you're moving, it's even a little bit more difficult there because if you're when you're moving from that area to full Office 365, they already have substantial familiarity with email and Word and Excel and things like that, but they aren't even close to understanding the differences between the two systems and the advantages and the extras and the other bounty brony points and stuff you can do that you couldn't do. Yeah. And that can be really confusing. Well, yeah, I've used this site, I've used Outlook forever and in Yata and so on. And why doesn't it? Well, it doesn't work here because it's a different place. And you've got different features and different functionality. It's the same, most of it is the same granted, but you've got different things in different places. And well, you've got it on the web. You've got it on your desktop. You've got it on your phone. A lot of that you couldn't do with the situation that it was. And there's a, in this case, not only do you have to learn, but you have to unlearn first. Yeah. Well, I was just thinking of how practically how we've handled this in the past.  Also, as those questions come in, and a lot of the questions are that mapping between the old and the new. And people ask those questions. You may not have it properly documented. The new system may not handle it the same way or handle it at all. Whatever that is, have detailed FAQs that are searchable that people can get through. Then on top of all that, one of the best tools in larger organizations, this goes back to my experience in the 90s, working for the phone company, was doing the Friday lunch and learns where you had users that had done the mapping themselves, and then they say, here's what I did and here's how I dealt with this change and this is what I learned about it and sharing that. And that was, and we had those, a couple, few companies, three companies that I worked for over the 30 years that did it on a regular, almost weekly basis, had the lunch and learns. And so people go in and they state like, here's the business problem trying to solve. And we had some people that, I just like, no one's gonna show up to that. Everybody knows how to do that. The room is packed. Like everybody has those same questions. So you can't, managing all this, assume that you know what questions people are going to bring. Document FAQs, have the lunch and learns around that and then modify your ongoing training. So you can learn from the people that have been trained, what else you need to add to the people that have not yet been trained.


Good afternoon everyone. As many of you know I'm Megan Garcia, New America's Director of Growth in California. What you may not know is what setting up a new office in California really is about for us. My goal is to help us harness the innovation of Silicon Valley and apply that model to the institutional thinking that pervades Washington DC and the think tank community. It's important for all of us to look at innovation as more than just the next gadget or new technology advancement and see it more as a mindset. A set of principles that dictates our approach to change, to people, to sustaining brilliant thought, and to encouraging a level of risk-taking that allows for the best ideas and policy solutions to emerge. So how do we harness it? We do it by bringing the talent of California's world-class ecosystem of innovators and inventors together under one roof and we give them a simple yet powerful goal. How do we make the world a better place through technology and innovation? Then we connect those innovators with our policy experts so that they can bring their ideas to scale. And I'm excited to announce that working with local partners in the San Francisco Bay Area, we're testing models of how we might best operate in California with the possibility of opening a California hub later this year. And it is with that excitement I'd like to introduce someone who has taken the startup model and applied what she's learned to the World Policy Institute's Global Entrepreneurial Ecosystem Project. She's also the co-founder of Foreign Policy Interrupted and a writer who focuses on global innovations and entrepreneurship. Please join me in welcoming Elmira Bay-Rasli. The next Steve Jobs lives in Pakistan. Yes, Pakistan. He or she lives in Turkey, in Nigeria, in Mexico, in Russia, in India, in China. He or she lives in Detroit, in Cleveland, in Pittsburgh, and in my hometown, Brooklyn, New York. In fact, Steve Jobs has lived in these places for many years. The trouble and the challenge has been, how does he or she emerge? For decades, anyone with a dream or an idea or the desire to build something went west. They went to Silicon Valley. In Silicon Valley, you could experiment. In Silicon Valley, you could build a product or a service that no one had yet imagined. In Silicon Valley, you could innovate. In Silicon Valley, you could change the world. Indeed, Silicon Valley has become the capital, the mecca of innovation, of the cutting edge, of the next big thing. It's produced world-class companies, Intel, Facebook, Google, eBay, and Apple. These billion-dollar enterprises that people have now come to call unicorns. Silicon Valley made it rain, and it became the end of the rainbow. Many people around the world have been eager to replicate this rainbow, to replicate Silicon Valley's success. Sorry, just pause right there for a moment. Replicate Silicon Valley says, governments all over the world have been talking about entrepreneurs and entrepreneurship as the key to creating jobs, to generating revenue, to attracting investments. They've implemented programs and policies that are encouraging people to launch startups, to have people take their ideas and roll them out. Universities are now teaching entrepreneurship. They're launching incubators and accelerators that are investing in ideas and helping nurturing the individuals behind them. The private sector is building funds that are investing in these startups. Communities all over the place, you hear about them. They're launching demo days and entrepreneurial fairs where men and women can come and present their ideas. Indeed, everybody wants to be the next Silicon Valley. Some have even come to say that they are the next Silicon Valley. I think we hear about it all the time. We hear about how Chattanooga, Tel Aviv, Istanbul might be the next Silicon Valley. Some have taken the Silicon name and applied it to themselves. In the Middle East, we hear about Silicon Wadi. In South Africa, we hear about Silicon Cape. In France, there's Silicon Sien. And in China, there is Silicon Dragon. For the past 10 years, I've traveled to over two dozen countries, looking to see how and if these places are becoming the next Silicon Valley. How are they doing it? What I found surprised me. And it wasn't about whether these places are becoming the next Silicon Valley. It's what I found out about what Silicon Valley was itself. Many of us assume that Silicon Valley is about technology.  It's full of these techies. And they're producing transistors, microchips, software, and all the gadgets that we're using today. The iPhone, the iPad, and soon the IWatch. What I found, and everybody's been trying to replicate this and looking for that, what is that special thing about Silicon Valley? What I actually found is that there's actually nothing special about what Silicon Valley is doing. The secret to Silicon Valley has been in its ability to take this concept of creative destruction, a term coined by Joseph Schumpeter, an economist, and destroy old business models and business approaches. It's the ability of Silicon Valley to shatter the status quo and to overcome obstacles. That has been what has made Silicon Valley its success. And they've approached it through openness, through transparency, by building trust. Silicon Valley ultimately is about community. And community is what Monus Rahman is building in Pakistan. Monus Rahman is a Pakistani entrepreneur who got an engineering degree in Madison, Wisconsin. He then got a job at Intel, working on microprocessing chips. It was his dream job. He loved being in that Silicon Valley world with all of the technology. He got an advanced degree from Stanford University. 10 years ago, he decided he was going to go back to Pakistan, not to replicate Silicon Valley, but to take the technology that he had and to do something in Pakistan that nobody thought could be possible. The internet, Jared Cohen and Eric Schmidt wrote in The New Digital Age is an omnipresent and multifaceted outlet for energy and expression. And he wanted to take that energy and expression and apply it to Pakistan so that Pakistanis could come together to build networks, to come together as a community. And that's something that's very difficult in Pakistan. Pakistan has many challenges. One of them is that there are very few public spaces for men and women to come together. The public spaces that they do have are the mosque, military barracks, and the cricket field. The other challenge that Pakistan has is that the country is split along geographic, ethnic, and tribal lines. There are four ethnic groups in Pakistan, and all of them are isolated in mountainous and rugged terrain that further impedes them by very poor infrastructure. These regions are led by strongmen, some by warlords. And it makes it very difficult for anyone to take an idea and to launch a business, let alone to scale it up. And so what Moniz did is what anyone would do, he went back to Pakistan and launched a dating website. He launched nasib.com in 2007 so that Pakistanis but Muslims all over the world can connect and to build friendships. And who knows, maybe even marry. It became so successful that he then launched his second enterprise, RosyPK. RosyPK is a job search website that he launched because he needed more engineers and technicians to help run nasib.com. Today, RosyPK is actually the biggest money generator for Moniz. But what these two platforms have done is that it served as a model for other Pakistanis to launch startups. When I visited the country in 2011, I found that Pakistan actually did have a very vibrant startup scene. There were many people launching technology companies. But when I asked Moniz, is this gonna turn into the next Silicon Valley? He had his doubts. He said the internet allows people to get information and to share and allows them to come together. But that is not what is going to change Pakistan. What will change Pakistan is the ability for people and the government to come together to collaborate and solve Pakistan's many problems. Indeed, what he said, what will change other places is not technology, it's the human experience, the inability to create a community. And we see that here in the United States with our own entrepreneurs. A number of e-commerce websites that launched a few years ago, Warby Parker and Bonobos, which is a men's retailer, both started out as e-commerce websites. And yet a few years ago, they both opened up showrooms. David Gilboa, who's the CEO of Warby Parker, the eyeglass maker, said that they did that because people still want a tangible human experience. They want connection. It's the ability to connect and collaborate that is the fundamental basis of any society. And it has actually been the key ingredient that has made Silicon Valley successful. People talk about the entrepreneur in the garage, you know, creating something, being alone. That's actually not true. When you look at what happened in Silicon Valley, it's the ability of people to come together and openly exchange ideas, to collaborate with one another, to easily get around, to work with the venture capitalists, to work with the engineers. But a place where people from all over the world eagerly wanted to come together. And it is that, not the individual or the technology that where the next Silicon Valley will rise, the next Silicon Valley will come where there is the next community. Thank you.


We're at three o'clock. Must be the rainy weather brought all the people out to this afternoon meeting. They didn't have anything else to do. We're meeting at three because of the annual historical society meeting at 5.30 tonight, so we met early so some of us could attend that. Are there any modifications to anything that's on the agenda? We've got one information item from Jeff Kilgore's office and I have a budget report that there's not time we can wait. It's not much of a budget this year, so if you want to put it on and we have time for it, that's fine. And if we don't have time for it, we'll do it after the minutes. Whenever you want, we can do everything at the end, things that I've talked about. Sure. Is there anybody here as members of the public for something that isn't on the agenda? I haven't seen the agenda. I thought probably you were here for the historical presentation. Okay. Hearing nothing else, first up is the concert schedule at Rusty Parker Park. Lynn, do you want to join us and you've got the schedule? You're ordering the weather? Yes. Sun every Thursday. So yeah, I'm here to request permission for the park from 6. 6 o'clock would be start time, 8.30, ending from June 7th to August 16th. So that's shorter than you did last year, historical presentation. I am, just like that. That's pretty good. Yeah. Okay. I just want to say I think we're having a fun time here today. Thank you. That was a great presentation. Thank you. So, it was really great. I appreciate it. Thank you so much. I appreciate it. So, I'm glad to be here. Thank you. It was a great conversation. Yes, son, every Thursday. So yeah, I'm here to request permission for the park. From six o'clock would be start time, 8.30 ending, from June 7th to August 16th. So is that shorter than you did last year? Historical presentation. I am just saying that. That's good. Yeah. And the charter, we love it. Okay. Hearing nothing else. First up is the concert schedule at Rusty Parker Park. Lynn, do you wanna join us? And you've got the schedule and you're ordering the weather? Yes, son, every Thursday. So yeah, I'm here to request permission for the park. From six o'clock would be start time, 8.30 ending, from June 7th to August 16th. So is that shorter than you did last year? Historical presentation. I am just saying that. That's good. Yeah. And the charter, we love it. Okay. Hearing nothing else. First up is the concert schedule at Rusty Parker Park. Lynn, do you wanna join us? And you've got the schedule and you're ordering the weather? Yes, son, every Thursday. So yeah, I'm here to request permission for the park. From six o'clock would be start time, 8.30 ending, from June 7th to August 16th. So is that shorter than you did last year? Historical presentation. I am just saying that. That's good. Yeah. And the charter, we love it. Okay. Hearing nothing else. First up is the concert schedule at 7th to August 16th. So is that shorter than you did last year? Historical presentation. Yeah. Hearing nothing else. 7th to August. So is that short?


the ancient Israelites talk about how when they would go to war they would always leave one escape route for the enemy open to mess with their head because they would undermine their commitment knowing they had a backup plan. Yes, that's the OG life hack. One, just contemplating another way to achieve the result, not actually pursuing it, measurably diminishes the likelihood you'll achieve your true purpose, your plan. And then the second part is it'll ruin your intrinsic motivation to actually achieve it.


Tana Hill fires left corner of the end zone. It's caught by Westbrook and Keeney. Is he in? He is. Touchdown! Tight! Throws, Cooper, did he catch it? Did he catch the ball? A juggle? And they are marking it as a catch and they're in the seven yard line. Wayfake. Tana Hill throwing deep downfield going for Birks. He's got it in the 20, at the 15, at the 10, at the five. And Birks, from Taylor Britt, inside the 25, throws the flat for Akakwo, one-hand catch. As Akakwo reaches the big point.


Twitter is probably one of the largest companies running Ruby in production today. And to give you guys some scope of how large we are right now, we've got about 110 million tweets coming in every day to the service. And because of the one to many nature of the relationship with Twitter, there's some multiple of this number actually being delivered every day. The record so far for tweets per second is about 6,939 tweets per second. They came in on New Year's of this year. They actually overlaid. You can't really see it too well because of the projector, but that's an overlay of a map of the world showing some of the epicenters on New Year's of where a large person tweet were coming through. That's pretty cool. We can check it out on the blog. Now, as far as my team put things in scope there, about 40% of those tweets coming in every day are coming in through a mobile origin. Now, it could be coming in through, say, a native app like Twitter for iPhone. It could be coming in through the mobile website, mobile.twitter.com, or it could be coming in through our SMS service, which is huge, especially in countries where smartphones are large. So today, what I'm going to cover is... I'm going to go on the architecture of Twitter in general so you can see where mobile fits into the big picture. And then I'm going to show some case studies from the mobile team, just three areas where we're currently using Ruby, because our team especially loves Ruby and we use it whenever possible. And they're pretty varying areas. And of the three, we're probably going to dive into most detail on the mobile website because I assume most of the people here are familiar with using Ruby and Rails for writing web apps, so we'll probably get the most of that. We'll wrap it up by showing some of the best practices that I think will be co-opted from Ruby early in development of the company. I think that Ruby was a major influence in the engineering best practices. And we have some time, hopefully, at the end to answer any questions people have. Cool. So let's just get started by showing exactly the architecture and where mobile fits in.  It goes into this big chart, which we're actually going to break down into three chunks.  If you're on twitter.com to our web frontend, which is running Rails, or to api.twitter.com if you're on a native app, which is also running a Rails frontend. Or if you're on mobile.twitter.com, that's actually a 100% pure API client. So it's actually routing itself on top of api.twitter.com, so it's just a nice layer of course. So then your tweet goes into the core code that handles all the business rules, who's following who, you know, all the lookups on the accounts, everything is in one relatively large code base right now, it's being split out into separate service, a service oriented architecture. And right now, we're still in the process of decoupling all the frontend code from the core code, but this is the eventual architecture that we're striving toward. And I won't go into detail about the services in this presentation that make up all the core code, but we have for instance our social graph service, which is powered by Flock, our open source graph database. Our user service is sitting on MySQL, we have a Geo service, Timelines, and if anyone's interested in finding out where, I'll be glad to talk in more detail there. Now this part of the graph covers the request response cycle. So you hit tweet, your tweet was routed to the main code, and then we verified that we actually wrote your tweet somewhere. It isn't actually delivered to users immediately. In the context of a request response, we just returned you a 200, and really we're dropping your tweet into a queue, so that we can handle it using the asynchronous pipeline. And the reason here is mostly performance and scalability.  So for instance, in the example of sending a tweet, we have the timeline demon, or the timeline fan of demon. So there's a demon there that's just popping tweets off the queue. It's making a lookup in our social graph to see, okay, who's following this user, and then it's delivering the tweet into all of those followers timelines. We have other demons which we'll get into like the SMS and push demon, which pops tweets off the queue and checks, okay, does this user have a device attached to his account, or is someone who is following this user, and they have a device attached to their account. We also have the streaming API consumer.  These are all just examples of three demons. We have tons of them doing all sorts of stuff. That's it in a nutshell. So we already touched a couple spots where the mobile team plays a role, like at the very top, the mobile website, or as I just showed a second ago, that SMS stuff. Who here has ever had to like interoperate with an SMS carrier or an operator? Okay, when I say that now, how many people here actually have to connect directly to the carrier? So you're not actually using Twilio, you're not using a third party. Okay, a smaller amount of people. And for those of you who aren't familiar, which I would say the vast majority of people here, it's a lot of work to turn on a carrier insofar as it isn't just you point it to a phone and you send a text message. We actually have to first strike deals with each carrier everywhere in the world. So in the US, AT&T, Verizon, T-Mobile. But there's literally hundreds of carriers in the world. And for each of them, we have to set up first a business arrangement saying, okay, here's what you can and can't do with users. And then we actually have to configure that carrier. So such as each country or each carrier may have a different shortcode. In the US, it's just 4044, but it could be whatever's available in that country. We have to internationalize all the commands and we have to also manage the disconnects file. So when you turn off your phone service with your carrier, that carrier has to write down your phone number. And every so often, they upload to us a list of phone numbers that are disconnected now and no longer wear their service so we can purge them from our database because they may not have shut off their Twitter account before closing down our account. And the reason they do that is down the lines that they may reallocate that phone number. And if you forget to turn off your Twitter account, someone else is going to start getting your direct messages, which could be pretty awkward. So we have to arrange how do we get these phone numbers and start purging our database. Lots of stuff like that. So as for the code base, it's actually about into two parts. There's the web service portion, which is getting incoming SMSs from the carriers and handling all the business logic around it. And then there's the outgoing code, which is actually a set of demons that are calling out to the carriers. In some cases, using like a SOAP-like interface. Some of them are using persistent connections. Again, it could be, you name it, and some carriers doing some completely different protocol for delivering SMSs to them. So the main code, as I said, handles a lot of product logic. In the case of the demons, some of the product logic, when we pop a tweet off the queue saying this is supposed to go out to a user, we check, okay, is this device asleep? And you can actually set, I don't want to receive SMSs like from midnight to 6 a.m., so your phone doesn't ring. Or there's any number of other configuration things that you have to perform at this layer before we push it off to the carrier. When we get an incoming SMS, we expose an endpoint so they can make a restful call just to post the incoming SMS message. And incoming SMSs aren't necessarily just tweets. You can actually issue text commands like follow, block, and all those commands. So we have to have that layer on top of it and know how to respond inside this app. The app itself is a Rails 2.3 app right now. The logic is only about 2,500 lines of code. And it's actually a 1 to 1.5 code to test ratio. So there's actually more tests than there is code. So what's an example, though, going back of some of the demons that we write to connect to these carriers? Well, for instance, when we wanted to turn on Apple push notifications, all we had to do is subclass our demon base class and override the method and put all the logic in there. We pretty much refactored all of our demon code into one set of poor classes that you had across the entire company. And the big advantage there is that our ops team now knows how to monitor and spin up new instances of these demons whenever we need to. So it's all pretty much just one code base that's shared among all the Ruby-based demons throughout the company. And as for the logic that's fit inside of that one method that you override in subclass, it's about 300 lines of code which handles both calling out to Apple and receiving the disconnects from Apple when people uninstall the iPhone app. It also has a number of sanity checks. So in a large distributed system, anything that can go wrong will go wrong. Among other things, we've had situations where plenty of duplicate SMSes have come in that they expect to send up to users. So you might receive, let's say, like five SMSes from the same user, who knows what happened. Maybe a demon was turned off at the wrong time. But when you're dealing literally with hundreds of demons at once, anything that can go wrong will go wrong. So if you put in sanity checks, I like to check against duplicates because in the U.S. you're charged per SMS you receive. And lots of sanity checks like that. We also had to put in the payload formatting for Apple's payload that they expect from us. And we also do the auditing so we can keep numbers on which carriers are receiving more SMSes lately. For instance, Cairo is pretty big lately and we're actually seeing graphs of what's going on there. It's pretty obvious. That's the SMS service in a nutshell. So we're actually going to switch gears. And that last example was mostly about how we use Ruby as a web service and how we're using demons. This probably is more relevant to people which is how we wrote a web front end using Rails for the mobile website. So what's nice about the mobile website? I think it's a great example of an agile product. And it was developed with two engineers and it completed about 51 days from the first commit to the public preview. And inside those 51 days we actually switched over the stack from the existing web apps that were running at Twitter which were previously running Mongrel and we moved it over to Unicorn. This was actually the first Unicorn app that was running in production at Twitter. And then actually shortly before production deployment we switched over one more time to Rainbows because of concurrency reasons we'll get to in a little bit. So unfortunately, it's nice to use something new and higher performance but there's a lot of operational overhead so you have to rewrite your public configs, you have to work with ops to see if they can update their monoconfig and how do we handle when this shiny new thing runs into this weird situation you haven't run into before. So we had to pay the penalty so actually pouring things over. Also in the 51 days of developing the app we had to develop a REST client from the ground up and we'll go into the details behind this client in a minute and why we didn't use existing clients out there and some of the design decisions but in those 51 days we did not sacrifice code quality. It's maintaining a one to one code to test ratio or code, code to test ratio and also the big takeaway with this project is that we were developing a framework for connecting to the API. So I don't think that you need to be a really advanced Ruby hacker to get a lot out of Ruby and it's more important to develop tools so that more casual Ruby users can be really productive without having to understand the details of the metaprogramming we'll get into. So actually what do I mean by that? So when we're developing our client we wanted to develop a domain-specific language around connecting to the Twitter API. The goal being that when API comes to us and says we have a new API endpoint it has this awesome cool new feature when can you ship this? We want to be able to spin up a new endpoint in a matter of minutes describing how to connect to it from all the user logic. So here's an example of our RESTful client making a call to the home timeline and current user is just an instance of the current user based on their cookie and we're making an API call there to fetch their home timeline it's pretty straightforward. This method is generated dynamically by making this call inside the class that will generate the finder. And this is an example of the domain-specific language we created. We're saying wire to this API path the home timeline API path connect to that endpoint you're going to take the JSON that you get back from the endpoint and you're going to map it to the tweet class and the name of the method is going to be find home timeline so it's as home timeline and collection equals true means this is actually an array of objects it's not just one instance of an object and you're not going to be quizzed on this but it's really easy to even now just glancing at it it makes a lot of sense exactly what it's doing and you know if you spend five minutes learning this domain-specific language you can quickly define your whole API in about one page and this is cutting back down hundreds, maybe thousands of lines of boilerplate code and the less fobbing and pasting you have to do the less chance you're going to make an error so this is described in the entire API that we're connecting to this only describes though half the challenge which is actually connecting to the endpoint how do we now map the API to friendly user or friendly Ruby objects so you have business logic wrapped inside of there well the solution is a domain-specific language and here we wrote an objectification module which also allows us to have access to a lot of class methods to describe how we're mapping keys inside the JSON back on the attributes on top of the Ruby object so here we're saying in exposes we're going to list off a bunch of attributes that we want accessors built into the object around the hash keys and below that we can also define custom behavior for certain hash keys so for instance if people here aren't familiar with the Twitter API we return for tweets a tweet object and embedded inside of that object is a sub-hash that describes the user so here under that objectifies line we say take the user key and take that hash and pass it to the user class which will turn the objectified with a similar DSL like this inside that class we can also pass custom parsing so in the example of created app as a hash key we just want to take the value we get back and we pass the lambda to say alright now parse that using time.parse so it's pretty flexible no matter what API throws at it we can pretty succinctly describe how we should parse it and turn it into a real object here's a fun topic at least a couple of years ago I remember a lot of people were asking about this I don't know if it's still big in the Ruby community scaling it's actually not that difficult I think for cq about operations in Ruby to go after a lot of low hanging fruit one example would be doing research about the libraries that you're using and I'm surprised by the number of people who use the JSON C library when the Agile C library for parsing JSON is literally an order of magnitude faster in the benchmarks we were performing and they actually have backward compatibility with the existing JSON gem so literally dropping in two lines into your code will give you JSON parsing an order of magnitude faster than what you had before again just doing research and looking to see what other libraries are out there that may be more suited for what you need to accomplish on the mobile website because we can't use recapture for our sign up captures we have to generate our own captures from scratch for low end devices and I think everyone hopefully at one time or others have run into manipulating images on Ruby and maybe they've used our magic or other libraries and I'm surprised by how people have actually seen GD2 and this is a cool image library it was actually extracted from PHP and it's now there's wrappers available for pretty much all major languages and it's at least twice as fast as far as generating images compared to image magic or at least scaling images and we have to run into any kind of memory leaks or any of the other problems in the past our magic has been afflicted with and we actually wrote a very simple wrapper around it so that you don't have to worry too much about memory allocation it's mostly abstract away from you and again this was only like a project that took a couple days to throw together the image generation portion so one other way that you can watch CPU on your front ends is actually being aware of object allocations and there's a misconception carrying over and over again that because you were using a garbage collector you don't have to worry about memory management that's all handled for you and you know it's talking to a really talented engineer from Google I asked him what about these systems like in Java that have a garbage collector can you really tolerate garbage collection pauses when you're talking about the super high scale systems and he said well actually yeah the thing is Java programmers that there's memory management still involved and they're still just is conservative about allocating objects and how we're reusing objects and you're still working understanding when you're allocating frame objects it's just you aren't doing it by hand and I think that you don't have to go in there and really drill in deep to see where I'm allocating objects but you just have to approach it from the mind am I being conservative with generating a lot of temporary strings you know what's the difference between saying plus string versus embedding one string and the other how many temporary strings are you generating things like that as long as you're mindful of it I think there's huge gains but we've talked a lot about front-end CPU but I think that it's kind of a red herring when you're writing a web app I think that the first way that you can scale your app is being aware of concurrency so you're actually spending your time in a request so let's say you've optimized your app so that you can parse JSON and perform all your business logic and spit back or render template in 10 milliseconds that's awesome what if you're spending 100 milliseconds waiting on API or you're waiting for your database and in that 100 milliseconds you're just sitting there twiddling your thumbs doing nothing the end user that's 100 milliseconds wasted your server is going to reach capacity even though it's not actually peaked in CPU it's important to actually go into your app and see how is most of my request time being spent and it turns out in the app when we're looking at it most of our time was IO bound depending on the load to the API we could see load times over 100 milliseconds even worse and it's amazing how much you can accomplish in 100 milliseconds you just batch up your requests and you could accomplish 10 requests in that 100 milliseconds while you're mostly twiddling your thumbs you now have 10 times more capacity that's a hand-waving math so Ruby unfortunately has a lot of people have complained about its concurrency model there's a great great great article concurrency in Ruby is a myth you should all google it so when you're calling out to C code your threads pause and lots of fun things like that and I won't go into too much detail but we researched different solutions and we settled on for connecting to our API we're using a library called Titus and among other things that this library gives us for making restful requests one, it's a lip curl binding so it's very performant specifically it's a lip curl multi binding and especially interesting is that it respects the global interpreter lock and it will not pause all of your green threads while it's waiting on responses from API so what we ended up settling on is we built our restful client around Titus and then we turned on multi-threading mode in Rails and we switched over to Rainbows and we're using green threads so one caveat is that it's very difficult to work with concurrency in general there's different paradigms then driven programming there's functional programming there's different ways of thinking about handling concurrency and in the interest of keeping things simple especially for developers who are used to more synchronous programming we wrote again a very simple I don't even know if you'd call it a domain specific language around issuing concurrent requests where basically when we issue each of these finds we're not blocking we're actually returning a proxy object and it's not until we call the fire request method that we drop down to Titus batch those three rest calls into one lip curl multi call and we suspend the current thread in Rails and while suspended other threads are still running and receiving calls from incoming requests so and then when this resumes those three objects will be filled into the actual responses from API this is actually a pretty simple way of thinking about concurrency rather than throwing lots of lambdas everywhere and worrying about syncing and all that sort of thing switching gears one more time to probably the least conventional way that we're using Ruby on the mobile team is as a testing harness and I don't know if anyone here has ever tried testing on Coco or on iPhone because I don't think it's nearly as mature as the Ruby testing community there's so many great frameworks available and great tools and one solution would be to import a bunch of these tools over to Coco that'd be cool or be really lazy and just run Ruby and it turns out that the core code that's running on the iPhone, iPad app and the Mac app is actually just vanilla Coco it's just straight up regular Objective-C and it's not dependent on iOS frameworks for all the core business logic like how do we handle user accounts how do we tweet that sort of thing and just so happens that Mac Ruby plays very well with Coco and it has a nice one-to-one mapping so you can treat Coco objects just like Ruby objects and matter of fact in the latest version of Mac Ruby they even have built-in support for I believe it uses mini-test so it was pretty easy to develop a test suite and treat these Coco objects just as regular old Ruby objects and slap on a simple UI using interface builder. It only took about an afternoon and we're working to actually integrate this inside of continuous integration so whenever we push a new update forget it'll automatically run on the continuous integration server the best thing I've heard about company culture is that it's like cement and while the cement's wet it's very easy to sculpt and change but as soon as the cement's dry it's not going anywhere and the way I interpreted it that was about general company culture but the way I also interpreted it was about engineering and we were very lucky the company early on had very enthusiastic Ruby developers and with the Ruby community had a very strong culture of certain best practices such as automated testing and agile methodology and also with the help of Pivotal Labs they drove home a lot of the agile techniques I think that a lot of the philosophies of Ruby are more important than the language itself I think one of the great things about Ruby is the community and we're all here to get stuff done and we care a lot about the product and I think that's influenced our team so this is just an example again from the mobile team each team might be different but we believe in having small agile teams almost like a start-up and on our particular team we have one product manager one designer and three developers all working on the iPhone and iPad apps and having worked at different companies which have a massive scale of users I've never been on such a small team that's had such a large reach for the app that they're working on I'm especially excited that we understand you don't throw people at a problem also it encourages communication when you're that small of a group I can just turn over to the guys next to me and ask them questions it's very easy to communicate among our small team we also practice iterative development so in this team every week we have an iteration planning meeting and with the help of Pivotal Tracker how many people here use Pivotal Tracker they're all awesome and if you aren't using it go try and account right now I think it's still free but it's totally worth everything it's a great agile tool and using Pivotal Tracker we write out the user stories and these are the features that we want to accomplish this week or we write out the bugs that we want to fix and then we can score them drag and drop them this is probably boring for people who already use Tracker but based on how we score these stories based on the points that are left so we're always fluctuating the release date and we aren't trying some rigid waterfall approach and it's really exciting so one last part about agile development is that we believe in continuously shipping code on the website on twitter.com and on the mobile site we are deploying every day on twitter.com desktop actually there's an average of 3 deploys every day I think the record has been about 7 deploys in one day and inside each of those deploys are between 10 and 30 feature branches going out so this requires a lot of process for this to work for instance our rigorous test suite which alone would take an hour to run if you were to try to run our test suite but we have it parallelized so it takes about 2 minutes because it's going to take 2 minutes of testing machines we have a number of processes in place around releasing products we canary deploys so they only go to a subset of the servers and we monitor metrics before we spread them out to the whole fleet but we have a lot of processes in place that ensure that we don't run into any problems with this but we believe it's all totally worth it to be able to write a feature in the morning get a code review and ship it that afternoon native clients are a little more difficult for continuous deployment because of the nature of the beast one compromise we've had is we do weekly releases internally to employees it's nice on the iPhone if you have an enterprise account you can do over the year updates so every week now after iteration planning we say alright let's cut that release and we make a build and we push that to employees on their iPhone so that we're constantly getting feedback so hopefully if I've done my job you're really excited about the company you don't know how you can work there and actually the second way though honestly one of the most important things about your engineering organization is who you have on your team and in our mobile team we care about a few things I don't think it's worth the bullet point but we obviously care about people who technically know what they're doing but beyond that we care about people who are passionate about the product and you can tell when someone uses the product every day and they really get it what we're trying to do and we really like those people we watch for open source contributors if you have GitHub on your resume you've got your foot in the door we love seeing open source and even if you don't do open source in a similar vein if you're working on personal projects you've got a really cool start up and you're just trying out to see where it goes doing cool weekend projects if we see that, that's a big flag that this person is probably passionate about what they do and finally I think that this is the first company where I've been at where culture is one of the major points people try to drive home when they're hiring we will gladly turn away an applicant who is outstanding technically but it's clear that they wouldn't click with the team we care very much about being surrounded by people who again are passionate about the product and really excited to work with but just in general people you want to hang around with that you like being around positive and a lot of fun cool so actually that is all the stuff I wanted to cover so that we could leave some time at the end to answer any kind of questions you'll have I have a question about what I believe is a new bug on the mobile site let me just say that I haven't text the mobile site in seven months I'm now working on the iPhone so it's been around for about seven months I have a mobile site I send a user there you might have to tweet this button and send them to mobile twitter.com status equals if they're logged in everything's great but if they're not it has to be logged in and then redirects them to the mobile twitter home and loses the status and there's no way for them to get back to my site without going back through like free changes just now I'm really lost okay I probably know what's wrong with you you've created a ticket for me thank you find me out the words and I'll help see if it's actually being worked on okay thanks please do a full stack mobile dev on your team what do you mean okay do we do a full stack interface versus core data or things like that sorry interface builder versus core data okay so do we do full stack mobile development do we use interface builder do we use core data is that a question do you split your team up and do you separate sub parts oh is a person a specialist in one particular area so in particular on the iPhone app and actually this is true of all of I think our developers we believe in generalists so in fact as I said until seven months ago I was working in the mobile website and they switched me over to the iPhone and we actually have another developer who's working in the mobile site who's been working on Android so I believe we certainly want to avoid the bus danger where one guy gets hit by a bus and they contain all that knowledge and we're kind of screwed so in general we try to rotate people around and I think that we all have at least some knowledge of the different areas but for instance I've been lately working on a feature that involves using SQLite and because I come from a web development background I'm able to write SQL pretty well so they're like alright why don't you work on that one thing that we didn't cover is and this goes into the generalist aspect is that in order to ship features we believe in code reviews so before you can merge your code into the main code base you have to upload a diff to a review board and a teammate has to write ship it on and part of it is also part of it so that you get two eyes on the same code so you aren't making an obvious mistake but indirectly it's educating other people on what's going on in the code so they kind of have to know what's going on in order to give you a ship it With respect to code reviews your significant testing setup how do you deal with the taxation involved in time So your question is in relation to all the process code reviews and testing how do we deal with taxation and time and I think that one it's dependent on the team so some teams are more reverse about unit testing than others so we to a degree leave it a little to the discretion of the team as a company we strongly encourage unit testing but I think that what's important is that these best practices have tools built around them so there's minimal taxation so actually when we are ready to push code to the main repository to touch twitter.com desktop we have a script called make review and it's an actual it'll prompt you each step and it will once you've finished each step it will upload your diff it will create a a ticket and JIRA for you and it'll automatically I think it'll actually CC your entire team for the code review absolutely everything except for clicking the publish button to actually send out the email so I think that the important thing is one just building all the tools around the process otherwise you know unless you can ingrain the habit people aren't going to do it and you know make it easy for everyone I think it's the key thing how do you manage to get the test suite to finish in two minutes as opposed to one hour right so I didn't go into too much detail we built a system that parallelizes the test suite so it'll divide it among a number of servers in our fleet so it'll evenly divide it so instead of whatever one server it's ten servers the math there doesn't work out probably just using numbers but it parallelizes the suite and it has so an interesting set of challenges because you know your certain test if you aren't careful your test suite may not be optimized to be touching the same resource you could be clobbering over each other so our internals we have this team focused on working on all the details and then when you want to run this automated test suite there's a utility in our bin directory that's called swoop let me say swoop test and it'll give you a nice progress in two minutes later you'll tell you if there are any errors so I'll go you first you talked about having small agile teams do you do air programming and the stand-ups I mentioned small agile teams that use stand-ups and we practice company-wide little agile on a per team basis some of them are much more into pair programming on the mobile team for a long time we were doing pair programming and lately we haven't been doing it as much shame on us but other teams they still embrace pair programming and it really works out for them so it's really at the discretion of the team and we highly encourage it yeah so it's a team by team thing how do you do authorization for mobile plans how do we do authorization for mobile clients what authorization do you mean? we're actually so on mobile.tv.com how do we do authorization so you hop on the mobile site again we're playing by the same rules as everyone else and we're using Excel when we first launched the app we weren't using Excel and it ended up being a lot more trouble than what was worth to try to use some internal system so we just ended up moving everyone over to Excel so there was one pipeline with the same team working on it on the API end what's Excel? so Excel is hopefully everyone's played with OAuth which is you click the link saying log me to the site and I give you a pop-up and you have to click the link there so Excel allows the provider or the third party app to just give a username and password input which then makes a call to our API or in terms of the OAuth token and the use case for that especially on native clients like the iPhone it can seem a little weird to pop up a web view and then putting your username and password there and the web view disappears and also on really really low end devices like feature phones and pop-up web view so in certain situations but it requires special privilege you'd have to talk to API about what is the break-up of the users between mobile.com I don't know that off the top of my head sorry, what's the break-up of users on the mobile website to the native app sorry do we do manual QA and all of us exist in two video continuous shipping strategy? do we do manual QA we don't do any insofar as we don't have a team of people clicking on buttons to make sure yes however what we try to do though is instead all the employees for starters you own your product and there's a danger if you do have a QA team is you can fall into the pattern of alright let's throw this over the wall and let that handle it especially on the iPhone app there's a certain level ownership among all the people on the team where at all times for instance we're running the latest current build so as soon as we run into a bug we're fixing it and we're expected to do all responsible QA ourselves hopefully they'll I made a testing system around it so it doesn't become so repetitive and then we have a roll-up strategy of rolling out to all employees in the company and we give them tools to report bugs and then we also have a system kind of decider so when we're rolling out a new feature in production we may roll it up to zero users and then we roll it up to 1% of users which in part is to see how does this affect system stability and also you know is there an issue in general with users using this feature so we believe very much in slow, deliberate roll-out of features and also personally taking charge of QA and having personal accountability around the code we write efficiently to changing needs changing demands team members whatever can really influence how your team are acting how do you ensure that these tools don't lock you in their process that make you less efficient and how do you allow them or have them allow you to experiment okay so the question is how do we make sure that the tools if I understand art having too much control on how we do things like we're not being locked into one regimen based on the tools is that correct and I think that again it goes back to each team has a certain level of leeway as to how they get things done so actually the mobile team is one of the most evangelical people to use a pimple tracker we use it for all of our iteration planning however for actual ticket management we use JIRA when we're actually pushing things out to production so that there's that level of accountability and tracking so everything up until we create the JIRA ticket it's up to us just make sure we have a JIRA ticket so everyone can sync together and I think that part of empowering people is giving them that flexibility to do things however best works for them and the results speak for yourself that said company-wide we're trying to get slightly more regimented like moving toward you know the mature development style like at a google where we're trying to push toward new things people might not be as familiar with especially from a start environment for instance we've been active especially in the last year on encouraging design reviews on major new features so you get feedback from your peers before you even lie down your first line of code it's probably overkill if you're making a copy change on the website but if you're developing a new feature it really is a huge advantage to just spend a lunch with your peers and show people what you're working on um there's a cell phone I think so we're trying to push people toward newer processes they might not be familiar with and there's a little to the degree of regiment there but it's a balancing act you guys use the iPhone UI animation what kind of herbs we've looked at it but we haven't invested the questions we're using the iPhone UI integration testing toolkit and I think that was relatively recently released and I saw some info about it but we haven't invested time to fully build up and to end testing on the iPhone but eventually we're going to do it so I think I think we'll just wrap things up right now and if anyone has any questions please find me I'll be happy to answer them I'd like to actually thank JR and Kobe everyone behind this group that's been really exciting thank you for having me


Machines make our lives easier in many ways, whether it's a smart thermostat that learns when to turn the heat on, or automatic brakes, machines traffic in the language of classical calculus.  To provide a richer description of these systems, experts are turning to fractional calculus. Mathematical models built from this more exotic, but more general form of calculus come pre-installed with a way of accounting for past events. This feature allows them to mimic the memory-like effects observed in real systems, such as the stock market, communication networks, and of course the human brain. But before fractional models are uploaded into our devices, researchers have to be sure not only that they're complex enough to reflect real processes, but that they're not too complex to render our devices unstable and therefore useless. Mathematically speaking, they have to ensure that a system that deviates from its rest state, room temperature for instance, in the case of a thermostat, can be controlled back to that state within a reasonable amount of time. To address this problem, a team of mathematicians asked whether such control could be achieved for equations called fractional stochastic differential inclusions. These equations describe some of the most unpredictable and noisy systems found in the real world, such as financial markets and quantum systems. They proved the existence of solutions for two forms of these equations, convex and non-convex. In math, convex cases are typically easier to solve when looking for the best way to control a system. Non-convex cases are trickier to solve. The ability to prove controllability in both cases is therefore a major advantage of this method. The mathematicians tested their technique numerically on a spring-like model. Although seemingly simple, the model was built using the same type of equations the team looked at before, making its behavior highly unpredictable but not uncontrollable. The team was able to show that the spring could theoretically be brought back to rest from any position it might adopt. Mathematical tools such as this will likely find increasing applications as fractional models become more widespread and complex. Scientists and engineers may do well to add these to their toolkits and ensure that their designs for new devices are both highly adaptable and controllable.