Live from Las Vegas, it's theCUBE, covering AWS re-invent 2017, presented by AWS, Intel, and our ecosystem of partners. Welcome back everyone, live here in Las Vegas, it's AWS re-invent, this is theCUBE's exclusive coverage of re-invent 2017, our fifth year covering Amazon, watching the explosive growth. I'm John Furrier, the co-founder of SiliconANGLE Media, with Justin Warren, my co-host here for this segment. Our next guest is Regine, the executive muscular and vice president of the data center group, and also general manager of the cloud service provider, part of Intel. Welcome to theCUBE. Thank you very much, it's nice to be here again. So, Intel inside the cloud has big growth for you. What's the numbers look like? Your earnings look pretty good. What's the business update? You know, it is growing, and it's growing in so many different angles, it's coming from multiple fronts. Part of it is just these killer workloads that are driving the need for change, artificial intelligence, immersive media, network transformation to handle all the data. It is crossing this real spur in growth from both the very largest of cloud service providers across the ecosystem. And one of the things when I look at the growth, I try to say, now where is it really coming from? Yes, Amazon is just insanely growing fast, and the big guys are doing well, they're a global expansion. But also, the next wave of cloud service providers and regions in different countries are doing really well too, they're actually growing faster. So the whole ecosystem is growing with demand. Yeah, it's certainly not slowing down. I can't see any way in which that it would slow down at all if anything is just going to get faster. I think so. And when I track trends like cap expending and analyst predictions on the future, consumer demand, everything indicates we're going to see another strong five to 10 years. Yeah. So how do the cloud service providers shake out because you look at this creativity of software renaissance, the interactivity, all the amazing use cases from Thorn funding, Missing Exploiting Children, to high-end HPC scientists, genome sequencing stuff, massive use cases. So this going to be a tsunami of software developers changing. How does that shape the cloud providers? Is it segment them into tiers? How are you guys looking at that business? Well, I think it breeds a lot of new opportunities. So yes, you see more and more new services and capabilities to enable developers to more easily develop code to more quickly to get better utilization out of their code. But I also think just that concept of unlocking performance and ease means new insights, new services, new capabilities, things that we just couldn't, a developer or we could not have done before. So it's that just like electricity, when you make it cheaper and easier to consume for that we use more of it, I think cloud computing is the same way. So we're seeing kind of a natural TAM expansion, market expansion around these things you just couldn't do without the cloud. And new workloads too. And new workloads. And all of those workloads, I mean Intel is basically completely repositioned itself from just being a chip company. It's like you're a data company now. So what are some of the things that Intel's doing to help people understand how they can make use of these new software techniques and these new tools and the capabilities of the Intel chips that underpin it all. So what are some of the things that Intel's doing there? So obviously it starts at the bottom with the best silicon, not just compute, but compute, network and storage and accelerators for all different workloads. We move up to the platforms. We do a lot of hardware engineering with the ecosystem, with our top CSPs, actually many top CSPs. We do direct engineering work to get better systems and architecture. We have a host of libraries we're creating that ease of use and unlocking performance out of it, reference architectures, co-partners and solutions. So for us, when we talk about being a data company, we can't do it from just even being a chip. We have to be a solutions partner with cloud service providers, enterprise, IoT edge solutions, we try to be there. You guys always enabled some very cool demos in the day. Even back on the PC, you always had interactivity. You pioneered multimedia. You always had that eye on the applications. Okay, on stage today, all this greatness is out there, NFL demos, all this cool stuff. That's really powering your business. So talk about your relationship with Amazon Web Services. What's it like? How do you guys engage with them? What's the relationship? Cause you guys are power engine for AWS. Well, we try to be. We want to be the best performance into their data center. We've had a many, many, many year deep relationship with Amazon. It started from simple co-development and engineering and is extended much more pervasively across their environment and upward sometimes into the services. We want to, one, make sure that what we're delivering today has been already optimized specifically for their unique environment. And that means we have to start a year or two before, if not earlier, to really understand where they're going and get their feedback so that when we either optimize a product or technology across compute network or storage or create something potentially custom for them, right? It takes a number of years of work. So our partnership has a lot of in-depth engineering. It has a lot of future and near-term enabling. And then we hope to see an expansion. What we really want to do is use our technology to differentiate Amazon. We want their services better because of the technologies or capabilities we put in. So wherever they want to align in terms of strategic investment and growth markets, we want to make sure the silicon can enable it. Intel always marched to the cadence of Moore's law. And you guys have always been a rapid machine execution. Amazon looks good. I mean, they're executing. They're fast. I mean, what's it like? Share some story.  What's it like working with Amazon? Well, I think, I mean, they are fast. It's amazing how quickly they can move and innovate, how rapidly those innovations roll out to the market. I will be honest, there are times where we miss windows because we are slow. And they just look at us like, well, we told you this four weeks ago. Here it is, right? So, what we've been doing for- Can't design a chip in four weeks. We don't have Kubernetes, no containers for chips. I mean, they're just, they have it down. So what we're trying to do is part of this transition from being a client company to a cloud and data center company and IoT company means everything we have to do is faster. So we have to design our chips more quickly. We have to put in more modularity for faster derivatives. And we have to move at cloud speed, not classic Intel speed. So what are some of the lessons that other companies can take from Intel's? I mean, it's a hardware company. It was originally a hardware company and now you've transitioned to being a cloud company and you're being pushed by Amazon to move faster and faster. So what are some of the lessons that you can share with other companies who are trying to start moving at cloud speed?  If you aren't understanding how the end customers, starting with Amazon's customers, but also then my customer, Amazon, how they're using and consuming technology, we can't really create good technology. I would say a lot of companies create a great thing and then try to go sell it at markets versus starting with the market and creating the specific thing. The other thing we've learned, I mean, Intel is a very data driven company both in our decision making as well as our company growth. This is a, and we talk about it from a developer behind, but it's same, iterate fast, fail quickly and move on. You don't need perfect. This is one of our learnings, right? Don't wait six months for perfect, move fast, get 70, 80% of the way there. I've heard governments say they get 40 to 50 way percent of there and make decisions best because they have to move that quickly for military or other exercises. So what we're trying to do is match that type of speed as well. It's the world of compute now. I mean, I was at Alibaba conference, they had their cloud cause, he's hearing it here, the same message, more compute. They're not saying I need more little chips, they're saying I want more horsepower. And you guys just announced the C5 instance recently, couple weeks with Amazon. What is that going to do?  What does that mean? Is that going to be an IoT edge opportunity? Is it all workloads? That's going to be like a pretty big deal. What's your take on it? Their C instant line has always been for high performance, growing workloads. We are seeing like for HPC workloads, anywhere from two to four and a half X, the performance moving from C4 to C5, right? This is an instance that can handle the most demanding workloads from high performance compute to artificial intelligence to others. So we have our latest and greatest Intel Xeon scalable process in there, a very high-performance one that we customize specifically for their environment. But then they do all this amazing software work and efficiency work around it to really unlock. I was really glad to see when they talk about those C5 instances launching, 25% on workloads, price performance, up to 50% price performance improvements on some others. So, I mean, once again, when you can take more compute and make it more cost-effective, it's just a lot more things people can do in the industry. So we're very excited about that instance. What's the biggest thing in the past five years that's jumped out at you and the massive change in the industry? Applications, startups, business growth, what amazes you? There's so much I would kind of combine it under what the industry calls digital transformation. You know, when I look at it, one of the points that always sticks in my mind is, CEOs, 70% of them have digital transformation at the heart because the data suggests that by the end of 2018, the top 40% of the top 20, 40% of the top 20 industries are going to be disrupted. So that to me, that amount of disruption happening and the companies trying to disrupt themselves, whether it's healthcare, retail, manufacturing, oil and glass, the use and pervasiveness of where technology and cloud can fit has really kind of astonished me. And I love, once again, I love that they're making what they do today better, but the new things that they're doing, I mean, in healthcare, right? It's just amazing, so. I mean, we used to use the word back in the day when I broke into the business in the 80s, data processing department. I mean, the cloud is one big data processor. I mean. It is. It is a computer power. We call it the brain, right? If cloud is now ubiquitous, right? It is from public cloud to private to edge and everything in between. It's that brain, right? That's enabling the compute. So we have to. There's always been the inside of everything. Congratulations on your success on the cloud growth. General manager of the cloud service provider and vice president of the data center group, Regene Schiller here inside theCUBE with Intel. I'm John Furrier, Justin Warren. Back with more live coverage here in Las Vegas for Amazon Web Services 2017. Re-invent more after this short break.


Okay, good morning. Good afternoon everybody and welcome to the latest Jenkins advocacy and outreach SIG meeting For a gender today, we've got three items I'll contribute to summit the video uploads and Join summer student presentation. Is there any other topics folks want to? Mention upfront I've not feel free to at the end of the meeting if something comes up Let me know and we're happy to discuss it. Does this look like it's recording to you? Yes, yes, it is. Okay, great Okay, so I'm deliberately for this meeting just not going through the actions. I know I've followed up Alyssa as a way currently and Skyler with the 4th of July she's not around today and But I've been following up with her on the blog post and we're a bit behind on those But still the intention is there to get them done, but we're taking the pressure off any kind of deadlines for now Okay, so contribute to summit One thing We have said is that they will we will do a contributed summit on August 12th such as before DevOps world Jenkins world and It will be similar structure to the ones we've had in previous years except this time will do it in conjunction with other CDF projects, so well While it will still call it Jenkins contributed summit will also have will also be calling it, you know CDF contributed summit and The other communities involved which are tacked on specifically they'll be also Kind of running a co-located contributed summit So what we think this means in practice because this is the first time doing it that this way is That in the morning We'll have some shared sessions across all the projects. So any shared topics related to continuous delivery Or maybe things like, you know joint outreach programs or events We'll talk about those all in one room and then after those shared topics I don't know if that will be mid-morning or after lunch. Then the different projects will Go break out into different tables and can do a bit more kind of unconference style So I Have a Meetup group page set up for that if I can find it So in the meantime, are there any questions or thoughts about what would be good to do there how it should work? Yeah, one question is about CDF governance So we'll be the special sessions about CDF governance and other things Because from what I see many contributors especially in Jenkins project They still don't understand how CDF works So maybe something like CDF 101 would be really helpful for the summit Yeah, let's put that down as topics So we can have a specific session on that and I think there's also Some work to do with CDF in terms of Jenkins Governance, so I think there's conversations we should have with Jenkins about how to What structure makes more sense going forward in terms of I think we talked about Sort of the replacement for the The steering committee meetings and a new kind of stewardship board made up of Jenkins contributors So I think that would be a really good discussion to have face-to-face So shall we put that down as one of the potential topics but potentially yeah, it would be interesting Okay, that's good Okay, so it's just going to show for now I do have the We have the page up on the the Jenkins developer and use a meetup group. So I'll copy this link So we've got a link to the event bright page It was also reposted from newtab.com if I recall correctly Sorry, could you repeat that? Oh, yeah, this is the newt here showing So yeah, so asking people to register so it's a free event But they do need to register just so that we can plan accordingly for numbers and food and Additionally, it's likely we'll have a An appreciation event in the evening again, it might be up leveled as a CDF appreciation event and we invite Every the other projects to come along But this isn't fully confirmed I think it was very likely to happen. So people should plan to attend but we're just Waiting to hear one final confirmation before we post it publicly. Oh and look we already have folks who can attend So that's great to be there mark any specific topic you might want to speak about or hear about I can do I Mean I think we talked about the Google summer of codes. Oh, look, will you be there for this? Well, it really depends on my visa Yeah, so we I think we had talked a little bit about in the in the Google summer of code SIG Martin and I will be there. So maybe we could do sort of a joint topic on that You also Martin will be there if I understand correctly. Oh Great and the Jeff Pierce will be there as well. So we get four or cut means Well, basically except me and Rick Rick's going to be there. I believe Okay, so subject to be as well Okay, well fingers crossed for a few but yeah, and I think Based on last year, I thought that the GSOC presence made a big difference in terms of Getting interest getting more mentors So it was definitely one of the kind of most active things that happen at the contributor summit and probably I would say I don't know for sure but contributed to to the high levels of participation and just a lot of momentum Speaking of it, maybe it makes sense to have session or maybe there's a few there for Whatever student programs intention problem programs because it's not the GSOC now it's also outreach. It's community bridge Which we are trying to kick off maybe Google code in if you participate this year, I'm not sure So yeah, if you have such session for whatever community program and talk to other projects because Yeah, I'm not sure about tick tone about Spinner care, but in this project, we have some experience and we can talk so maybe we could have something from the CDF later eventually Yeah, I think that that's a great idea and I've noticed like I'm trying to keep tabs on what the tecton folks are doing and like they were keen to do a mentoring type program and In some ways that's similar to community bridge. I think they're looking at the ones Kubernetes does which is pretty I think big wide scale in terms of just connecting up mentors at any time at any point Regardless of internship program, but making sure people can get connected to folks who can help them so It's also something we could do So yeah, we do it at Focke now. So when people come to whatever we try to guide them But yeah, haven't got something more formalized having country points for that would be nice Yeah, I think that would be good to figure out as well if we can work together and Yeah, just share resources be nice I have a question in the event bright page for the computer contributor summit Sail has ended. Oh I don't know if that's supposed to be a say Also, there's no way to Yeah, sign up is not available. Okay. That is good debugging Yeah, let me get that Skylar and I'll cut to fix that except. Oh, she's away today. Okay. Let me make a note I can I see you I hear you So Tracy are you still there? No No, she she dropped should I pause the recording? Oh If you have access to that, of course, I Just didn't know what there is such fish like Yeah, there there is a feature for the host and I think because I got here At a certain point there you go Tracy you're back. Yes, sir I think my connection is flaky. I'm just gonna go off video, but is that still recording? It is still recording. Okay. Perfect. I think I said it to record to the cloud bits. Hopefully It will keep doing that okay, so I Don't know where I dropped off, but I think had we finished with the contributor summit So just video uploads just to summarize the discussion we had just before the meeting started I need to get access to the share to the YouTube channel so I can upload this videos directly and so paying one of the folks who has access and in the meantime if I can make those available on Google Drive or public place like I guess the zoom cloud for this recording then perhaps mark wait or I can help and Then we'll have a playlist Oh and Oleg says playlists can be configured to automatically add by labels. So that's awesome Okay, so on to the next item, which is the joint summer student presentations Yeah, I'm scrunching for you right now so that you get a bit of an bite Thank you. So who's is this your topic Oleg? Do you want to lead the discussion or is that you Maki? This one Yes. Yes, so basically It's what we were discussing In yesterday the GSOC meeting Now we have five active GSOC projects in the Jenkins community plus to our three students plus we kickoff community bridge and plus there are Summer students working on GSOC and other projects. So there are at least three organizations Which also have students who work on Jenkins related things And the idea was that what if actually what if we do a joint session so that we just Bring every day It might be useful for community bonding. It might be useful to just share meetings between the stakeholders and We do all the recordings and events in GSOC anyway So for us, it's not a bit ever not to be covered but yeah we could host it as a part of second or third face evaluations in GSOC What's the timeline for that? I think this would be great The second evaluation I will be from 20s to something like 26 of July and The third evaluation is a bit more tricky because third evaluation collides is Jenkins both if I recall correctly so it's mid-August Okay, I was thinking rather about the second evaluation and this would be a virtual event. I expect yes Yeah, it would be visual event So yeah, we can just run Jenkins online meet up or whatever Have two sessions one morning in European Union for Europe and the Asian and Pacific region Another one is even session so for my request and the European And yeah, we just run several demos during each sessions plus Kone. So that's how we used to approach your GSOC Over previous years and it works pretty well Yeah, I think that works well, and I think I saw that you'd already communicated that with the folks doing outreachy and Matt sicker seemed keen like he was up for it Yeah, I don't remember that but yeah, maybe we communicate it Yeah, I think in principle it seemed fine. So do we I mean, I think we can go ahead and Plan that as a job What do we need to do to work out what date to do I mean, I'm happy to all this Doodle so once I get confirmations from GSOC or convenience that they're fine So far I get a plus I would plus one from Markey, but other or kidneys haven't applied yet They'll start to do once we have agreement and doodle just Great just for an online meet up what I Don't know why this matters, but it seems to in terms of the title. What would we call it as an event? This is a good question Wanted to represent Yeah, all the should we do it as a kind of an Outreach like part of this advocacy sig or why not or we can just call it something like summer project demos Yeah, so final name is need to be there fine But here you just have the draft of the meet up announcements so that all the event managers will be able to review it Yeah, why don't we we could go ahead and just start creating a draft now Gonna try sharing my screen again, okay, so I'll stop check You can take over. Yeah, let's see if I can share And hopefully Internet holds up Okay, so I'm in Jenkins online meet up and let's just put a placeholder for it. So this would be Jenkins summer Demons So he's put gsock and outreach Yes, just a cow three shift plus. Yeah, maybe sliding is interesting to join because he already contributed Some needs for jcask. So yeah, maybe you let come you into bridge, but we can start from there Or maybe just do it like this Yeah, I drink some demos Gee and Let's placeholder for now and so rough dates it will be this week 20 second Yeah, basically, it will be 23rd 24th or 25th because 26 is evolution that line for jsock. We don't want to postpone it in the very last date and Monday just gives Students and mentors some time to the driver Okay I'll put it just there for now. I know this. I don't know what time zones these end up and Would it be an hour? We're gonna do two different sessions, right? Yeah, it will be probably 90 minutes or maybe one hour. So it really depends on how many speakers we get because yeah, I will need to Reach out to three communities, so it's CCRM for see and Eclipse if I recall correctly Yes, we will just need to talk to them if they join the sessions then Okay, so what else I mean, we don't have to fake Do this all now, but it'd be good to know what to follow up with do we have something we can use for this? Or do we should we maybe We have some Jsock artwork, but you're for the sessions probably you'll need to create something else Okay, so I need some artwork just to and then a description and So this would just depends once we get the speakers You're right. So I can prepare this content. It's not a big deal Okay, so I have a gender and then the location will just need to have the And we'll need a host for it. Yeah, awesome because for YouTube life, we are still limited by 10 participants Okay Usually it means that if you have three or four students plus we have at least one or could mean Who runs the meeting plus we have mentors and yeah, here we are capacity limit So maybe no family uses on for these sessions Yeah, so can you assume the other thing we could try although maybe this is not the right first time to try it But there's a platform. I'm seeing some other communities use for webinars, which is called crowdcast and that Although I think that might work better for just a couple of people presenting But they're things like you can have a green room and you can pull in speakers and Kind of sopping out speakers and there's a nice chat integrated But actually maybe zoom will be simpler for this time Okay, I'll add you as a co-organizer And then we'll see who else we can get Yeah So basically you can just see if it is draft and we will clean it up later So I have access to this page. I can edit it on my own Great just to do. Okay. So yeah, so let's get that in place and then I'm just gonna write this in Okay, we can survive with these graphics because we use some on me top Yeah We need the host and We need the time and we've got their graph which I can link to Nope Yeah, no word is so these things are private anyway So on the event and access to them and the event managers know where to find them Okay, and we're likely. Yeah, you said we'll do this too Good and then we're reaching out to So GSOC outreach II and if you said eclipse GSOC as well potentially So eclipse CDC realm and Fosse so for foresee everything is organized because yeah, I'm a mentor there So it's not a big deal for me Then for two other projects, I know the context of So I can just send a message and see what they reply Yeah, I'm happy to reach out to the eclipse GSOC folks But yeah, I think if we set our date once we set a date based on The Jenkins GSOC students then we can just reach out to them and say hey if you could join it would be great to Add you to the agenda. Well, we can just send a little to them So then they can also vote by the way maybe it's So yeah, it's yeah, it's eclipse foundation Okay All right, anything else on this Presentation That's fine. Yeah, it'll be good to just get I think we've we haven't had any online meetups for a while So it'd be good to just get those going again and You do most Mostly do them a set of meetings Okay, cool Okay, next topic We have get a channel for newcomers Very created but need people to help with balancing the load Yeah, so basically it's what we were discussing a couple of weeks ago. So it's just Item for update my plan is to create this channel Before that I will send a message to the development team. So if I find Few people who are interested to spend some time there. Well in GSOC we regularly Receive messages because apparently many newcomers end up in a GSOC Gither channel. Yeah But yeah, we can Just creating a channel for it Like I can help in that channel Then we can just create it Great. Have you sent the email already? Is that something I should reply to or not yet? I will send it Okay, I'll keep a lookout and yeah, I can add Just volunteer visibly in front of people and hopefully it will encourage other folks to to join in But yeah, no, I think there will be nice kind of entry point for people So I will read the documentation so that it points to right place because now we have for Jenkins. Ioslash participate page But this page doesn't really provide any entry point contact for interested contributors Yeah six months ago, so I added Contacts to many pages. So all Jenkins are your pages support referencing Gitter or select channels via metadata Once the channel is created, I'll edit there as well Great Yeah, that'd be good to have another topic If you're done on that one, I think that will be really good to do and just kind of get people more and more involved and Maybe even find a way for once people are Kind of on-ramp. They're happy to help. I know some schemes you kind of have it So people are encouraged to you know, give back once they are more and more up to speed Which seems to work well for outreach in GSOC. So maybe a more general one would be great I'm I was just gonna bring up the Jenkins IO Sorry the github to take a look at the How we're doing on pull requests there could be better, but So we have a few pending blog posts The most of them are from JSOC we lent them So Do you is there anything I can help with on these No, JSOC blog posts are fine in the blog post from them and I in the top I mean the second pull request from the top is also ready to go. I believe you will mash it today Okay, but Merge now like we shouldn't like people are planning to do it Yeah, so for so the biggest problem is about the documentation pull requests But yeah, this documentation Pull requests really need some assistance from documents special interest group So No for pull requests in the bottom This one yeah docker image in documentation. It's something it's a lot left over from Java 11 So yeah, I'll just be in people because I work with other and so I'll just check the And so three other pull requests. Yeah, they came from country Utters whom I do not really know But yet the pull requests seem to be reasonable and it would be great if we finally integrate them So ignoring the on-hold ones this one I was tempted to just Unfortunately close it because it's been a while and I don't think This it's been sitting there ever since I've been here, but I don't think unfortunately I think people have moved on Is that something we can do or just ask people to reopen it And so maybe we can bring up it again to model the documentation special interest group Okay, it's interesting to take it over there. Maybe we could Okay, measurable in the current state Yeah Yes, you do that and that that otherwise I'm gonna recommend we just close it and Let the folks know involves say if they are Interested and they reopen it then we can get somebody to work with them on it. I Can help connect them with somebody Okay, so Yeah, I think just see what we can do to help keep things ticking over Yeah, it's really great all the the GSOC posts coming out now. So and as you if you could keep letting me know in the Gitter I can keep tweeting them out Okay, I think that's everything I had from my side Okay, anything else for today of anything Great. Well, I think this is really good and lots of exciting things coming up So, yeah, thanks everybody and yeah follow up offline Thank you for anything meeting And hopefully Hopefully you'll be able to discuss more GSOC topics. I'm sorry not to do this whole topics next time so yeah, and the next meeting will be at kind of the later time and Hope to have some folks back in like Skyler and Alyssa Okay, thanks everybody. Goodbye. Have a good one


I'm happy to introduce James Sariawini. He's a software engineer working at Amazon Web Services and he will present a talk with the title, Riding Redis in Python with Async.io. Give him a hand please. Hi everyone, welcome. Thanks for being here today. So as mentioned, my name is James Sariawini and today I'm gonna be talking about Riding Redis in Python with Async.io. A little bit of background. The reason I started this project in the first place was, really I just wanted to learn more about Async.io. I had heard a lot about it. Everyone was talking about it. I've read the docs, but I didn't really understand how we could use this to write something a little more realistic. And at the time, I was familiar with Redis and I thought, you know, Redis has some features that looks like it would be a really good fit for Async.io. So I wanted to see how far I could take the idea. So I started exploring Riding Redis in Python with Async.io. The reason I wanna share this today is because I think there's some useful things that everyone here can take away from it. I wanna show how you can structure a larger network server application. That wasn't very obvious to me from reading the network, the Async.io documentation. And then I wanna show you a couple of patterns that happen in Redis that I think apply in general to various types of network servers. So the basic request response structure, how that looks like. And then also a couple of other ones that are interesting, so publish, subscribe, and then blocking cues. So if you're familiar with Redis, this would be the BL pop and the BR pop, the blocking left pop and blocking right pop. By the way, if you're not familiar with Redis, that's okay, I'm gonna be explaining all of the specific features we're gonna be implementing, so don't worry about that. But I think that these patterns apply to, say, a chat server or into any kind of your own task queue or job queue. I think in general these would be generally applicable. Before we get started though, before we dive into it, I just wanna give a little bit of a disclaimer. I'm not an async IO expert. As I mentioned, I wanted to learn more about it. So a little bit about me, I work at AWS and so I primarily work on, these are the libraries I primarily work on or have written a substantial portion of. And the point of this though is that I am more, my area of expertise is in Python libraries and supporting multiple Python versions, that kind of thing. So writing an async IO network server that only runs on the latest version of Python is not something that I would consider myself an expert in, so I'm just a beginner trying to share what I have learned.  So just as a warning. Okay, so the one minute introduction to Redis.  The one thing that's kind of interesting is values can be more than strings. So you have your basic case here where you can set foo and bar and you get a value back and then I can also get a value and it'll give me that value back. But in addition to that, I can also have lists. So in this case, I can say R push, which is a right push onto the foo list, the value of A. So this is like foo.append A in Python and I can do that for three elements, A, B and C. And then on the right hand side, I can say L pop, which is left pop from foo. That's gonna return me the first element A and then I can do L range from foo zero to two, which is basically a slice. So that's gonna give me the elements from zero to two, which in this case is B and C because we popped A previously. So that's the first thing I wanna shoot for and I'm gonna look at how we can do a basic request response. So at the end of this first section, I wanna be able to send a get request, interpret it, figure out how this is gonna work and then send a response back. So I did, I think, what most people would do, start at the documentation. If you haven't seen the documentation, this is the async.io docs. You start at section <IP_ADDRESS>.2, the obvious place to start, which is the TCP Echo Server Protocol. And so I changed one line here, but this is essentially straight from the docs. And by the way, I'm gonna be showing a lot of code here. I'm gonna highlight the specific parts that I think are interesting, but it's okay if you don't, you don't have to follow or understand all of this. I'll put the slides online so you can look at it more in detail later. But there's three parts to setting all this up. The first thing is we're gonna do is create an event loop. We're going to then call create server and give us the Redis server protocol. So this is the class we're going to write and this is going to give you all of the logic for handling the Redis protocol, parsing, figuring out how to call into our database, all that kind of stuff. And then this will set up the listening sockets. The next thing we're gonna do is call run forever. This will wait for a network IO and then call the appropriate things into our protocol. And we'll look at that in a second. And then the last thing we're gonna do is just clean up. So once we wanna exit, we close down the server and call loop.close. But I wanna look at this Redis server protocol. I should mention real briefly that there's I think two other ways you can do this. You can use the low level sockets in the async IO loop and you can also use streams. Streams are a little bit higher level. What I found is streams while they were really nice to use, the performance for me was, it was substantially slower. And I think protocols for me was a nice middle ground between still being fairly expressive but having pretty good performance. I'll show some performance numbers at the end. So here's a protocol. So the idea is you have connection made, data received. You get a transport for connection made and then when data is available, you call data received. So I looked at this and I kinda understood what it was doing. I think that there's some history there from some of the other frameworks that influenced this but I still didn't really get how protocols worked. And for me, going one level deeper into async IO and trying to understand how all this works really helped illustrate for me. So I'm gonna go kind of briefly into async IO code to understand how protocol and transports work because that's really gonna drive a lot of this implementation. So first thing, start with the Redis server protocol. And in this loop, what's gonna happen is whenever a connection is made, async IO, notice in the top right, this is async IO slash selector events. So what's gonna happen is you have a callback. Whenever a connection is made, you call except connection two. And what's gonna happen here is notice a protocol and transport are created for every connection. So every time you have a client connection, it's gonna create a protocol and a transport. And this part here with the protocol factory, that was the Redis server protocol class we passed in. And notice it was a class objects, not an instance. So it's gonna call our class that's going to return an instance. But the main thing here is that there's one protocol and transport per connection. So if I had three clients connected, I would have three transport protocol pairs. And the next thing now is let's see how connection made and data received are gonna be called. So if I look at, this is the selector socket transport, still in the async IO code. When you create it, there's this first line here, which is self.protocol.connectionmade. Connectionmade is this thing that we're going to write and we can see how it's gonna be called. The loop.call soon, I found that kind of confusing. It's really just loop.schedule this thing to be run or loop.add this to your to-do list of callbacks. But essentially what this is asking is for this method to be called and then the arguments to pass. So notice that the last argument is self and the class we're in is a selector transport. So we'll see that again when we go back to our protocol class. The next thing here is the read ready method. So again, another callback and look at the last argument, it's read ready and we'll dive into what read ready is. Still in the async code again, just kind of showing you some highlights here. And the main part of this is this protocol.data received. Again, this is the method that we're going to write. So what's happening here is whenever there's data on a socket, we'll read from it via socket.receive and then call data received. So the main thing here is that these are just callbacks. So they're not co-routines and not anything fancy like that. We just saw, they're just simple callbacks that get scheduled whenever a new connection is made. And so if an event loop was like this going from left to right, if I had four connections as data comes in, I'm gonna call data receive onto those protocols. So hopefully that gives you an idea of how protocols and transports work. So we can start fleshing out the basic get and set response here. This is how more realistic data receive might look. So the first thing I'm gonna do is gonna get data over the wire. We're gonna call into our parser to parse the wire protocol. I'm simplifying things a bit and I'll discuss that at the end, but we're gonna parse the wire protocol. And so this will give us, say, a list where the first element is the command set, next one is a key, the next one is a value. And then once we do that, we're gonna look at what command we were provided. And then after that we'll call into the DB layers. So we'll say self.db.get and self.db.set and we'll look at how that looks like. And then finally after that, we're gonna take a response, ask our serializer to serialize it to bytes and then use transport.write. So a transport is really just an abstraction over a socket. It allows you to write data back to the client. So whenever we have something to send to our client, we say self.transport.write. And so this is the basic overview for the first part. And also noticing connection made, I should mention we're storing the transport. So that's how we're able to write back to that client. So it's one connection.  This is the Redis serialization protocol. So it's a text-based format. This is just how it would actually look like. We'd get data over the wire that looks like this. Stars like the list type and it's three elements. And you can kind of see there's a set foo bar there. And then this is what we would return. So just to give you an idea of what we're actually sending back over the wire. Okay, next thing, the DB class. So this is where all the logic happens for the Redis commands like get and set and all the list manipulation. As you can see here, it's just essentially an abstraction over a dictionary. But the main things here that I found the DB being a separate module, so it doesn't know anything about async.io. That's really helpful. So you can imagine how you'd write a unit test for this, right?  So very straightforward to test. Here's how some of the list commands would work. Again, just, hopefully you can kind of just get it just to this, right? For R push, we're essentially gonna manipulate a list. For the L range we looked at, it's looking up that key and then slicing based on the start and stop. And then same thing with L pop or popping from the front of the list. By the way, use the collections.dex so you get the constant time pop left. But I'm just as an example, we're gonna use a list here. And you can see how you can start to integrate that into the protocols here. So realistically, you'd probably extract this into a command handler class. But here I'm just adding FL statements here that are gonna say, if it's R push, we're gonna figure out how to construct the appropriate columns to the database layer. And we keep that nice level, that nice separation between the async code and R logic. Okay, so at this point we've covered the basic stuff. We know how to get, we know how to respond to a get and a set and a list command. You could use this to flush out the other types. So sorted sets, hashes, all that kind of thing. You can get a lot of the commands implemented this way. But I wanna cover two more interesting cases that I think took me a while to figure out. So the first one is publish, subscribe. What we wanna go after here is we're going top down. So we have a client that connects and says they wanna subscribe to a channel called foo. And then we have another client that wants to subscribe. And what's gonna happen is at some later point, if another client comes along and says, I'm gonna publish on the foo channel this message, we wanna be able to write to every subscribed client for that channel, that message. And then just to give you something a little more concrete here, it's that same example that's actually using Real Redis and Real Redis CLI. So what you'll see here is I'm gonna subscribe to two channels. And then on the third channel here, I'm gonna publish. And the main thing I want you to see is when we publish in the bottom thing here, you should see the top two get the message. And that's what we're shooting for. So we're gonna publish two messages, here's the first one. You can see how both got that message and then we do it again. And so then they both get the next message. So that's ideally what we're trying to implement here. So how do we do this? If we remember that we had one transport and protocol per connection, we need something a little bit different. We need to be able to, from a given transport protocol pair, somehow communicate to other transports that are interested and write data to them. And what we're gonna take advantage of is this protocol factory. So I showed you some async IO code where we were doing protocol factory and instantiating it. We're gonna take advantage of that in a second. But the actual pub sub part is pretty easy. We're gonna create a new class and whenever you call subscribe, we just have a dictionary of a channel name and a list of transports. So you give it your transport when you wanna subscribe to something. And then the publish is equally straightforward. You look up the channel. You have a list of transports and you iterate for every transport in transport. You just call transport dot write. And hopefully that's pretty straightforward. There's not really a lot of async code there. It's actually pretty simple. And the way we integrate this, same thing, we're going back to our Redis server protocol. So in this command here, we're just calling our pub sub object dot describe and notice we're passing our self dot transport. And then same thing with publish. So this part's pretty straightforward. The thing that was kind of tricky is how do we actually get all of the stuff wired up together? And so you remember this very first line we looked at, which is we're passing a class object here, not the instance. We're gonna change that slightly into a protocol factory. And now we're gonna pass the factory to the create server. Now all this is gonna do is just store a reference to the class and it's gonna store args and keyword args to pass. So this is also basically like functools dot partial if you're familiar with that. This is just kind of making this a direct concept in the code. So now whenever this gets called, we're able to pass in a shared object. And so the thing that was useful for me here is that instead of having transport and know about other transports and having to coordinate all that, you just tell the pub sub object and the pub sub manages which ones it needs to call out to. And so they again, keeps your logic very simple, very easy to test. So that's publish.subscribe. Last thing I wanna look at, that's probably the most interesting one is blocking list pops. So this is what we're shooting for. I'm gonna subscribe to BL pop, which is a blocking left pop on the foo key and zero is just a timeout, which means wait forever. So if I do that with two clients, notice I don't get a response right away. But now if another client comes along and does an R push on the foo list, I'm gonna pick that value, the bar value and send it to whatever client's been waiting the longest. So this is essentially like a queuing system, right? And again, just to give you something more concrete, we're gonna do the same thing here. Create two clients. We're gonna have both of them block and we're gonna publish two messages. And this time you'll see one client gets the first message and another client gets the second message. So hopefully you get what we're shooting for. We only wanna give it to one. So now we need to manage which one's been waiting the longest. So how do we do this? This one was also took me a little bit of time to figure out, wasn't intuitive to me, but we're gonna use the same idea or we're gonna use a shared object. So this is a key blocker object. And the way we do this, we'll start at the bottom and kind of work our way up. So remember that database object, which is not supposed to know anything about async IO. So what I did here was the same idea where, if there's something in the list, we don't have to block, we return right away. But if there's something that we do have in the list, or if there's nothing in the list, then what we need to do is block. And instead of having the database object, know how to block and start integrating and coupling async IO code, it just returns some sentinel value that says you have to wait, right? I don't have any data available. Whoever's calling me needs to figure out how to wait. And if you go up one level of the stack, we can see in our data received, same kind of thing here. We're gonna pass in our database and our key blocker in our loop. But here we're gonna call it BLPOP, same stuff as before. But if we get something that indicates we need to wait, this is kind of the new part where we're gonna look at async and await here. So I'm gonna call key blocker, this new object dot wait for key. I'm gonna get this thing back really to co-routine. And then we're gonna say loop dot create task, this co-routine. So remember earlier I showed you how the data received was a callback. And one of the important things there is that you cannot block that. If you block that method, you block the entire event loop, right? And everything stops. So the best that we can do, if we wanna block for something, is create a new task. So if you're familiar with threads or something, create a new thread conceptually and ask the event loop to run it. And that thing can block. So that's essentially what we're doing here. We're creating a new co-routine and we're asking the event loop to run that in its loop. Okay. And then now that we have the corresponding blocking part, we need the push part. So whenever data comes into their push, we're gonna tell the key blocker about it. And that's this part here where we're gonna say create a new task with data for key. And so now let's look at how the key blocker looks like. And so there's a couple of new things here. There's the new async and the new await stuff. And we'll go over how this works. So the first part is the async def wait for key. This is new to Python 3.5. This is creating a native co-routine. And the important part here is that we're gonna use an async.io.q. So if you're familiar with the q.q that's used in a multi-threaded scenario, it's kind of the same idea. And essentially what we're gonna do is block. So we're gonna say value equals await q.get. And what this is gonna do in this co-routine is it's gonna sit here and wait until there's actually data available. And because we're using the async.q which gives us the FIFO semantics, it will unblock the one who's been waiting the longest. First in, first out. And so once we get our value, we can then do the transport.write for that single, only that single transport. And then we have a corresponding thing for data for key. We don't technically need to use it. There's a q.put no wait, but I just wanted to show how you could also do that. Okay, and I found async and await, while conceptually it's kind of easy, you just put like await where you would wanna block normally. I wanted to know kind of how that worked. So here's a very, very high level overview of how async and await work. So what's gonna happen is we did the call soon, right, for our task. So we have wait for key happening in the event loops. If the event loop comes along and says all right, we're gonna call this method. And then remember we called q.get. And one of the things that I found, I realized was that whenever there is an await somewhere deep buried in the guts of async.io, there is a yield somewhere that is the only way to stop Python code from executing and essentially save the state so you can resume it with a code routine.send, right? So there is a yield somewhere deep in async.io. And at this point, we go all the way back up thanks to the magic of yield from and we have this object future. Doesn't really matter. This is kind of like the, what it's called, but what happens at this point is this is essentially frozen. So the event loop knows about this call stack and it's frozen. And then something else comes along when data actually is available for the key. And we say data for key, which is gonna do the q.put. And that q.put is then gonna have a value associated with it. And again, I'm kind of glossing over the details here to simplify things, but what's essentially gonna happen is this is going to unlock the future. So this is gonna say that this future is done, which will then schedule it to be run in the event loop. Then this value here goes over to the yield and resumes and then now you get that value back from your q.get. So at a very high level, you can kind of see how that would work with this async and await here. That's essentially what's happening here. Okay. So that is basically how you would do blocking list pops. You do the same thing for a BL pop and BR pop from the end of the list. There are a few additional considerations. There's things that I didn't really have time to cover that actually changed a lot of how the internals work. So first thing, the real parsing is more complicated. So I made a big assumption that we're getting all of the data as a single request. Realistically, you would get partial data, you would feed the parser and it would tell you when it had a complete command for you to run and you could potentially get more than one command if you're doing pipelining or that kind of thing. Pub stuff can handle clients disconnecting, right? So there's also another method on the protocols that we didn't look at, connection lost and that's how you can handle disconnecting. And there's also like some advanced pattern matching that kind of stuff. And then the last thing is that blocking queues can actually wait for multiple queues. So I can say BL pop, you know, foo and bar and then when data's available on either of those it will unblock. And so for that I couldn't use an async.io.q, we actually had to go a little more low level and use the async.io.futures. Okay, last thing, performance. I was curious, so there's a Redis benchmark program you can run that comes with Redis. These are the parameters it did, so just the basic set and get. And on my laptop, Redis server, we got about 82,000 requests per second for PyRedis server. I got 24,000 initially. I was at Yuri's talk with, he was talking about UV loop, so I thought, let's try it out, see how well it does. And just plugging UV loop with no changes brought it up to 38,000 requests per second, so I thought that was really cool. We're a little more than two times as slow. I'm very unoptimized code. When I profiled it, it was mostly in my parsing code, which can be optimized quite a bit. It's inefficient how I'm doing parsing now, but I thought that was really cool. So we're pretty close. So just a summary of what I covered here so far. Looked at transports and protocols, hopefully showed you a little bit more in depth, how they work, how they pair to a single connection, and then we looked at request response and then some other patterns for how we can share a state or how we can communicate state across various transports. So looking for publish, subscribe, and then blocking Q-like behavior. So I'm gonna put to these slides online. I don't know where they are, where they're gonna be yet, and I'll put the code online as well. I don't have links, so I guess for now, I'll tweet them out eventually. So that's probably the best place for more information. But once again, thank you. Thank you very much, James, for this talk. We take a few questions. Hi, great talk. Why did you write your own path instead of using pre-existing pathers? There's, for example, the high-readest pathers. Is there C interface to the C pathers of Redis? Good question. Mostly just to see what the overhead would be, just kind of learn more about, I mean, mostly this was a project to learn more about how Redis would work and how to implement it. But I think if I was actually going for performance, that would be the next step is either try to clean up the parser myself or to just use the high-readest, which is a C library to do the parsing for me. And I think that would get pretty close performance to the real Redis. So yeah, that's something I want to look into. Okay, some more questions. Hi, great talk. Do you made the tasks with Reds with persistence or not? Oh, are you asking about if we persist the data to disk? Yes, yes. So that's not something I'd looked at. I don't actually know the best way to do that. So with Redis, it's just as the fork and exec and then writes it in the background. That would be my first attempt. It would do something similar, try just the fork, exec, and figure out the serialization, the RDB serialization, and see how far that, how well that works with AsyncIO, but I haven't actually tried it. Okay, thanks. Okay, one more. One last question. There it is. And what do you think about AsyncIO? Did you like it, the experience, or what's your conclusion of? So I think one of the biggest problems for me was just, it was hard to figure out how to do things. And I think, not coming from a background of writing a lot of AsyncIO code, I think the docs, and I think I heard this mentioned earlier, the docs seemed a little bit of work. It was not, I could understand how building blocks worked and how coroutines worked and tasks and all that, but I think how to fit things together, and I think just having more examples, having more documentation about that would really help. As far as the internals, I found it a little confusing. For the people familiar with AsyncIO, how you have futures and tasks, which really should be called like coroutine driver and how that schedules things and you get futures that then have callbacks and the way that kind of works, I found that a little bit hard to understand. I've been looking at some of the other ones. The other, I think, frameworks like Curio was one, that seemed a lot simpler for me to understand. But I think for me, the biggest problem is just having more examples of how to do things and then it's a great framework. So thank you very much for your talk again. Give it my hand. Thank you.


Guys, we're doing a P.O. Box stream today. We got quite a few things today. Man, my room, I gotta clean up again. Okay, ugh. Okay, so this was shipped to my house from China by FedEx, and I don't know what it is. I have no idea what it is. So... I just figured I'd open it on stream. Some of this stuff is carried over from the last P.O. Box. Which one should I open first? The China Box? Yeah, well, this wasn't even in my P.O. Box. I don't know where this came from, but it was sent to my house. And it was FedEx. Whoops. Okay. Yeah, and it says... The box says S-Fan on it. Right here. It actually says S-Fan. Do you see that? Don't leak address. I covered it all up. Don't ruin it. Good knife cutting cardboard. It wasn't cardboard. I just cut the tape. It says S-Fan. Okay, it says S-Fan on the inside, too. What is this? I don't know. It says S-Fan on this. Again. Razer Black Shark Version 2 USB Soundcore Special Edition Multi-Platform Wired Headset for Esports. Wait, hashtag ad? Wait, Razer sent me a headset? Dude, free headset. It's in a bag. Wait, how many slots is this bag? Not so small bag. It's only six slots. Trash. Is this good? Is this a good headphones? These headphones are awesome. They have my name on them. These are great headphones. Not even wireless. Yeah, I normally use wireless headphones because I walk around a lot. I might try these out and it's good to have an extra pair of headphones. I might try these out. So are these brand new? Did these just come out? They came out in the 70s. Nice. Poopoo from Ruchotify. Okay, thank you for the note. Fake poop. Eight different pieces. Fake poop. It's gummy in before ban. It's not going to be banned if it's fake. It doesn't look fake. It's fake. I know it's fake. It's plastic. I'm touching it. Eat it. I'm not going to eat it. Why is this one yellow? What is that? It's self-made. I'm. Thank you, Ruchotify, for the lovely gift. I sent you some of mine. Did you get it? No, J-Dog. I did not get any of your poop.  Look at this one. The Frick. It's not homemade. It came from Amazon. It's factory-made poop. Okay, let's see. What is this? Inflatable Halloween. Wait, this is cool. Happy Halloween, my best friend. Fun stuff for your streaming room. Many, many hugs from Granny Nani and family. Okay, we're going to set this up right now.  They're sure to bring a smile to everyone's face. Just follow the easy step-by-step instructions below to get started. Okay. Okay, so where should I put this? It has a zipper. Is there something inside that I have to pull out? Rise. Rise, my minion. Rise. Dude, I have the VR chat thing in the middle of the room. Oh, oh. Nice. It's like a pop-up for football. That looks good. That looks really good. Thank you, Granny Nani. Thank you so much. That is perfect. What a good gift. In a dark corner, thank you for the four months, man. Carmina Gaming. Thank you for the two of your hosts. Thank you. USB cable. Oh, frick. S-Fan. The TV. Look at that. Look at that, dude. Looks good. It's a little Paladin symbol. I got to put it with my feels good man or my feels okay, man, that Jenny gave me. Let's take this and put this right here. Let's see how it looks. I got to get a longer cable. Ooh. Pink. Like Paladin. Does it show up good on stream? Maybe this is better to put on here for now like this. Seriously? Any primers? It's only $5. You got to be kidding me. You put this on an apron for me? You've got to be kidding me. Shame. Blessness, dude. Church Mills. Thank you for the two-prime. Five months. Thank you, dude. Okay. We got one. We got a prime. Any primers? Next. Gachi bot. That's for me. Yeah. Thank you, creepy uncle. Thank you, dude. Beer bottle opening hammer. Dude. Wait, this thing looks sick. A righteous Paladin should be armed to the most epic of weapons even when opening bottles from high. Look at that. How do I open? It's childproof. Why? Ready? This is what it sounds like when he pulls. No, it doesn't. You know what it sounds like when I poop? Dude, look at this hammer. Wait, so what is this for? What is this screwdriver for? Oh, oh, oh. It has a little hole. Thank you, High Life. Thank you, dude. Thank you so much, man. A gift from you from Bed Bath and Beyond. Happy birthday, S-Fan. Here's something to make those long days and nights gaming a little easier. The world's only go anywhere. No pressure seat cushion. Uh, step one. Lay the cover flat with the gripping side down. Fancy yourself a scientist? Try the egg test at home. Any seat cushions two inches tall or more can pass the raw egg test.  This is important and put the egg on the cushion and sit. Put the egg in a plastic bag to keep any accidents from happening. If your seat cushion is shorter than two inches, we don't recommend the test. Apply lube first. Trust me.  So we have our egg right here and we're going to put it in the egg wrap just in case because we don't want to have an accident as they say. Chat. Hold. Three. One. Zero. Look at that, dude. No way. Put it between your cheeks. No, I'm not doing that cheater.  No, it didn't. How do we know it wasn't hard boiled? It's not hard boiled. I just I gave it the note. I gave it the perfect amount of pressure to where I can crack the egg without causing a mess. Okay. It was calculated ready. Hold on this thing.  You said it's impossible. That was easy. What do you mean? I did it wrong. I did flat palm. I literally went like that. Oh, no, I'm going to hold in the bag. Okay. So this is what we did. We bought this hammer. We want to see if this hammer or now we bought. We got this hammer. We bought the sprite. Ready? Very good. 77 delicious for three months. Yeah, it's very good. Yeah. If you guys keep wanting to send stuff for me to open on stream, I will do this every month. I missed it. I do it again. I can't. There's only one cap.  A Danny DeVito blanket. This is so soft. Wait, is this a towel or a blanket? It is so soft. It's got to be a blanket. Dry your junk on Danny. I think it's a blanket, dude. This is cool. Thank you whoever sent this. I didn't see a note. Very good. What a stream today, man.


I'm presenting to you today from the comfort of my home in rural Alberta Canada where yet another winter storm seems to be rolling in. Today I'm going to chat to you about what it is to be a moderate programmer. Why becoming one will make you a better programmer and maybe more importantly a better teammate. First a little bit about me. I'm the front end lead at inclusive where we do public relations analytics for large brands such as Lyft, Airbnb, etc. I've been using Ember for over six plus years since pre 1.0, since pre Ember CLI, and still using it for work and for pleasure today. This is my fourth time at Ember conference and my first time speaking at it. My first time attending virtually as well as it is for all of you too. So first off what do I mean by moderate? I mean avoiding extremes of behaviour. I'm just going to tell you a little story about why this matters to me. I've been a programmer now professionally for over 20 years and I've been through a lot of changes in the field especially front end programming. And I've done almost every single one of the things I'm telling you to avoid today. And I think it's really important to share this story because especially junior developers and other developers who are coming up, you're likely to make a lot of these. And I'm hoping that that I can shed some light on my path to what I hope anyway is becoming a better programmer. First off this is not moderate. I do not recommend jumping into your tub. So I don't know if it's in our DNA or if it's in our culture, but extreme behaviours seem to be endemic around programmers. And we have to work harder than others at actually curbing that behaviour. It's so much in our culture, we even have extreme programming as a programming paradigm that's been used by a lot of people over the years. So yeah, like I said, we have to work a little harder at curbing those behaviours. So this quote moderation and all things including moderation for everything I'm going to tell you today, there's of course reasons to ignore my advice. And sometimes we do have to go all in on something and do an extreme behaviour depending on the thing. However, I do think that we should be fighting for the middle ground, not only in programming, but really in life itself. So this quote was brought to you by my mother. Well, actually it was Oscar Wilde or before that apparently this quote can be attributed to Socrates. So the idea of moderation in society and as human has been around for as long as we've had society. So first item is the moderate programmer names things appropriately, but with enough detail. So let's dive into some examples of that. Here at the top we have a not specifically enough named array. So we have an array of items that contains a name of Canada, which I guess is a country, although it just says name, so I'm not sure. Code is, yeah, well, what kind of code? Well, let's go into a two specifically named example. Now we have country ISO code objects, which is better, but that's pretty long. And oh, look, we're repeating country ISO twice. So that's a lot to read a lot to look at that's going to make it more difficult for someone looking at it after. In the moderate way on the bottom, we have all countries name is Canada, and the ISO code is CA. So now we have a little bit more information about that that's going to be easier for someone to read. Let's go through some rules about naming things. You should name things three words or less, whether that's a variable or a function or a class, any more than that is probably two verbose. We should avoid jargon and metaphors. Try not using the words factory or object or class. All of those things are repetitive and do cause problems in the code. You should also use plurals where appropriate. This would be in the case of arrays or collections or maps. It signals to your teammates or for you looking at your code in the future that this in fact is multiple items makes it easier to understand. Also for Boolean's prefix with is has can. Again, this is another trigger when we're looking at it at the code later. It's much more clear that this is in fact a Boolean if it has those words in front of it. For functions use a verb the noun like get users find admin things like that. Number two, the moderate programmer writes small functions but not too many. So again, here's another example. This function is doing far too much. We got set country for user and we're passing in an ISO code and a save user, which I think is a flag. It's a little bit hard to understand though, not that this function has too many lines. It's only about 10. But that's is doing conceptually too many things. So let's break it up. Oh, we've broken out far too much. Look at this. We got now four functions where we have one before. So now we got to get user get ISO code, save user get country. And then the main functions start save user. Well, this isn't quite right. It's going to be really difficult to because every time we get to a console user equals this user. Okay, what does this get user get? So now we jump up have to look at it just becomes a lot more difficult to understand. Here's a nice moderate example of writing small functions. We have now two functions, both do one thing conceptually. The first one gets the country code or gets the country name from the ISO code. The second saves the user we check if the user is valid. We see if we get the country set it to the user save the user and set a user flag. So much more easy to understand this code than the last two examples. So again, some rules, a function should conceptually do one thing. Be aware of how many flags you have in arguments. Any flags, even one is probably a sign that your function is doing too many things. And you may want to reconsider that. Be aware of how many arguments you're passing to a function. Again, too many arguments, I'd say even more than three. And that's probably again, a good indicator that it likely should be refactored. Components are functions to and this is really important. So everything I said here also applies to your component creation. Because conceptually, components are functions as well. So your component should do one thing. You should be aware of how many flags you're passing to your component, as well as how many arguments or in the case of components properties, you're passing to it. So the number three, the moderate programmer writes tests, but only enough. So you've probably seen this before, eat food, not too much, mostly plants. This is a nice moderate take on eating by Michael Poland. And in programming, we have this. So Guillermo wrote this a while ago, write tests not too many, mostly integration. Now, if you take the idea that writing tests is like a nutritional balance for your program, they have some similar parallels. So writing tests, because if we don't write tests, we're starving our program, not too many, because too many and our program is going to become bloated. And we mostly want to do integration. Now, why do we want to mostly do integration? Because integration gives you the best balance of knowing enough about the program to find regressions and not enough that it's going to become tripped up over changes that aren't important, that don't need to be caught, integration test.  So I'm not saying that you need to do test driven development. I'm saying that you should make sure that you have tests in your own PR right away. Embraer integration tests are your best bet. Again, they're the perfect middle ground or the perfect moderate ground in where you're going in the best bang for your buck when it comes to testing. The unit tests know too much about the implementation. I'm not saying don't do unit tests, they're important, just as important. However, you should be using them sparingly and only when it's important for a regression to be caught. Acceptance tests can also become too fragile, and they don't really know enough about your program. So they're often catching things where they shouldn't be. And it becomes that you're fixing tests more than you're fixing problems. So don't forget, you have diminishing turns after whatever percent of code coverage. This is different for every team and every project. I consider talking to your team about this. See just how many tests do you need to catch those regressions to catch any bugs that you might introduce. So the moderate programmer refactors, but only eventually. This talk by Sandy Metz in 2014 at Rails Conference was really great. And she talked about refactoring. You don't need to be a Ruby developer or even a Rails developer to understand this talk. I highly recommend it. She says one thing in that that really resonated with me. And she says duplication is far cheaper than the wrong abstraction. She even has a blog post specifically about that. And I think it's really important. But above that, I've written right ugly code. And that's the one thing, if you're only going to take one thing from my talk today, I want you to take that.  Because unless you write ugly code, you're not going to know what beautiful code is. You're not going to know what the right abstraction is going to be. You also might make it harder for your teammates to read the code. So it's really important that at least at first you write ugly code. It's okay to write those for loops instead of a filter. It's okay to write the if statements. It's okay to write the ugly code.  And I thought this kind of summed up what I was talking about fairly well. So you can see at the top, it's a much more complicated thing to understand than below. So again, some rules rule of three abstract on your third duplication. If you're going to refactor, it's okay. If you leave some duplication in your code, that's not a sin. You absolutely can do it. In fact, you probably want to do it up to your third time. At that point, you should be considering what the right abstraction should be. Well, future me or my teammates understand this better. And when I say future me, I don't mean future me that has been working on this code for three days and knows it inside out, but future me who's been tired, who is underslapped and who is now stressed because they have to figure out that code. That's a different person than you are when you're writing it. So you really want to consider that way the costs of the refactor. Is this going to be better for the program? Or is it just clever? Like, is it going to stop bugs from coming up? Is it going to be easier for my teammates to read? And I think that that's maybe the most important key is you should be airing towards easier to read rather than easier to write. So the moderate program gives up, but not forever. So there's a number of examples about this. Let's go through each type of giving up. First off, give up and ask someone for help. And this is probably the most important versions of the giving up. If you're struggling with trying to get some code written, if you're not sure and you banged your head against your monitor for three hours, it's definitely time to give up and ask someone for help. That could be a teammate. It could be online. It could be in Discord. It could be just a Stack Overflow search, but it's really important to stop and do that. Also, I found that while asking for someone for help, within minutes of trying to explain the problem, I've already solved it in my head and I don't even need to talk to someone anymore. So it's more important that we need to do this and not just power through when we're struggling. Also, give up and take a break. Take a walk, play a guitar, do some yoga, whatever it is that takes you away from programming, even if that's just for 15 minutes, that will clear your mind and make it easier to solve your problems. And finally, give up and throw out your code. It's okay. You can write your code two, three times. And sometimes you have to struggle through it once or twice before you find the right way to do it.  You probably want to get stashed at first. You'd be surprised at how many times you might need it. So let's sum up what we've been through today. We talked about the moderate programmer names things, but with enough detail, writes small functions, but not too many, writes only enough tests, refactors their code eventually, and the moderate programmer knows when to temporarily give up. Thank you, everyone. I really appreciate you listening to my talk and please reach out to me on Discord or wherever you can find me, and I would love to chat to you more about this. Thank you again.


Awesome. So thanks everyone for tuning in. My name is Amanda. I'm excited to share some work that I led and then developed with my colleagues at the Harvard Business Review. Our data and operations are essentially closed source, unfortunately, but we for this project use a lot of the learnings, tools and tactics from the open source world. So I'm excited to share that. Excuse me. Some background about me. I have a degree in public relations basically, which will make less sense to you. I think the more I talk, but that's all to say my data career started sort of after my formal education ended, which is why I'm really also thankful for a lot of the open source resources, open data and research stuff that really lets you kind of learn on the job. And so I ended up working in public health research for a while at the NIH and then I pivoted to industry doing survey and product research for public relations for public health campaigns and then TV, film and now digital publishing at HBR. I'd love to know more about y'all in the chat or Slack after. I think if you've ever related to Hackjob Panda, this talk might be up your alley because it's sort of our DIY solution to something that may have other solutions, but this is ours. And maybe you're here if you're familiar with HBR, in which case that's awesome. I'll have some HBR data trivia for you that you won't find anywhere else. If you're not familiar with HBR, briefly we are a web and print digital outlet dedicated to improving the practice of management through like really expert research content. We have a paywall and subscription business model and we release a print magazine six times a year and have a pretty big online presence at hbr.org on most of the social media platforms and we have several email newsletters and a growing podcast network. Today I'm mostly talking about our articles, so in the magazine and on the website. You've probably seen an article, but when I say article, I'm talking about sort of all these components you see, including the author bio, which is usually at the end. And then the blue sort of highlighted pieces are things that we have readily available in our web analytics platforms, which is what I primarily work with, but then things like the summary and the actual body, those live elsewhere, and we would have to join that in separately. And here's sort of a snippet of the limited view you kind of get with web data. It's often, you know, you're mostly working with the headline and events that happen oriented around the headline, like did they view the page? Did they view a paywall while on that page? Did they start subscribing what in a visit that included that article? So that can be kind of limiting depending on what kind of question you're trying to answer. And as it happens, we started getting more questions from our colleagues that they came to us as a web data team, but it kind of started to feel like we needed to know more about the content of the article to bring that into the fold to better answer questions, like how people are navigating. The headline doesn't really explain why someone may have left the page or maybe tell the whole story about maybe they clicked an article that had a similar title immediately after. And the content is your bread and butter, so it kind of made sense to want to bring that into the fold. So I was like, great. Let's just go figure out where the sort of raw article content lives, invite it to the party, and then we'll get our drawing on. As it turned out, it wasn't that simple with our infrastructure and the available resources. In theory, we had access to everything we needed to make that pipeline happen. But it was more that like the time and resources weren't going to be allocated in a timeline that worked for us. So our team was kind of just given the blessing to like do it yourself. And, you know, other teams like engineering would try to fill in some gaps, but there wasn't going to be sort of a dedicated sprint or commitment to sort of making this data available in a repository that made like professional data engineering sense, if that makes sense. So our team ended up resorting to a lot of open source tools and practices and tutorials to kind of package this full article data to make it useful for us and for others. So until that point, the ways we knew how to get our article content was scraping our own site, which is not optimal for a lot of reasons, the Web Analytics platform, which doesn't have the full content. And then in our world, every article is also a product because we can sell a print of it. So it also lives in this product information system, which is just a really gross place, no API, this Kluge UI that you could scrape, but it really wasn't worth your lifetime of effort doing that. So we had to kind of go back to basics and ask ourselves, like, what even is an article? What is the purest form of the article where we can like start from there? And as it turns out, it was XML, helpful developer helped us crack that. I think a lot of people here are probably familiar with XML working with like government documents or whatever. But essentially, it's a lot like HTML, you have tags that have a semantic meaning, like title or author bio. And then you have the schema document, which is an XSD file, which is like the code book for all of those entries, where you might specify like the title should be string and not a date. And maybe you want to impose a character limit, and you can spell that out in the schema document. So we figured that out. And it turns out behind the scenes, you know, there's this XML and like an editor will write in a text editor, that'll become XML. And then it goes on these parallel tracks to our print publishing platform. So that becomes the magazine. And then also to our content management system for the website. So we figured that out. That was great. I plan to do some cleaning and wrangling to make the data sort of tabular for each article. So each article would become a row and all of these XML fields would be columns. But I also started thinking sort of what is the whole data life cycle of our article, because we're going to want to use this sort of more downstream. And so we kind of cover this content management silo on the left, again, mostly XML. But then once the article kind of goes out into the world, you know, it's in print at the store on the internet, there's a lot that goes on to measure it and then analyze it. And there's all kinds of sort of browser data constructs like cookies involved and analysts are working with different analytics software and maybe doing stuff in our Python or Excel. So that kind of balloons the formats. And then the end goal ideally is that, you know, we're taking these insights and making new content or features. And these things probably sit on top of the article so we never get baked in or hard coded. And the platforms we use for those kinds of features we're often serving parts of the article, maybe reformulated or targeted using JSON. So we have this XML again, and we knew we had this end goal of like, we want to make more data driven decisions using more of the content than we could before. And we have to kind of bridge this gap. And in another world or universe or another company, there's more or less a straight line you could draw that's like proper data engineering. But again, that's not the timeline we were in. So we were just trying to come up with something that was reproducible, debuggable, and kind of could be made why they accessible. So in our world, my solution was making our data package that would convert that did the work of turning the XML into sort of a table, which people are familiar with. I did a lot of like oral history ethnography to figure out what all the fields meant and what was generated by a platform versus a human and what the fields meant. And I built in built that into the documentation and offered a lot of export formats for people using different tools.   And in our case, that stuff was having a datathon with our team. And then taking that show on the road and our team calls a roadshow anything where we're presenting either at a meeting we host or we show up at other teams meetings and say look what we did. And so that helped kind of get more advocacy and enthusiasm around the data.  I'm comparing it here to the New York Times annotated corpus, which is a common benchmark for text analysis. I should say, our full archive actually goes back to the 1920s, but we haven't digitized all of that. So this reflects what is online mostly. So generally, we cover more time, we have fewer articles or articles are longer and more complete with summaries and tags. And our vocabulary is a lot smaller than the New York Times, which makes sense because we're more of a niche outlet and not covering world news. At this point, I had done a lot of data janitor work. And I was honestly kind of exhausted. And I never want to look at the data again, because it took a lot of just almost detector work to figure out what was going on. And that's how I knew it was time to sort of bring in the team and get some fresh eyes on it. And so that brought us to our datathon, which we did a really compact format compared to what you might be used to. There's like full day, full weekend hackathons. But ours, our goal was to just dedicate more time than we usually do to sort of dig deeply into data, figure out some QA and work out any kinks that maybe I missed in the data cleaning step, and then maybe make something interesting, try to tackle those questions that I showed earlier. And it was great since people could use whatever tools they wanted, Excel, Python, R, and in fact, we have people doing all of that. One of the interesting products out of this format was my colleague Abby. She made a shiny app that was basically Google Trends for our content. So you could compare terms or ngrams in the article text and or the author bio at the bottom and see how those frequencies changed over time. So we got to look at fun, kind of trivial things like, you know, the rise of Uber is a brand in our content versus us using it as an adjective. And then looking at how our contributors changed over time in terms of how we describe them. So here kind of looks like there's a convergence or balancing of the professor author, whereas in the past, we had people, fewer people identifying as professors, and maybe more independent authors, but now that's sort of converged. One of the more immediate and interesting these cases after we presented at one of our roadshows, Abby's tool was really gin up a lot of excitement. And someone came up to us, it was our localization product manager. So he was like, Hey, I'm working on Spanish machine translation for articles. And I'm finding that there's some problematic titles, where I have to override what the machine translation says. So in these cases, it was sucking up can become kind of vulgar, if you let Spanish do its thing in automatically. And then the word cool can end up meaning refrigerated or interesting in different contexts. So he was having to override a lot of these and was just kind of catching them as he noticed them. But with a tool like Abby's, he could sort of get ahead of that. And in fact, cool and suck make up a real percentage of our articles and sucking, I guess, is maybe more prevalent in some of our recent stuff. Although it's a little misleading because 2020 isn't over yet. So that's not sort of a full denominator. But I don't think you need me to tell you that maybe sucking is on the rise this year compared to last year. But what I will tell you is that what we learned is that open source tools and practices, concepts like a hackathon or just like committing the documentation and even the tutorials we use to build the package up to this point helped us sort of rescue what was almost closed data within our own enterprise. We have these other tools that are disposal, but not really. And making space for people to make custom tools allowed us to bring more people to the data table. And that was awesome because I think it brought people who weren't necessarily data people or data wasn't a big part of their role on paper. But it got them interested in able to self service some questions that otherwise would end up in our backlog. And maybe we wouldn't get to it in a useful time period for them. And another thing I love about sort of being able to use R and stuff like that is that, you know, there are enterprise analytics tools, but you know, those cost money and there's license seats. And I've noticed like some people don't even nominate themselves for access to these tools because they think it's like not in their purview. But if you're using something free that you can spin up and just send a link to and like eliminate a lot of that friction, I think it opens doors for people. And we found that the state of can be useful for onboarding and just engaging people with what your product or bread and butter is. So for us, that was articles. We don't expect people to sort of read our whole archive when they start. And that's not a useful way to get to know the product. But like a tool like Abby's again is a great way to sort of get a sense for how things have changed over time. And I do want to acknowledge that we weren't like totally dead in the water. We got this far because we had experienced our developers and helpful and excited colleagues who were able to fill in a lot of gaps in our organization. So I do want to thank those people more specifically my colleagues at HBR. And also the CSP conference organizers and anyone presenting and listening today. I have an appendix of the slides which you can see at the link if you're interested in how we ran our data on both in person and remotely and some of the our links that led us here. So with that, I'm happy to answer any questions and thanks everyone for your time. Yeah, thank you. We had a lot of smiles and laughter and comments on your suckings on the rise this year comment. I think it may be the quote of the conference.  It's true. Yeah, so one question that came up in the chat was about the code. So I think you said the code is closed. Is that right? Or is there a plan to open it up eventually? I could probably do something with dummy data or we have a limited feed that's basically the same XML like you can get 10 or 100 articles a day through that feed. And then I guess the process it's the rest of it could be open sourced. I haven't prepared that but I can I could do that. Yeah, it was kind of well documented elsewhere and maybe I didn't put all those links in the appendix but yeah that can the process can be open sourced. Yeah, there was also the comment that HBR is really known for clear, concise case studies and it's awesome you just delivered a very clear, concise case study. Oh, thank you. That didn't even register for me. I think we have a trademark case study so I didn't put it in the title. But yeah, thank you very much.