Hi everyone, my name is Irka and I'm a part of the technical writing team at Red Hat in Brno. My talk today is about teamwork, collaboration, that holy grail of software development and especially so in open source. Primarily though, I will be focusing on where documentation and technical writers fit in all of that. To be more specific, I will start with outlining some of the challenges to teamwork that you may run into at one point or another when working on software. Then I will go over two teamwork methodologies that aim to overcome these challenges. First, the well-established waterfall method and then a more agile variant that we call DevDocTest. By the way, because DevDocTest is a bit of a mouthful, I will be abbreviating as DDT. So keep in mind, I'm not talking about the infamous chemical pollutant. So I will describe the theoretical basics of DDT, but I will also touch upon some practical downsides that could prevent you from using it successfully in your team. Finally, I will introduce some best practices that work for us in getting the most out of DDT. And because it wouldn't be a proper talk without a quote, let's start off with this one. The funny thing is, if you Google for quotes about teamwork, you will find very many that talk about how it makes the dream work and how it divides the task and multiplies the success and how it's generally great. And all of that is true. But you will find very few quotes that talk about how hard it can be to actually pull off teamwork properly. You see, teamwork is usually not something that just happens, right? It's something that you have to put thought and effort into.  Let's take a look at how that works in software development. If you're flying solo, work on some pet project of yours, then you do the code, you do the testing, and you do the documentation. If there is any at all, which it should, by the way. Naturally, coordination is not an issue here, unless your golem or something like that. When you're working in a small esteem, perhaps with two or three other people, you are all probably a bit of everything. Though some might generally be slightly better at one thing or another. Among the few of you, it should not be a huge problem to work out how to get things done. But now, when your project, your software thing grows in size, it naturally requires more developers and probably also specialized testers to make sure all that fancy code does what it's supposed to. At this point, there is also probably an end user of some sort in mind that is not other developers. So the documentation should follow suit and provide information that is useful, but also comprehensible for those end users. And that is where tech writers come in. This is also where documentation or, well, generally communication coordination might start being a bit troublesome, as it won't always be clear to everyone what needs doing and when. Finally, when you reach the highest level of project complexity, which is usually what happens at the corporate level, for instance, it mayhem, almost. Developers have their specialized teams and subteams with leads, product managers, program managers and so on. Your end users probably require a lot of attention, so customer support and sales specialists come into the picture. Testing is, of course, a lot bigger deal and there is probably a specialized quality engineering team to do it with its own structure, of course. Finally, pretty much the same goes also for documentation. The problem is, to get things done on this level, all these people have to be on the same page about what's to attend to, how and when, which can get pretty messy. To tackle this complexity, there are multiple approaches to collaboration in cross-functional teams like that. Perhaps the most tried and tested is, and arguably the most widespread also, is the waterfall method. I'm sure this doesn't need a lot of instruction. It's a no-nonsense, intuitive process based on sequential progression. I do my thing, then you do your thing, then he does his thing and so on. But the problem with this is that it tends to drag on for a long time, especially in larger teams. Oftentimes, this also means that towards the end of the release cycle, there is a lot of work left, especially for testers and documentation. That results in crunch time, meaning long hours of being overloaded and stressed, or it can result in delays. Honestly, it's a bit of both, usually. So this has also been described as the hockey stick problem based on the visual representation of the workload, as you can see here. All right, to illustrate, let's take a look at a sample waterfall cycle. First, based on data from customer support and sales, the PM and the dev lead plan to address a specific user problem. They task the developer with working on the code. And when that is done, QE ensures that the code works properly. Afterwards, it's time for the writer to step in and create some user-facing documentation. To do that, however, they usually need the assistance of a subject matter expert, an SME, and that's usually the developer. However, at this point, the dev has likely moved on to something else and probably does not even remember much about the thing that is being documented. Still, let's say that they powered through somehow, and now the document should be reviewed by a tester, a quality engineer, to ensure that the described instructions are consistent with what the software actually does. But the tester is probably busy with preparation of the impending release and might even have time for some very rudimentary sanity testing, if anything at all. Finally, when they're done with the tests and the documentation has therefore received the green light from all the necessary stakeholders, it might very well be far too late or the people involved have clocked in dozens of overtime hours. So things get done, but it's not ideal. The traditional counterpoint to Agile, sorry, the counterpoint to waterfall is Agile. Nevertheless, every implementation of Agile is slightly different and my team was now exception. Because it was not really viable for us to jump fully onto the bandwagon that is Agile, we came up with a bit of a hybrid that we call DevDocTest or again DDT. So the essence of a DDT workflow is that documentation and test plans are put together synchronously with the code rather than after it. This aims to make creating and reviewing documentation easier for both the writer and also the subject matter experts involved and also to make testing more efficient for your quality engineers. Ultimately, the intended effect is quite simply better results and more predictability with less crunch and no hockey stick. So let's see how a DDT cycle would differ from what we saw in waterfall. The first big difference is that testers and writers are present in the planning phase. This helps them scope the user stories much earlier and be better prepared for them. Afterwards comes the most important part, which is also called the DDT phase, DevDocTest phase. During this phase, the testers and writers work in lockstep with the developers and they prepare a test test plan and write the documentation respectively while the code is being created. All these they can actively cooperate in shaping the software as it is being developed. For example, testers can point out problematic code interactions while writers can suggest improvements for usability based on their own experience. When this phase is done, testers can then follow the created documentation to test the software, which means they review both the docs and the code at the same time and have an easier time doing it too. As a result, the overall process is faster, more efficient and also give testers and writers more say in the final form of the software. And at the end, there is much rejoicing. So that much at least is a theoretical high level version of DDT. But let's see how that translates into reality. As a tech writer in the DDT workflow, you're essentially like creating a painting of the night sky while the sun is still in its zenith shining bright. You have a reasonably good idea what the night sky looks like and perhaps you even have some astronomical data about the colors, shapes and positions of the heavenly bodies. Still, your imagination has to do most of the work, so you will admit a struggle. Apologies, I need a drink. So you will still struggle. It is not easy. But if you do get it more or less right, you will have ample time to perfect your painting when the stars eventually do appear high above. Also, you won't have to worry about finishing your work before the daybreak. But much like painting the stars at noon, DDT may in some cases not work really well or not work properly and not just for documentation. Sometimes your astronomical data may turn out completely wrong. You see all new code measures in development. So the feature that originally got planned and that you had in mind when you started writing could end up being something utterly different or it may get deep prioritized and pushed off. Also, from time to time, it can happen that a giant meteor just comes hurtling from space and instead of the night sky, you have Armageddon. So, which is to say that that's the feature that gets crept completely. So you need to keep your astronomical data as current as possible. The essence of DDT is collaborating in lockstep synchronously, which is significantly more demanding in many ways than if you go sequentially. You know, it's no longer, now it's my turn to work on the code, then it's your turn to test and then it's his turn to do the documentation. Instead, it's basically everyone's turn almost all the time. And finally, perhaps you're still making a painting of the sun and you cannot possibly start another painting just yet. So what I mean by this is that transitioning from waterfall to DDT can be pretty tough, especially if you're a writer or a tester. This is because you probably have a lot of back loaded work, like left from the old system. But at the same time, you suddenly have to take on the very much front loaded work that is connected to DDT. So this can cause long, long spikes of work where the hockey stick problem just keeps going and going and going. And so the effect is that contrary to what DDT is aimed at, what its goal is, it actually creates a lot more work and a lot more headaches, at least temporarily. So as a consequence of all these pitfalls, quite a few teams I talked to were a bit reluctant, let's say, to abandon what's been working reasonably well for them and move to DDT. One specific reaction that stuck with me was, get the fudge out with this bulldozer, except they didn't say fudge and bulldozer. Anyway, when you're a writer or a tester and some of the teams you work with on a product are agile-ish, using DDT and some are not or are somewhere in between, it basically means you are always in a crunch, either because you need to be taking care of the front loaded agile work or because you're on the hockey stick.   Firstly, writers, as early into the planning phase as possible, work with your product managers or anyone else who is in charge of the user experience and figure out the, well, the intended user experience. What is the issue that the user is facing? What would they like to achieve with the software? What will the changes in the code look like in the front end and so on? This will make your writing a lot easier, but it will also nudge the product person to figure out the UX as thoroughly as possible, which is always good. This goes hand in hand with being generally in the loop. If there is a major change in the development direction and the writer learns about it like six weeks later, then you can't reasonably expect the documentation to be on point and on time. So if you're a developer or a PM and you're using some tracking system where you can name-take somebody, like, for instance, JIRA, keep your writers in mind and toss them a line when something important happens. Next point. If you're using a granular Timebox milestones system, like Sprint, you can try using them to keep the individual teams in strict lockstep, but it's easier and honestly more likely to succeed to use them for priority sorting. So it doesn't have to be like the development exit of this feature is targeted at Sprint 10, so the documentation has to be finished at Sprint 10, but perhaps if development of feature A is targeted at Sprint 10 and feature B at Sprint 15, we should document feature A first, right? Finally, as I mentioned, creating documentation stories very early can lead to them being inconsistent with what the code looks like in the end. So taking part in so-called test batches or hackathons with your testers or with your quality engineers later in the cycle can reveal many of those quibbles and also give you an opportunity to test and discuss them in real time with anybody who's attending, who could be like also support people or it could be your PM, anyone. So basically, this is something for everybody to do, not just writers on second thoughts. Now, I'm well aware that none of the tips that I've talked about here are exactly groundbreaking, but as I see it as a tech writer, even the most complex and cumbersome methodologies should be made of small, simple steps. Ultimately, I don't think we have DDT all figured out quite yet, so you can take this as an in progress experiment review of sorts. Perhaps you could take up the same experiment and have much better results even. Maybe DDT would make perfect sense for your team or maybe you even have no choice in the matter. In either case, I hope this has given you some insights into how to successfully synchronize with your developers or synchronize your developers with your tech writers and your testers and ideally also without much of the sweat and tears that this very often can involve. With that said, I'm not sure if our time is already up, so if you'd like to clarify something available for questions here or later in this code. Thank you, Erika. It was a great, great talk. I don't see any questions in the chat, so probably if anyone has a question like you said, you will be available in this court. Yes, there is one question from Vendula. How exactly do you synchronize your knowledge with DEF and QE? Okay, that would be probably for another talk, maybe even longer than this one, but generally it involves meetings and asking questions and being trying to be in the loop and not just relying on them keeping you in the loop because oftentimes they don't. So I guess being proactive, which I know is a bit of a buzzword, but it sums it up fairly well. Yes, there's one more comment about that because Vendula is saying that it means, it seems to her that it's a lot of meetings and well, yes it is, right? It can be, but given that it's either that or sort of being in the dark and having to spend long hours figuring out what's actually going on, it's the lesser of the two evils, I would say.


Bella is not my sister. I don't know how many of- mmm. Let me backtrack, let me backtrack them, and go to the purpose of this video. I had an excellent time in Chicago having fun on a formal date party, okay? This does not mean that I'm getting married. This does not mean I'm even dating this chick. But I looked very nice. I had a suit on, I went out and got this suit, it's a very nice suit. I matched my belt with my shoes, although you can't see my shoes, and I had a good time. End of story.  Apparently that's, apparently that's outlandish. I expected some comments along the lines of, Wow, Matt, you look great. Thanks, guy.  Thanks, guy. Wow, Matt, you look really fancy. Glad I'm a fan. Nope, not even close. Let's go through the wonderful comments you guys left me when I just wanted to have a good time. Let's just go through them. So, let's just start right at the top. 716 likes. He dumped his sister for his cousin, shaking my head. 715 likes. There are some of my YouTube videos that don't have 715 likes. And you guys are going to put 715 likes on, hey, dump this sister for his cousin. Let's see the replies here.  Ha, ha, ha, true, bro. Ha, ha, ha, for real. Look, MMG found his other sister. How many sisters do you have? Incest is WinCest. I think you need to move to, okay, it's kind of funny. I will give you guys this. I will give you the Bella thing. I will absolutely give you the Bella thing. I look like Bella.  Me and Bella look similar. If this is my sister or my cousin, then Shadow Jocinco is my brother, because this chick is literally like six shades darker than me. Look at my skin. Look at my skin in this picture. Do you think all white people are like brother-sister? Yeah. It just doesn't even look like me in any other aspect.  You know what? I love my nose. Shit's huge, and I get it. Because we're white? Okay, then by that logic, then LeBron James and Serena Williams are brother and sister, I guess, because they're both black. That's crazy! I guess I'm also, by this logic, Selena Gomez is my sister, and if that's the case, then she should hit me up, and she should have some family bonding time with me. Bell's skis forever. I knew we were going to do this. Guys, just because I put up a picture with a girl does not mean anything about Bella. It doesn't even... It's a picture. It's literally an Instagram picture that is so irrelevant, you guys. I'm not saying I'm dating Bella. I never said I was dating Bella, and I also never said Bella was out of the picture. The other thing, guys, I'm not going to date Bella for the channel. You guys always say bring Bella back. There's a reason we broke up in the first place, and there's a reason we're still friends, you guys, but I'm not going to give back with her for YouTube. I think... I was talking to Vic about this the other day.  They never broke up, and they never got back together.  When I, me and Bella broke up, that was serious. Okay, I made a super short video on it, and I didn't talk about it for, like, six months, because that's my business. Like, I do love you guys, but I'm not going to tell you about that. So when you say, like, bring Bella back, guys, I'm not going to change how I, like, live my life to bring Bella back to the channel. That's not real life. I love you guys, though. I will stay friends with her. She still likes making videos. Totally cool with doing that, because I know you guys want that, but it doesn't mean that I'm getting back into a whole full-fledged relationship for YouTube drama. All right, whatever, that's my rant over. Now that I've gotten all of that out of the way, let's continue with these comments. Fuck's not, Belski's forever. There's so many of these, guys. All right, I'm going to go through all the Bella comments and just get them out of the way right now. Um, Belski's forever. Fuck's not, Belski's forever. Dang, peace out, Belski's. Uh, hashtag RIP Bella. Bella back. This account name, by the way, this account is MMGIsGayNow69. I see your formal date a mere six feet five to your towering six feet nine. Guys, it's really, really basic math, okay? I'm six five. We know that, that's obvious. The girl in the picture is an average girl height five, six, but she has eight inch heels on, right? So five foot six plus eight inch heels makes her six two and me six five. That's why she's about three inches short of me. It's really simple, Matt. See, look at this. Look at this kind of soul. Look at this man who's actually looking out for Papa. Matt, I feel bad for you because you have to see all these I'm not supposed to use that word anymore. I'm not supposed to use the retarded word.  I love your channel and I probably watched every video like ten times. You know what, Ty? I'm saucing you a follow, bro. Call me. Yo, yo, yo, look at this meme I got tagged in though. Look at this meme I've got tagged in. Real OGs in the channel will remember this is like the years of my life too. This was senior year of high school. There's Ashley. I think some of you might remember Ashley. There's Bella when I was a freshman in college and then that's I'm not dating this chick, you guys. This is a picture that I posted with it. Am I dating my mom because I have a picture of my mom on my Instagram?  Yeah, you guys are literally the absolute worst. I had her tagged.  Guys, chill. This girl got literally massacred. It should be a hate crime. It was illegal what you guys did to her. So I instantly switched the tag out and made it at underscore or I made it at your underscore mom LL because I'm hilarious. And you know what I've done is I've went and done that on all of the pictures on my Instagram.  You could have these hot shakes on Instagram but you do this to yourself and you will never have them. But he pulls a lot more than just Tori Holtz now. That is an old school joke right there. I appreciate that. I appreciate that and everyone who liked it, who got that reference. That is literally a two and a half year old joke.   Yeah.  The sun is setting.  I know she likes black ripped guys that are 6'5". Yeah, it's me. I bet you won't respond to this comment. I bet you're right. I guess his favorite dating website is Ancestry.com. Ouch. Where do you guys get this stuff? Where are you finding these roasts? It would take me like 25 weeks to think of something that funny. You gotta wear the Gucci Christmas robe next time. Not a bad idea. Tell me I wouldn't be slagging out at a formal date party with the Gucci Christmas robe on. Here's a really high quality one. She flay. That's actually a really good point. Next time I see her, I'm going to make sure I tell her, hey, you flay. Oh my God, look at this. You two look cute smiley face. Austin, dude, I love you. Austin Fitz, dude, you're the goat. I hope you see this video, bro. I hope. There was definitely a good reason for that. And I'm tempted to vlog this one now. Freak you, Papa. You better go back to Bellskies and slap her across.  I literally can't. I'm not finishing that. I can't read the rest of that out loud.  That's honestly, that's refreshing. That's a new one. I'm not used to that one, I guess. This duo would drop at Flush Factory.  We're 100% dropping it tilted. Don't at me.  Sped. I'm ending it with this comment, sped. Great comment right there, sped. That's all. That's literally all it is. I don't know what that could possibly mean. I really hope that doesn't mean something. Moral of the story, I love you guys so much.  I still will vouch that I have the funniest fan base on YouTube, other than maybe Tyler1. Tyler1 is the only person whose fan base competes with mine. That's a fact.  Or don't actually, that might be better. Bye!


So people I am back with another death stole video and today guys We're going to talk about why I may just return for the witch queen now. I've been putting it off thinking about it I'm like shall I play I haven't played destiny such a while. I'm so far behind I just don't know but today we're gonna talk about why I just may return For the witch queen How's it going guys? My name is D.P. Jen if you enjoyed the video leaving a like really helps out anything like what you see I want to see more be sure to subscribe I've got bits all over me know what it is. I got a hoodie for Christmas. No, you don't wash it You just put it straight on and on top of your t-shirt took it off and now I've got bits All over me man. I'm doing my head in but yes people the witch queen comes out Is it 22nd of February? I mean this is off the top of my head 22nd of February Let me just quickly check back on that screen Yes, it's a 22nd of February and To be honest it looks good.  There's gonna be a lot of content there for me Even after having quite a break from desktop I mean, I think I played last season this season now I think I'm like a level four and the season passed the season before that before that and before I didn't play since the water glass Literally didn't play since before the water glass came And I just jumped back on The start of the season. I'm just like I'm so far behind my power levels like 13 21 tops people tops I mean, I just don't get the whole game's changed. So I last played it I mean all the mods the weapon the rose. I'm looking at weapons thinking what the hell is that that mud? I've never seen that in my life and it's just all overwhelming.  Well, I'm still only a 13 21 I'm coming back. I'm gonna be so far behind even if I played now up until the witch queen I'd still be I'd have to put in quite a grind to get my level up that high and like I just Might just pass. I mean we've got dying light to we've got tiny teenagers We've got quite a few games coming. I mean, we're not always sure destined to into there But I was like, I just don't know I don't know But then one go released a twab last night and let's skip to the scene. Oh, yes We can sit here. This is the start of it.  I'm not interested in none of this rubbish. I'm interested in this right here Let me just do you mean a bit so you can all see that shit Starting on February 22nd, the power floor will be updated to 13 50 any player who signs in will be at appropriate power levels to start the witch queen campaign Even if they've taken a recent break from the game That includes motherfuckers like me now Don't get it twisted me and Bungie don't see eye to eye and many many things But this is a great great cool for players like me Who feel a little overwhelmed jumping back into the game after taking a break thinking on my levels are just too low And this and the other yes, I understand It's it sucks for people who've put in that grind now the people who've played the game constant day in day out I guess the only advantage you guys have are the better old weapons and all the weapons that I have definitely missed Which is fair enough. I reckon Bungie should definitely reward players who've grinded Considering and leave people out like me who've taken a break. I mean because if it was the rules reversed That's what I'd expect.   I'll be pretty upset What you could be just like I don't care.  I'm talking for the other people here But even still I think that's a great great thing and like I said It definitely I'm sitting there thinking now. I'm playing which green I'm playing which green hundred percent. Will I cover it like I used to cover desktop before? Probably not. I'll probably cover it here and there depends how much I like it. I mean, I've got a little bit I'm still gonna jump on now.   So I'm sitting there thinking now We've got what what's the date today.  So we got basically a month of Me playing getting the weapons getting what I want get those perfect old old weapons and just sitting on them And that's what I'm that's what I'm thinking.  That's how that's how hard times are Gaming wise, it's just nothing for me to play and although actually I'm enjoying call of duty Yeah, I say the first time in 40 years. I'm enjoying call of duty But paper it seems as though I'm gonna call it beyond like that. What is it which Queen which Queen? It may have just brought me back Now I can't say how long I'm gonna last play me But I Think it could be quite good. I think it could be quite good But we will say people we will see No, I didn't you know I jumped on Twitter today and saw Palomine ease now talk about this and I'm like What's he on about? What is he talking about? So I put a joke tweet. Where is it here?  I'm got people writing. I believe so.  I was joking people So, yeah This is good news in my opinion.  I don't even know it.  What is this? analyzer void analyzer Omni telemetry Benchy special. What is that? Benchy special sounds quite sexually farmanist But I don't know that's what I'm saying.  Are they even a thing?  I mean, like I'm saying, I just don't know what's going down here. I don't know what's going down here I've missed so many exotics. I'm guessing as well Ah people people but yes guys. Yes, yes, yes I thought I'd make a quick video of dating people and whether I stand with destiny I don't think I've done one where I've spoke about Witch Queen and if I'm coming back to play on that, I know a lot of people have asked that question but Yeah, I think I might. I think I just might. Thanks to release right here. Where'd it go?  That's a good picture.   Yes people There we have it Uh guys, if you enjoyed the video and want to see more desktops from me, be sure to subscribe and hopefully guys I will see you on that next


Okay, we're back here live at SiliconAngle.tv's theCUBE, our flagship program, we'll go out to the events and extract the signal from the noise. We are New York City live at Strata Plus Hadoop World, put on by O'Reilly and Cloudera. We have a special demo walk-on presentation we grabbed from the hallway. As always, we go where the action is and the action is in New York City for Big Data Week and we're excited to bring really the talk of the hallway conversation was a big demo. Yesterday Microsoft did, a lot of people came out raving about how awesome it was and we checked it out and we brought in Microsoft to do a kick-ass big data demo, showing how easy it is for mere mortals to use Big Data, which has been a big theme of this year, analytics, higher processing power, ingesting data, all that tech geek stuff under the hood, that hard to do stuff, it's getting easier and easier and we're going to bring that demo to you. I'm John Furrier, the founder of SiliconAngle.com, I'm joined by my co-host. I'm Dave Vellante of Wikibon.org and we're here with Mike Flasco of Microsoft. As John said, Mike yesterday gave the hot demo of the show and everybody's swamping them after the demo, he's out of business cards. So Mike, welcome to theCUBE. Thank you, thanks a lot. That's a good sign when you're out of business cards, Dave. It's like, business is good, you know, you get all your cards, it's a sign of good insight there. So take us through the demo, set up for the folks out there, what happened yesterday, what you did in the event, in terms of what was the program and then about the demo, then we'll go right into the demo. Sure, sure. So what we did was, we announced a couple of things. One was a preview service in the cloud for Hadoop as well as kind of the complimentary thing that runs on premise. And the demo that we did was, what I considered to be kind of the wheelhouse of Hadoop, which was, we took a large amount of log files, processed them in the cloud using Hadoop, you know, did some aggregation on them, did some shaping on them, et cetera, brought them down to, let's say, 20,000, 30,000 records. And then we pulled them into Excel so that we can do a little more ad hoc analysis, as we wanted to kind of self-service analysis. And the whole idea was that we had log files tracking our online business. And in our demo, it was an online bike shop. And what we were doing is we wanted to see the traffic patterns. We wanted to then mash that up with data we had in our enterprise databases about some of our sales records and promotions we were running on those days. And then finally, once we'd done that, enriched the data a little bit more with demographic information so we could understand how our traffic was going versus the kind of marketing impact we wanted to have versus the demographic of the user that was seeing our site. And the idea was to show how easy you can do that, you know, just cutting up to speed with a cluster in the cloud and just using Excel. So do you want me to just take you through it? So just take you through it, all right, great. So what I've got up on my screen is HadooponAzure.com, which is a preview service that we announced yesterday. It lets you get up and running with Hadoop service in the cloud just a couple of clicks. But then the other thing that we talked about was, you know, sometimes after you set up a cluster, sometimes your team's running their production cluster or whatnot, and it's nice as a developer to be able to just get up and running really quickly, have a really nice debugging experience, have a really nice iterative experience as you're refining your query, getting going. And so what we did is we shipped the exact same thing that we run in the cloud as a one box set up for a developer. So literally in two clicks, a developer can be up and running with a pseudo-distributed Hadoop cluster on their laptop like I've got here. You can see the address here of my dashboard page is localhost. So I'm running the same experience we run in the cloud on my laptop. So as a developer, I can test everything out very quickly, get a nice debugging experience, et cetera. And then when I'm ready, push it up to my company's production cluster or whatnot. So what we did yesterday is we kind of went through that loop, and then what I used is this thing called the interactive console. It's something that we have that lets you kind of easily interact with Hadoop cluster across the web. Think of this as like a web-based command line for what we're doing.  This is just a really simple kind of web UI over top of Hive. And what I had done, I'm not gonna show you all of the lead-up demo.  We had them up in the cloud. We created some Hive tables over top of them. We processed them in the cloud, and that brought us down to kind of some aggregated data off of those weblog files. And so what I showed people was I said, select star from weblog results. Let's say limit 10, just so everybody can see what our results had looked like. And let's evaluate that. Actually that might, let's do this first. Select star from weblog month results. Have to remember what I actually did yesterday. From that. And basically what we had done is we aggregated all this data into a set of data that represented the log hits from this month for particular countries we were interested in and particular product categories we were interested in. So you can see what we're getting back is country information, category, components, or bikes, or clothing, whatnot. So the category of product we were looking at. And then IP address showing where the hit came from on our website. And so that we had used Hive in the cloud to kind of aggregate all this. And then I said, well as a developer I want to iterate on this data really quickly. I want to understand what I want to do kind of on a regular basis for my team. So I'm running all this locally. And then my next step was to say, okay, let's take this, let's mash it up in Excel. And let's figure out what insight we might be able to get out of this. So what I did is yesterday on stage, what we did is we opened up Excel 2013. And what you're looking at right now is a set of records that I've pulled in from a database running locally on my machine. I figured everybody would trust me that Microsoft knows how to get data from SQL Server into Excel. So I didn't show that step yesterday. I just said, look, here's a bunch of data. It's normalized data, so it's got that shape to it, right? And so what this data is, is it's per day what type of discount we were running on clothing versus bikes versus whatnot. So we were running a 25% discount on 1017 on clothing on our website. And then also per country we have and what kind of channels we were using to advertise those discounts. But then kind of the net new thing was, okay, so I got my data from my enterprise database. How do I combine that with all this kind of data I'm getting out of Hadoop, all this unstructured data, and how I bring this together? So what we did on stage yesterday was we said, we went data explorer and we said, look, you can go from other sources and now you've got directly an option from Hadoop distributed file system. And so if you say from Hadoop distributed file system, this brings you up this kind of explorer window. And again, because I've got all this running as a developer on my local box, I can iterate really quickly without relying on anything else to be set up around me. And I asked the system to go and interrogate my local HDFS using WebHGFS, which is just one of the standard rest interfaces of Hadoop. So it works against anybody who's running WebHGFS. I can see I've got a result set called month results here. And let me just hit the binary tabs so we can look at this a little bit better. And what it did is it said, look, we just saw that you've got some text files under the covers. It looks kind of like this. It took a sampling of it for me. And I can say, okay, that's what I want to work with, but I don't really want to work with it like that. That's hard. So parse this into a table. The delimiter I had was a comma. And you know how semi-structured data is, right? It's not always a nice square. And so you get to say what you want to do if you don't see a nice square and you're trying to treat it that way. So I said, well, you know, I know that the first few columns are what I want. So just truncate the rest if it's kind of jagged. I don't need that. So I say apply. That goes, okay, now you've got it roughly as a table. So let's make this bigger so we can see what's going on. We don't want this column, so just drop that. We know that this data represents our country.  So what we did is we renamed this to country. We know our column here looks like our category type. So I renamed this just to item. We just have two more of these to do to kind of clean it up. Let's say this is our date. And finally, this is our IP address. Remember, all of this is coming from what we processed with our Hoop cluster earlier. So this looks kind of roughly like what I'm after. Let's rename this whole query to be our traffic query. And then what we said is, look, this represents our data in Hadoop, but now let's join that with the data we got from the database so that we can do some further ad hoc analysis on it. So let's get related data. And this is the other data I've already got sitting in my sheet that came in from my SQL database earlier. And so I can say, well, this date matches this date. And this item up here matches this category down here. And it tells me 113 out of 200 rows are gonna be combined together. We're gonna miss out on about seven rows. So let's join that together. Now let's look at that table we joined to. And let's say we want the percent discount to be brought into this. And maybe we want the channel's information to be brought in, which is what channels were we using to run the discount. So great, I brought this in. Now I've combined SQL data with Hadoop data. And you can see it joined pretty well, but now I've got these nulls here. Like nulls is a database thing. I don't know how the heck I'm gonna visualize that. So let's clean this up a little bit and say, well, if the value was null coming out, that means I wasn't using any channel. So look for the nulls and just zero them out. So now I got rid of the nulls here. I don't want those. If I have a null in my percent discount column, then that just means I wasn't running a discount, so just do zero. That means there was no discount running. I was running that product at 100% of the price that day. So let's apply that. Great, so now I've kind of cleaned up that data. I've joined them together. It looks roughly like what I want. And let's go ahead and say done. Now, a couple things to notice here. Let me just add a new sheet so I can kind of zoom in on what we're doing. What we did is we just formulated a query. And that query is something that's gonna pull from the data that was in Hadoop. And we're gonna join it with the data we've got in Excel as it comes in. And I'm looking at a sample so that I could do all this quite interactively, right? I've got about 30,000 to 50,000 records sitting in Hadoop that I'm gonna pull in. And so we do it over a sample so you can see and kind of visualize this is what you're gonna get, but you get to do it interactively. And when you're ready, you say add to sheet. Add to sheet is kind of that moniker that says, okay, look good, go ahead and do the big pull from the Hadoop cluster. Go grab all those records, pull them in. So now I just sucked all those records in. They've been joined together. So now I've got it all together in a sheet, right? The thing I think is really neat about this, at this point, if you know Excel, you know what to do, right? Like, all those things you used to do with Excel data now just work here. And so we always joke that this is kind of the most used information worker platform so we were really trying to get it into the tools that people are already comfortable with as soon as they do it, yeah. So then what we did is we said, okay, this is great, you know, I can look at a table of data but how do I get richer visualization on top of this thing? So what we did with that in Excel 2013, you have all the standard Excel visualization that you're used to but there's also now something called PowerView. PowerView lets you do kind of ad hoc visualization, interactive visualization. So I'll take you kind of through the arc we did. It takes about a minute. What we did is we said, okay, we've got data by country so that's often best visualized on a map. So let's click country. Brings up our country information here. Let's just make this a little bit bigger so we have some real estate to work with in a second. And great, I've got countries listed that came out of my data but let's turn that into a map. Great, so now I've got little bubbles on a map. Okay, we're getting there. But what I wanna see is actually the hits by region. So let's go IP address and let's make that the sum of the hits. So now you can see the bubble size change, the biggest bubbles in the US, smaller one in Canada, et cetera. So I'm starting to see where the traffic was going for country. That's interesting but if we go and say, okay, well let's make this richer, let's not just do a bubble, but let's actually superimpose a pie chart on top of the region. So now I can see per region, the size of the bubble represents the number of hits, the pie chart represents how the hits were dispersed across the various product categories that I have. So now I can kind of at a glance see how I'm doing day to day.  Now I've got kind of a little interactive visualization that as I click the dates, my pie chart is gonna change based on the traffic for that day. So let's do a little bit more real estate here and just enhance this one more time. So let's go like that. Last thing is, so we've got our hits, we've got our date. Now what we wanna understand is how well our discounts were running against our hits and to see, when we was running a discount, did that affect my traffic on that day?  Let's maybe look at that as a card, not a table. Go here. So we've got this, I'll do an abbreviated version of yesterday, we did one more thing but we'll leave this for now. And as you can see on 10-1, we were running a 5% discount on components, no discount on anything else. That's what our traffic looked like over here in Canada and in the US. Now if I go, you can see the green section here representing components is maybe a quarter of my traffic. If I go over to like October 5th, 10-5, all of a sudden the green section is representing over 60% of my traffic. Well I was running a 45% discount on components that day. So hopefully there's a correlation between the sale I was running online and the traffic jump and components that I'm seeing here. And so our next step was say okay, our running hypothesis is that this sale worked, right? Where was I running that? Well if I add this into the mix, now I can see okay, where was I running that sale? Well I was running it in our social channels that we use and at a local event. Okay great, so now I've zoomed that in a little more in terms of how things are working for me. So my hypothesis is that that particular campaign worked. Let's get a better understanding of the people that actually drove this traffic due to that sale. Because I think I'm hitting maybe the 20 to 30 year old crowd with my website, that's what I'm going after, but let's validate that. And so what we did is we went back to our source data and we said okay, our running hypothesis was that on components in the United States, just one more filter to go, on October 5th, so let's zoom in to October 5th. We were running that discount and this is just a sampling of the IP address, so let's just grab a few of the people that hit me that day, right? I'm assuming these people are representative of the people that were really driving my traffic, we could do them all, but let's just do a sample to, let's all copy it over here so we can zoom in on exactly what's going on here. So now some of the people who hit me that day drove my traffic, so let's highlight this section here. And what I wanna do is I wanna enrich this data to get demographic information about these people. And what we do is we have another service that we run in the cloud that we call Data Market. Data Market is just that, it's a marketplace for data. Just like they have marketplace for apps to go buy apps, there's some free, there's some for pay, there's a data marketplace, some data's free, some for pay, et cetera. Inside of the marketplace, there's a data set that lets you, given an IP address, give you back more information than you could imagine, right? And so what I did is, in Excel 2013, there's an app marketplace for information worker apps. And they're called Apps for Office. And what I did is I went and wrote an app for office that talks to that data market in the cloud and says, given this IP address that I've just spelunked to get, enrich it with the data you have in the data market and visualize it here so I can get a better understanding. So you're matching up that data in rendering. Exactly. So at the end of the day, we've got some enterprise data, some Hadoop data, some data from a third party. Awesome. So we went, insert apps for office, I built a little web log companion app that talks to that. I got three options to add average household income, number of households, et cetera. I hit go, it hits that service, it enriches back. Let me just expand these so you can see our titles. So that's public data. This is public data. Anybody can go use it. So it's private and public data, matching up together in real time on tools that everyone knows, exactly what people want. Now we're seeing, look, the median age of the people that was driving this traffic, I have a 50 and a 52. So it's somebody with some cash in their pocket to spend on my high-end bike components, right? And so you can see what they're... The baby boomers buying it for their kids or themselves. Exactly. That's the next question. And so... That's big questions. That's Cloud Air as a motto. Exactly. So now you've got this insight that says, wow, it's actually an age range that's a little bit different than we thought. Yeah, it's cool. How can we take this further, right? So that was the demo we did yesterday on stage. Mike, that's fantastic. Thanks for coming on the queue. I want to say I'm glad we could squeeze you in because I think this is exactly what people want. They want the ability to do things simply. My question on your quick question is, is it only limited to Azure? Can you use other Hadoop? Was it just, what's the lock-in? Is there a lock-in or is it only Microsoft? Good question. So basically, our whole take here is that we're pushing against things that are 100% Apache Hadoop. So all of the Hadoop stuff that we run is 100% Apache Hadoop. All the tools we build are interacting with clusters using all of the standard REST interfaces in Hadoop. So what you saw was I was pulling data with web HDFS, issuing queries with Hive. So anybody who is supporting those interfaces that are directly in Hadoop, these types of tools work on top of it. Okay, so I'll give you a quick question and then drill down on that because as I use case, hypothetical. So Dave and I play with HBase a lot. We're like HBase geeks. So I've got an HBase database. I'm sucking in all this information from the public. Data source, and it's sitting there. It's just hard to get out, right? So what do I do? What do I use this tool from that kind of developer? Right, so today, the first integration point we built was through web HDFS. Tomorrow our goal is to go and do things over top of HBase as well with these tools. So today, the direct HBase query won't, we don't have that plugin into Excel, but that's the evolution of where we're going is to support all of those projects in the Hadoop ecosystem, whether it's HBase, web HDFS, et cetera, H Catalog for metadata over time so we can get an even better experience. And you're using this just to be clear. You guys have a partnership with Hortonworks, which is, again, 100% Apache Hadoop, but this is your, their distro is what you're using. Correct. Okay, cool. Well, great demo, fantastic. That's the kind of future we see. We see mere mortals using big data, and that's at the end of the game. That's where the market's going. A little bit complicated right now, but it's rapidly accelerating ease of use and simplicity, so congratulations on a great demo. Thank you. We'll be right back with our next guest right after this short break. All right, awesome. The cube is this conceptual box, if you will, and we bring people inside of the cube and then we share ideas. The cube is a comfortable place. It's a place where people feel happy and are happy to share their knowledge with the world and we're happy to be ambassadors of that knowledge transfer.


Organization Development OD is the study of successful organizational change and performance. OD emerged from human relations studies in the 1930s, during which psychologists realized that organizational structures and processes influence worker behavior and motivation. More recently, work on OD has expanded to focus on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management and transformation of organizational norms and values. Key concepts of OD theory include organizational climate, the mood or unique personality of an organization, which includes attitudes and beliefs that influence members' collective behavior organizational culture that deeply seeded norms, values and behaviors that members share and organizational strategies how an organization identifies problems, plans action, negotiates change and evaluates progress. Organization development as a practice involves an ongoing, systematic process of implementing effective organizational change. OD is both a field of applied science focused on understanding and managing organizational change and a field of scientific study and inquiry. It is interdisciplinary in nature and draws on sociology, psychology, particularly industrial and organizational psychology, and theories of motivation, learning, and personality. Although behavioral science has provided the basic foundation for the study and practice of OD, new and emerging fields of study have made their presence felt. Experts in systems thinking, in organizational learning, in the structure of intuition in decision making, and in coaching to name a few whose perspective is not steeped in just the behavioral sciences, but in a much more multidisciplinary and interdisciplinary approach citation needed, have emerged as OD catalysts for tools.