Hey everybody, we're about ready to get started with my April office hours Somebody wouldn't mind just dropping something in the chat. Let me know that you can Hear me you can see me and We had an issue last time I did this where there was a little bit of a audio video sync issue that was going on with the With my With the full camera view like this that you see right now think I got that figured out Only way I really know is by getting feedback from you though because I can't watch this while I'm actually streaming it So bit of a live stream noob Here so yeah, just let me just see something in the chat to see that people can both see and hear me My volume is quite low. I Wonder if other people are saying that too Kurt so if I see one person said my volume is low everything sounds perfect great So Daniel you might just need to turn your volume up a little bit here Did a mic check before we started this and everything seemed okay, so hopefully everything's great I bet I see other people are saying it's good. So For those of you who are new Welcome If you're watching this recorded and you're not watching it live welcome to you as well I do this all Good Daniel. I Do this trying to do this or I do this on a monthly basis the third I Do this on the third Wednesday of every Of every month at 11 a.m. Eastern time which I believe is 4 p.m. GMT or UTC My goal here is to just kind of share a little bit of what the the biggest news That I've been following in the Microsoft 365 space that I think that you should know or sorry the Microsoft 365 development space That I think that you should be aware of and then also I am I'm gonna share a little bit of that but then I also want to answer any questions that you have I only had one question that was Submitted to me ahead of time and in fact, let me just make sure that that that I have those The chat up as well just to make sure that I can see any questions that pop up here Yeah, so that's what I end up doing with this So I'm gonna go ahead and dive into this Dive into the stuff in my presentation or my slides That I want to show you I'm gonna go ahead and dive into this right now So let me switch over to my screen here and let's bring me up. Oh, I'm in the corner there Let's put me over there makes me a little bit easier Oh god bring me back. Where'd I go? Bring me back. Where am I there? I am Okay, so let's go ahead and kind of dive into this I like to cover I'm gonna cover three main topics today. So I'm gonna cover The SharePoint framework some stuff about Microsoft Teams I don't have any things specific about SharePoint embedded But I will talk a little bit about it and then some Microsoft graphty kind of stuff and then handle any of your questions That you may have as well So let me go ahead and get started with this. I have a bunch of links that I'm gonna share with you to along the way So first thing real quick is let's just look at the State of the Union and the SharePoint framework Current version that we're still on version 1.18.2. That's the current production version We've been on this for a little while. So I don't gonna spend any really any time here If you're interested in learning, you know anything about anything about what's going on With the SharePoint framework, I'll go ahead and drop a link That's not correct. I've got that link and drop that in the chat here This is the latest version of the SharePoint framework where you can take a look at it I've got an unboxing video on YouTube. That's the link that I just dropped Where I kind of walk through what's new and what's what's interesting. I haven't seen anything that Microsoft has Anything that's really drastic about the next version actually let me switch over to that real quick I haven't seen anything drastic about the next version that Microsoft is releasing Related to the SharePoint framework So I'm not sure if we're gonna see it like there's nothing that's really all that big it's in the latest version of it There's there's one or two things they've added kind of in the timeline as you see here from 1.19 Let me grab all these links and drop these guys in we've got the SPFX the beta for 1.19 beta zero is in there I've got a Link for bot-enabled aces and I've also got a link For an article or for a tutorial on something news. Let me drop all these links in There we go, so just dropped in oh and this one where'd the other one go Aces wait is that all of them one two three aces beta zero data visualization enabled aces This one is what I wanted Let's drop this one in too Okay So a couple things with this the beta they released the beta about two weeks ago three weeks ago or so It's beta zero. There's nothing really interesting in it except for there's one thing I guess that they've added And that is the ability for us to a new template for adaptive card extensions That allows us to do a little bit of like simple charting. You basically give it A a series of data and it can do a little line graph chart It's early. It's a preview. So we'll see where they end up going with this They give us more capabilities with this, but it's nice little data visualization that they've added to the to the product here That we can potentially take advantage of in the future They've also added some other stuff too. That's relatively new And just in the timeline so support for bot enabled adaptive card extensions I got a link to that which I've dropped in the chat and the designer for Viva connections cards for adaptive cards There we the other big things that I've seen in the in the latest version That they've done is they they did they've they've decoupled Some of the dev tooling from the actual release itself. So you'll find Some packages in a beta project You'll see some some of the packages in the dev dependencies are referencing a version That used to match the exact version we were on but now you'll see some that are a little bit ahead like 1.20 They've just said they're making some improvements to it and they've also committed to making an update To the docs to kind of explain what they're doing They haven't done either one in about in a couple I guess it's now about a month So kind of waiting to see what they do there part of this. They did upgrade the version of webpack to webpack version 5 That's new But it doesn't really there's not a whole lot that really that that's going to do for us One thing that they did do that I have noticed is they've actually addressed a lot of it missing things on property pains property pain controls So specifically they've added things like about Actually, let me pull up our the release notes here where our here we go So we have in the release notes. We have a couple things that they've added here. So new features We've got oh here you go. Here's a view of what that card looks like That I was telling you about so this is the the data visualization for the card template and You can see where you got a little stock price or you have multiple series that are on it as well And then here's some stuff that they've added Right here, so they've added in on the property pain controls We've got things like aria labels and descriptions were missing. They've added in some disabled properties to like the pick the Icon picker and the thumbnail picker They've also added in an alt tag for the choice group for images And then an inline label as well for the toggle. So they've added in a couple little things there that For the property pain controls that were some gaps I noticed a bunch of these bugged a bunch of them a bugged some of them And it looks like they found additional ones and they just kind of got out in front of it And went ahead and added in to have a little bit more of a focus on accessibility Concerns I mentioned this data visualization card. This is the link that I posted in the chat a minute ago This is a tutorial that was mostly written by this guy Mohan Which is the engineer on it But I was involved in kind of editing it and also improving it a little bit And it walks through how you can go about creating one of these using this new template the data data visualization card template And once you add that you can see you get like the standard kind of a visualization How does it work? Well, you are gonna have this same thing. You're doing it a basic card view And now you can pass in this new thing of a data visualization card view parameters And it allows you to specify the kind of component So in this case here we have a component called a card bar and it has a title associated with it and in the body you tell it the Component name that you want to use which is a data visualization The kind is a line Type and then you have a series of data. So this implies we're gonna have other options Maybe bar charts, maybe pie charts, etc And you also have the ability to define things like a header and also a footer And I think I have a picture of that too So if you have a header and a footer, then you'll see the chart goes side by side with the header in the footer Where the footer allows us to define things like quick views to where I click on this and it opens up another visualization You're gonna see more about this coming soon right for right now It's fairly basic stuff that Microsoft has provided with this Lucas PO I see your question about Teams toolkit and SPFX. I'll come back to that in just a little bit Let me run through some of the news and we'll talk about some of that stuff They've talked about some of the other things you see down here at the very bottom Copilot integration scenarios aligning with teams packaging improved experience for third-party Cookies that are being blocked tooling improvements. I haven't seen a whole lot there They've talked about it on their their Bi-weekly calls that they do for the community, but they don't do a whole lot more than that So we're still kind of waiting to see where they go with this Let's talk a little about Microsoft Teams, what's new with teams well a couple things they have we have a new API based message extensions and the teams developer portal This is me grab that link so you can have that more easily Let me drop these two in here. So there's one and then what is the other one the API? Oh, I jumped my slides too quick. Let's go back The teams toolkit. I wanted this one and then I also wanted This one is that right? Yeah, I think that's right There we go. Yeah, so there's three links. So a couple things with this so today when we build Today when we build our What are they called our? Message extensions we have the ability to use a the bot framework or the teams AI library to build our message extension There's a new model where it's going to allow us to use an API based one as long as the API is is based or is implemented using the open API standard This allows to go into the teams developer portal add an entry for the eight for the API Exposing different methods are available on it and it's kind of like a no-code way of creating a message extension There are some limitations to it. We can't use it for co-pilot plugins We also it doesn't support any kind of an authentication scenario There's a couple limitations to it But then in the in the latest most recent post that I'm linking to here They say hey, make sure you pay attention to build so the conference coming up in later in May about a month from now They've got some stuff that seems like they may be giving us an update on this with some new things that they're covering with us To make it even easier to build our message extensions or to implement them They also did a release an update to the teams toolkit version 5.6. That's not new What's new though is that this link that I'm giving you There's a guy named John Miller who is the from Microsoft who is essentially the owner of the teams toolkit The teams toolkit And he did a three-minute video that kind of walks through what all the stuff that's new in which you could take advantage of There's two other things you should be aware of as well One is that the team the CLI that we were using used to be called the teams FX CLI That's been renamed. So we have a new version version 3 It is now been renamed to the teams app The CLI is now called teams app. And so you don't automatically get the rename is an automatic Like if you're currently using teams FX CLI you can uninstall it because you're gonna have to install the teams app CLI it's a brand new package That you're gonna have to use So that's where you go in PM install At Microsoft teams app dash CLI you're probably gonna install this globally as well So you push a dash G on the end of that Another thing that's kind of continuing with some an announcement that Microsoft made last summer in 2023 And that's they were standardizing a lot of their names of their different products On a new domain called cloud.microsoft I Think that's right all of a sudden I'm looking at that and that makes me I kind of get it almost a feel for that Feel that that may be I may have that backwards. Oh just lost everything. There we go Here this is the link about the API message extensions kind of walks through didn't I didn't show that The new domain. Yeah cloud.microsoft. That's correct. So what this is is that if you've created like a teams app You are Probably right now you may right now on your web app For the tab that a tab based app that you've created with a web app for Microsoft teams You may be using the X frame options or CSP headers the content security policy that says I'm only going to allow the this page to Load within an iframe that is hosted on a page at teams.microsoft.com. That's what we currently use today Good way for you to make sure that somebody isn't using that web app or that web page Outside of teams you want to only be used inside of teams Well, one of the things that they are doing is they're standardizing their domains For a lot of the cloud. They're specifically their user folk their user That are primarily user used How do you say that I'm trying to figure out how to say that where it's going to all be on an app on a Domain called cloud.microsoft. So we're going to have teams cloud.microsoft instead of it being teams.microsoft.com This is going to happen at some point in June or July You want to make sure that you are ahead of this so here It's like your app used to be using this everything worked But now it's going to start opening up in teams. cloud.microsoft You just want to make sure you update those domains to make sure all those changes are done I think they said in here that this is going to happen in I think I thought they gave us a specific date Let me see is it June Yeah, so it's going to go live in June of 2024 We got a couple a couple more months. All I would say is just you can go in and you can add a new Head it just add a comma and then throw the domain on the end of it because you can usually use like a CSP policy Or a or the x-frame options. You can put multiple domains in there I would just put teams.microsoft.com As another one in like a CSV list later you will be able to remove your teams Microsoft.com Okay Let's see. So what else do I have here with the slides? Oh, I totally messed up that part. Let's get that fixed real quick There we go. Now I'm in good shape All right, so what else do we have here? Two more things I want to share with you. This is relatively new ish And that is we have a thing called the the the sites that selected permission Oh, let's see. So someone asked a question here some ordinary user will they extend the domain name change Changes to SharePoint at some point. That's my expectation. Some more ordinary user Is that they will extend those changes to all Properties at some point in the near future. So all cloud properties in the future I don't know when that's going to be but that's what my expectation is One of so there's two other things I want to cover here before we start diving into Yeah, we start diving into questions Trying to get through the news in the first like 20 minutes of our office hours Let me jump over let me let's first talk about the pages api because the second part about sites that selected really goes into Somebody's question I passed in and I'm just going to apologize. I always have a hard night hard time pronouncing his name, but Slovaks If I know I'm mispronouncing your name, and I don't think you're going to be able to join us live But you're going to watch the recording It's your question. So I'll come back to you in just a minute so the Let's look at the api changes here. Okay so What this is is that microsoft has released this new api called The pages api for microsoft graph And what this is going to allow us to do is going to allow us to programmatically interact with share point pages and news posts It's going to allow us to do things like Get a list create edit publish and delete pages With this api Now right now with their v1 I want you to read this one. It's kind of interesting here where they say like with the v1 release We're introducing a targeted scope of features based on comprehensive research use case analysis and product scoping You can also go through and work with some of the out of the 14 out-of-the-box web parts that we get from microsoft In addition to any third-party web parts and they list them out Now why do I just call this one sentence out? I will tell you that this has been an area of like that's hotly contest contested by the community There's a lot of people in the in the community who are Have been waiting for this for a long time of giving mic given microsoft a lot of feedback for a very long time And let's just say that I think that the implementation of v1 It doesn't really do what A lot of people were hoping for and been asking for so when they say comprehensive research use case analysis and product scoping I think I might add based on the conversations. I've been in is that There's a lot of stuff that they didn't do with v1. So At least we have a start at least we have Something that is that has finally reached production And we have like something that's kind of standardized so far My hope is that we will see some roadmap stuff with this and that they will talk a little bit more about it Maybe at the upcoming conference the microsoft 365 community conference In orlando if you're not if you're going to be there. Let me know. I'd love to meet up if you're thinking about it Then I've got a registration code for you that you can use and save a hundred dollars off your registration If you use connell 100, uh, I'll be there. I've got to I've got a breakout session on using the teams ai library For message extensions and bots in teams apps And then I've also got a full day workshop on friday About building all sorts of things with microsoft teams It's not a workshop in the sense that you're going to build stuff with me But what I am going to do is I'm going to I'm going to show we do a lot of like instructor led demos And if you want to try and follow along you can but it's going to be hard to kind of keep up especially with conference wi-fi I'm going to give you everything that I do at the end of the session. You're going to be able to download the code the slides And um, I'm also going to give everybody who attends that workshop access to my next teams On-demand course that I'm I'm working on right now. All right. I'll come back to that in just a little bit Let's also may have a question about it. So I'll explain more about that in just a bit Okay, um now so that's the that's some stuff with the pages api Hopefully we'll see some more about it But I've got some I put some links in the chat where you could take a look at that and you can learn more if you're interested Um, you know, let me talk about the other piece here that we have about sites.selected Now I had a question here from um, slovox Um, about this sites.selected Let me give you a little bit of kind of full context over what you would want to do here because he's got it He gave me two questions and I'll read back those questions in a minute, but let me explain what this Let me explain kind of where we are with this In in 2001 and about this time two years three years ago Microsoft released a new thing called sites.selected a new scope that we have for microsoft graph Now the goal of this what we had was previously was that if we wanted to do Anything with an app that was going to talk to the microsoft three. It was going to talk to the the um talk to share point through microsoft graph We had to get we gave our app a permission, but that permission applied to the entire tenant So we would give it a permission and say like um Let's just say like write to a list Well, that permission was going to give that that permission was granted to that app For all lists and all site collections or the right permission would be there for everything across all site collections And that's not what a lot of people wanted. People wanted to have a lot more Of a focused thing. Oh slovox great. Good to see you here um People want to have a little bit more focus. They want to say I want to give this app permission But not to the whole tenant and all site collections in it instead. I want to do it just for the the the specific A specific site or like a handful of sites So what microsoft did about three years ago is they introduced a new scope on microsoft graph graph called sites.selected um Now when they introduced this back in 2001 I think it was february 2001 it only applied to To app only scenarios meaning that our app was going to obtain an access token from from a azure ad or microsoft entra and What it would do is that when it goes to get that token It then is is going to be operating On behalf of as an app, but not in the context of a user But what it did allow me to do is it allowed me to focus and say that this app Could only have these permissions on this particular site Okay now Um, the way we did that the way we granted it that permission Is as you can see at the bottom of this slide in the blue area. Let me pull my laser pointer up here Let me pull my laser. Thank you There we go So the way that oh great now my laser pointer won't work So the way it would work is I would go to the site's collection and pass in the id of the site on the graph dot microsoft.com And go to the permissions Endpoint which is a new endpoint and I could add in a new permission Now for an app only permission I would specify a role to be either write read or owner so owner being I have full control And what you're doing is is you're giving this intra app This intra application access to This site collection with the role that you define Okay, so this app can now operate Without any user interaction and have those permissions on that site collection Right and you can do that in multiple site collections if you want it Now what just got announced in the last In the last just recently Is they've added in another capability And now we can have delegated permissions or delegated scope. This is this is what you see on the slide This is what's considered app plus user So when I first go through and I grant my app sites that selected just like the the app only option When the permission was granted The the app would have no access To any site collections. It was just says I got the sites that selected permission Now what I can do with the delegated permission Is I do the exact same thing that I do with the app only permission Where I'm going to go to the same endpoint you see here on the slide the permissions endpoint for a particular site collection And I'm going to grant a specific app So that's the same thing you see down here on my in my slide right here 89 ea. So that's the intra application I'm going to grant it a specific role and if I'm using delegated the app plus user It can be a read write manage or full control These permissions are in line with what you're familiar with with like working with SharePoint So when you grant a user are they read permissions write permissions Manage permissions, which is kind of like managing lists and and the designer roles in SharePoint And full control, which means it can do anything You're granting that app those permissions in the scope of This particular site, but because it's app because it's a app plus user It is the the app can only have though the permissions It can only do this the things that the current user can do So it's not like a user who's a reader can use this this app To do stuff that only a manager could could do or could add data read and write data or have full control over the site It is the the the lowest permission is between the app and the user is what the app can do on behalf of the user Okay, so it's the lowest common denominator All right doesn't it's not a way to bypass any user permissions if you wanted to do that That's where app only comes into play because then it's not factoring in the user So you might ask well, what happens if the app has both a site not selected scope for app only and for app plus user because when you look at the code at the bottom here of the post that I'm making to go grant that permission You're not seeing anything that really indicates that this is app plus user or app only and that's true because what defines that Is the access token that your app obtained if your app goes and gets an access token As an app only access token using the client credentials flow where it uses a client id and a client certificate Which is what's required for a SharePoint online when you want to app only Or if it obtains a access token on behalf of the current user That's going to define that's going to dictate the access token How it was obtained is going to dictate if it's app only or if it's app plus user So you get both of those options So that's what's new here. This is a big deal now The stuff that we don't see yet and microsoft has said that they're working on some things It are a little more granular scopes like giving us maybe this permissions endpoint All the way down to say a particular list or a particular document library And that way you could just give an app permissions to read and write to a specific list right now Your the the most granular scope that we have is a site collection We can't go down to the actual Artifacts down inside of a site collection I was trying to think of another word than containers because that's now a real thing Thanks to SharePoint embedded I wish they'd come up with different names here. So no, this has nothing to do with with SharePoint embedded SharePoint embedded works totally differently within that Now Solox had had two questions here. So first question was Um, I'd like to hear about a real life example for the usage of site dot selected In a delegated scenario. Is this something that we can use in the SharePoint framework? Uh development projects. So the answer is kind of Well, not really because if you think about it This is gonna I don't think you're gonna be able to do that because the app is gonna have to authenticate. I mean, maybe You know what I haven't thought about that actually now that you say now that you asked that I what I don't know is can the app Can you ahead of time Can this id actually that you used to have this permission could this be the SharePoint online Uh, the special SharePoint online intra app that we get with that we use for like the ad htp client Or the one that we use to talk to microsoft graph Um Maybe you can maybe you could do that with app plus user You definitely can't do it with app only but you might be able to do it with app plus user Because you can't grant that you don't want to you would never grant an app an app only permission uh to a um To that special that special SharePoint online app um So you might be able to do that so To to finish the to to give that give you that entire thought here is that I could Go create a SharePoint framework component that would use the site that selected specifically to call a particular site by id and read and write to a SharePoint list once Well, I guess either before or after that app is deployed I would have to have A manual step where I go in and I add a permission for that SharePoint online app to grant it app plus user of one of these Permissions and then the user using the app would then have access to be able to make changes to it I don't know if that is entirely possible. I have to test it out But I think that you I think theoretically that should work um If it doesn't then you would end up having to create your own app to go do the authentication to go Grab that but I think that might work. I think that might work. I'm trying to think back to a demo that um A guy at microsoft patrick rogers did he may I remember I was watching this demo that he did just recently and I can't remember if that's what he showed or not um I was half paying attention. So sorry. I I was listening to it, but I was multitasking and now that now that you ask I'm not sure But I mean as the the real example for this though is that Uh With a site with a user you're gonna have to have the app's gonna have to be able to have access to it. So I think that you could Yeah, so I think that you I think that I think it could work Uh, so you said if you remember correctly at the beginning we had read write full and that was only followed by making Uh usable the manage role and finally they've been announced support for delegated app principal scenarios I'm confused as to what the manage role is used for manage. So owner is like everything Manage is um Manage is is for like managing lists, uh, and the designer role of like creating pages Full control has access to do everything. So there was no like that that manage didn't really exist Um on app only uh because app only was just had full control for everything um Daniel says is there a tool or ui to allow a non dev tenant admin to grant permissions for an application Uh on a given site collection non techie customers might struggle to grant permission to sites otherwise So there is no gooey for this Um, so no, there's no ui for this um, however, uh, I think I don't know if anybody knows this up top of your off the top of their head. I'm looking at the I'm look I don't I can't tell everybody who's connected to the to the chat um But you might be able to do this with the cli firm microsoft 365 Um, I just saw somebody post something. Let me see if that's let me go back to my browser Yeah, for now in case you're given site. Yep. So let me check one thing, uh, anthony. Oh, thanks anthony anthony says There are p and p commandlets. So you could do that with powershell Um, but I wonder see a lot for microsoft There we go Uh, and let's look at commands for share point Wasn't there one just for a site is a spo Site app permission. I bet that's it. Yeah, that looks like it's No, not yet. How about ad That might be it Yeah, so you add a permission you pass in the site because there's the site itself And then the permission is reads that's really the role And then the display name is the name of the app, but you really have to pass in So you could do the the display name or you could do the id So display name Or the client ID of the app. So either one Um, so yeah, it looks like you probably you could use the cli for microsoft 365 to do that or as anthony said you could use The um p and p powershell commandlet. Uh, yeah, and so Daniel so both anthony and soulbox are both saying that so I would trust both of them They both know what they're talking about and it looks like it looks like a tube from the cli for this Cool So um, so actually another question here. You said is the managed permission level suitable for app only permission scenarios to what would happen if Manage role in an app only scenario. Here's what I found. Yeah, so no, I think I kind of went over that What you're asking is can I use manage For app only and the answer is no the only three that are available for app only are read write and owner Whereas for app plus user it's read write manage and full control um I don't really understand why there's not Why they didn't do the same for both of them. Um, but They didn't so it is what it is um Yeah, okay, cool So Let's see. So I've got one question. I don't think I have anything else on my slides. Nope. I don't so Let me just say this in case people ended up dropping off The next office hours is set for may 15th, uh, which is the third um the third Wednesday of the month This is right after The conference in Orlando the microsoft 365 community conference that is the last week of april beginning of may So we'll have some stuff to definitely talk about and that one Um, the I'll have a link I have a special link that I've set up For the office hours that you can use to make your life easier and tracking You know when the next office hours is it's coming up. Um, we put that in the chat It's just vtns.io slash office hours office dash hours Right now it points to the one we're watching right now when this one's over I go right in and I'll update the link to point to the next one. Um, so it kind of rotates You can always have one that says The current one and it's almost always like the very next one that's coming I just didn't change it for this in case people are traveling a hard time finding it. It's like, well, just go to that link And I'll take you to today's makes it easy um So the next one will be on may 15th Um, I have a I'm I'm curious for those of you who are watching this and give me some feedback on this Okay, put it in the chat put it in the comments Um I stopped sending out a regular email just to remind you hey office hours is coming up And instead what I've done is let me switch off this Instead what I've what I've decided to do is uh, I've got it. I always post the next one that up on um, on youtube And you have youtube's get the whole setup to where you can click on the you can hit the the thumbs up on it So you get notified but then also there's a notify me button That will send you an email when it's when the event is coming up and when I go live So you can be notified when the office hours goes live to me. That's a good I think that's a good enough kind of heads up that hey office hours is coming Um, and I also post about it in our in the discord channel the void tonus discord channel Which you can learn more you go to void tonus.io slash discord and um But With that being and when and from there you can download an ics file I have a discord the discord event has an ics file that you can download and import into your calendar um And I'm at the bottom at the bottom of all of my emails that I send out to and all the the the newsletters That I send out There's also some information on when the next office hours is with a link to where you can go To the the entry and on the youtube channel where you can then click notify me But I wonder I mean I've had a few people ask about I know one or two of you who are on the Or are with me right now.  Like the monday before the office hours in the past I did it always did one on a monday and the the like a week and a half early and then just three days early But I don't want to be like I don't want to spam people so One thing I can do is I can definitely set it up to just say I'll subscribe to be notified whenever this is done. Um, I won't I don't know if I'll do like a I'm not going to auto add people I don't want to auto add people to the to that email But I can set it up to where you can be notified when the next one's coming If people want to rely on on getting it through their email Okay, so yeah, so rick. Uh, yeah, I recognize you. I recognize you rick So I know rick you you definitely asked about it. So you'd like that. I think another one was Um, I don't want to call him out, but there was somebody named I start let it first name Letter of first name starts with a k. I think that you might I don't know if you wouldn't like it But if other words work, yeah curt so there's just need nowhere to check Oh, this works email notification a few days. Good reminder. Okay, you know what, okay I appreciate the feedback. I will I will add something to go through to do that where you can you can add in as long as you have an ics You can enter just have it on your calendar I would you know what I could just do the ics file and put a link Uh in the email to where you can then add it And be notified of when the next one is so just just for Just for kicks. Let me just show you one thing and show make sure everybody is aware of one thing Let me switch back to my screen Um This is the discord channel. So if you come down into the office hour section Um, actually, you know, everybody should have access to this, but I wonder does everybody does everybody have access to it No Oh, I know what but if you come over here to the events So if you if you come into our events, you'll see the one that's happening So like for the next one you can come to this right Select it and you can say add to calendar and you can download an ics file just like this So in the discord server you can learn more about the discord server if you just go to My website botanist.io you click community And then here's a link to where you can join the discord server Um, and then once you're in You can see a list of events and then say I want the ics file for the next one You see I've already got the youtube live already set up for the next one as well Okay Let's see go back to my chat So let's see Uh, yeah, and lucas. I hope you're still around because I saw your question I will be happy to answer that so lucas lucas had a question Uh, you intend to take an online course on the team's toolkit similar to the sharepoint framework Uh course model You said that is a question. So I think you're asking let me or I think I think you're asking Do I intend to create an online course About the team's toolkit similar to what I've done with the sharepoint framework And the answer is kind of uh, let me show you what I'm doing Let me switch back over to my screen real quick Um, if you come over here to courses you see there's a sharepoint framework section Then there's a section here for teams. So right now what I'm doing is with the team I have an accelerator program that I run Um, this is a bigger course a couple of people who are on the chat right now I've actually been through this But this is a multi-week Based accelerator and the goal that I have with this is to Really get Have a much more high touch to work with customers directly And and teach them the microsoft team's app development If you're interested in learning more about this There's a little three minute video up here at the top that you can watch that I kind of explain what this is There's a bunch of featured things that we go over I would recommend that you come down to the very bottom and look at all the people Look at all the testimonials that I've received from people who've said some great things Including I know kurt is one of our is is joined us right now kurt gave me a really good quote as did Where is he there's don I saw don was in the chat as well. So Don don did it as well And or don gave me a great comment and then adam also gave me a really good one just recently as well That has some great comments as well, so This is a this is a cohort style where I meet we do a live class every wednesday for two hours where I teach so those Kurt and don and anybody else who's done this so far based on all y'all's feedback I've changed the schedule a little bit But I teach for two hours on wednesday we teach a topic and then I assign some homework It's totally optional, but that gives you we will then meet the following monday for office hours For two hours, and that's only available to the people who are in the cohort max of 15 people And you can see it's it's done a pretty good job of selling out as well I've got nine seats available to that as well if you want to learn more about that Please reach out to me Send me an email my email is andrew at void tonos.io Or just join the waiting list. You'll get an email back asking you to kind of share some context And I will follow up with you as well And I'd love to talk to you about it and make sure that you're a good fit for it But also make sure that did you would get what you're looking for out of it? The other thing though is that right now I don't have any self-paced courses on teams app development They're planned I plan on doing it. I'm actually working on them right now. I'm working on a bunch of different courses right now so I'm working on one on sharepoint embedded and working on one on a couple of four teams as well I'm not going to do one specifically on the teams toolkit, but all of them will use the teams toolkit So I'll like I'm going to have one on building tabs both personal apps and channel apps One on bots one on message extensions And then there'll be a bunch of other stuff as well. So the the best thing I could I would say is that To stay in touch and keep in keep in touch on what I'm doing with this Check out my Check out my My newsletter so subscribe to my newsletter because that's where I'll be communicating Everything first I'll also put it on some of the social medias and stuff So follow me on linkedin. You can get all those links from the bottom of my site Kurt thank you very much for the comments about it. I appreciate that West coast guy, you don't have access to the discord forum. Everybody has access to it Everybody has access to it. There are a couple of things that are available just for That are just for students or for students of a particular courses The office hours one that I showed you a second ago in discord That is available for my ultimate the ultimate bundle the students of my master's degree framework ultimate bundle And students who have gone through my team's app dev accelerator It's a place for them to ask questions They get like a priority response because they're like they're my they're some of my my most loyal my best customers They're not it's not available to everybody. So those are the first questions I like to tackle when we do our office hours However, you can always go in and jump in on the chat and ask a question and I answer those That's what I've been doing today as well. So you're not I don't want to say you're missing out You're just not getting the priority kind of access And I'll be completely honest. I mean if some people have bought my courses and they've supported the work that I've done I I I want to give I want to I owe them something and I owe them a lot more attention to someone who is just asked who is not Has supported some of the work that I've done But I do want to reach out more people and that's why I do these office hours Public I move these office hours from being a private thing just for those students to be more public so Students of my courses get first access to first priority to get their questions answered in these office hours And then I do it for everybody else. So west coast guy, you can definitely go in and join the discord There are forums there to ask questions as well, uh, even if you're not a student of one of my courses Um Don't know how to say your name mind double o zero killa mine killa You said trying to show a share point online page and a task pane Sharepoint add-in and I end up the following behavior the frame loads And a new browser under opens and navigates to So i'm guessing there was more to that um So the problem that you may be having with that so it sounds like you have a problem um I'm guessing one of the reasons why you may be having a problem with it Is because you're not getting any Uh, you're having an authentication issue and that's because the app has to go has to log into sharepoint online um Sharepoint online doesn't want to run in an iframe And that's what a task pane is in all of the office clients. So I don't think that's going to work Um, I think what you need to do what you need to do instead Is because remember the task pane is going to only give you like a small rectangle and sharepoint online Is not designed for that small little rectangle experience If you're looking to it depends on what your scenario is But if you're trying to show like data from a list or to interact with a sharepoint list Or to work Then you would want to use like microsoft graph Or the sharepoint rest api To reach out and to get data Back and forth from your your task pane PowerPoint add-in If you want to show a web part that's not going to work that's simply not going to work Um, and it's probably it's because the authentication piece You have you're not sharepoint just is I don't think sharepoint I think sharepoint's got a csp or x frame headers that says it doesn't want to run in an iframe For just gen general sharepoint pages and plus I even if it didn't do that uh, I'd question the The use case because again, you've got that limited vertical limited width vertical experience With the task pane that you can't change the size of it You would always get the suite the microsoft 365 suite bar at the top the navigation on the left hand side You guys can get a lot of stuff so I'm not sure that's the that would be a Having a sharepoint online page for your task pane is going to be a great experience That's it for me. I got I'm a lot of questions Does anybody have if anybody has a question, please we're going to feel free to post it in the in the chat I'll just share one thing that since the last time we got together Proper web app is the answer you bet absolutely mind kill it. I would definitely use I would definitely use a A a web app a standard just web app to to to host your Your task pane Because you're also going to want well first for a bunch of reasons first of all you want you want you want You're going to have a better experience. You have a better ux experience that you have control over. That's one two you're going to have You're the that you're not going to be getting a all the page weight That sharepoint online has every page tons of java script the sharepoint Framework run time You don't need and want all of that loading in that task pane. It's going to be a really slow task pane Consider that that task pane has to load within and initialize within five seconds Or the office client's going to throw an error and say sorry this add-in doesn't work A web app I think is just a better experience and that's what I think that that's definitely what you the direction you should go in Um So let me go back to what I was saying a second ago Since we last got together I was at microsoft I they microsoft had me create a course and teach a two-day Immersive course to a bunch of customers related to sharepoint embedded Great experience great experience had a really good interactions with a lot of customers Who were there? A lot of really good questions I'd keep an eye on sharepoint embedded. It is a really cool option For storing content for your custom apps Specifically, I love the idea of using it for as a door as a data storage option for your microsoft teams apps because Some people when they create teams apps like well I want to show all my store all my data in sharepoint because I've already got a sharepoint site It's like well. Yeah, that's great But every place you install your app like every every channel All the data is siloed just to that instance of the app inside of that site collection There's no like global site collection that everybody can use easily There are ways you can do it, but it's it's you're working too hard to make it to make it work in my opinion However You could do that if you were going to use sharepoint embedded and you could store all the file based content inside of a sharepoint embedded container For your app or multiple containers for your app So that's one option, but then there's a lot of other really great options And opportunities for using sharepoint embedded. I'm I it is one of the areas of In the share points development space That in my 20 years, I haven't seen I've rarely seen The the updates I've rarely seen the what do you call it the um The interest in the excitement around something in the sharepoint space like I have with sharepoint embedded Kurt says, what are the latest pricing sharepoint embedded? I can show it to you Unless they've changed it So if you go to aka I'll drop when I get to this link. I'll drop in the chat if you go to the sharepoint a developer documentation and go to sharepoint embedded and if you look under terms of service and search for pricing So here you go payment terms and grab that link I'll drop that in the chat for you It is you are you pay on three different types of meters You will pay based on The amount of storage So this is in a gigabytes per day. So this is kind of like Um It's a lot of way way azure does this. It's not exact. So At some point during the day It does like a snapshot and says How much storage are you using across all the containers for this app? And it says that's the amount and it adds it all together and that's the amount of gigabytes per day So you get charged based on that That is charged at a little less than Seven tenths of a cent per gigabyte You also pay That doesn't look that number doesn't look. Oh, yeah, you pay based on api calls and you're paying that much per api call For from microsoft graph. So if your app is when it's reading and writing data back to the containers It's doing that using It's doing that using the microsoft graph. So you have to pay based on the number of api calls And then furthermore if you're doing data egress pulling data out of share point embedded You're also going to be paying That much per gigabyte that of the data that is being taken out So if you're like putting large files and downloading large files, it's going to be more expensive than if you were You are not that data is that data is only applicable. So a couple things about this the graph api transactions It's only the api transactions that you make. So if your app So for example, if your app shows a bunch of files from your share point embedded container and A user goes to open one of those files in outlook or sorry in word on their desktop client or word in the web client You only pay for the ones that you pulled out the data Um You only pull out the you only pull out the the The you only pay for the api calls that you used to go call graph to get that list to display it on your web app But when word desktop or word the web app goes to open the file, you don't pay for that That's microsoft opening the file if it's You as the developer opening the file and doing stuff. That's different. You pay for those Um, the the data egress as well Is only for only applies to the data that is that you're pulling out that your app is pulling out of the app so like It's not and it's they're keeping an eye on things like video streaming and stuff like that but Anytime that your app is pulling that data out. That's the only thing that could be impacting it You said pricing is for preview wonder if they plan to increase it Don't know they haven't said anything about that yet um the ga for SharePoint embedded is I don't think they've set a public date um I don't think they've set a public date for but There are two big conferences coming up. There's the microsoft 365 community conference in orlando in like two weeks And then microsoft build is in seattle Uh, actually, I think it's on one of my own one of the um Once I opened up which one was that or one of the pages at the top build is may 21st through the 23rd so Those are two opportunities for them to go ga that would be close to the timeframe they shared back in november. So I would May looks like a pretty good bet Um based on what they've shared so far but as far as like any updates to pricing I haven't seen anything about it Um to where that's going to change Uh, I don't see any other questions And we are actually right at time. Look at that. We got less than less than a minute I want to thank everybody for joining me for a sec or for my Third new office hours. We're doing this on as a live stream. Love to get your feedback Anything that you liked anything you didn't like any constructive Uh criticism. I'd love to hear anything that you've got. Um Please just drop them in the comments below Um, only way I know only way I can make this better and make and improve upon it Is with your feedback? Um Right now my only action out of this is to create a way for people to sign up to email notifications when the next office hours is coming up Uh, but I'll definitely do it Thank you very much Hope you enjoyed it. I hope everyone has a great rest of the week and I'll talk to you In a month ish Bye everybody


So, as we have seen that prokaryotes are simple genomes in comparison the eukaryotes are relatively complicated. So, eukaryotes have larger genomes, they have tandem repeats in them and their introns are there and they which are in between the axons which are the protein coding regions within the genes. And eukaryotes have a little bit complicated genome, so in the chromosomes the DNA is grouped as heterochromatin densely packed region and eukromatin which is lightly packed region. Here is the typical eukaryotic cell, so we see it is pretty stuff as compared to the prokaryotic one. So, we have those nucleus in the middle, we have those lines or networks of channels coming out of the nucleus called as endoplasmic reticulum help in transportation. For protein senses, we have those ribosomes for energy senses, we have mitochondria, we also see cytoskeletal, so that supports the makes the structure of these cells intact and Golgi operators they are concerned with the secretions, so large complicated membrane bound organelles are present in the eukaryotes. So, here in this diagram we see the connection between the DNA and the chromosomes, so here towards our left we see a DNA strand which is a 2 nanometer wide strand, so the DNA wraps over the protein complex molecules which we call them as histones, so here we have those histones these are histones 1, 2, 3, 4, 5, 6 and the DNA wraps over them and this kind of structure in which we have a histone and the DNA we call it as a nucleosome. So, then these histones they turn around and then they make some wider structures, so we come up to a 30 nanometer filament right here in this section, then this these sections they these structures they they super coil on themselves to make those further bigger fibers and then up till they reach into the chromosome the width is 1400 nanometers, so if we look into the chromosome now we can recognize there are kind of different arms in it, so which we call it as sister chromatids, remember this is just one chromosome but we have two chromatids and then somewhere in the middle we see a constricted part we call it as centromere, so the middle part of the chromosome we call it as centromere whereas the terminals we call them as telomeres. Please remember these nomenclature while we are discussing the heterochromatin and eukromatin parts, so chromosomes if they are stained with their dyes they give different coloring patterns we can come up with dense heterochromatin region and we can come up with light regions which are called as eukromatin regions, so dense regions obviously they are densely stained with the dyes and they take more color and light regions take less color, so if we look into the gene expression the heterochromatin regions are packed, so the enzymatic machinery cannot reach there, so there is these regions are poorly transcribed or poorly expressed whereas if we look into the eukromatin regions, so they are mostly highly expressed. Here is the diagram in which we can see the relationship between heterochromatin and eukromatin, so you look into these nucleosomes which as we have seen earlier these are the combinations of DNA and histone, so they are quite jam packed with one another, so obviously the enzymes it is hard to go and to access the DNA which is embedded in between. So there are different modifications on the DNA or on the histones that bring about those structures here we see them, so for example there are histone methylations in which methyl groups are added to those histones and in that case the system moves towards this side, so it becomes eukromatin and same way we see that there are some other methylations on some other amino acids that can move back into the other direction also. So histone deacetylations and histone methylations and there are some other complex proteins which get attached and may give us this heterochromatin region and whereas the reverse processes they give us this eukromatin region, so here we see in the eukromatin region the histones are quite spaced and DNA can be accessible, so this is the region why we see that this eukromatin part is expressed more as compared to the heterochromatin part. So eukaryotes are distinguished by the presence of prominent nuclei they have larger genomes and their chromosomes are relatively complicated.


President of the United States. Thank you very much. Secretary Verity and Jim Abner, ladies and gentlemen, I know you were expecting the Rose Garden. This isn't the Rose Garden. It's a little drier than the Rose Garden. But thank you very much and welcome to the White House. It's an honor to be part of the Minority Enterprise Development Week and to recognize your contributions to this nation. This year's theme, Business to Business, Strategies for the Future, reminds us that the national as well as the global marketplace offer great growth potential to all businesses, but especially to minority enterprises. You all know that I used to work in Hollywood and it's some place some people call a dream factory. Well, the days of the Hollywood dream factory are long over, but this nation is full of dream factories. They're your businesses, for you take your dreams and make something tangible out of them. You employ people to make the things you sell and that fulfills their dreams and you sell the things they make to others and fulfill their dreams. Now we know that there are Americans, unfortunate Americans who've lost the capacity to dream. These souls may live in neighborhoods where they must fear to walk, their children go to schools where the teachers fear the students, and when they walk down the hallways of their apartment buildings, they hear the crunching of crack vials beneath their feet. Well, like all Americans, they hope for a better life. They hope for a better life for their children and somehow it seems that better life never comes and then the hopes fade and the dreams die and when that happens, disappointment turns into despair and despair becomes a self-fulfilling prophecy. And that's where you, minority business leaders, come in. For you prove to those who are less fortunate, who need examples of success, if they're to hope for better, that yes, there is hope and yes, there is a chance and yes, this is a nation of bounties. You prove that hard work and determination overcome all obstacles and that despair is unnecessary. What is necessary is self-esteem and self-confidence. You can teach them the ability to see that the future is contained within the present, that work itself is its own reward, and that with work comes dignity and eventually the fulfillment of your dreams. During Minority Enterprise Development Week and throughout the year, we can all be deeply thankful for the economic freedom that enables all American businessmen and women to seek their vision of a better future for themselves, their children, and their country. So thank you, America's minority businessmen and women. Thank you for what you've done for yourselves, what you've done for the economy, and what you've been doing and will keep doing for the nation as a whole. Thank you and may God bless you all and now, Bill and Jim, I guess we have a few awards to present. Mr. President, it's with pride that I announced the award winner for this year's 8A graduate of the year. Mr. Ronald O'Sherro, of Ron's Construction in Honolulu, graduated from the program in 1985. Since then, Ron's construction has continued to expand and is now doing over $5 million in sales each year. Mr. O'Sherro is an outstanding example of the success stories that exist amongst the graduates of the program. Thank you very much and we certainly appreciate everything you've done for us because with the help of the government, you know, it certainly made things happen for us. Thank you. Mr. President, the Female National Minority Entrepreneur of the Year is Lillian H. Lincoln of Centennial One, Inc., Lanna, Maryland, Ms. Lincoln. Mr. President, the Male National Minority Entrepreneur of the Year is Gabriel Ahiri of Santa Cerve Company, Indianapolis, Indiana. Here's your administration and the new administration that will be forthcoming that the Young Entrepreneurs will have the greatest opportunity ever in this world under the guidance of the administration and we want to thank you. Well, thank you very much. Thank you all. And again, congratulations to the winners.


Check one two, okay Guys ready hi folks Thanks for showing up. So We're about to start Today we will do a session with you called OpenStack and Hadoop 101 Sorry if anyone will not see our Black slides here. I was actually I was actually thinking that it will be like usually in OpenStack summits with the large projectors Anyways, I will distribute them after it after that. So if you don't see anything, no need to take the picture all the time So today we will do the session We originally called it getting big data cloud Getting big data cloud downright Eventually, we renamed it a little bit colon or getting big data clouds Building blocks in place. I will explain later. What's the difference here? So We will discuss a couple of things We will discuss the main use case for a virtualized big data solution on top of OpenStack cloud. What's the difference to running on bare metal? What are the building blocks on OpenStack layer? What are the building blocks on Hadoop layer? So we will do an overview of Hadoop ecosystem an overview of OpenStack ecosystem and how these two map together technically, so how Which you can get out of agenda, but how Project Sahara puts them all together and how it works in practice And we will do a discussion about some of the best practices and some of the known problems and questions which exist in this area So some of the gaps which are not yet closed for OpenStack Some of the gaps which are not yet closed for Hadoop and some of the testings and certifications which we are still yet to do So I'm Dmitry Novakovsky. I'm product manager ex-presales at Mirages and together with me. I have two more folks today My name is Sergei Lakhianov. I'm PTL of Sahara project in OpenStack and I'm principal engineer in Mirages leads in the big data track Oh, it's working. Okay, great. I'm Trevor McCabe from Red Hat. I'm one of the Sahara course I've been working on Sahara for Two and a half three years So right folks to ask question about Sahara, that's for sure Before we start I like to do a quick quiz. So can you please raise your hand if you're running OpenStack cloud in POC or later phase today in your organization? Okay, now, can you please raise your hand if you're running big data workloads on top of OpenStack? Great, okay So the rest of the folks are here to see how it's how it's actually can be done. That's good so Some a couple of words about use cases. So those of you already running will probably know the words Here on the slide you see like two dimensions. So one dimension is technical use cases, right? So how can the big data solution be consumed out of OpenStack cloud? The basic and the simplest use cases obviously to whatever provision Hadoop cluster on top of Virtualized OpenStack environment consume it directly like you would consume it from any other solution any other vermetal deployment Another use case is elastic data processing or EDP a separate API and point exposed by OpenStack Sahara project, which we will talk about. We will discuss in details later which allows to create OpenStack Hadoop clusters on demand on top of running OpenStack cloud. So not not pre provision it and then run the work The data processing job on it, but actually provision it on demand for a job that you already have The third approach would be to tap on the workflow engines on top of pre provisioned on top of pre provisioned Hadoop cluster on on OpenStack And that's the one dimension from for the technical use cases for the business-friendly use cases for those of you who need to whatever justify introduction of big data solution on top of OpenStack. There is a really nice white paper from cloud era and here is the link for it Called 10 Hadoopable problems. So those of you who are still struggling to get your management to approve the project for it might benefit from using Exactly this one. Now We will transition I will hand over to Trevor to do an overview of Little bit of a history of Hadoop ecosystem and its current state today And then we will get into showing you how the Hadoop ecosystem maps to OpenStack ecosystem today All right, thank you So this is my timer here to make sure I don't go over so we'll start the countdown And if it goes off then I'm done Let's see There we go. Okay. I can work the buttons. All right. So what 101 class would be complete without a little history, right? We always have to have a history lesson Hadoop as we know it today really began at Yahoo in 2005 They had a lot of searching to do a lot of pages to rank and they started looking for a solution so it turns out that Google had published some white papers in the early 2000s on Distributed file system also on the map reduce algorithm and Yahoo picked it up and started working with it So things go on 2008 is a is a big year for the project Cloudera is formed in a separate effort Yahoo moves Hadoop to Apache and is now a full Apache project 2011 comes along another big year Heart and works is spun off from Yahoo to focus 100% on Hadoop and Apache releases Hadoop v1 in December 2013 comes along and we have version 2 With some improvements that we will talk about soon. So that is kind of the The beginnings it grew up pretty quickly. So today we have a choice of multiple Hadoop distros There is the Apache project itself is the upstreams the foundation for everything you can find all the pieces there You can pull that stuff down and run it but we also have some commercially supported Distributions and these are the the biggest ones out there You have CDH which I believe stands for the Cloudera distribution including Hadoop from Cloudera Hortonworks data platform from Hortonworks and you have the map our distribution All of these are supported by Sahara by the way So you can run vanilla Apache Hadoop or any of these other three Let's see. Okay. So what was the primary problem that they were trying to solve you you probably know this But we'll go over it anyway. So the problem was how do we search or run analytics on data sets of ever-increasing size, right? It's not megabytes and gigabytes anymore. It's terabytes and petabytes, right? So how do you do that and the answer is divide and conquer? This is not necessarily a new idea if you go back to the 80s and 90s If you remember, you know all the development and excitement around quicksort, right? Same idea break stuff down into smaller and smaller pieces solve the problem and reassemble that is exactly what map reduce does in In a parallel fashion you take your data set chunk it up throw it out to a pool of servers Get intermediate results and then reassemble them and anything that is Anything that you can decompose into smaller Chunks and process independently is a good candidate for a map reduce type approach Okay, so what are the common components in Hadoop? There are basically three and we'll talk about the fourth one in a minute, which is really just a refinement Hadoop common is Sort of like Oslo and open stack it is it's the kitchen sink for all the common libraries, right? So anything common across multiple modules Goes in Hadoop common and is not you know reproduced elsewhere HDFS is the distributed file system All the data and executables are pushed around a Hadoop cluster using HDFS Map reduce is the data processing platform itself Originally was I think Java only but there are a bunch of different language bindings now so you can write MR programs in in C++ and Python and probably other things yarn is a resource management and scheduling component You'll hear if you do some research about Hadoop version 1 and Hadoop version 2 and so the distinction between those is really this initially map reduce Handled the data processing platform as well as all the resource management. It was one big monolithic piece and in v2 Apache Broke the resource management and scheduling out of the map reduce piece and put it in yarn The advantage of that is that yarn now is general purpose. You can actually use it for other stuff It can run any framework you want it to as long as you write a application masterclass See what my timer says. Oh great plenty of time Okay, so what does HDFS look like? This is a pretty simple diagram a high level view of HDFS Basically you have a name node The name node tracks where your data is right Your data exists on data nodes. It's replicated across multiple data nodes for high availability When you have a client that wants to interact with HDFS data, it asks the name node Well, where's my data the name node responds tells the client Oh, your data is on these data nodes and then the client goes out and reads and writes the data nodes directly so that is HDFS and a nutshell and a HDFS really underlines a lot of things there are there are different processing frameworks We'll talk about a couple of those later on but really this HDFS component of Hadoop sort of Undergirds a lot of them. I'll also mention that HDFS is as much an API as it is an implementation So there are things like Gluster FS and From Red Hat map R has their own implementation of HDFS The map I forget what the term is but it's the map R file system so there are a lot of Possibilities there We'll also talk about the yarn architecture here a little bit There is a single resource manager. This is very analogous to what we just saw with HDFS the resource manager talks to node managers on each of the execution nodes and Then for each framework that you run you'll see the little purple Circle there or oval That's the application master for a framework and so what the application master does is it goes back to the resource manager and says hey I need you know this amount of resources to run this framework and then the resource manager allocates it and the node manager is kind of the middleman managing all that MapReduce itself in Hadoop has its own application master That's included so obviously you don't have to write that yourself and it's deployed for you But that's really what's happening under the hood Alright, so this gives you enough to write the typical hello world Hadoop app which is it's probably word count right take a book Read it in and then find out how many times each word in the book appears right that's sort of everybody I know that's their first Hadoop app. That's very impressive But you need you need more right that's not enough we've got the core, but we need other stuff so in production environments You want things like cluster monitoring? If you have complex workloads that are you know are expressible as a graph You might have synchronization points where okay, you know tasks You know one two three you have to finish before task four can go on You've got migration out of legacy data systems or back and forth across frameworks So those are some of the the kinds of things that you need and writing MapReduce jobs is not always easy You want high-level scripting right you want more expressive things so you can get your job done faster You want a sequel like front-end so that you can query Big data databases as if they were sequel right so all of these sort of desires and things people found as they started Using to Hadoop led to the development of a pretty rich Ecosystem And we'll talk about just a few of those things here There are there are a lot of them. So this is just a handful Pig let you write scripts. There's a language called pig Latin and this is sort of freeform scripting over Hadoop data sets It compiles them into as many MR jobs as it needs and it runs them for you So that's very nice hive is a sequel like query language You've got Uzi which is a job scheduling and management system. Sahara actually uses this under the hood It supports pretty complex DAG type jobs if you know what a DAG is it's a directed acyclic graph and basically multiple jobs with dependencies zookeeper I think a lot of people here know about zookeeper choose than other instances, but you can use it for high availability and synchronization and distributed config H base gives you I believe it's a I always get this wrong. It's millions of Millions of columns by billions of rows or it's the other way I can't remember but it's really big That's the point.  There are many many more people are making new projects around this stuff all the time And there's also stuff outside of Apache like Hugh Nagios ganglia other things that are being added So there is a very rich ecosystem here So wow, you know, how do I choose? How do I know what's what's going to overlap? How do I know that it's going to work together? What version should I use have I forgotten anything? Well, this is this is where the distros help you out They answer that question. So here's a typical HDP deployment You can see you've got multiple Methodologies under the data access layer. There's pig there map reduce. It's got storm Of course HDFS and yarn are still there. You've got things like security In governance around the outside You can see Uzi on there. So this is a subset that this looks like HDP version version 2 But this is a subset that that Hortonworks put together To give you all the all the capabilities you want Likewise, there's something here from CDH very similar In that it has HDFS and yarn at the core, but they'll they'll choose Services a little bit differently to give you the same Capabilities like for instance they have in Paula if they're doing sequel, right? So distros are your friend. They help make the choices for you And as obviously as new services come online these companies release new stuff And the beautiful thing about this is Sahara gives you access to all of them Simultaneously, so you don't have to choose you can have everything so Great, I hinted at Sahara a little bit, but then the next obvious question is how does this all map to open stack? Right, how am I gonna get this on my cloud? Hey, that's my alarm perfect So I'll hand it over to Sergey at this point Thanks very much Okay Okay, let's take a look on it from the opens tech point of view So what's the place of big data in open stack? Obviously It's a it's a workload on top of open stack. Actually, there is another place for big data for the for the cloud infrastructure itself for example for Logs processing and management, etc. But let's take a look on a workload part so In the open stacks big data implemented by the Sahara project It's the codename of the officially integrated data ports in program and open stack So is it was already say that some there are two main goals of Sahara project the first one is a Vision and operates the data processing classes and the second one is a schedule and parade data processing drops and From the Sahara point of view. We're working not only with the Hadoop, but with the data processing frameworks to Support the new ones on the mend with a pluggable Approach so for now For in Sahara data processing means support for the Hadoop Spark and storm So how Sahara works actually it's fully integrated inside the open stack and Like if you'll go from the user point of view. We have a Spreaded dashboard inside of horizon That exposes all of the functionality of Sahara through the UI Teases spite in Sahara clients and Sahara integrated with one second a day and using heat for for the underlying resources provisioning such as virtual machines instances IP addresses volumes, etc So started with me duck with a Heat usage for provisioning will be on the option. So We've no way to use directly the service without heat So For for the external data storage Sahara uses swift as a default object store in open stack and the Sahara integrates with swift to To enable it as a file system for Hadoop to be Transplantly used as a file system from the Hadoop jobs. So some some words about the current state and Ecosystem of Sahara in open stack. So as I already said, it is an official open stack project for like last two releases, I think so it was initially started why these Merantys and the Red Hat and Hawthorne drugs joined And so so we have now much more contributors and For example, we have vendors contributing their own drivers for example map are fully supporting their plugin inside of Sahara Okay, so both the provisioning part the main and mostly interesting part from this Provisioning stuff is that Sahara provides and users ability to manage all of the configurations layout For the Hadoop clusters. So in Sahara, there is a template mechanism That provides users ability to specify the whole layout and configurations manage them and Create the clusters from the templates Many times so you can have a template and reuse it for example for the bare metal and virtualized deployments There is a concept of not group templates in Sahara that Consists of the specification of the Process that will be executed on a On a node it's in fact a role for the cluster node it could be for example master worker and It could be least of the process of Hadoop that should be executed in this one To it contains open stack flavor to specify number of CPUs RAM and etc. That will be used for the virtual machine creation and there are some storage and networking specific configurations such as the syndrome volumes or Net network that should be used to allocate the IP addresses for the cluster The another template type is a cluster template and it specifies list of not group templates with the numbers of instances for each of them in addition to it it specifies some cluster-wide configurations like affinity for the Processes inside the cluster or some cluster level Hadoop configurations like replication factor Then Sahara checks Check the configuration before actually starting the cluster. So it validates number of services configurations for them, etc Based on the already created cluster you will be able to scale up and down the cluster using Sahara interface just by changing the number of instances for the node groups Here is an example of the template for the HTTP cluster It consists of the three node groups the master node group contains the name node the source manager history server and a bunch of additional Services like for example unbarry that is used for the actual cluster provisioning And it could contain some some other master sources for example like each base each master The another node group is a is a worker node that contains actually the main Worker processes for the Hadoop is data node for HDFS and not manager for yarn And we could have a secondary name node that will Like it will be another node group for a secondary name node for Hadoop and for example for Uzi server and client so this Three types of node groups build there the cluster topology and we could reuse it To create the clusters of the exactly the same configuration to reproduce same configuration So a few words to the supported distress right now we in Sahara support three vanilla plugins. It's named vanilla because it's just a upstream package versions of Frameworks, it's Hadoop spark and storm and we support three vendor distress. It's a HDPE CDH in mapar actually all of the plugins Supported or initially supported by by the vendors for example HDPE plugin were written by Hortonworks initially Cloudera plugin were written by Mirantis But now very actively supported by Intel folks who are working with Clouder and mapar fully contributed and supported by by the mapar team So how it grows with the ecosystem There are many new frameworks in a Hadoop ecosystem that are joining Apache and Hadoop world for example Cloudera and Hortonworks bring in some new frameworks each their major release and By the very pluggable architecture Sahara seamless support Could support any new products in framework actually any framework could be implemented through the plugins And that's actually the intention of the project is to Support the new frameworks based on their own demands and integrate between them to to provide Data ports in stack. So right now It's a provisioning part is a fully pluggable. So any any framework for the data processing could be implemented as a plugin in Sahara and the EDP part is not so pluggable right now, but it's becoming more pluggable for the Transparent support the new job types and data sources as a plugins and our actual plan to to extract all of the EDP stuff as a plugins to make it even more flexible and to make support Easily for the for the new data sources for example So right now Sahara is mostly Hadoop centric But it's already contained as some spark and storm and growing to support other data frameworks as well so Go yeah, so Okay, so you heard a little bit about How do PECA system you heard a little bit about how it maps into open stack via Sahara project So the next question which we usually hear at Mirantis and Redhead. Okay, how do I put this all together and in this section? I will share some practical tips or some practical measurements which we did on By running big data workloads on some of the test clusters and some of the best practices we came up with I Mentioned at the very beginning that we changed the title of presentation because to be honest We wanted to bomb you with some more statistical data and performance data But what happened is that the lab which we are using to Which we were using to do the measurement it became unavailable for us to some period of time So we had to postpone some of the statistics publication. We will do it over Current cycle still we will give you some we will give you some best practices now and we will also open for questions at the end so before again before I start Sahara as Sergey mentioned is now the part of Sahara is now part of official all the official open stack distributions including Mirantis open stack including Redhead open stack platform including canonical so you can get a pretty also installed pretty much anywhere automatically now As we go to actual dude actually doing the notes So first question we usually hear is how to compare the how to compare running hadoop jobs on In virtualized environment as opposed to running it in bare metal How do vendors and like cloudera and Hortonworks were usually recommending positioning for you So the rule of thumb the number which we usually say is that KVM as a hypervisor introduces roughly 10 or lower percent of overhead for KVM itself, of course it varies depending on the On the specifics of how do jobs that you are going to be running But again if you're talking to your manager if you're talking to a customer 10 percent so lower is usually the safe number to expose Obviously you need to I you need we recommend and you need to isolate the networks So separate storage traffic from open stack management traffic at least to make sure that Traffic that hadoop workload will generate will not kill your open stack cluster. It's really not nice on the storage level for On the storage level for hadoop VMs themselves The best practice recommendation is to actually use not swift not default LVM driver But to use block device driver So the driver which allows to you pretty much to do a direct pass through of a block device into virtual machine and Avoid ice-cazi overhead, which is significant Especially if you attach 10 to 12 volumes to a single virtual machine avoid LVM overhead, which is not significant But with LVM you cannot bypass ice-cazi. So you'll have to do like this and in such way you will pretty much utilize the full Hardware disk and pass it through into virtual machine yielding the best performance Two ways to do it today. At least if you're using mirages open stacks So one is like the guy here in the link described Configuring it manually. Okay, it doesn't work like this anyways Second way is again if you're using mirages open stacks, there is a fuel plug-in Which will make which will make public publicly available soon if you want to test it earlier shoot us an email We will get it will get you to test With which you can configure your open stack cloud at the deployment time to support block device driver and allow VMs to be created with with persistent storage like that and One more item here is scheduler hints passed by Sahara. So the way how Sahara works by default is that it hints Nova scheduler to Schedule Hadoop VMs in a way when they will run on the same Compute node as the volume is going to be created So in such way you preserve the main rule of running Hadoop workloads is to keep your compute as close to storage as possible So Sahara does it by default strives to do Sample configuration so that's the sample the sample hardware plus virtualized configuration for for Big data cluster which we once came up with for one of the customers So here you see open stack compute hosts. It's quite beefy. It's has two sixteen core two sockets worth of sixteen cores CPU to two hundred and sixty six gigs of RAM Simple simple simple pair for Open-stack and host operating system and well for tele for terabytes hard drives in j-boat mode You used via j-boat the rest of the configuration are virtualized Are virtualized Machines for the Hadoop components themselves so for the manager not for master not and for worker nodes and A couple more nodes so on the networking sites on the networking sites. Sahara uses standard networks to When it provisions big data When it provisions big data clusters with Hadoop So again, make sure you separate not only storage but also tenant networks if possible on different physical links obviously Because again, nobody likes killing open stack because of placing too much load on the Hadoop itself on RAM Please don't oversubscribe if you're subscribed CPU. It will just get slow if you're subscribed from it will never finish running so well usual practice with KVM, but especially relevant for big data workloads and And Last but not least as we were discussing with folks today With all the practices of which we which we can list here You still are in the chance of running into poor performance or running into killed open-stack cloud The key idea is that you will still most likely for real production workloads We will use some consulting from your big data vendor whether it's Horton works called there are power somewhere else so again on the open-stack site we can optimize for more or less generic usage of Hadoop on open-stack, but still it will very much depend of what will you will run It will depend much more on what you will run compared to when you're running generic stuff in individual machines So if you're deploying big data get ready to get some consultancy from folks who know big data really well not only open-stack Some of the open problems and questions which are relevant for Sahara and for big data workloads today So first of all, we're all working in the community to get Not only virtualized virtualize Hadoop for you, but also Bermetal Hadoop for you So Sahara natively obviously integrates into open-stack capabilities to do Bermetal via ironic Though there are some specifics which one needs to account for so for example here right now We have the patches in the review, right? What already merged patches for in the current cycle of Sahara to do some smarter things with ironics So for example when Sahara would be provisioning in Hadoop VM into a Hadoop Image into the Bermetal node. It will also automatically take care of provisioning All the available disks to be properly used by this node So by passing nice Kazi by passing pretty much and everything including Cinder itself So one more thing in works is to allow Different Cinder volumes to be used separately for HDFS and intermediate storage results and Things which we are also looking at which I mentioned that will be presented in our next development cycle is Certify some reference architectures on the hardware So this is the distribution which we use this is a Hadoop distribution this we used this is how much performance we get This because this has been taken too long So we've been focusing a lot of developing Sahara developing plugins and so on so forth and not so much on the hardware Certification we're closing this gap right now and final last but not least New my capabilities have been merged into Nova lately mostly for any three workloads We'll also got to take a look on how to properly leverage them for a big data including things like socket affinity So pinning the VMs to a specific CPU socket to avoid cross socket switching That's already gets us beyond the level of having Sahara as an obstruction layer and having support from for different Distributions that gets us to a point of actually running it in production and making sure it get we get most of the performance Available from the hardware that is used within the cloud Questions Sergey Why do we recommend with what why is our default recommendation with goes with block device driver not with swift? Actually the main reason is that when you're working in a swift it's implemented in a Hadoop in a way that you will you will access data in a swift through through the proxy nodes and it means that you will have a bottleneck on this note and Like a performance will be not not so good as using directly data and When a Hadoop sites and mostly all of the workloads are very data intensive So you need like very performance storage It could be dozens of terabytes of data and so that's like the default option for us is a ability driver with a direct attached disks for it and Swift could be used for some some input or results storing for example results could be much much like the size much could be much lower than the inputs Or you need to store the result that would be much smaller than the inputs seems to be used for it like as a for example When you have no Hadoop clusters right for example, there is the issue of running So We would keep sender volumes backed by BDD think of it as a block device pass rule directly into VM I think we're supposed to use the mic for questions we should have mentioned that earlier. Do you want to restate your question from the mic for? Posterity so just curious about the Scaling the cluster up and down if you know HDFS replicates the data as you scale the cluster up and you add data nodes The cluster has to rebalance. Is that something you find happens quickly and easily or like how often would you typically see? People rebalancing or scaling the clusters up and down so While you scale scaling down the cluster secure doing the decommissioning before they disable the data node. So it's in fact, it's The state of the note put as a maintenance mod like on the read only and Here are in it Starts the decommissioning process that moves all the data blocks to another data nodes. It's a full scaling down and for scaling up rebalances like we're not doing rebalance because it's like it could be endless process in Hadoop and the real There is no real advantages of rebalancing when you add in the new data nodes because because if you will start Writing new data, it will be put on a on a free note first It's not for HBase So it's only for HDFS for HBase you will probably need to to run rebalance manually HBase is a very specific case We have our object storage service based on SAP RGW, but not based on SWIFT So when can we integrate with the SAP? instead of relying on SWIFT So there are two options, basically the first one is to use as a rudder's gateway for SAP Is it exposed as SWIFT API, but it's it will introduce a proxy nodes and the performance will be bad Because you all data will go through the rudder's gateway There's another option is to use a self file system that is doing exactly the same, but as a file system and the performance will be most performance There is no like native self support for Hadoop and we're evaluating writing the Hadoop plugin to support this as a self natively, so we're thinking about Now let me make sense and how it's implemented We actually run a step in In-production with the SWIFT API and then Hadoop cluster on top of of it with with SWIFT as the data store for that So there's that we're gonna give a talk actually tomorrow on that So if folks are interested just to see how we're doing that come by So does the Sahara API Have planned to support the scenario where you know You have a separate user for creating a cluster while different group of users are you know only mind-submitting jobs and getting the results so in Vintaka we have a Probably in the end of Liberty release we added the ACL support So you could you could mark the cluster public And it will be publicly available turns In open stack and the users will not be able to operate the cluster itself, but we'll be able to run the EDP jobs on it Another question about the API so for some of these emerging spark like interactive Analytics right it'll be desirable the user can query the API and get the response right away if let's say you're doing a Query on some big data But does is there a plan for the Sahara API to support that kind of model? So we don't have such functionality right now and about the plans probably It's not in a roadmap for now Probably in the veto API we'll think about supporting the interactive queries So the BDD driver. It's in fact It's a very very small driver that just Using the VRTIO to attach the Block devices that does hard disks from the node to virtual machine in fact, it's just Saying to to leave weirds to add the VRTIO device The VRTIO performance is just a few percent slower than than bare metal so the VRTIO is like Like doing only a very small overhead it's like it's directly bypassing the disks to the virtual machine So on a scale yes, it it will like in total the overhead will be the same 12% for the performance You know, it's not introduced on the huge He is a huge difference for performance of your client actually, we were running tests and On a long running tasks, you you will never see this one two percent