Much better. So welcome. My name is Eric Chabelle. I guess you wanted to do that, didn't you? Go ahead. This is Eric Chabelle. I think I think he knows what he's doing. Yeah, please welcome him. I've done this before. I work at Red Hat. I've been in open source for probably 20 some years now. I'm older than I look. I'm told. I've been at Red Hat 10 years. This talk is very different from most of the stuff you've seen here. Well, I guarantee it's different than most of the stuff you've seen here. It's also something that I've only started doing the last couple of years and it's been a featured keynote to open up conferences. For example, I was in Split Croatia, if you know where that is, but it's like a vacation destination. Think of Greece, same idea. And I was in the National Opera House on a stage. This is the coolest place I've ever talked. Have you ever seen when they do those shows on TV like Cabaret's or Comedians and it's all golden boxes and stuff around them? And that's what it was like. So I came out on stage and I was kind of like, I was kind of looking at everything more than they were looking at me. I was like, holy god, this is cool here. You can find that online. It's pretty much the same title. It's on YouTube. You can you can watch that a little bit. Or you can pay attention right now, whatever you want to do. What we're going to talk about is kind of fuzzy stuff, right? This has got nothing to do with technical stuff. But every single one of you, and I should start with who are you, like who's a developer here? Who does nothing with technology at all here? Hey, there's always one. Who has been in technology or any kind of work around this area for say more than in ten years. Five years, a couple of years. Who's a student? Okay, so so we got a nice mix of everything.  So let's ask a few more questions. How many people here write? I don't meet papers for your present for your for your class or whatever, but blog posts, do they twist your arm to do it or is it a volunteer thing? Has anybody here written a book? That's kind of way out there, I get that. Who's written an article in the last month? See how there's like no, oh, there's one hand. One hand only. So we've got probably, I'm just guessing, several hundred years of experience in this room working in IT or something related to that. And there's one person that shared something this last month. What's wrong with this picture? Where are we standing right now? We're in a university. What do we do here? We share. What am I doing right now? How many people here have given a talk in the last month? That's a little better. Why do you do that? To share something, right? When I did this as a keynote at a place, it was a technical day for Red Hat. We had about 500 people in the room and I don't think when I asked about the age groups and the different levels of experience, I think most of them were gray-haired old men that have been working a long time and there was only three that raised their hands that even did any kind of sharing and reading or blogging and stuff like that. That's criminal, right? You don't want to be the old guy in the corner at some place like a bank that has the seat molded to his butt and does not share anything because that's job security. You ever heard the phrase security through obscurity? That's awful. That's not what you want to do these days, right? Where do you think open source comes from? Sharing. You look at anything and I've been living this since day one. I got really lucky. I went and I'm from America originally. I was in the military, got out overseas, stayed in Holland, met a girl, that's why I ended up there. I studied there. My teacher was talking about, heard of him? So he does this thing around Minix and operating systems, but you can't really install it anywhere because there's no TCP IP stack. So we use Linux a long time ago. That's how I got into it. From that point on, everything I did I shared and it's open and you can always find the problem. You can always go further. All those things you always hear about code and stuff. So I basically applied that to everything I do. And the best day of my life was when Red Hat called and said, hey, you want to come work on open source. You want to get paid to work on open source. When I tell people what I do, we sell free software. You see, you know what I'm saying? It's like, think about this. These are not oxymorons. What we're going to talk about today is three things. First of all, you, I've already started poking. But it all starts with you. Everything about what's going to change in your life needs to happen now when it comes through you, right? It has nothing to do with anything else. I don't care how cool the software is. I don't care how cool the company is. If you don't do it, it's not going to happen. So remember that. Second thing is really easy sharing. How old are those two kids right there? That's where it started. Trust me, you've been sharing since you were very, very small. Why'd you stop? Society isn't mind, mind, mind. Is it? Is it? I don't know. In America, maybe it is now. And it's about growth. It's about growth. It's an easy path. It's really easy. Trust me. So when we look at you, I've asked a few questions already. I can't stress enough. Wherever you work, I don't care whether there's an open source or a closed source or a super private IP company or whatever. You have colleagues next to you. You have colleagues behind you. You have bosses. You have kids at home. You have days at schools where you can talk about your job with your kids, right? You have all kinds of places you can share. Do you do that? You have pizza evenings where you guys hack at work. Heard of those things? Meetups? You got to go to a meetup? Have you presented in a meetup? I don't care what you do either.   You know lawyers at Red Hat actually do stuff like this? It's really funny. It's like watching a turtle come out of a shell. They're really, really not about sharing, right? Then they've got to come out and explain all these really complex laws to us about the IP and what these new things mean. Everybody, I don't care what you do, has something to share.  So tell me about it. Write about it. Tweet about it. Whatever you guys are doing. Snapchat about it. It's whatever it takes. I mean, it's so ridiculously easy. So these, I've never, never thought in my life I'd be given this talk at a university. So it's really funny when I put this up because yeah, you remember university? Yeah, yeah, yesterday. So it all started here when you're kind of like that original curiosity. Everything's new. Everything you're running into, and in my case, open source quite quickly. Everything I was doing was like Google something, look for it online, find somebody's blog.  Anybody know what slash dot is? I don't know if that's still a thing over here, is it? Is it still kind of active or? It used to be huge, right? It used to be the thing. I did a very small Linux install course. I don't remember what version it was.  Somebody slashed out at me. Didn't know what that was at the time either. But my website is being hosted by Skippal Airport in Amsterdam. And a little company that had two servers. And on the server where mine was was also the two biggest news sites in Holland. And they crashed it because they didn't use caching. So we're talking like gigabytes per second or whatever of traffic and it just completely holds a server. And then that means the news sites were also offline because it was a mess. Everything you're doing in the beginning is learning from other people.  Their first experience would be looking at a blank screen. It used to be like that. Anybody remember buying Microsoft products in a box? Yeah, you got a CD in a book and good luck. Really, that's all it is, right? Is that how you want life to be? Not me. Everything I have you can choose. You reuse, share and do anything you want to. So I hammer a lot on the word open source but apply that to everything, right? I mean, it doesn't have to be a religion.  No. I mean, I have an iPhone. I want stuff that works, right? Open source is about getting into that mindset and realizing that sharing is caring. It's really that simple. The more open stuff is, the more standardized stuff is, the better we're all off. And time and time again we learn that. Who here remembers when containers started? There was one container engine. What was it called? You're not that old. Come on. Yes, Docker, right? Anybody here work for Docker before a hammer on your company? Nah, just teasing. They came out with whatever they were doing with these containers, right?  Everybody was on that one. That was it. It's a really good lesson on how not to do it. What's the first thing that starts happening when somebody gets a little bit of a disagreement with a company that owns technology like that? It's time to make a standard. Now we have an open container initiative, OCI. That means we define how an engine is supposed to work and people program an engine based on the standard.  And Docker's becoming irrelevant as the only tool set to manage your containers, right? There's a little bit more behind that story.  They tried to open source but then only little pieces and that's not really sharing. That's just trying to manipulate.  There's people much more religious about this than me. And it all goes sideways. So in the beginning, you're digging around, trying to become professional, right? Anybody here a programmer? The guy's doing.  I started pretty late at school. I think I was 28 when I first started studying for my masters. I hadn't touched a computer at all. I was a marine so that should tell you kind of where my mindset was, right? How I got into computers was more of an accident than anything else. But the thing that caught me was day one. I knew I was in the right class because we were programming. And there's nothing cooler than trying to solve those puzzles, right? Some freaky language. You don't know how to do anything. It's really, really cool. But then, all of a sudden, you're going to roll out of school for the people that haven't done it yet. And you start getting anybody done like part-time jobs in a real company. They come in there and they say, hey, write a web service. You're like, oh yeah, I can do that. I can do that on my laptop all the time. You do that in a company. What happens? It's a whole different ball of wax. So much more complicated. You got all these buzzwords, containers, all kinds of stuff. My first job out of school was at IBM, believe it or not. Part-time. And I only did it because I expensed my school. You got to understand school in Holland is about 3,000 a year. So I got lucky doing school over there and not doing it in this building. No debt. I was expensing my school and my books and working part-time. And it was a DB2 logistics area of IBM. We're talking about tables with like, if you do a join, it is so big. It's worldwide to parts and stuff. It's so big that it does not come out of the printer. And you're like, what happened?  And everybody in your department knows these are older guys.  And I'm shitting myself thinking I'm going to get fired for spending all that money on paper. You program something and you push it to dev and it disappears. It's not on your screen anymore. It's gone. I mean, these are complicated things are in real life, right? It gets way more complicated. You're constantly learning, constantly trying to prove yourself. Anybody here that's been doing IT for a while, have you ever stopped learning? Why are you here on a Saturday? I'm impressed. This is the busiest room I've seen so far.  This is the best session you're going to see. I'm not kidding. I mean, it's based on feedback. If you take what you hear today, continue to do this. You're going to be highly successful. You're going to get job opportunities and stuff that you never thought you'd get. I don't go hunting jobs. People come to me. You know why?  You help people. It's why people want to be nurses and doctors and things like that. It's not to be a famous nurse and doctor. It's to help people, right? Nothing makes me happier than someone sends me a little email or a note and says, oh, that's great. Thank you.  I'll tell you a really funny story. I've been doing it for a while, working in the middleware BU of our company, doing BPM and BRMS mostly. So that's business rules, business processes, that kind of stuff. I'm a huge fan of the very simple basic steps first. How do you install it? How do you start it? How do you make a project?  So I fill all these buckets. This means that our sales guys can go out and they don't even have to go out. The customers find the stuff, play with it and then come to them. I had not gone down to South America because I live in Europe and it's a long flight to go down there and like train people and do stuff. And who doesn't want to go to Brazil? Who doesn't want to go to Peru? Who doesn't want to go? So all the other people in my team wanted to go. I said, go ahead, I'll just stay home. So I didn't go for about five years and they kept coming back and telling me, hey, they're really, really, really big fans of you down there. I'm like, yeah, yeah, yeah. They're just jerking my chain. And I finally went down after about eight years and the whole team of SA's met me at the door and in South America they're a little bit more warm I guess you could say and men when they greet each other they kiss on the cheek, at least in Brazil when I was there.  That makes my day. You know what I'm saying? You don't need to pay me, do that. That's really, really fun. Okay, so what do you want to do with this IT stuff, right? And I don't think just constantly learning new technologies is exactly why you're in it. Is it? You want to be rich?   Cool shit, right? And finally I get to use this slide in Boston. I'm a big Red Sox fan. I play baseball my whole life. I write for a fan blog. Not so much anymore. I'm a little too busy, but I really like it. I was there last night. Great stuff.  And you want all the coolest tools, right? You want to do all the fun stuff and you want to have the coolest toolbox. And you want the playground to play and you want to be able to use all these nice little tools and these open source things, right? I see the stickers on all your stuff. But most importantly, you want to have impact. That's what I've been describing. Whether my impact is based on software I wrote, I have no illusions. There's no software I'm writing that's even going to live past three years old, probably. Ah, one exception. One exception. I worked at a bank in Holland. And I wrote one of my first services there was a zip code check. It's one line. You can't optimize this. And really funny enough, one of my friends who used to work with me there went to an outsourced company outside and got hired back in like five years later. I was digging around and found our original 1.0 document for that. They're still using my service and my name's still on it. So maybe just maybe if the bank doesn't die there might be something floating out there for me a while. All right. So that's about you guys and where you're going and what you're looking to do. Now how are we going to do it? Very easy. You've been doing it since you were this big. Just share. You guys have... Does anybody here have a blog themselves? Some place you consistently write at or whatever? What's the copyright on that? Do you know? Do you care?  I've said it before, right? Open starts with you guys. It's about how you want to position what you're doing. If you look at the bottom of my website, my blog or whatever, since day one I've had like this creative comment. That's the most... It's in the art section. It's meant to cover art originally. But writing I guess falls under artistic stuff. And I want it as free and as open as I can get it and that's as far as I can get it I think.  I don't care. Everything I've done since 2005 I guess. Have fun. Tear it up. It's fine with me. When you look at where you work, not everybody has the luxury of what we do at Red Hat where it's like it's a thing. So how many people here work in a company that's got more than 10,000 employees? Now keep your hands up. That's three of you. Of you three with your four with your hands up, how many of you have written an email telling your CEO directly he made a mistake? More than once. Red Hat is a meritocracy they call it and it's just like open source. Anybody ever worked on an open source project? Somebody doesn't like what direction it's going. They all jump in and tell you how shitty your decision is, right? Sometimes not very nicely, but most of the time it's not too bad. Most of the time it's not too bad. That's exactly what it's like inside of Red Hat, believe it or not. Our whole company functions like, and this blew my mind when I came in, I subscribed to probably about 50 mailing lists because everything in my company runs like an open source project so everything's a mailing list.   I'm not egotistical, don't get this wrong I'm not better than anybody else that is not what it's about. It's a bad decision. I wasn't the only one telling him which is funny he just keeps hearing the same thing all the time it's a good one for you to use it. These are some of the things you'll find on our walls and all our offices so it's hard not to be motivated at work when you see stuff like this and I could talk about it now IBM just bought us they bought us very carefully what do we own? Nothing, no buildings, no software every patent that's been done by somebody from us goes into the patent pool where everybody shares it's protected the only thing they own is the people that are still working there. They paid $34 billion for us, right? Sounds nuts, right? They'll probably earn that back on us in about five, six, maybe seven years it's not so crazy then is it? They're risking one third of their shit to get our stuff because of these ideas because of the way we work inside they're doing all they can not to change us not to merge us, not to acquire us not to have too many of the IBMers get involved and mess it up they want our culture to remain the same they want our culture to spread I don't know what it's going to result in maybe it's all crap in two years I'm not working here no more or somebody else has destroyed Red Hat but you don't even see IBM as a company of Red Hat a company of nothing, nothing has changed same HR, same everything it's crazy my favorite is we could put a big ole exclamation point at the end of this kind of how it works who doesn't run into open source in their companies now it's pretty weird if you're a CIO and not doing open source at all it's just not happening and that is so different even ten years ago when I first started you know 90% of my I was a solutions architect the first three years 90% of my discussions are about trying to convince you why you shouldn't be using community stuff because I went through it at the bank we did our first implementation with community stuff tried to do the second implementation updated one component and all kinds of crap breaks who hasn't gone through that in an architecture yet besides the students wait till you get out there it sounds like a good idea at the time but when you get to actual enterprise stuff like what you're doing in organizations it gets complicated and has legacy stuff it's really nice to have components that are open source that are stable for three years that's guaranteed the APIs aren't going to change that developers can count on that and continue to program against that and everything I deploy still works the libraries don't get flipped the next release so as you're growing, as you're sharing you start bridging the gap to these people you're going to start noticing that you're basically a teacher everybody should be a teacher so you're teaching about how to underwater basket weave because that's your thing bicycling, baking cakes it really doesn't matter look at YouTube man every kid in the world thinks he's going to be a rich YouTuber because they're sharing about how to put on makeup at 12 years old maybe so but they're doing the same thing it's absolutely viral why aren't you guys doing it watch your kids man put your bank account online there's a limit to what you want to share but it's really up to you it's really, really up to you I can say this all day long and you go back home and you do nothing different than you did yesterday nothing's going to change it's really funny I can tell in this talk there's almost nobody like in their phone and not paying attention everybody's looking at me so you're all thinking about it like shit this is so obvious so here's some tips along the lines of how do we do this sharing and stuff write, write, write I don't care if you don't like it or not keep trying you know what's great about this stuff when you want to do a blog or whatever or medium.com or any place you want to put this stuff you don't have to write a book nobody's going to read any farther than one screen anyway so make it fit if you've got more to say make it part two, part three now you've got three articles if you get a fourth article you're now written for a month see how that happens it's real easy I have a so after they crashed my blog with Slashdot I went over to Google's blogger thing just because they're the best at hosting all the shit and cash and stuff and SEO and I don't have to worry about it and manage it myself all I want to do is write so anytime I have an idea I open up a new article I put draft at the beginning maybe whip out a quick title again and I'll come back to it later that's just my bookmarks for ideas to write about I always have about eight to ten rolling at any one time right now I have right before they started all these slides will be available they went online ten minutes before we started that was stage stuff I put up before I knew I was going to publish that stuff that is not all of it but that's all that would fit on the screen you can see the number of blogs I did for articles I did per year I'll be honest it all started out me being a developer and just bookmarking work that was so complicated I didn't want to look for it again still one of my top three visited articles you never know you never know when you put something online what it's going to be maybe nobody looks at it maybe it becomes the most famous thing you've ever written I was working on a piece of code it's two classes one class basically takes any object you give it zips it up and puts it in a database can it call that and then you have to unsealize and unzip it when you take it out so you can use it in your code those two things are the top three people are still looking that out it was based on Jboss code from like 4.1 or something we are so far beyond we're like at 8 now or something I can't imagine what people are doing with that code I hope they're not running Jboss 4 but anyway another stupid one was fixing WordPress would give me a blank screen at some point and it was some stupid version and it was a bug and if you did something with a flag it would fix it people are pounding that how to make an ISO remember burning stuff to CDs somewhere in that crap probably way back by 2008 and 7 is an article about how to do that on a Mac I got my first Macbook and I was playing around with it and couldn't figure it out and command line to do all that shit properly those are my top three stupid but I don't have to look all over the internet to find the stuff I was doing at the time I could google what I was looking for plus my name and it popped up on my site that's pretty cool you'll see later here about 2011 2012 and it starts getting focused and that's the tip I can give you up ahead of time I was all over the place in the beginning I was writing about cycling that I was doing I was writing about baseball I was writing about all kinds of crazy stuff you're not going to get an audience it's kind of chaos so from 2011-12 if you look at the top of my site it says very distinctly in the small description what I talk about there and that's all I talk about there if you want to talk about baking which really doesn't apply to IT make a baking website it's a lot easier for everybody to find and focus on what you're doing same thing for your social media I have like four Twitter accounts some of them are baseball I started writing I think around 2013 when they won the World Series I did it on the one that was already famous for all the IT stuff I had about 5,000 followers on Twitter you post a photo that's really nice at Fenway Park with a sunset you don't realize how big the Red Sox Nation is worldwide it's huge it's like within five minutes you have 200,000 impressions or something on that that's great when the game's going on but then all the IT people are like what the hell is this? and then when you go back to talking about the other stuff then the baseball people are like what's this? so I kind of mishmashed it all and then later pulled it apart so that's a tip focus on what you're going to talk about anything you're writing about or reading about or working on doesn't require you to write an article like if you're right now coding in my class on something here or in my session and you're thinking this is kind of cool so I put a little thing on Twitter and say this is a link to the code I just wrote I'm going to publish it later every week you are doing something that somebody probably could read about even if it's just 100 characters not everybody has time to write every week I get that maybe some people don't like to maybe you write in other languages I have somebody I know that writes Japanese for me he takes my stuff because he doesn't want to generate the content but he doesn't know how to do the Japanese so I let him translate the stuff I'm thinking pretty soon before you know you start getting crazy stuff like this where people are just like oh I love all these stuff maybe this is an example I use from some internal stuff we have we generate like t-shirts with slogans on them and stuff stickers that you see on the laptops but maybe your thing is podcasts or YouTube videos sitting down and talking about stuff I mean there's nothing you can't come up with there's all kinds of mediums maybe Instagram is your thing there's people that are millionaires on there I don't know how they're doing it but if you could figure that shit out you wouldn't be here listening to me we'd all be off riding in your Ferrari so we talked about you we talked about sharing this is where it leads to the growth I already mentioned one growth aspect here is you're going to get some recognition around that you're an expert in a topic an expert or not that really doesn't matter that's a relative term but you're somebody willing to share the experience for good or bad and anybody you're helping to get further along in what they're doing that's the growth you're a teacher you let your kids run across the street in traffic why do you let the developer sitting next to you at work do the same thing right it's a mindset there shouldn't be this fear that if I help him he's going to go past me the fastest way to promotion is make everybody look good around you seems weird doesn't it it's worked for me you don't climb over people you lift them up it's a completely different philosophy it's got nothing to do with socialism it's got nothing to do with communism it's not political there's no need to be people that act like that shit they always get bitten in the ass eventually I can sleep at night I can look in the mirror so you've been here for three days I assume you've been here three days you just come on Saturdays that would be really hardcore you blow off the weekend and just come on the weekend how many people live local to Boston who drove more than an hour who had to fly to get here Jesus look at this very cool that's dedication so think about it think about Sharon please, please, please, please, please I hope to read your shit one day too right? maybe you help me figure out how to change the oil in my car third time you're seeing it that means you won't forget it ever you share, grow very simple, very easy any questions? it's so shocking nobody has a question they all stunned it's that easy Sharon if you want to raise your hand I'll get you the bank right here thank you I can repeat this question I listened to you I remember this part where you talked about subscribing to a number of mailing lists so my question is how do you keep up reading? mail filters and experience over time there's a lot of shit I don't read so when I go on vacation I put a note up because I read this once that I think Volkswagen does this they automatically put your out of office to delete all email they think that the health of the employee is important enough that you do not need to be coming back on Sunday night trying to figure out what your big pile of email is if it's important and everybody knows that in the company so it's really important I'll get you again so I put a really clear out of office message when I'm gone, I'm gone I'm deleting everything and reach out when I get back mailing lists are filterable so my inbox is you're a pretty select person if you get into my inbox so all that other crap can pile up and I can browse it when I've got time but it's a technique it's an art form but it's open source man keep doing it and you're going to keep doing it we have one internal legendary artifact it's called the memo list anybody that's Red Hatter is giggling right now I shit you not most people don't believe this I was on that thing one hour you're automatically subscribed back when I joined it was just over 2,000 employees I was the 15th solution architect in IMEA that shit was already off the rocker I mean like within the first hour I think I had 500 emails in that thing and I'm not kidding it's the hardcore Linux I'm sorry very deep like religious about stupid shit I mean it is unbelievable what they get pissed off about it's embarrassing and there's no filter I mean actually this should be used to figure out who you fire in the company in my opinion but that's not how it works so my opinion is one small opinion in all these people but I can unsubscribe and I did and I've never gone back and I haven't missed anything we have in the j-boss side something called the core which is the same idea but it's like listening to the 3 year olds in daycare versus the intellectuals right they do complain about stuff but it's based on things and it has discussions and that's stuff I can live with and that's maybe 10 a month it's a big difference big difference but the memo list oh my god yeah thanks for the talk but I had a question regarding sharing stuff basically if you have something radical or controversial to say about the topic what if the company that you're working with doesn't like what you're writing about or somebody from somebody doesn't like what you've written in the past and they come back to taunt you taunt you okay there's a couple things there one if your company won't allow you that's something you agreed to when you signed up there I am not a fan of anybody filtering my stuff but that also requires discipline and maturity on your part to not write like a juvenile child and call people names and you know what I'm talking about consider every email one of the best piece of advice I ever got earlier in my career was consider each email each message you send in the company whatever as if it's going to be on the front page of the New York Times you do that and you'll be fine you won't be on memo list but you'll be fine that doesn't mean that everybody let you working at the bank was hard to do stuff like that when I was writing they were pretty open about it but it's up to you what you want to do with it I can't tell you to quit your job you know what I'm saying that's the easy way out for me personally it's a hint I don't want to work here I've actually done job interviews where you sit down and based on the feeling I'm getting and the questions I'm asking I mean if you've been around a while you know what you like and when are we happiest at work when we go to work with a smile when we want to be there things are easy when I can do a little bit of what I want to do and a lot of what they require me to do and the more we can turn it the other way around the better right if I like programming in Java I don't want to do Pearl if I have to do 20% Pearl and 80% Java maybe I can live with that but if it's 64 to the other way around I probably want to leave so in job interviews you start asking questions or when people approach me for stuff and I've actually twice stood up in the middle of one and said thanks for the coffee and left completely freaked them out they don't like it when people and I'm not being arrogant I'm not being an asshole it's just we can better stop now think about the process as you go through for a job if it's difficult already and you're not even in imagine how messed up HR is already it's just going to get worse and if you think back for the experience guys that's proven out more times than not when you get into these jobs so I don't mess with it anymore it's not worth it so I would say it really makes you pissed off about that stuff don't sometimes you can arrange that stuff that you write on your time don't use their hardware and they have nothing to say about it right but I mean you're all young right I don't know how many of your parents now have been putting you on Facebook since you were born you know there's people that are doing that there's stuff out there now about you that maybe you didn't put there and you do have to own whatever you write and is on the internet forever and ever and ever and your bare butt on spring break hanging off a bar is going to follow you for a long time you know and writing really nasty stuff or making comments like who hasn't seen people getting in trouble for stuff they probably how many baseball players this last year that when they were 16 in high school said some racist shit in Texas and are now being held accountable because me too is a big deal over here now guy missed the all-star game because of it are you accountable can somebody use common sense if I was hiring somebody like that I'd sit down and talk with them and I don't think they should be held accountable their whole life or 16 year old one statement on Twitter you know what I'm saying but that's me not all bosses are going to be like that I can't warn you enough man think before you push publish but that being said I mean I have opinions on stuff and I've published about them I own them that's what I think I'm not abusive I'm not a shock that's not me maybe that's you that could be your brand see where it takes you the guy in New York City the DJ Howard Stern or whatever he's like crazy out there right I mean but he's that's his thing he's doing just fine got more money than I do probably whatever makes you happy and that's the big key thing on this if you're sharing and you're helping people and you're doing what you love paths will open up and you get to choose between A and B pick one of the two and go don't look back you ain't going that way does that answer your question in a really long way everybody's got an opinion anybody have an opinion on what he's asking here's your chance share anything else it's all common sense really right I'm just sharing what I've gone through and what I've seen up close on the lucky path I've been able to take you like it pretty sure you guys did have you yet to have an audience watch this one and like be bored it's really funny to watch your faces you should stand up here and see how the reactions are going it's really cool I mean there's enough talks I do where you're talking about technology and half the people are in their laptop shit like that it's really nice when you see somebody touching a nerve on people that you know it's helping them get to some place they need to be I feel like this is pretty close this is one of my favorites to do like it any last minute comments we've got like two minutes probably yeah I had I that's another thing take this please please take this if it talks 45 minutes if your story is 30 take 30 okay and let me go if your talk is two hours and you use 15 minutes and let me go but you've told your story your audience is going to be so much more happier than that you're doing this crap and trying to look at your watch and stretch this shit out to 45 minutes okay I almost never look at the clock I think my first talk today was about half hour maybe was supposed to be longer but I was done and it's based on what your reaction is and what you guys are doing I can't turn this into a whole discussion if you don't ask a question right see boom oh yeah yeah the titles are kind of funky yeah so my real title is consulting level technical marketing which is complete mumbo jumbo it's just something that they use in the hierarchy in there so I use global technology evangelists most of the time but the last year and a half two years I've been doing a role that's called portfolio architect and what I do basically when I'm not doing this is kind of interesting it's what the solutions architect in the field don't have time to do and what consultants don't have time to do they do engagements and do really complicated solutions with our technologies for example omnichannel integration so integrating everything through their stack microservices and all the stuff so they can talk to their customers and all their channels that's pretty huge they'll have a big repository document somewhere and then they move on and the essays don't know where those are and nobody can align three of these customers to look at how they did this with our technology so that's what I do so I took three different ones a bank an airport and third one is it's like a year ago and I look at that and then turn that into a generic architecture with diagrams based on all three so it's not any specific one but it's generically how you do that so that then the solution architect can talk to a new person that's interested in this area or pieces of that architecture and have a story around it to look at based on customers so it's not we like to say that there's a lot of marketing out there around products and solutions where people talk about all the cool shit they can do this is fact based so nothing that appears in mind is a marketing so it's really funny the reactions internally is all the marketing people are like no no no if it does that my product doesn't know it doesn't this is what your product does you want to market something else that's fine with me that's what I do another long story short Eric do you really have another talk in like seven minutes oh shit yeah I got to run to the next one that's your fault man you all put all my stuff in one day so thank you so if you all want to run over to the other thing on the other side here we can do some fun stuff on the stage and actually code something for you yeah I think that's the conference auditorium yep so let's go thanks


Hello everyone, I'm Colline Royaux. I work as a PhD student at the French Natural History Museum in both the Borea Research Unit and the French Biodiversity Data Hub. And this video tutorial is related to the written tutorial available on Galaxy Training, Compute and Analyze Biodiversity Metrics with Pampa Tool Suite. So I'm going to share my screen with you. So first thing to do is simply to go to Galaxy Ecology through the URL ecology.usegalaxy.eu, so I'm going to write it. Make sure you are logged in with the user menu just here. So I'm logged in, that's okay. You are obviously not obliged to be logged in to use Galaxy Tools, but you will have some restrictions and your data won't be saved. So it's better if you create an account and log in for this tutorial. So firstly, the Galaxy interface is composed of four parts, the head banner where you can log in, find data and workflows and so on. Then on the left, you can find all the tools available on the instance separated in several sections. And you can search keyword as well to find tools quicker, like I don't know if you want to manipulate FASTA files. You'll have all tools containing FASTA in their names or description. So then on the right, you have your current history containing all your data and every output of every tool you use. And finally, on the center of the screen is where you currently have the welcome page. But there is also where you get the interfaces of the tools, the visualisation of the data, etc. So you can expand this part by reducing the history part and the tools part on the site. Just using these arrows on the down corners. Then to start training, you just have to click on the little graduation at that is on the top banner to access the training. Then you are on the welcome page of the Galaxy Training. You can also open the Galaxy Training in another page just by clicking here and opening in another page. But it's really, really handy to do it this way, so I think it's better here. Now we can open the Ecology section. And our story is the first one here, a computer analysed biodiversity matrix with Pampa tool suite. And you can open the end zone just by clicking on this computer logo. And here it is, you have your training ready. So let's take a quick look into the tutorials overview. So in this tutorial, we'll learn how to evaluate properly populations and communities, biological states with abundance data through computing and analysing of biodiversity matrix on Galaxy. We'll use true survey data that are available on the Datras portal, carried by the International Council of the Exploration of the Sea. We'll try to assess how those exploited populations of fishes are doing over time in Baltic Sea, Southern Atlantic and Scotland. As we use raw data from the data portal, we'll start by pre-processing to make it fit into the proper format. I've avoid mistakes on the analysis, then we'll compute metrics at population and community levels to analyse it through a common statistical model called generalised linear model, also known as GLM, to respond to a classical ecological question and finnally generate interesting visualisations from it. So to do our wonderful analysis, we'll use a workflow called Pampa, which is made of five tools. This workflow is dedicated to ecological analysis, and it is divided depending on the biodiversity level you want to study between community level and population level. So to clarify these terms, you can expand this frame here that contains the definition of population and community. So a population is a group of individuals of the same species that are interacting with each other, whereas a community is a group of several species, so several populations interacting with each other. So here on this first figure you can see the organisation of the workflow graphically. So each part of the workflow starts by the computation of metrics, then the modelisation parts, and finally graphical representations. So the details on each tool are written just here on the text just below. So for this video, we'll go through the details while we're using it to avoid too many repetitions in this recording. Then there are also some technical details about the models we'll use in this tutorial if you expand this frame. So here is the technicalities on the statistical models. So I'll explain all this quickly when we go through this part on the tutorial. So let's dig in into the end zone. We can now start by creating a new history. So here is the step we are currently doing. You will just have to quit this pop-up part just by clicking anywhere around it on the gray area.  On the top right of the history panel.  Just as you want to name it, no obligations here, it's just for you to find it easily later. So you just have to click on the unnamed history, on the name of the history really. And I'll name it as it is proposed in the example here, so I just copy paste this part of the tutorial. So I opened it back just by clicking on this on the logo and I get back exactly at the same place. So this is a really, really useful tool. So I just copy paste the name in the tutorial and just paste it right here. So I have my wonderful name here. So you can also add tags and annotations to your history to find it easily or to add background on your analysis, anything you think can be relevant to add here. Now to import your troll survey data into this history, start by retrieving the Xenodolinks provided in the training. So you go back to the training. Here you have the links provided. So there are other ways you can retrieve the data that are explained just down there, but we'll do the easy way for this video. So you can either click on upload data here on the top of the tool panel just under the search banner, or you can click on load your own data that appears on every new histories on Galaxy. Here you have white window that just popped up so you can click on paste fetch data here, and you have an entry that you can write in. So just paste your three prylings in the frame that just appeared and just click start. Here you have your three data frames appearing. So first they are gray as they're waiting to be charged onto Galaxy, then they'll probably become yellowish that's that which means it's charging. And hopefully it becomes green as the uploading works as well so yes yellowish one here. If it goes okay so it's green but if there is any error, it will become red don't panic if anything goes red, it happens a lot. So don't worry just try try again maybe or try another way to upload your data that are explained in the tutorial, just no panic here. If you want more informations on how to upload data in other ways, you can go to the tutorial Galaxy 101 for everyone. So this tutorial is. So I'll just quit this one. Go back to the welcome page, then the introduction to Galaxy analysis topic, and you can find many, many useful tutorials to learn how to use Galaxy. And I advise you to go through these ones if you want to see other ways to put data and other options there is in Galaxy as well. So we're waiting for our data to be downloaded so it can take a while. If it seems to you that it takes a really long time maybe it's better to refresh your history. And there it is. It was charged but the interface wasn't following on this one so just refresh your history works here. So now we have our data properly charged but we have to check it. Maybe checking the headers that must be survey, year, water, area, AFIA ID, species, land class, CPU number per hour. So we'll also check the other ones. So it's alright so sorry I didn't say it but you can just click once on each and you'll have more information on your data sets and overview of your data sets. So it's a really handy way to see if everything looks alright. So let's see to the other one. Here we see everything looks okay. Here as well. The format is CSV each time so it's what we need. And we'll check if there is that separators are commas so to do so you'll have to click on the visualize these data so it's the chart logo you have here. And there are many visualization but I like to use the editor to see the separators. If it's not point commas for example so yeah. Our separators are right on this one. So let's check the others so I just have many visualization here. To do bar charts and everything you can search for the editor by writing just the three letters, the three foot lasers of the word. And then you go on the editor and you see everything looks right here. Also, just the last one. And that's okay so our data is properly formatted everything is right. So you can see there is three frames that has been retrieved from with different methodologies so we have to keep our analysis of the tree that I said separate so I said in the tutorial. I'll have to go back there. Sorry. Okay. And so on. And there we are. So I said in the tutorial, the CPU equivalent area is the survey. So it's a French survey on 1000 authentic bottom troll from 1997 to 2018. So we'll just add tag tags to each file to make sure we don't mix it up. So to add tags you just have to click on this logo here. Then here and you'll have to write hashtag and the tag you want to add so it has to be in one word. So here we have a void, then the second be file corresponds to SWC IBTS, which is a Scottish survey. So this survey is on Scottish Scotland, West Coast, bottom troll from 1985 to 2010. And the last see file is the by be it s survey, which is an international survey on Baltic sea trolls from 1991 to 2020. So I added an hashtag each time here. Because I want the tags to be spread through the history but if you don't want to do so you can just add a tag without an hashtag for this particular case. So I advise you to do so as we'll have a lot of data sets that are going to be generated. So it will be mixed up pretty quickly. So in Galaxy best is to have your data sets in tabular formats to avoid any problems. So you can convert your data types by clicking on the pencil here at the top right of the data set box. So then you click on converts. And here you can choose a target data types or there is only tabular. And then you can click on create data sets. And here a new box appears on your history with the target that has spread with our data frame. We're waiting to be launched here as it spray will become yellowish and hopefully green finally. So if you want to convert to we have to compare the two other that assets. So you can do exactly the same manipulation. But you can also click on the converted on the new that I said that just appeared click expand it. And click on this round arrow that means to run this job again. And here if you just click on this icon to select multiple that assets. You can choose your two other that I said by holding your click. And then sliding down. So that in one execution of your tool, you get the two other converted so it's quicker. And it's another way to run tools on Galaxy so good thing to learn. So now we just have to wait it to be converted. So here you may be a little concern about the fact that we don't have the tags appearing here but you just have to refresh the history, and it will appear once again on the new converted data. So now the data is properly uploaded onto your history we have to prepare the data to make sure the workflow goes well. So let's go to the tool we want to use and see in the help section. So, the tools we'll be using are in the species abundance section. So here we have compute GLM on population and community data. Here, we have calculate community metrics and presents up some stable and finally created plot from GLM data so as we are going to use. First, the calculation of metrics tools. I'll just go to the calculate community, community metrics so you see it opens in the center of the screen, or you can go as well. And just opening your training and going up there and also click on the highlighted on the two names highlighted in blue. So it goes right to the tool you want. It's also very handy way to use the training here in the pop up screen. So let's go to the help section to see what kind of input we need. So we see we need a tabular file with observation data. So it must contain at least three columns, a first one named observation units, which may seem a bit mysterious right now. So GLM actually identifies a unique observation on a particular geographical and time points. So it's just one event of observation. So that's more information on this field by going back to the training, going to the pre-prepare the data section and just expand what is the observation unit field and the observation unit nomenclature frames. So all the information you need are just there. But we won't bother with this column for now. We'll just use the year and location column. So as you can see, there is a possibility to use either these columns. So then we need a species code column that's identified the observed species and a number for the abundances of the species. So let's go check the other tool on presence absence table see if there is any difference and we see there isn't. So let's start the preparation of the data. So as said in the training, we'll just start to concatenate the data sets to head with cats. So here we just have to click on this and select these three data frames. So you can either on the multiple data sets option, you can either just click on the slide or click all your control button and click on each of your data sets. So then you just execute the tool. So the new data sets will normally have the three data frame reunited into one. So it is here confirmed by the three tags that has provided to the new data frame. So we'll just wait for it to be done. Don't hesitate to take breaks during the computation of tools. It can sometimes take a long time. So don't hesitate to do so. So now our new data frame is ready. So you can have just a preview of it. So it seems to have the same format at the previous ones. Just it has many more lines. If you check here, many, many more lines than we had in the first ones. So it's a good thing. And we can see there are the three tags here. So the best is to suppress these tags and just add a new one. Okay, so this tag is to identify these concatenated and not mix up with the other ones again. And now as we did naive concatenation, there is good chance we have the three headlines on the three original data frames in our current content contentated data frame. So we'll have to check it by using the tool count occurrences of each records as it is said in the tutorial. So just click on the highlighted tool, select your last one, your last generated. That I said so it's must be already selected here, and then just choose the first column as it will be detectable in anyone in any column. Your file is delimited by tabulations as we converted it just before. And we want to sort the results with the various values first. So let's execute the tool. So now our data frame is created. We'll just have to see if we were right and view the data by clicking on this I logo here. And there is three occurrences of the survey so the banner of the first the header of the first column. So we'll have to try and delete it, delete the two other headlines that would make mistakes in our analysis if we keep it. So we'll use the filter data on any column using simple expression on the concatenate data sets. So the seven one in mine, not the counts of course as it is just a report we won't be using it further in this training. And with the flowing condition so see one means first column then exclamation comma equal sign and the survey. So here we ask for filtering any line that has a survey in the first column. So this exclamation points and equal sign means different from and survey is our character string. Then we indicate there is one headline to skip as we still want to have one headline in our file. If we don't do so we won't have other, we won't have a line at all so that's not what we wish for. And then we just have to execute this tool. So we can refresh the history to have the concatenate tags that appears. So here it's maybe better to remove it from the count as it isn't data frame that will use in the future. And we'll check it again on this last filtered data frame if it works that we have really suppressed the lines we wanted to suppress. So here it says it's kept 100% which is, which is kind of weird but we have a lot of lines, maybe it went through the check. So we'll just open the count, rerun, run the job again with the round arrow, change for the new data frame, the filtered one, and just do the same exact computation to see if the headers were really suppressed from the data frame. So in the meantime, let's get back to our training. So here we are doing this step to check if there is really just one headline, and then we'll just start by formatting the data files. So we'll have to look into our generated filtered data frame and see if we have material to create year location species code and number columns to use with our current columns that are not in the proper format for now. So our count just finished and we have just one survey in the first column so the headlines has been removed. Yay. So in our new data frame, the year and species column are in the right format it seems. So we just need to change the column names into the column names right for the compile workflow. For the number column, the abundance seems to be just here the CPUE number per hour column. So we'll just change the name of the column as well. And the only particular case will be for the location column. We could just use here the area column to do it for of our current data frame, but there is a risk to mix up areas from several different survey as this concatenated data frame is composed of several surveys. If they unfortunately use the same names that we don't know that so we need to create the location fields by adding dash at the end of the survey column, and then we'll merge the survey and the area column together to avoid any mix up. So at this point of the tutorial, there is another subtlety will proceed these modifications, not only on the concatenated file. So the one I've got under my eyes here. Just make the tags appear so it's clearer. So the concatenated file will use is this one and we'll also use that three with surveys alone. So the data frames as well. We do so because as the community analysis has the possibility of being analyzed separately from each other. This option doesn't exist for the population analysis so we'll need to keep the data frames separate to avoid analyzing several populations of the same species. We'll need to interact as one only population would do. So first we'll have to use the tool column regex find and replace. So that is just here on the end zone. So if you haven't understood what I said here I know I don't have the best English, and maybe some are clear so please just pause the video and go and look into the written tutorial. Maybe it will be clearer for you. Don't hesitate to do so. And we'll get back to the column regex find and replace tool here to just add the dash at the end of the survey column. So to do so we'll use regular expressions, which is a little tricky as well. So I'll try to be to explain right here but there is a lot of documentation on regex online so don't hesitate to go there. So we start when we get here just to switch to the multiple data sets option as we want to do the computation on several data sets. So first the filter, then we hold our control key and select the three converted data frames. So here we have four data frames selected. We'll choose to use the first column as it is the survey column, and then we click on insert check to do a character string manipulation with a regular expression so also known as regex. So I hope this won't be confusing. But if you are interested, I just advise you to type cheat sheet regex on your browser and there are many effective documentations you can find this way. I will try to explain what I'm doing clearly but it's not that easy to to get how regex work. So I just erase here the in the find entry, I just erase in the replacement entry and come back clean. You don't want to bother knowing how regex work. I totally understand you can just copy and paste what's proposed in the tutorial you just don't don't have to understand this but it's not the center of this tutorial but I think it's still really interesting so I'll try to explain how I constructed this regular expression. So first, what you have to do is to look into the structure of the survey names we have in the column. So, luckily, we did our count on this structure so we know a bit on how it is constructed. We know what is everything in this. So, we can see our survey names are composed of uppercase letters. Only for the SWUC, IBTS, there is a dash so you can just ignore this dash we'll just use this part of the expression to construct our regex. Because we want the dash to be added at the end of the expression so it doesn't have to match the whole character string. So we can see that our survey names have a minimum of four characters here with the bits and the IBTS also. So we start by writing what stands for all uppercase letters in regex, which is uppercase A dash uppercase Z. I write it in the replacement, which is really not the right place to do it sorry. So, these have to be between square brackets. And here we have the expression standing for any single character that is an uppercase letter. So if I want to indicate there is three letters uppercase letters that I need. I put a three in between curly brackets just next to it. So here I have what stands for three uppercase letters characters. And then as we need to select character strings of four or more uppercase letters, I just copy paste the uppercase letter expression, just paste it after the three between curly brackets. I just write a plus sign after to indicate the prior expression. So this is for one or more characters. So here I have the proper regex expression to point out every character strings of four or more uppercase letters. So here it will go through this one, this one and this one. Okay, so we simply want to add a dash after the string. So we have to indicate we want to keep the original character string by putting it between brackets. So this way, I put the all expression between brackets. So it can be called back in the replacement entry with the backslash one expression. So this means first expression between brackets in the find regex entry and just add a dash after it. So it will likely add a dash into our four data sets. So we'll just have to just take a look if everything seems all right. It's the right for data sets on the first column, that survey column and our regex is good. You can check back in the training if you want to be sure. And then you can execute it. So here we have the four new data frame that appear. So it's a first good point. We'll just wait for it to, for the tool to replace to add the dash at every lines. So here we have one up here, everything is right. So we can just check for the few first column if our replacement worked with the by viewing the data with the iLogo here. So here the wonderful dash. Here it is again. Wonderful. And finally, yes. So it worked just fine. Now we have to merge the columns survey and area. Just to use simply as said in the tutorial, merge columns. So just refresh the history again to for the tags to appear. So it's easier to see it. You'll have to select the multiple data set option as well. Again, just click and slide on your four data sets and select to merge the column one with the column four. So it's really handy to have a preview here. As you can see, we can just check each time if we are in the right, if you selected the right columns. So that's amazing. So everything seems right here. First column, fourth column, just execute the tool. So again, we have four data frame that pops up in the history. Okay, so refresh again. Okay. Everything works fine. That's wonderful. So if you have any issue at this point, don't worry, everything will be fine. Just try again, maybe check back all your settings and I swear I have a ton of histories and there is more error than a good part. So don't worry if anything goes red. It's normal. So now if we checked our for newly merged column data frames, we can see there is, there is a nice column that's formed that is a survey area with our survey names dash the name of the area. That's wonderful. Same here. Same here. And same here. So good job. So now we just need to change the column names to have our data frames ready. So we use the regex find and replace. So regex again. I'm really sorry for the one that doesn't like it, but this one is actually easier because we don't have any expression. It's just character strings, just as simple as it can be. So just click here. You can do the copy paste thing again with the the tutorial. So I'll do the first one this way and then I'll explain you how I did this.  So you can copy and paste here. Each arguments. It's okay to do so. But you can also just look into your column names. And you'll have to add a check each time. Each time you want to change another character string so with the insert check box here. So then you have the species. You want to change into species code. And then the CPU e underscore number underscore per underscore hour into simply number. And finally, the survey area into observation. Into location. Sorry. So here we have our four column that are going to be formed. That's amazing. And the four data frames here that we selected execute. So, after this, we'll just have to check if the columns are okay. So we'll have to wait for it. But if you want to use this workflow on other data of if you just want to preprocess in other ways, these data there is million ways to do it. It's just one path. You can do it anyway you want. I just added in the tutorial a frame that you can expand with tool names that can be really useful in proposing data. So don't hesitate to to look into it. There are many tools available in galaxy to do this preprocessing parts that are very efficient, you just have to search for a bit and when you find the right one it's. So now are for you and other frames are ready it seems we have here species code number and location. So here I looked for the large view but again you can go to the preview year species code number location. Okay. Same here. Just all right. And it's okay. So everything seems right again refresh your history for the tags to appear. We're done with the preprocessing of the data. So I know this part can seem very long and laborious but we wanted to give you a real life tutorial on this one. So the preprocessing is always a large part of the analysis in science. Very often it is even the longest part of the analysis, and it is very important to get to know your data set before analyzing it, just dig into it to know its biases and flaws to analyze it with with care and extract the best and list bias conclusions out of it so be careful on that to do good science really. That's the final objective. So now we can do the more scientific parts of the tutorial we can now compute our community and population metrics. And we'll start by the Catholic community metrics tool. So just by clicking my the highlighted in blue tool here. Here it is. And you can choose the concatenate data file as you as we talked already before we can do the community metrics on the concatenate concatenated. Sorry, concatenated file. As we can do the analysis by separating the surveys. It's okay to do it this way. And for the community metrics we want to compute will select all because present substance issues richness, Simpson and Shannon index. Also, Piel and he'll indexes are of interest so we'll just compute them all. So it seems alright we have our community metrics calculated just checking if we have everything. So we have the total abundance of the community of at each year and location. We have the species richness is Simpson, the inverted Simpson index, the Shannon index, the Piel, the heel index, and we have a new column named observation units. So we'll talk about it just later. If you refresh your history, you can see there is the concatenate concatenate tag that propagated here which is completely normal we just add a community tag here to identify our metrics file. And then we can do the same to compute our population metrics. So let's go back to the training. Click on the calculator presents have some stable tool. Then we select multiple data sets. And we select our three survey alone. Yes, so the 21, 20 and 19 execute. So just have to wait for the metrics to be calculated. And the next step of the workflow is to compute the statistical model. So with the compute GLM tool so let's just have a look into it. So we see we need an input metrics file so the metrics we are going, we are currently creating so it's, it's now down. And then we see we need an unit ops information file. So for now it's. We don't have such a file so let's see in the help section what is this file about. So here in the input we see the population we need with the population data. And then a second file tabular with observation unit data which contains at least as much columns as use explanatory variables in addition with the observation unit column. This file is to give information on each observation, such as where it has been made, when what type of habitat was it, etc. It can contain any important information that has been noticed on the field when the observation has been made, or how the observation has been made by whom, etc. So as I said before, an observation unit is a single event of observation at one time in one place. So the observation unit field is in the files that we can see here, and that we will need in the observation file as well. So we can check in the population metrics and it is there as well. So, yep, it has been created in each one we have to have obviously the same formats of the observation units, as it represents actually key to link the metrics with the observation on on the field and the audio we need actually. So if we, so if we have to build this observation unit file, we need to have it observation unit field that is formatted this way. So with the year of sampling underscore and the location name just next to it. So here in the tutorial, we have the nomenclature here that is detailed. So again, if it's not clear in my in this video, don't hesitate to pause and go and look into the written tutorial I just say quite the same thing but in another way. So don't hesitate to go there. So to format this new file, we'll just have to add the underscore at the end of every year values. So here with the column regex find and replace tool again. We'll do just as we did to add the dash at the end of the survey column just it's an underscore and it's the year column. So we'll do so on the concatenate data file to have the most informations. So it's the 22 here the data frame we used actually to calculate community metrics that still contains every information we have on the area of observation. So the column we want to modify is actually the year column so the second one. So we'll insert to check and in the final regex part will write the expression to point out any number from zero to one so it's a zero dash nine between square brackets so it has the same form as for the uppercase lasers. Then to indicate we have a four of this number. So the format of a year we had a four between curly brackets. And finally, as we did for the location column we put this expression between brackets to call it back in the replacement entry with backslash one and just add the underscore after it. So now we can execute this tool. Now as we did before we just have to use the merge columns. To. On this last data frame that is waiting to be generated. So we'll just have to check if it worked. Yes, it worked. We see there is there are underscores after each year here. So we just have to say we want to merge column two with the location column nine. Execute refresh to get the concatenates just to get the tags back and just wait for the columns to be merged. If the columns are merged, we can say we have our observation unit field ready. So let's see at the end we have our observation units that are properly formatted. So that's amazing, but to get a proper observation unit file. We need to remove the underscores from the year column with approximately the same manipulation we need to add it. So again go to column regex find and replace, select the last the last data frame we used column two insert check. And then again zero dash nine between square brackets four times with an underscore just after. And this time we'll have to select only these parts to put it between brackets. So only this part is between brackets, not the underscore and just call it back with backslash one in the replacement part and the underscore should be gone. So now if you look to your data sets, the underscores after each year value has been gone. So now we have to cut the columns that gives details on the species, the length class of the species, the number of it. As our observation units file and needs to have informations only on the time and place of sampling. So not the species, not the observed biological individuals there. So on the last generated file without the underscores will go back to the training will cut will use the tool cuts columns from a table. So selecting the last generated file, then ask to keep the column will add in the list of fields, the limited by tab. It is still a tabular file so the limited by tabs and cut by fields will keep our first column survey second column year third quarter fourth area and then will just keep location and your location. So nine and 10. So here we go. So I do exactly has on the, on the hands on so you can just refer it to there if you're a bit lost, don't hesitate to do so. I'll just, I just don't read everything in the tutorial for because it will be a bit boring, I think. So, yeah, again, don't hesitate to get this video on pause and just relax a bit and see if you missed anything on the tutorial on the written story. So now we have our file that is properly cut and we have the observation we want. So now we likely have a lot of repeated lines as we suppressed the information on species. So we'll have to suppress the duplicated lines and get unique values in the output. So in order to do so we'll use as said in the tutorial sort the short data in ascending or descending order tool. So these two not only permits to sort your data so just select the last data frame, but you'll see a bit further so I'll use the six columns so you as you can see I selected one headline. Then the columns to select to do the sorting according to is the next one. So the last one representing the observation units. So it's supposed to be the core of this file. Then we select ascending order alphabetical sort. So it's not this part isn't really that important. The most important part of using this tool is to check this particular option to output unique values. It's the outcome we want for our file to shorten it at the maximum so please check the yes box here and in your case we keep it this way. So just execute this tool and hopefully there will be less lines in our new file. So our data is finally sorted and we have unique lines and we see there is far less lines so here around 100,000 lines and in our unique lines we have just around 600 lines so we were right on the fact that we had a lot of repeated lines. So we just have now to change the column names with regex find and replace tool so I it's the same as column regex tool but just hasn't focused on one column so one or the other is really has its own use and not always proper to use on each on the same for the same manipulation so it's according to you according to the expression you're looking for and so on. So let's choose our last sorted and unique data table insert a new check find the year location and replace it by observation dot unit. So make sure you start by modifying this year location so that the first check is on this particular expression because if you do it after the second one. That is location to replace in sites. If you do this check first you'll have no more your location of course it looks for location you have it here it will change it into your site and you won't find it this way so just be sure to use it as the first check. And then execute we check we had yes we have the right data frame and execute. So if these steps works we have our observation unit file ready and we can just start to compute some models. So, we did a pretty good part here it was, I hope, not too heavy. So let's see. We have site and solution unit. It's perfect. We just had a unit of tag to identify it I don't put a. Nash tag here. Because I, it's the last version of our observation unit file so it's not necessary to make it spread through. So, the problem is the tag won't appear here when the data set is not expanded. So, it's problematic for the visualization but when you enter it in in a tool you can see the tag anyway so it's where it is the most important to see it I think. So okay, let's go back to our training. We'll start by the community analysis apparently. And that's when I have to talk a bit about generalized linear models so in these tools you'll be able to perform either a classical generalized model. And that will test the effects of your site and or habitat on any metric you choose. Or a generalized linear mixed model that will be able to handle pseudo replication of year and or site values by taking account of temporal and or special sampling replicates. I hope this is understandable. Explain this way. It's really statistical technicality so it's it's normal if it's not. If it's hard to comprehend so don't worry. So to achieve to this we have to set random effects. So that's why it's a model that said mixed. So we have to set random effects either on year and or on site. In the data we currently have there isn't informations on the habitat so we'll test the effects of year and site only in our models. There is often geographic pseudo replication in this type of sampling so there is often replicates of maybe the same individual that will be sampled in two areas so to take this into account will only set site as a random effect. So you'll have to know that if you want to put an effect random effect on year as well, you won't have effective results of your model so you can't have all effects in the GLM that is random. So we have to hear test year as a fixed effect. For the community analysis we use the compute GLM on community data that is here on the first hands on. So we use our community metrics file 23. That is tagged concatenate and community. The unit of it's the last we use. Of course, it's just here we just did it, it was. So then for the response a variable from metrics file. So we have a big choice between species. So total abundance species richness, Simpson index inverted Simpson index Shannon index peel index and he'll has the species richness is easier to understand because it is into interprets. Quantity of different species at a given time and location so it's just a count. And it's located in the first column so that's the one we choose. So to avoid the mix up of the analysis of the different surveys that analysis are of course separated by the survey column. We'll have one. One analysis per survey. Which is located in the first column of the unit of data frame. So then as we said before we set the year and site only so I suppose the habitat effect here if you want to add it back you just have to click on the entry and click back to habitat.  Just very easy. And a random effect on site as it is in default parameters. So you can specify advanced parameters. In this tool but we won't do it this time, you can get more informations on these advanced parameters just in the tutorial in the. In the frame here, if you expand it you'll have details on it. So don't hesitate to go through it. And then we just have to click on execute and wait for the models to be ready. So here we have our results. So there is three outputs. The first file is the GLM results file. It often has a weird display. Yes, there is. It's quite hard to read in Galaxy. So there is some ways to avoid this type of problems. You can go to the you can click on the enable the scratch book visualization on this logo here in the top banner. And if you click again on view, it should be right. Yes, you can now see it as a well formatted data file. There is another way you can see it better. You can just as it is said on the tutorial, you can use it's an optional step. You can use transpose rows columns in a tabular file. Just select the GLM results and execute. And it must display nicely for you to look at it better and make it easier to visualize. So both options are good. Just choose the one you prefer. So there are many, many, many details on the produced outputs in the written tutorials. So if you want to know what does each little part of the files mean, you can go onto the tutorial, read everything, open frames with details and you should know everything about what what's happening really in these files. So here our transpose just disable the scratch book and make you see our transpose data frame. So we have a lot of analysis results and indicators that are indicated in this results file. So on this first output GLM results from your community analysis, you can see there has been a four models produced. One global, one on bits, one on EVOE, one on SWC, IBTS. So as we asked for just the first one on global data, we won't look into it as it's not relevant for many reasons we already explained. We can see that the distribution set is the Poisson distribution. It has been set automatically by the tool and it is the proper one for using on species richness as it is counted data. So the Poisson distribution seems just fine. If we look only the significant parts of the results. So I know it's not readable this way. Don't worry. The representation will do next will be far more efficient to see it but I want to have a look on in these in these raw results for you to get to know it a bit. So here if you look into each CINIF lines on the transpose or each CINIF columns on the classical GLM results that isn't readable other anywhere else in the scratch book. So you can see there is always a no significant result, no significant. So it means that the community hasn't really changed over time in these places. So it's always no, no, no, no, no, until the end of the file. So no significant results, no significant effect of time on the species richness. So now, usually we're a bit disappointed, but we'll, we forgot something that is a major issue without data frame. It contains only part of the species found in the samples. All the rarest species has been removed from the data frame to get a smaller data set to use in this tutorial and avoid a too long runtime of the tools. So the data set we used isn't proper to make community analysis as it contains data only on part of the community. So here it's for the example. So it's not a big deal as we only wanted to show you how to use these tools. And now you know how to do a proper community analysis. So that's pretty good. Just the data isn't fitted to do so. So we can still move on to the second output simple statistics on chosen variables. So this doesn't have many informations really on the, on the, the model itself. But it's more on the interest viable we use such as we have statistics such as the maximum value of species richness we have in our data sets. So here we see the maximum is seven, which is very, very low for a survey on troll fishing. So it only confirms these data set isn't suited to make a community analysis. So the third output is actually my favorite. It will give you more keys to evaluate your model and give an indicative rate to your model. So here we have a global rate of 3.5 out of 10, which is pretty bad actually. But we have to see the details on every analysis. So I'm just going to zoom out a bit so we can have it on a same line. I hope you're still seeing. So if we look on the details on every analysis because the global rate is quite an indicative rate so not to be trusted too much, not to make real conclusions from. The best rate we've got is four out of eight on the evo a analysis. So this model has a complete plan. So each criteria is set here and defined just down there. And if you go to the tutorial, you'll just have details on the analysis rating file here in this frame and just under read and interpret road GLMMs output. So if you want to know more about all these criteria, just go there. So here we have a complete plan, which is very good, but it's not balanced. So a bit problematic. It has a few not not attributed values, which is good. But it seems dispersion and dispersion and uniformity of the drills isn't okay. So these two criterias, if they are not checked are pretty bad. Actually, it means that the Poisson distribution doesn't fit our data. So it's really not good for the model itself. Then in the last part of this file, we have rect flags and advices so major red flags you have in your analysis. And you can have some advices on how to make your analysis better. So, for example, you can try to ameliorate your model by trying another distribution law. But as we stated, the data wasn't proper to make a community analysis. So it probably won't help much in our case. So these advices are not to take and not always good. Just a little help if you need one, but it's not solving data problems actually. So we can still try to visualize our results even if these won't probably be trustworthy. It can be interesting to see what does a plot of a bad model looks like. So let's go back to our tutorial. Just go down there. There are all the details you need on the how to interpret every single part of these three output files. I won't do it here. It's too long.  I won't do it here, but just look into it. It can be also useful if you want to just interpret statistical models out of these tools, even if you do it on R and everything. It can be really useful, I think. So let's create the plots with the creator plot from GLM data tool. So the input is the GLM results file. So note not the transposed one. Just be careful about that. Then the metrics data table used for the GLM. So here is a really handy way to remember which metrics table you used. You can see in the GLM file name, you have the data you used to perform your analysis. So the 32 is the unit ops. So here, the metrics you'll need, the metrics data is the 23 and 32 for the unit ops table. So really a handy way to make sure you pair the right GLM with the right metrics and unit ops table. So let's execute and see what it looks like. So here it creates data collection containing four PNG files. So first I'll zoom out because it's in the high definition. So here we have the representation of the global analysis. So as I said, it's not relevant to look into these results as it gets data from several communities together. So it's not relevant. Let's look at the other ones. So the SWC IBTS isn't really interesting as well. So we see every lines is pretty flat. We have very big confidence intervals. So to give you more keys here is in blue representing the species richness variation estimated by the model through time. So not much variation from one year to the other. If any value would be significantly different from the first year of the temporal series, it would appear right here. Obviously it doesn't appear white. Then the lower plots represents the mean species richness for each year without the modernization and we still observe not much variation. You can get more detailed interpretation of this representation in the return tutorial, but I won't talk more about it here as we already stated these analyzes are of poor quality. So I'm going to zoom back too much. Okay. So here we can see we had the same results when before so we did the same thing. Here under the plots you can have a detailed interpretation I just talked to you about and we can go back to the population metrics. Maybe we'll get luck here with the population analysis this time. So we can go directly to the compute GLM on population data. Same as before the input just we have to do the visualization on several metrics file as we have several surveys. So here are the three files we need. Here there are bits SWC IBTS and EPO. And then the unit ops file is the same as before with the community analysis of course. So for the response via from the metrics file will choose just go back there will choose the abundance so the fourth column number of the species. So just as before we set year and sites as explanatory variables and sites with a random effect and same we won't specify any advanced parameters. So let's execute these models right away. And now we have a lot of outputs three per survey so nine in total. So we'll get the same outputs as for the community analysis. So you can go through it and try to analyze this as we just did for the community analysis. It will be computed for each species individually. So I'll just go through the SWC IBTS analysis in this video tutorial. So you can check for the interpretation on the by BITS analyzes in the written tutorial. And for the evoy analysis I'll let you go through it if you want and try to make your own interpretation of it. So here we have all our analysis so to know which one is which survey just refresh your history simply. And here you have all the tags you need to see. So as I find the plots easier to interpret and almost as complete as the row results, I'll do the interpretation directly on the plot. So I just open the create the plot a plot from GM data to again open multiple datasets of course what GMM results as well as for the metrics data table. It's in the same orders or it won't be a problem at all and then the unit ops entry just execute. So while I wait for the plots to be produced, I just will take a look at the rating file of the SWC IBTS analysis to see if there is anything I should preferably look into. So we can see all single species analysis has the same rate. So four out of eight, which is a medium rate. But of course we have to look into the criterias and we see there is the same criterias validated or not each time. So we can just do a meta analysis of all our analysis are all our single species analysis here. So we can see the sampling plan is never complete or balance residuals aren't uniform. However, dispersion is regular so it's hard to make conclusion on the distribution, but it doesn't seem to fit very well. Let's see how those the represent graphical representation looks like now. It's easier to know. So I'll have to zoom out again. So here we see the representations are the same as the communities. So we'll just look into the if there is any significant analysis. So here we see, as we don't have stars, significant stars on the global trains that there isn't a significant effect of time on the abundance of Gadus Moria. Same here we don't have significant effects. No significant effect of time. Again, there's no no significant effects. And finally here we have a little star, and we have a significant effect for us Chrome birth Chrome Bruce, which is simply the Atlantic mackerel. So it shows a significant augmentation according to the model. However, when we look further into the craft, we see there is a pretty doubtful augmentation of more than a million percent individuals from 1985 to 2010, which is pretty doubtful actually. We see there is quite an outstanding augmentation in row balances between 2006 and 2007 2004 and 2007 sorry, before coming back to average values. So it seems there are there has been some bias introduced or something. Let's take a look to the other ones if we see the same thing. So with the there is also an event like that with Gadus Moria between 2006 and 2010. Here we don't see such an event. It seems there is a problem also with this species. And what about the last one? No, we don't see it. So there are three species that seems to have issues and strange row abundances take off really. This kind of patterns often appears in bias data sets with a shift in sampling methods, for example, or mistakes while containing the individuals or typing the data. So maybe we'll get better results if we cut the data from 2004 to 2010. So you can try it if you want to after this tutorial. We've already seen tools to do so. So you can do it. I'm sure. And I hope the models will be better. So for the three other species we observed variations are smoother. The three variations are just seems okay, even if the model isn't significant. But we can observe some strange behavior sometimes in the plots as well. Like here we have a major augmentation then a major going back. There seems these two points are quite. So maybe there was mistakes from 1992 to 1993. We don't know about it. It happens a lot in this kind of long term surveys. So that's why you have to get to know your data before making any conclusions. So because many ships can have occurred in material methodology, people working on the project, the way we count, many things can happen. So make sure you take all these modifications into account when you do such analysis on data frame that you don't know a lot about. So here it is that we just did this tutorial. I'm just going to zoom back because you must not see a lot. Yes, we did everything we had to do today on ecological analysis. So it has been made on both community and population levels. So now you know how to preprocess abundance data on Galaxy, compute ecological metrics, construct a generalized linear mixed model and interpret it. So it's not that easy to get immediately. So I hope I didn't bore you too much with technicalities and that you enjoy this training with me. Don't hesitate to ask questions on the events chat or by email at this address. So my professional address, colin.royo at mnhn.fr. So if you have any suggestions also on how to make this training better, do not hesitate to contact me as well. And also don't hesitate to use the feedback form that is at the end of the training to give me your feedback. Thank you very much for watching.