All right. Good afternoon, everyone. This is Michael at the Nebraska Library Commission with my cohorts, Krista and Laura here. And we are into the next hour of Big Top from Small Libraries 2014. We have been recording all the sessions. We are continuing to record for the rest of the afternoon. And we will be posting all those recordings and the PowerPoints next week after the conference. The next session we have up here is how to make money on the Internet for your library. So we've been getting donations. We've been buying used books and now we're going to make some money ourselves. And speaking on this topic is Mary Raimi, the director of the Pioneer Memorial Public Library in West Virginia. She has worked at the Baltimore Museum of Art Library, the Milton S. Eisenhower Library at Johns Hopkins University. It is currently the director of the Pioneer Memorial Library in Harmon, West Virginia. Mary, go ahead and take it away. Thank you so much, Michael. I am so excited to be here today. I have been energized and inspired by all of the presentations today. And I just want to give a hello and a shout out to all my fellow librarians because, obviously, we are not in this for the money. We are in this for the love of libraries and lifelong learning. The photograph is my cute little library. The Pioneer Memorial Library is a small library in the mountains of West Virginia. We are wild and wonderful. And we are in the middle of the Manangihila National Forest. It is a beautiful national forest. It's about a million acres of rural playgrounds that we have here. Our library population is about 2,000, maybe a little over that. We are open 30 hours a week. And our entire yearly budget is about $35,000 per year, including me. There are two employees at our library. And I am a part-time director. I work 30 hours a week. And our library is closed on Fridays.  I am at my house today feeding the wood stove and listening to the snow melt. So I started as the director of the Pioneer Library about a year and a half ago. And I brought some skills to the table that I have been sort of honing and working on for the last 10 years. And the title of my presentation today is How to Make Money for Your Library on the Internet. And I have a little help from Clint Eastwood. And the theme here is for a few dollars more, because honestly our entire budget for buying books, CDs and DVDs for the year is $2,000, which is not a lot. So my first goal was to figure out how I could raise more money from my library without having a traditional book sale or a cakewalk or a fundraiser. And this is how I did it. This is my story. Okay. Does your library need money? If not, this presentation isn't for you. But I'm thinking your library needs the money. The state of West Virginia has a bill up in the legislation to cut our aid money by about 10% this year. And that would be about $1,000 for us. And that $1,000 is a chunk of change that we need. Libraries are facing cuts all over the country because our economy is not so great right now. What we're going to talk about today are ways that you can set up on the Internet so that you can have money coming into your library potentially from all over the country. Really doesn't take a lot of time. And with a minimal amount of investment, you can set up these continuous streams of money that come into your library without you really having to do a lot of work. PayPal is one of my favorite places to go. And I know a lot of people here are concerned about the security of PayPal. But if you have bought anything on eBay, you have used PayPal. eBay owns PayPal and PayPal currently earns about $5.6 billion per year as of 2012. Everything is encrypted and very secure via PayPal. Once your information reaches PayPal, it resides on a server that is heavily guarded both physically and electronically. So it's really safe. And how I suggest that you use PayPal is by putting a PayPal donation button on your website. Now, let's just go, there's the donation button in the lower left-hand corner. And when you join PayPal, they give you this little bit of HTML that you can place on your webpage. It's really super easy. Our website is set up so that it's a WYSIWYG. What you see is what you get. So you really don't have to know a whole lot of HTML. You don't have to be a professional web designer. I am the person that alters our website. And I put a donation button right on the front of our library website. Let's see if I can go there and show you what it looks like. If y'all can see this, it's down very small actually at the bottom of my page. But anyone from all over the world can click on that little button. They can use their credit card. They can make a donation to your library. That money goes right into your bank account. And the person who donated it actually pays the fee. So there's really nothing that you have to do. It's kind of like a solicitation for donations that is always on your website. And it works for you. Now, we also use Kroger to earn income for our library. Kroger, according to their website, is the largest grocery chain in the country. And it is the only grocery store that we have, really. It's 23 miles from Harmon, West Virginia. It's where we do all our shopping. Now, Kroger has this thing, and it's called the Community Rewards Fundraising Program. They also call it Kroger Cares. There are 2,424 Kroger stores in the United States. And what you do is you set up this gift card program that you and your patrons can use to benefit your library. So how does it work? Basically, you go to the Kroger and you buy gift cards from the Kroger. You have them all preloaded with $5. Then you sell these gift cards to your patrons for $5. And every time they go to the Kroger, you load up the gift card with cash at the customer service desk.  And 5% of everything you purchase comes back to the library.  It could be your school. It could be your local community service. The good thing about the Kroger card is that you really don't have to do a lot. You promote the cards. You sell them for $5. You watch the cash roll in. Kroger sends you a physical check when you accumulate enough cash for them to cut a check. I think the cutoff is like two – but you have to accumulate $200. They send you a check. The negatives to the Kroger gift card is that patrons have to remember to go to the customer service desk to refill their cards. I know that sometimes when I go to the Kroger, it's all I can do to keep that list in my head – milk, bread, eggs, milk, bread, eggs. So it's hard for me to remember to stop at the customer service desk to refill my card with money. And also, sometimes you have to wait in line. Now, I also have started using Better World Books. When I first came to my library a year and a half ago, I would say that my collection had not been weeded in many, many years.  I weeded at least 2,000 materials from my library and, you know, we're a small library too. And I can't house 2,000 books for a book sale. And I certainly don't want to schlep around 2,000 books in boxes for a physical book sale somewhere else. So Better World Books is just a great option for me. One of the pluses of Better World Books is that it keeps books out of dumpsters. Before I had heard about Better World Books, I was taking books and, you know, literally throwing them in the dumpster. There was a story in Fairfax County, Virginia, and I'm just going to click on this link. It's a hyperlink. And it's a story from the Washington Post. The Fairfax County Library had revamped their system, and they had literally gotten rid of a quarter of a million books. And they threw them in dumpsters, which outraged the patrons, as it well should, because, I mean, honestly, as a small library where every penny counts to see that kind of waste is really kind of appalling on several different levels. So Better World Books is great because – let me just go back to my PowerPoint. Better World Books sells books for you on 50-plus internet outlets. You know, that's a lot. That's a lot of different places where they are selling books. And I'm sure that if there are librarians out there amongst you who buy used books, like I do also, that you have purchased books from Better World Books if you don't already use their servos. Better World Books is cool because your library gets 15% of what they sell, plus 5% gets donated to a charity. You get to choose the charity that you would like to donate that 5% to. Currently, Better World Books has eight literacy charities from which you get to pick, which is kind of nice. Even better Better World Books pays for your books to be shipped, so there is zero cost to you for Better World Books. They even send you the boxes. So what you're doing, the minuses are you have to pack up the books and call UPS for the pickup, and you have to provide the tape. And you don't get as much profit as if you sold books on Amazon yourself or on half.com. So 15% really isn't a lot, but then again you think the book is not a dumpster, it's getting into the hands of someone who really needs it, and you're also donating to another worthy literacy cause. Okay, the next resource that I use for bringing money into my library using the internet is my blog. I love to blog. I am a little compulsive about it, and I recommend that you start a blog for your library. I'm in library school right now, and they refer to this kind of behavior as library 2.0, and some people call it library 3.0. I don't want to put too many flashy tags on it. I just think it's the way to communicate via the internet right now. You can use your blog as a platform to reach out to people literally across the world, and to me that's the exciting part about blogging. The blog that I have is called Rural Librarian, which yes, it's very difficult to say, which makes me laugh. And I use Blogger. Now I have another hypertext link, and I'm just going to click on this and give you a peek at my blog. This is what it looks like. It was free to set up. I took the picture of the mountains in the background. I try and keep it updated that you can see. I am woefully out of date. I have not updated my blog since January. And the way that this blog makes money is, I don't know if you can see them, but there are ads on the right-hand side of the blog, and there's ads also here at the bottom of the blog. And these ads are provided through Google AdSense, and this is what they call monetizing your blog. It's really easy to do. It doesn't take a lot of time. And when patrons click on the ads, you get so much money from it through Google AdSense. And I can't say two clicks equals $2. There's no easy math on how you make money, but the more people click on your ads, the more money you make. So if you look down here at the bottom of the blog, this is a review of the book that I just finished reading called The Good Lord Bird. I recommend it. I really enjoyed it by James McBride. At the bottom of my review, I also included these little units that connect directly to Amazon. Now, how I have done this is by becoming an Amazon associate. And if someone clicks on any of these little links and buys either the hardback, the softback, or the audio book of The Good Lord Bird, I believe it's 4% comes back to the library. Okay, so now I'm going to go back to my slideshow. Hang tight. Okay, so we've talked about the blog. WordPress is also a great place to go. Now, another thing that you want to do with your blog is you want to link your blog around. You want to have your blog linked to your website. So if you go to my library website, which we did earlier, you will find a link on the left-hand side for the Rural Librarian blog. And let's see. I've also, when I write a blog entry, I include a link on my library's Facebook page. You can include it on Twitter or Instagram. And really, it's just using social media to promote your blog. And you never know who's going to enjoy what you've written, and they might share it with people. And you might get more page hits than you could ever imagine. The thing that I really like about blogs is it allows me to communicate with my patrons who are more internet savvy. Now, I will say that I do serve an older population. They're not necessarily a tech-savvy population. Like a lot of the people I've heard speak today, we don't have a reliable internet in all areas because we are located in rural Appalachia. We're about 2,500 feet up. There is one cable company. It's very expensive. And so, like I said, when I'm communicating in my blog, I'm probably not reaching a large portion of my patron population, but I am reaching some people. So for example, a lady came into the library about six months ago, and she started talking about my blog and greeted me like an old friend. It was as if she knew me because she had been reading my blog for about a year. She brought her young son into the library, and six months later, when we had a board member leave us, she is now one of our library board members. So this is kind of, I guess, a happy side effect of starting a blog. You never know who's going to read it. Now, Google AdSense is the way that I have monetized my blog. And these are all new words to you. I apologize, but honestly, it's really not hard. I love technology. My background is in graphic design. I love computers. I love software, but this is very easy stuff to learn, and it's very easy stuff to do.  You have to write maybe half a dozen blog entries, and they have to be relatively typo-free and grammar error-free. They want to see that you're reputable, and they will prove you as a Google AdSense blog. Now, Google automatically places the ads on my blog. I really don't have a lot of control over the ads that they place on my blog. However, you can edit the content of your ads. For example, we serve as a public library, but we also serve as the school library to next-door Harman School. It's a pre-K through 12th grade public school, and I chose not to have dating or alcohol ads on my AdSense blog because I just thought it didn't seem appropriate for my audience, for my patrons. You want to write your blog entries with a meaningful link or two embedded within your blog. You want to include your source materials, and you also want to have good SEO. SEO is search engine optimization, and it's really all about getting Google to rank your pages better. When you do a search on Google, and you're looking for a public library in West Virginia, if your page is highly rated by Google, your library is going to be at the top of the list when you do that search. It is a mysterious Google algorithm that they use to prioritize the page hits that you get in a search. It's a little mysterious. They give you hints about how to get your page better ranked, though, and it's through using these search engine optimization tools. It's all about the page hits. You really want to get people to read your blog. You want people to connect to your blog. You want people to relate to it. And when you learn how to write using good SEO, your blog will get more page hits. So I was going to show this cute little video about what is SEO, but I understand that the bandwidth might not be so great for all of y'all, so I'm not going to run it. Instead, I'm just going to talk a little bit about it. SEO, search engine optimization is really about considering how a person searches the Internet. So if someone was going to look for a review of a book, they might search, for example, the Good Lord Bird James McBride. They might search fiction about the Civil War. They might search historical fiction about John Brown. There are many phrases that people might use to search for your blog entry, and you want to include those phrases in your blog. You also want to update regularly. That's part of good SEO. So my blog has not been updated in over 30 days, and that's kind of an SEO no-no. It means that my page will be ranked lower. But it's about using more adjectives and using more words than you generally might. So instead of saying, like, I really enjoyed this book, you would say, I really enjoyed the Good Lord Bird by James McBride. You're including more descriptive words. So I showed you on my blog the Amazon associate links. You have to apply to Amazon to become an Amazon associate, and you can have those neat little, they supply you with the HTML to plug into your blog page. And it's really a cut-and-paste type of procedure on your computer to put those cute little blocks at the end of your blog entry. And every time somebody clicks on one of those links and buys a book, you get 4%. Now, the more recommendations, the more people who buy things from your Amazon associate blocks, the higher the percentage goes. I believe it goes up to 7%. Now you also want to develop backlinks to your blog. Backlinks are places that will link to your blog. So there are some awesome librarian sites out there. You might be familiar with Librarian and Black, the not-so-silent librarian. And I have written to many of these people and asked them to include a link to my blog on their site, and very kindly many of them have. I'm trying to encourage more libraries, especially in West Virginia, to include blogs so that we can have a page that unites all of the library blogs together. I think that would be very helpful and beneficial to everybody. You know, and like I said about finding the board member out of nowhere, the more people you bring to your blog, the more people literally will come to your library. Board yet? I hope not. Relying on Clint Eastwood for a little boredom breaking up here. And this is just a little bit more about the SEO. If you're writing a book review of Janet Ivanovich, which I've done on my blog, you want to include the phrases that people are going to search for, like American Mystery Writer, Funny Mysteries, Good B Treat. I have even found that putting typos and intentional misspellings can yield page hits. Now, this kind of contradicts the good SEO, but at the same time, you know, when I look at my Google Analytics, I do see that I am garnering page hits from typos. So occasionally I will do this and it's more for, I guess, my pleasure to see if people respond to that or not. Okay, now Google Analytics, I heard somebody talking about that earlier today in one of the lightning rounds. I'm just very happy that other people are using this as well. Google Analytics is not going to bring any money into your library, but it does inform your blog rating. And it's really important to know which of your blog entries are the most popular, which of the backlinks is providing you with the most page hits and links to your blog. It's a little scary, I will admit, Google Analytics is a little scary and intimidating because it's all numbers and graphs, and honestly I am not a number and graph person, I went to art school. However, if you look at these statistics long enough, they really are very informing and helpful for you as a writer and as a librarian. I was also, let's see if we can go to Google Analytics here, and I'm just going to show you what my page hits look like today. I have a couple of blogs, but this is my rural librarian blog. For some reason, I had disconnected from Google Analytics for a while. The code had been erased accidentally, I'm not sure how that happened or why, and I just replaced the code last week. But if you look here, there's all kinds of reports. There's an overview, I can see, let's see, there's my little graph. You can see if someone's a new visitor, if someone's a returning visitor. You can see that the average, this is probably just the last few days, the average visit duration, 35 seconds, that's actually not so great. I have a bounce rate of 76.92%. That means that three-quarters of the people who look at a page are like, that's not what I'm looking for, and they go somewhere else. That's okay too.  Let's look at the acquisition overview. You can see that some people found my blog by doing an organic search. That means it's just a Google, probably a Google search. Some of them found me directly, maybe they have a link. Some of them found me through social media, and some of them were referred to my Rural Librarian blog. I love looking at my Google analytics, sometimes I obsess about it a little too much. Let's see if we can go back to Clint here. The other way that I make money from my library, and I have to confess, this is my favorite way to make money from my library, is by selling books on Amazon. One of the gentlemen who spoke early on about buying used books online, I was like applauding him in my living room because I do the same thing. With my very tiny $2,000 book budget, I buy books used on Amazon. I go to my local Goodwill and buy books. But then I also sell books online. And the books that I sell are typically books that are donated that I don't want to add to the collection. Or they are books that are weeded out of my collection, and they're still in good condition. Like I said, I had a $2,000 book weeding job, and I had the time to really go through them and figure out which books had resale value and which books don't. Okay, now how to get started with selling books on Amazon. First of all, you need someone who's going to commit, who's going to say, yep, I'm going to try this. I'm going to do this. You want to invest in bulk bubble mailers. I typically go through a company called ULINE.com. You can buy them 100 at a time. You can buy 500 at a time. What you don't want to do is go to your little dollar store or post office and buy them one or two at a time. You really want to buy them in bulk so that you are making the most profit possible. There are some seller obligations on Amazon. You want to make sure that you mail your sales quickly. Within one or two days, you want to make sure that you mail them via media mail. And if you're selling a book that's worth over my limit is $40 or so, you want to get delivery confirmation so that the person doesn't say, hey, I never got that $40 book, now refund me because that hurts me. It hurts my library. Now I also tweak my Amazon inventory once a week or once a month and typically I tweak my inventory on Friday.  If you're the lowest price and then they show you, if you're not the lowest price, they show you what the lowest price is. And typically what I do is if I am not the lowest price on a particular item or book, I will, I mean I hate to say this word, but I will lower my cost. I will undercut the highest bookseller by about $0.10 so that when someone is looking for that book and they go to buy it, they choose my book because it's $0.10 cheaper. And I know that sounds crazy, but honestly that $0.10 makes a difference. Now the plus is about selling books on Amazon. You will get reimbursed $4 for shipping, which is great. However, you have to put the money out upfront to ship that book. So it's, you know, it's this leap of faith that you make in cyberspace. And I think it's twice a month Amazon deposits our profits right into our bank account. So it's not like I receive a check and then I have to deposit it. It just magically appears in our bank account and I, you know, it really is magical to me. Let's see. Let's go on to the next one. The downsides of selling on Amazon. Amazon takes a big cut. They are going to take between 20 and 30% of your profit. They are not, Amazon is not very transparent in how much they take out of your profit. I resent that. However, you know, I'm able to make money from my library from the middle of the mountains in West Virginia. And I appreciate that and I appreciate Amazon for that. Customers may be unaware that they're not buying directly from Amazon. When you buy a book on Amazon, I think a lot of people think, oh, it's coming from this giant warehouse in Amazon land being shipped to my house. Well, that's not the case. You know, there are a lot of what are called resellers like me. I am a book reseller who sell on Amazon. So a quick story, I sold a book to someone and got a call about 7 o'clock in the morning. I had put my first and last name on the return address and they called me to tell me they were not satisfied with the book they had received. And then I had to explain to them, hey, I'm not Amazon. I am a, you know, a person, a real person who just sold this to you. Sorry, you're not happy with the book. And I learned a valuable lesson to never put my, if I personally was selling a book, never to put my real name on the return address of a book. The other minus is you have to go to the post office regularly. Now for me, that's not a big deal. Honestly, our post office a lot like Rachel's post office is right around the corner. It's less than a quarter mile from our library. So it's really not that far.  A book inventory can take up a lot of space and it has to be managed well. I have about, I would say a hundred items on sale right now on Amazon. And it's taking up maybe three shelves in my, in the back room of my library. You know, and that's, I guess that shelf space that could be going for other needs. But right now for me, this is okay. I have enough space that this is really not that big a deal. Now there are other selling venues online. I mean, if Better World books can sell to over 50 venues online, there's got to be just a ton of them. I have also sold on half.com, which is a fun place to sell too. And the cool thing about half.com is that they take a much smaller cut than Amazon. So you're going to make more profit here. But the catch is that more people use Amazon than use half.com. So you're more likely to sell on Amazon than you are on half.com. Now, you can also list the same books on both Amazon and half.com. And I have done this before, but you have to be really vigilant about deleting that book from either half or Amazon once you've sold a book because I have also made the mistake of not deleting inventory and then accidentally sold it twice, in which case you have to write a very apologetic letter to the person who bought it saying, hey, really sorry, my bad, here's your refund. Okay, now I've sold books online for 10 years. I sometimes refer to myself as like a book bottom feeder. But there is some skill to this. Most books have little to no resell value. And it can take, like I've said here, it can take years to hone your book buying and book selling skills. But I have a few tips here such as nonfiction generally sells better than fiction. Bestsellers, like if you're going to try and sell Stephen King online, you're going to see on Amazon there's a million copies for a penny. So bestsellers usually have little or no resell value unless it's a signed first edition. It's very different. Small press and academic presses, books, they tend to sell better because they have smaller press runs. Therefore, there's fewer of them. There's a million Stephen King books and Daniel Steele books out there. But there may only be a few thousand of small nonfiction type books. That's why they sell better. There's just fewer of them. Textbooks, I generally try not to buy or sell unless they are very current within the year or two. Textbooks that are older than a year or two immediately lose their resell value. It's one of the ways that textbook publishers make a lot of money is by updating those textbooks every year so that you have to buy a new one. Another little tip for you if you're selling on Amazon is don't sell large heavy art books. You lose money in the shipping. When you take a book to the post office and the post person puts it on the scale, they're weighing that book and you're paying per pound. So if you have a very heavy book, it doesn't matter to Amazon. They're still only going to give you that $4 in shipping. So people ask me all the time, how can you sell a book on Amazon for a penny and make money? Well, it's in the shipping. If you have a teeny tiny book and you can ship it for $2, you have not just made a penny, you've made $2 and a penny. Now, it doesn't sound like a lot, but if you have a book inventory of 10,000 books and you're selling literally hundreds of books a week, that money really adds up. So my next tip is don't sell books that are marked up or in questionable condition. I also smell books when I'm evaluating them because some people are very sensitive to books that are kept in, say, the households of people who smoke. Some people who buy those books will immediately return them saying, you know, I have allergies and I'm returning this book. And it also causes me personal pain as a librarian when someone writes in a book because it immediately diminishes the resale value. So I'm kind of coming to the end of my presentation and I'm sure that you're all asking, well, how much money did you make, Mary, in one year at the Pioneer Memorial Library? Amazon's selling use books last year I made $1,000, which is going to make up for the money that we lose from the West Virginia state government. From PayPal, I have made nothing. But this is what we call in the marketing world a long tail investment. That PayPal button didn't cost me a dime. It's going to sit there on the front of my website indefinitely. And someday someone's going to see that little PayPal button. They're going to click it. They're going to make a donation. And I'm going to be doing the happy dance on the circulation desk in my library. Okay, maybe not. But I will be very happy. Kroger, I have maybe two or three patrons at the most who are using this program, myself included. And last year we made $500. I mean, out of the blue, Kroger sends you a check for, you know, $200 a year in there throughout the year. That's awesome. It makes me very happy. Now, my Google AdSense blog, again, this is a long tail investment. Last year I made $8. But I also am using this as public relations. I'm using this as a way of communicating with my patrons. And this is, when you blog, there's a cumulative effect. Next year I hope to double the money that I made this year. Amazon, as an Amazon associate, I'm very sad to say I have made zero income. But again, those reviews are going to sit on my blog forever and ever. Someday someone's going to click on those little links. They're going to buy a book. I'm going to be very happy. And, you know, I'm also sort of educating my patrons about the books we have available and what books are about. Okay. Now, this is something I have not done before, but I am planning to do later this year. CafePress.com is a great place to create items for sale. You create the merchandise. Now, for me, I have to say I have a little advantage because I used to be a graphic designer and I can design things in Photoshop. For me, it's not so hard. Maybe you have a patron, however, who uses Photoshop, who could set this up for you. People buy library-themed or region-themed merchandise directly from CafePress.com online. And as they are ordered, CafePress creates the merchandise and mails them. So these are like virtual products that you never see that exist in the world of CafePress.com. And, you know, magically, money appears in your bank account. Your library gets a chunk of change. You don't need space for the inventory, and CafePress does all of this for you. Okay. This is my last word here. There are many other ways, I'm sure, that you all out there are using to make money for your library on the Internet. You know, be creative with the ways that you use to promote your library and make money all at the same time. And Clint, of course, has his closing message there. And I want to – I have a last page of website references. If you want to connect to Better World Books or Kroger Cares, you want to buy Bubble Mailers and Balk, if you want to read more about selling on Amazon, I have links to all of that stuff here. You are also free to email me at my cute little library. I love helping people learn how to set up these revenue programs for their library.  So, you know, I would say at this point, it's like 3.45 my time, do you all have any questions? Are there any questions or comments out there that I can address? Yes. Thanks, Mary. We do. We've gotten some questions and comments, and I'm going to kind of take a point of moderator privilege and ask the first one myself. Anybody there? Did you have to work or run any of these blogs? Can you hear me? Hello? Uh-oh. Mary, can you hear me? I can hear me. I just can't hear anybody else. Oh, this is weird. Hold on. Can you hear me now? People on the line have said participants can't hear. Okay. Unfortunately, however, she can't hear us. Mary, you cannot hear me at the moment? Well, now Laura Murray says we can hear you both. What? But if she can't hear me, she can't hear the question. Sorry about that, guys. I must be lost in somewhere. Mary, can you hear me now? Yeah, I still can't hear anyone. Okay. I don't know how we're going to ask a question. This is funny. The audience can hear both of us, but we can't hear. And I can hear her. She cannot hear me. Oh, there you are. Oh, can you hear me now? Yes, I can. Sorry about that. Okay, great. It was weird. Everybody could hear each other except you can hear us. I don't know what's going on. I'll start back over. Thank you. Very, very practical advice. And as a point of moderator privilege, I'm going to ask the first question myself, which is, in setting any of this up, did you have to run this by or seek permission from anyone from your municipality as this being kind of an external revenue stream for the library? Quick answer? No. That's one of the beautiful things about a small library is that there's not a lot of bureaucracy. Okay, great. Yeah. Yeah, I've just, I've heard of libraries who said, you know, we went to our city to ask if we could do this. So I just wanted to kind of check on that. One of our viewers does observe that forgiveness is much easier than permission. Oh, I'm all for that too. I just, I don't know why anyone would object to raising money this way. Well, yeah, there might be legal reasons. I, you know, I don't know what, what the actual reasons were and the indications that I heard of. But anyways, so questions from the audience comments. Gosh, we have a lot. Okay. Someone says, do you have someone help with this? It seems time consuming. I don't have anybody who helps me with this. I do this all by myself. And it is, it can be time consuming, but once it's all set up, it's almost like, you know, a little perpetual motion machine. There's really not, it just takes a little bit of work a week. And honestly, I probably spend about an hour a week on all this stuff. That's great. Someone else does observe that this is the sort of thing that friends might do. Yes, absolutely. Yeah, I wish I had friends and volunteers who would help me with this. It would be awesome. I would truly love that. And these are the kind of tasks that you can give to the volunteers in your library. I mean, I imagine that this kind of revenue earning model could be used in larger libraries to great effect. Okay. Not a question, but a comment. You might have more success for your PayPal button if it were at the top of your homepage with an explanation of how a donation can benefit the library. Yeah, you're absolutely right about that. That was kind of thrown on there when I first came on as director, but you are absolutely right. It should be at the top of the page. It should be bigger and there should be an explanation of how it works and where the money goes. Let me see. Someone observes that Zazzle is similar to Cafe Press. Yes. Yeah, Red Mubble as well. Someone says that you are amazing and you're very inspiring. Thank you. Well, you know, someone asked, have you had any feedback from your community regarding your participation in third-party advertising? I have not. I have not had anyone come in. In fact, I never even thought that there would be ethical concerns about it until one of my instructors raised that as a potential issue. You know, should public libraries be advertising? And my feeling is, yes, I think we should be advertising. I think that in some ways we need to be using the corporate model in raising money for our libraries. You know, we're not just a nonprofit. We're providing an incredibly valuable service to our communities. Okay. Well, let me see. Someone else is asking, do Amazon or half.com charge just to list the inventory? No. Okay. And they'd like to know what size mailers you use. You know, I'm not really sure. I don't have like the measurements in front of me. I tend to buy one size that would fit most larger size hardbound books. That way, if it's a little paperback, I can always put it in there and double it over. And that way, I don't have to buy an assortment of sizes. I would say, you know, just do a little research and figure out what size an average hardbound book is and go for a larger size. And figure out what size an average hardbound book is and go for a bubble mailer that will fit. I think from my experience, that's a number 10, but don't hold it too. Don't hold me to it. Yes. Something like that. Yeah. I have a hard time keeping numbers in my head. Someone has asked if you've ever written an article for American libraries. And they think consider writing an article about this. That's awesome. I would love to write an article. Yeah. Exactly. Yes. In my spare time. I did have one comment on Twitter. Susie says, I have a blog and a personal blog both monetized, but no one ever clicks in the links. I've never made anything off them. Yes, but you won't if you don't put the links in the first place. That's right. Yeah. And just keep going. You know, keep blogging. I find it very cathartic.  I'm in the middle of, you know, a forest. I have very few colleagues. So this is kind of the way that I keep myself going. Somebody has asked if there are any other measures outside of Google analytics to tell you how well all this is working. Well, yeah, I mean, for example, I know what my most successful articles are. My most successful articles are the ones that I write about the history of West Virginia. And that really helps to inform what I choose to write about. West Virginians are very into their own history. They're into genealogy. It's a very interesting history as well. And our library is a depository of a lot of genealogy information. And so it really helps to inform who my long distance patrons are. There are a lot of people who have moved from West Virginia to get jobs, because honestly, we're in the middle of Appalachia. There's not a lot of jobs here. So I am actually reaching people who are former West Virginians, but who miss being here or who have a family connection. And looking at Google analytics and seeing my number one blog article is John Alexander Williams, the history of West Virginia. That says to me that I need to be writing more of those kinds of articles. Ah, yes. Someone observes that when they sell books, when the library sells either donated or weeded books, sometimes the city, the money will go into the city coffers instead of the library coffers. And I can answer that one. That's why you have a foundation or a friends group. If you have it done that way, the money doesn't go back to the city. It goes into your friends or foundations coffers for the library's use. Yeah, our libraries are not set up that way. We are a quasi-governmental organization and we're very independent. I mean, we receive funding from the state, but we receive no funding from our local municipality, the city of Harmon. We don't receive any funding from your county. We do receive funding from our county commission. Yes. Well, yes, someone else has observed that the friends can bypass the municipal funding restrictions. So you do have to think about that. And that is one really good reason to have a foundation and a friends group. It is not a dodge. It is making the sure that you're getting things are going the way they need to go. Right. Yes. If you have time, maybe for one more question if there is one. Sure. I think we've kind of, I've tried to sort of, well, I've sort of tried to combine a few of them here because there are a lot of questions. I think you've really opened a lot of people's lives today. Well, thank you Mary. That was a wonderful presentation. So lots of practical advice. And as everybody can see on the screen, there's Mary's contact information. So by the fact that she's put it on the screen, I'm going to assume that means it's okay to contact her. And every now go click on her blog. Right. Yes.  Thank you very once again. And that's going to wrap up this session. This presentation of Big Top from Small Libraries.


Live from Boston, Massachusetts, it's theCUBE. Covering IBM Chief Data Officer Summit, brought to you by IBM. We are wrapping up theCUBE's coverage of the IBM CDO Strategy Summit here in Boston, Massachusetts. I'm your host, Rebecca Knight, along with Dave Vellante. It's been a great day here in Boston at the CDO Strategy Summit. Yeah, I mean, I like these events. They're packed with content, very intimate. You know, not a lot of vendor push. Well, one vendor, I guess, is pushing. But I like the way, you know, we're talking to Chris Penn about earned media, and owned media, and paid media. This is all media. It's really the quality of the content that differentiates those media. And IBM always has really solid content here. A lot of practitioners, a lot of, not so much how-to, but hands-on stories, use cases, maturity models, things of that nature. And I think we are seeing the maturity of the CDO role from a back-office function to one that's sort of morphed into or evolved into data quality and part of the whole data warehouses, King, you know, push, and that meant a lot of reporting, a lot of compliance, a lot of governance, to one that is really supporting a monetization mission of the business. And when you think about monetization at the simplest level, there's two ways to get there.  And now, you should be careful. Not all of these companies are for-profit firms, but in a commercial sense, those are really the two levers that you can push in a lot of forms. Productivity, time to market, time to value, quality, things of that nature. But at the end of the day, it comes down to spending less, making more. Right, exactly. And I think you made a great point and that data was the back-office. It was sort of something we had to worry about, manage a bit, but now it's really front and center in the organization and thinking about using it to make money and to save money. And I think that that's what we're learning about too. And what I've appreciated is how candid IBM is being, frankly, about mistakes that it has made and it's saying, this is a blueprint because we've learned. We've learned where we went wrong and here's what we have to offer other companies to learn from us. Well, it's interesting too, if you take my little simple model of how to get value out of data, from IBM's standpoint, it's really a lot of opportunities to cut costs in a huge organization, 300,000 employees. So we heard from Jim Cavanaugh and Inderpal Bhandari today how they're applying a lot of their data-driven expertise to not only capture that data, but understand how they can become more efficient. We haven't seen the growth. The idea is, you know, everybody talks about the string of quarterly declines in terms of revenue. The good news is the pace of that decline has slowed. That's sort of the best you can say about IBM's top line. But the bottom line seems to be working. And IBM's such a huge machine that you can actually squeeze a lot of cash flow by saving some money. And there are a lot of stories about, you know, IBM in the supply chain and making that more efficient, which as we heard was a main focus of a lot of the CFOs, you know, or CXOs out there.  You know, it's going to be interesting to see, you know, if IBM really will be perceived as a data-driven company, you know, they're pushing cognitive, you know, there's a lot of blowback about Watson and how its very service is led. Having said that, IBM's trying to do things that Google and Facebook and Amazon aren't trying to do. IBM's trying to solve cancer, for example.  So, you know, got to give props to IBM for that effort. The social innovation piece, I think. It's really, it's part of this company's DNA.  Honestly, I'm not really qualified to address that question, but, you know, IBM tends to take capital and pour it into long-term, you know, businesses and eventually gets there. So it's not there yet. And so, but if IBM can use the data to become a more efficient company, be more responsive to its customers, understand the needs of its clients better, that's going to yield results. And I think the other part that we've heard a lot about today is the cultural transformation that's needed to make these dramatic changes in your business. As you said, IBM is a huge company. Hundreds of thousands of employees dispersed across the globe. So teams working across time zones, across cultures, across languages, that is difficult to really say, no, this is where we're going. This is our blueprint for success. Everyone come on board. Well, and you've seen some real cultural, you know, shake-ups inside of IBM. I mean, I was mentioning just a very small example.  This is a corporate executive office. It's an open area with open cubicles. They're nice cubes. Believe me, the cubes are nicer than your office, I guarantee it. But they're open. You know, you can see executives, you can talk to executives in an open way. That's not how IBM used to be. It was very closed off and compartmental. Or everyone was working from home. I mean, that's frankly. Well, and that's the other piece of it, right? They said, hey, guys, you know, time to create the beehive effect. And that's created a lot of, you know, dislocation, a lot of, you know, concerns and blowback. But personally, I like that approach. If you're trying to foster collaboration, you know, nothing beats face-to-face contact. That's why we still have events. And that's why the cube comes to these events, right? No, you're absolutely right. A growing body of research has really pointed to the value and the benefit of an open office to spur collaboration, spur creativity, to get colleagues really working in understanding the rhythms of each other's interpersonal lives and work lives. And really that's where the good ideas come from. Yeah, so, I mean, those decisions are tough ones for organizations to make. But I'm presuming that IBM had some data related to this. I hope they did and made that decision. And, you know, I mean, it's way too early to tell if that was the right or the wrong move. Again, I tend to lean toward the beehive approach as a positive, you know, potential outcome. Right, exactly. So the other piece that we've heard a little bit about today is this talent shortage, this skill shortage, because you made this great point when we were talking to Chris Penn of shift communications. So much of all of this stuff is now math and science. And that's not what you typically think of as someone who's in marketing, for example. We have a real shortage of people who know data science and analytics. And that's a big problem that a lot of these companies are facing and trying to deal with some more successfully than others. Yeah, I mean, I think that the industry is going to address that problem. Because all this deep learning stuff and this machine learning and AI, it is largely math. And it's math that's known.  Right. Okay, but these are concepts in math and algorithms that have been proven over time. And so I guess my point is, I think organizations are going to bring people in with strong math and computer skills and people who like data and can hack data and say, okay, you're a data scientist and I'll figure it out. And over time, I think they will figure it out. They'll train people. The hard part about that is not necessarily the math. If you're good at math, you're good at math. It's applying that math to help your organization understand A, how to monetize data. B, how to have data that's trusted. We heard that a lot. So the quality of the data. C, who gets access to that data. How do you secure and protect that data? What are the sort of policies around that data? And then in parallel, how do you form relationships with the line of business? So you got geeks talking to wallets. Right. Like how do you deal with that? You need the intermediary who can speak both languages. And then ultimately the answer to that, I think is in skill sets and evolving those skill sets. So those are sort of the sort of five things that the chief data officer has to think about. Three are in parallel or three are in sequential and two are in parallel. Well you also mentioned the trust in the data. And you were talking about it from an internal standpoint of colleagues agreeing, all right, this is what the data is telling us, this is clearly the direction we go in. But then there's the trust on the other side too, which is the trust that the company has with customers and clients to feel okay about using our data, using my data to make decisions. Well I think it's a great point. It was interesting to hear Chris Penn's response to that. He was basically saying, well, we could switch suits, but it's not going to have the same impact. I'm not buying it. I'm really going to keep pushing on this issue because while I agree that IBM doesn't have the same proclivity to take data and push ads in front of your face, it's unclear to me how you train models and somehow those models don't seep out. Now, IBM has said, we heard some IBM executives say, no, they're the customer's models. But you know how ideas get in people's heads and things happen. And that's just one example. There are many, many other examples. So think about internet of things and the factory floor and you've got some widget on the floor that's capturing data. And that widget manufacturer wants to use data for predictive analytics, for predictive failures, sending data back home. And then who knows what other insights they're going to gather from that data. Whose data is that? Is that data owned by the widget manufacturer? Is that data owned by the factory? It's their process. It's their workflow. Now, of course, if I'm the factory owner, I'm going to say it's my data. If I'm the widget manufacturer, I'm going to say, well, that's my data. And you're both right. I mean, I think that's the problem here. So there's no real arbiter to make that determination. Yeah, and I don't think these things have been challenged in court and certainly not adequately. So there's a lot of learnings that are going to occur over the next decade and we'll watch that evolution. But Jim Cavanaugh's right. We are at a real seminal moment here for this explosion in data, which is really changing the role of the CDO and how it fits in with the rest of the organization. Yeah, and I think the other thing to watch is how everybody talks about data-driven organizations, digital businesses, cognitive businesses. What are those? Those are kind of buzzwords, but what do they mean? What they mean, in our view, is how well you leverage data to create competitive advantage. And that's what a digital business does. It uses data differentially to retain customers, or attract and retain customers. And so that's what a digital business is. That's what a cognitive business is. Most businesses really aren't digital businesses today or cognitive business today. They're really few and far between. So a lot of work has to be done before we reach that vision. Yeah, but it throws out the Ubers and the Airbnb. Those are sort of easy examples, but when you have giant logistic systems and supply chains and ERP systems and HR systems with all this stovepipe data, becoming a digital business ain't so easy. No, no, and we are really at early days, exactly. So that's something to discuss next CDO strategy summit. And I think there was a lot of discussion early on when the CDO role emerged that they're essentially going to replace the CIO. I don't see it that way. There's a lot of discussion about what's the growth path for the CIO? Is it technology or is it business? But I think the CIO is okay. I think the CDO, I think actually there's more overlap between the chief digital officer and the chief data officer because if you buy the argument that digital equals data, then the chief data officer and the chief digital officer kind of one in the same. So that to me is a more interesting dynamic than the CIO versus the CDO. I don't see those two roles as highly overlapping and full of friction. I really see that chief digital officer and the chief data officer should be more aligned and maybe even be the same role. And it gets back to the organizational politics that are involved with all of these massive changes taking place. Well, again, the starting point for a CDO in a for-profit company is how can we use data to create value and monetize that value? Not necessarily sell the data, but how does data contribute to our value creation as a company? So with that as the starting point, that leads to, okay, well, if you're going to be data-driven, then you better have measurements. You better have a system. I mean, do you use enterprise value? Do you use simple ROI? Do you use an IRR calculation? Do you use a more sophisticated options-based calculation? I mean, how do you measure value and how do you determine capital allocation as a function of those value measurements? The vast majority of the companies out there certainly can't answer that across the board. They might be able to, you know, the CFO's office might be able to answer some of that, but deep down in the line of business, in the field where decisions are being made, are they really data-driven? They're just starting. I mean, this is first, second inning. Right, right, right, right. So there's much more to come. Great. Well, you have watched the cubes coverage of the IBM CDO Summit. Thanks for tuning in for Rebecca Knight and Dave Vellante. We'll see you next time.


You need a financial advisor with the freedom to focus on your specific needs. Whatever your goals, Reeve Potter can create a game plan tailored to you. Call Reeve at 432-0777 at Pikeville, Kentucky to learn more. The City of Whitesburg has seen a resurgence of small businesses sprouting up all over downtown. It has been our small business owners who have really been the ones to keep our economy alive. Much of this boom can be attributed to the Business Retention Program, which incentivizes business owners to set up shop in Whitesburg, Kentucky. From medical spas to fine art photography galleries, there is no bias in the BRE. If it helps the local economy, then it's a good fit for the program. For these people to take a chance at opening up their own businesses here in town, that's been a breath of fresh air for us. To learn more about the Business Retention and Expansion Program, visit cityofwhitesburg.com. Reporting for Mountain Top News, I'm Jameson Killen.


Hello everybody Good afternoon. We are going to start with the next talk which is about audit and namespaces by Paul Moore So please help me welcome Paul Hi, everybody. Can you hear me? Okay in the back? Yes, good. All right great It's my name is Paul Moore and I have to I have to say this presentation that I'm giving to you today Wasn't actually created by me. It was created by somebody else on the audit team Richard Briggs who works for Red Hat some of you may know him Unfortunately, he couldn't be here this weekend for dev comp. So They asked me to kind of step in and do the presentation, but it's a good presentation And Richard actually gave it at the Linux security summit this past summer Up in Canada and it was great. We actually recorded the videos for the first time this year So if you want to see Richard actually give the presentation and do all the Q&A You can go to that YouTube video. So can I really get your cell phones out take a picture of the URL? Really, I only see one person. All right. Anyway You can also if you just go on YouTube and search, you know audit namespaces like security summit, you'll you'll find it It's very well done video Actually has some production value in there. So anyway, I would encourage you to check it out It's 45 minutes in length. I think so it's not too long And it's it's much more interesting probably than what I'm going to be able to give you today So anyway, um this slide I actually added I kind of thought about giving Richard's introduction slide to see if anybody would know the difference between me and Richard But I figured that would probably be kind of a joke lost on many of you But anyway, so if you watch the video this slide is different Anyway, this is me. That's my Twitter handle. I don't use Twitter an awful lot But if you do send me a message or whatnot, I will respond to you Those are my email addresses. I tend to use both I use the paulmorend.com one a lot for the upstream things But you also see me use my red hat one quite a bit too. They're the same person For the past several years, I mean I've been doing well, I've been using Linux since maybe the Mid-1990s I've been actively Engaged as a Linux developer since maybe 2004 I think And I'm currently acting as the sce linux audit and labeled networking maintainer for the kernel I also created the libsec comp project Happy to talk about any of those if you want during the conference, but today we're going to be talking about audit and namespaces Um, actually a quick show of hands. How many people are here for containers? Okay, now how many people are here for audit? Wow all right I was thinking in my head last night when I was going over the sides of being completely the opposite I was expecting everybody raised their hand for containers and like two of you to raise your hand for audit And that's primarily because there's my boss and my boss's boss here and I figured they'd be excited about audit Anyway, all right. So good news. We're talking mostly about audit not so much about containers and namespaces Because quite frankly that it's an interesting story So anyway, let's get right into it So first of all, what is audit? Hopefully this is not a new concept to you guys and based on how many of you raised your hands I think you know the basics of audit, but for those that don't Audit was created back in 2004 by rick faith who worked for red hat at the time I'm not even sure if he still does. I don't know but It was a red hat project Auditing on linux actually started before this Susie had an implementation which for whatever reason wasn't able to be merged upstream But audits we know it today started in 2004 Dinosaurs weren't roaming the earth then but we at least remembered when they did at that point in time A lot of people like to talk about audit as syslog on steroids I don't necessarily Like to think of it that way, but if you do that's okay. It's it's not completely wrong The whole point of audit is to have a more secure logging mechanism in the kernel The reason is about the time that audit came into existence We were starting to take linux through a lot of third party security evaluations things like common criteria Which is a very rigorous Security evaluation and I see some common criteria people in the back that smiled when I said that so It's a very difficult very painful process at times because you really have to go into the whole Operating system in very fine detail explaining how everything works and making sure that it works the way that you said it does So it's very interesting thing and one of the things we realized when we were doing that is we didn't actually have A good security guarantee around a lot of our logging that happened We've got syslog and syslog is great But syslog ends up in var log with a lot of other application log files and it turns out for practical reasons It's very hard to secure that log At least secure it to the sense that there was enough integrity around the log That lawyers could feel good standing in a court of law and saying that you know based on these log file entries You know bob did this and sue did that at this point in time on this system So we needed a better mechanism and that's why we created audit It works well with sce linux and it works well with sce linux for two two reasons One it's a great tool for sce linux to report on things, you know, and there's an sce linux denial I'm sure we've all seen a navy sea message, right? Wow, okay. I also do sce linux. I'm really happy when there's not very many hands that go up for that But anyway, so it's a great reporting mechanism for sce linux It's also very good because of the way it's structured You know all of our audit logs go into its own directory So it makes it very easy for us to Tighten down and harden that directory because the only thing it goes in is audit logs So we can lock that directory down using sce linux For other applications which helps guarantee that integrity that we're looking for Audits a bunch of different pieces. There's obviously the kernel component, which I've been talking about But it's also there's a very important user space component to this as well There's obviously configuration tools and there's a lot of neat little tools For parsing the logs and doing cool reporting. In fact, I'm going to do a shameless plug as steve here He'd be very disappointing because I'm actually going to plug his his lightning talk If you go to steve's lightning talk, if you look on the board, you'll see there's a big audit announcement It's kind of cool. I won't steal his thunder since he's not here But if you have a chance today, go go check out steve's lightning talk. I think you'll like it So anyway, so there's there's that part of it But for the things that we're going to be talking about today one of the more important things for Auditing is the audit daemon This is the component that sits on your system That takes all of the log information from the kernel and either writes it to a disk sends it out to the network It's actually it's a pluggable mechanism So if you have your own special Log aggregator or whatnot, you can hook directly into the audit daemon and get those and get those log records out of it We also have the normal, you know configurable kernel filters I mean you can obviously do filtering after the fact on the logs, but Most people who use audit heavily generate gigabytes of log information on a daily basis And you can imagine as your enterprise grows the logs only grow So one of the ways we can kind of combat that is with these kernel filters so that you can specify Only what events you're interested in You know if you wanted you could log every single syscall invocation on the system It's a great way to crash your system But you know you could do that So anyway, and finally the last point on the slide This always seems a little silly to me, but I some people do Think audit does something differently Audit doesn't actually enforce any security policy The one exception I'll tell you in a minute Mostly it just reports on what's happening on the system And you know any violations from the other security policies on the system The one exception to that and I doubt anyone here in this room has this enabled. It's definitely not the default You can set it up so that if something happens to your system Or the audit logging mechanism such that it can no longer Safely record audit records to disk You can panic the system The idea being that there are certain groups of users and certain groups of system administrators That if if they can't have a log of what is happening on the system They don't want anybody to use it. It's kind of like if I can't play with my toys Nobody can play with my toys sort of thing So anyway, that that's the only bit of security policy that audit actually enforces Everything else you see in the audit log. That's that's somebody else's fault not ours. So don't blame us All right, so anyway, so I'm going to talk about namespaces But before I do now that I got the quick audit introduction done I like taking questions while we go through the slides I think it's a lot easier than asking you to remember all your questions until the end after I've thrown all this stuff at you So feel free when I get to the end of the slide or if I'm in the middle of slide Raise your hand and I'll try and answer your question And also remind me to repeat your question because we're being recorded. I'm really bad at remembering that So name spaces In short Richard's personality here is you know kernel enforced user space views and That's a great way of looking at it One of the ways I always kind of thought of it Is that we're all familiar with virtualization, right? You know, there was zen and there's kvm q emu and whatnot and that was That was hardware based virtualization and it gave us different views of the hardware Namespace is the same concept except instead of giving you different views of the hardware We're giving you different views of the system Okay, so We've got let's see what do we have we have seven different namespaces here Um You know if we look at the first one the mount namespace which is kind of interesting in its own, right? Um, basically that just gives you a different view of the file system Okay, so every every application on the system if it was running in its own mount namespace Could have a different view of the file system, you know, you could have different home You could have a different user different var different temp It's it's kind of a neat thing especially when we start talking about some of the other uh namespaces that are on here But the first point Richard's kind of got these segregated You'll notice into peers and hierarchies and this is an important distinction If we see the top four those are all peers and what that means basically is a very flat Structure, you know, we were talking about the mount namespace for example So what that basically means is if we've got, you know, four different applications each running with their own mounts file system They don't necessarily inherit anything from each other, okay They're all see a different view and they all have a different representation. They don't inherit anything, okay There is no hierarchy. All right. That's why we call it kind of a peer And we've got that with the mount namespace. We've got that with the uts namespace Which if it didn't say host name up there, would any of you know what the uts namespace was? Yeah, okay. I'll be asked when I first heard a uts namespace. I was kind of what what is uts That's the host name domain name information and whatnot The ipc namespace. This is all the ipc mechanisms, you know Shared memory all the sys 5 stuff the unix domain sockets, I believe When richard was first putting this presentation together over the summer for the Linux security summit We were going back and forth and he was kind of using me as a dummy to do the presentation He presented that and I I asked him I said, yeah, I know about the ipc namespace But I I can't think of anybody that actually uses this can you and he kind of laughed and he said, you know I went and looked at it too when I was putting this slide up. I can't think of anybody that uses the ipc namespace So it's there and you can see it's been there since 2619 It just doesn't appear to be very common. We're not sure of anyone that's actually using it Then there's the network namespace that's also a peer namespace. It's very cool This I think it's easy to think of how you might use that Especially the way it's implemented every namespace gets a fully independent network stack Which is kind of kind of nice and handy The way it starts off you create a network namespace and all of your network interfaces Live in the initial namespace then you have to go ahead and move them Into the individual network namespaces The only way you can talk between network namespaces is to actually talk out over the network and come back in Now we have some neat little things We have some virtual ethernet device pairs so that you can you know set it up to talk between the two interfaces But you know all your normal things like you know, you can set up a firewall between applications on the same system, which is kind of Interesting if you think about it You know ipsec stats, you know all that stuff It's kind of a nifty little thing But anyway, uh, and let's see and that came in 2624 all those have been around for quite some time Next we have the hierarchical hierarchical namespaces Probably the easiest one to think of at least for me anyways the pid namespace We all know on the system, you know For better or worse, everyone became very aware of of of process trees with when system d came about You know and it runs as pid one and then it just it goes on from there Well, the idea with a pid namespace is you can create your own namespace And the first process in that pid namespace becomes pid one and then it goes on from there It's hierarchical in the sense that You create a pid namespace and let's say you're running bash You know, it's the first process you start up in that namespace so that gets pid one But it's hierarchical if you're in the initial namespace So if you're in the if you're in the host system before you started that namespace, you'll still see bash But you'll see it with a different pid Which is a bit odd if you think about it So this one process can have a different pid Depending on which namespace you're in, you know, if you're in the initial namespace Might be pid 3000, but if you're in That bash instance, which has its own pid namespace if you ask it, you know Hey, you know echo dollar sign was a dollar sign dollar sign. I think I can't remember anyway It'll show up as pid one So it's kind of interesting and it it has some very interesting ramifications for audit Because if you think about how are we going to start logging actions for that bash instance if it does something security relevant We want to tell the user. Okay. It was running with this pid. Which pid do we use? Do we use the pid from the namespace? Its own namespace or do we use the one from the global namespace? So that presented some interesting challenges. We'll talk about a little later Then there's a username space which Depending on your point of view. This is either the greatest thing to come along for unique security in years Or it's a source of tremendous terror and fear I tend to fall in the later camp of Terror and fear, you know, some people have those dreams where you wake up and you you're falling I have these terrible dreams or wake up and I'm in a horribly mismanaged user namespace Judging by the amount of laughs I'm assuming you have those dreams too maybe Anyway, so that's that we'll we'll talk a bit more about that later And then we have c groups and this I believe was the last Namespace that's come along. I'm not aware of any that are in development Maybe you do So the basic idea around c groups right is to to put resource constraints on processes and groups and whatnot The c groups namespace Allows you to have a hierarchical representation of that So basically you can you can sort of hide these limits from the things that you're running in a container Which can be very handy All right, so I talked an awful lot on this slide does anybody have any questions for me move on I'll know everything there is to know about namespaces All right So what are containers so It's probably a lot of you that aren't gonna like what I have to say here But from a kernel point of view containers don't exist They're a fiction. Yes Right, they don't exist It's it's kind of I like to think of containers much like Santa Claus Right people get all excited thinking about Santa Claus people get all excited thinking about containers But if you go around you talk to people really talk to people about Santa Claus You get different answers from each person, right? You know you go talk to a kid in America He'll tell you one thing about Santa Claus you come talk to a kid in the Czech Republic He'll talk he'll tell you something totally different You go talk to a kid in brazil tell you something totally different about Santa Claus. That's kind of containers really That's how I think of it anyway The kernel has no concept of containers and as such it's up to user space to define what a container is And as we all know there's plenty of container engines out there, right? I mean dockers the big one that everybody always talks about but they're definitely not the only one doing containers And depending on who you talk to you're going to get a different answer as to what a container actually is However, it seems to be common across the containers and even different configurations using the same engine Is that it's some mix of namespaces Some form of sandboxing seccomp is kind of the popular one, but se links you can also think of as sandboxing wavedan And c groups right because you know especially for people that provide these as services to people You definitely want to make sure you Restrain them so that one container doesn't go crazy and steal all your cpu or network And so anyway, and richards probably if you listen to his talk, he's a little more polite about this He put the kernel has no concept of containers halfway down the slide. He kind of buried it A little more direct, I guess So anyway, the kernel has no concept of containers, which is interesting as we get a little farther into this presentation talking about the audit impact it's really up to the user space container manager to define what the container is And report on it and track on it because the audit subsystem in the kernel really we have no clue You know people want people People have all these requirements for around container logging and at this point in time, you know, it's like we what's a container? you know so it's definitely a challenge and One of the ways we're looking at solving it this last point on the slide Is creating some form of container id or a set of namespace id's to somehow Uniquely identify actions that happen inside the kernel as being associated with a container And this is still an ongoing discussion. You'll see as we get going All right, so What's the problem and when we talk about audit and namespaces and containers? There's really kind of two issues that we worry about or two issues that we that we run into all the time One is as I talked about earlier, we have the audit daemon. Okay, and that's responsible for Basically doing all the audit stuff that that people really care about that actually generates the audit logs And so, you know Where does that run? Is that running in a container? Does that have to run on the host system? Then there's also the issue of all of the Various trusted applications that run on your system You know for example when you log into your system Pam generates an audit record saying hey, you've logged in Vsftp, you know if you were to log into vsftp It you know generates some audit records and mentioning both those because we'll see later in the talk that those actually cost That's a good deal of problem But basically its first slide in highlander there can be only one right now There can really be only one audit daemon on the system We can talk about this more but in practice that doesn't actually seem to be too bad of a restriction because it's kind of It's fairly rare on most container workloads that you actually want to run auditing inside the container Most applications that run inside the container have some very good logging of their own It's usually only when you start talking about, you know trying to manage your whole system That's usually when audit gets involved. And so just having that run on the Host basically in the initial namespace is usually sufficient for most workloads The mount the uts that's the host name the domain name namespace and the ipc namespaces We have no problem. We've never encountered any issues with that with audit Things were pretty much Wide open everything was working well. It's about three seven rc one time frame Then we kind of started noticing problems Some of these were problems that existed for a while It just happened to be that the way user space was evolving the way the kernel was evolving They they didn't actually become real real issues First one was the network namespace The issue was that the All the communication between the audit daemon and the kernel and between the trusted applications in the kernel You know things like pam and vs ftp happen over a netlink socket Okay, netlink for those of you that don't know Effectively relies on the networking infrastructure to deliver messages between the kernel and user space well if you If you had a separate network namespace You know and you you were trying to log in via pam the issue that you had run into Is it didn't necessarily work again the more than i'm getting ahead of myself, but that caused this problem We fixed that in 314 Which was a long time ago. So you guys should be good now We had another problem with pid namespaces We'll talk about that in a little bit The username space as i told about you know this, you know, you either think it's great or it scares you There are some possible security concerns. We're mitigating those right now explain how but eventually we're going to have to address those An audit d basically right now Uh You have to run audit d in the user Initial namespace and the pid initial namespace Everything else, you know, you can you can run it however you want But it's got to be in the initial namespace for pid and for user All right So now we're namespace. This was started to explain on the past one. So, uh, you've created a new network namespace And you're trying to log in Let's say out of using ssh or you're just trying to log in the console It breaks And you want to know a great way to upset developers users Linus, you know, basically prevent them from logging into their system It's a real great way to get a lot of angry hate mail real quick um So anyway, uh When you break something for such a wide group of people you get a lot of proposals about how to fix it Um Because this was pretty wide ranging that the least complex of the simplest and quickest fix went out And it turns out it's It's not a bad fix. Um Basically the problem was that When pam started up, you know, it tries to open a net link socket up to talk to the kernel to say hey Bob logged into the system On a system where you don't have audit configured and it's not running You would get econ refused and so pam pam is coded out in such a way So if tries to open up this connection if it gets econ refused, this is okay You don't have audit configured. That's fine. Ignore audit. I'm just going to go keep going and log in well when we Quote quote fixed things for the network namespace Basically that all of a sudden allowed That connection to be visible and The problem was there were certain there were certain cases Where instead of now getting that econ refused you would get e-perm And well the way pam was configured because this this code was written long before anyone was thinking about namespaces Um people looked at and said oh hey, okay, whenever we get e-perm back That means that audits configured But something has happened that's denying us to generate an audit record And so therefore this person is doing something bad. We don't want to let them log in So in the process of fixing our problem for network namespaces, we pretty much just locked everyone out So We much as you do in real life when you find yourself in awkward situations Sometimes you tell a little lie, right? It's not good. And so that's basically what we do here We sort of lie a little bit and in these particular cases When you're not in the initial pit or username space um We we basically lie and we say econ refuse so that it you know pam still operates correctly And life goes on so Also in real life, you know when you do give these little lies it comes back to haunt you later This is going to we know this is going to be a problem later that we're going to have to solve It it continues to haunt us um But the facts are we have to solve a lot of other problems first before we can get back to solving this one Pid name space, um So last slide I talked about problems where we broke pam now. I'm going to talk about the slide where we broke vs fdp Uh, basically vs fdp auth is a way of sandboxing Created its own pid name space before it would do the user authentication And for a variety of reasons and as I'm looking at my watch, I'm Going longer than I anticipated. So I'm going to start talking I'm gonna gloss over some details. Um, that was not working So basically we uh were there's capabilities and all this good stuff Um, we fixed that and perhaps more importantly what I wanted to talk about and alluded to earlier was the whole pid name space and The fact that you know, you can have one process with multiple pids if you're in a separate pid name space and so process of doing this we realized that we weren't necessarily In the kernel always recording the right pid what we want to do Is we want to record every all the pid information relative to the initial namespace Because the majority of the time when the administrators come on they set security policy and they you know Have all the logs generated.  How you know, how does that happen? Um, so we had to clean up a lot of the code in the kernel To make sure that we were always translating pids and logging things correctly User namespace This is As we talked about this is this is one area where things are still ongoing There was a lot of patches posted in 2013. There was some good discussion You can google the linux audit linux hyphen audit To get to the mailing list and we have archives and you can go back and you can check it out if you're interested Um What's going on time to go into all the details, but um, there's a lot of good discussion There was even some talk about creating a audit specific namespace Ultimately everyone decided that that was probably not a good idea For the simple reason that you you don't want to have audit records Off in their own namespace because you want to be able to have visibility to everything From the host system. You don't want someone to be able to hide in a namespace That at least that's that was what convinced me But anyway, so we're working on that Right now as I said, you have to run audit d in the initial username space. There's there's kind of no way around it Eventually though, we want to work towards a system where we can have multiple instances of audit d One for each user namespace or at least allow you to have one for each user namespace We wouldn't force you to do that And that presents a lot of interesting things Especially when we start talking about in the kernel because everything in the kernel right now is architected around one audit demon And so if we start having multiple audit demons, we need to look at it And you know, we we have a queue of audit records that the kernel builds up and that we push up to user space As you do with any sort of communication channel But if we were to start namespace if we were to start allowing multiple instances audity We need to start looking at okay. We can't have a single queue anymore We're gonna need to split that up because otherwise you could run into a situation where one container Could overflow the queue and start choking out others There's also we talked in the beginning about kernel filtering For some of the audit records Well, we would need to have in addition to your own queue We would need to have your own rule set because what audit d in One container cares about is obviously going to be different from what another container cares about And the host audit d is probably going to have a completely different set That's independent of everything because they're probably going to want to watch individual containers as well as the own system So it's It's actually quite a tricky thing And we also want to make sure that if you try to configure The kernel in one of the namespaces That you're not making a configuration that's going to adversely affect other audit audit configurations that are set up in other user namespaces As well as the host system. So for example, I told you about the panic We don't want somebody in user namespace to be able to configure The entire host system to panic if something goes wrong With audit because otherwise, I mean imagine that you could have a you could have a container Somebody sets the system up to panic when something goes wrong with audit And then that person just floods the system with audit records and wedges it and then the whole system goes down for everybody The great way to get your container provider upset with you So anyway, and those are just the problems we know about now, you know as everyone understands software development will tell you You know There's the problems. You know when you start And then we get halfway through there's another set of problems that you had no idea even existed Um as referenced by the fact we locked everyone out of pan if he has fdp So I'm sure we'll undercover more as we start going down the prototype path, but That's at least what we're aware of at the moment And so the namespace id Way in the beginning of the presentation I don't know 40 minutes ago Talked about how one of the ways we need to reconcile the fact that there's no such thing as a container Is we need to somehow Come up with something a namespace a set of namespace ids or a container id Something that we can assign to each audit record So that system administrators when they look back in the logs can say okay This event happened in this container and this event happened in this container And there's been multiple attempts over the years I think the first one started back in 2013 Where we were using some proc i nodes to do that and the namespace folks were a little upset with that idea The checkpoint restore people were very upset with that idea because what happens if you were to freeze a namespace Or sorry freeze a container Move it to a different host and restore it Yeah, the i nodes aren't going to match up And that that pretty much that reason right there is what knocked out the first three things on the list Richard came up with an idea of a serial number a monotonically increasing number starts at one and each time you created a new namespace It would increase But the same sort of problem once you started moving things across hosts You know well what what does somebody has the same number? You know, how do you resolve that you can't easily change that number without the user space running inside of that container Having some issues Then there was some some talk about similar to what they were doing with the proc i node They wanted to do it with the namespace file system But they included a dev number with the i node so that it helped a little bit But you still have the problem when you start crossing hosts in a in a whole enterprise So that was kind of difficult one of the things let's skip ahead just a minute. Yeah, okay um So that's pretty much it The namespace id we haven't completely ruled it out, but I think we're definitely trending towards and I want to say my mind is almost convinced that we're going to have to start going over to a container id Container id is basically It's going to be a token a number of something that the kernel is going to have absolutely no clue what this is Your container engine will set it your container will basically tell the kernel it's like all right this This process has this container id And we want all the children processes from that to go ahead and inherit that container id Unless of course you start up another container engine inside that you know and you pivot it off. That's fine And we actually we have some precedents for doing this in the kernel today With session id's for those of you may not be familiar with audit Session id when you log into your system It gives you a new session id and everything you do from that point on on the system You could you could sudo you could su you could change se linux roles You could do all sorts of things, but that session id remains the same So it's an easy way for your system administrator to track everything you do on the system I didn't mean to make you paranoid that sounds really bad now that I said that Dan Like how the way I said token Oh, sorry. Remember I said I was really bad at repeating the question So dan asked and stopped me if i'm wrong, but you were asking basically does the id have to be an integer Could it be a uu id or something like that some arbitrary value, right? okay So no, I mean we obviously we haven't created this we haven't coded it up now The one concern I would have with uu id every uu id I've ever seen is really long And a lot of the audit people want really short things because they have gigabytes of data per day And uh, they they don't like spending a lot of bytes for things like that, especially for something It's going to be an every audit record Okay, so Dan was saying that he he doesn't He doesn't care about particular uu id He just wants to usurp this and use it for other Devious things Because that's what dan does devious devious things um He's laughing But anyway, um This is all possible. I mean just second So All up for discussion. We haven't actually designed this or coded it up yet It can't be huge. We're not going to let you abuse this too badly. I mean people Yeah, people are going to abuse this but we're going to try and limit the abuse But this is the point is this will be a token This will be a value that the container engine is responsible for setting And the container engine is responsible for managing basically We'll just we'll just carry it along in the kernel and do the right thing that way but Use space left tell us so I'm sorry. What's your question? Okay, so the question the question is basically I want this Two months ago when when can you get this done?  I wish we had it done There's only so many of us, right? Um If we get a lot of people yeah, yeah, I know The the person sitting next to him happens to be my boss's boss. We really want this too. So Um, all I can tell you is that there's a lot of people that want this solved You're you're not the only one It's a I'm sorry say what? Hey, it's all open. Oh, sorry. The question is how can he how can he help and All right, we'll kind of run out of time. I'm going to skip the conclusion for right now Because I think we sort of talked about this There you go Camera phone take a picture of it get the links. There's the mailing list um We're on github We actually have a github track or tracking this issue number 32 Um, we have a mailing list the the mailing list i'm posting up here is the linux audit mailing list Um, I'd imagine eventually as we get further down the line We're also going to talk to the container folks on their mailing list Um, but since we haven't quite sorted everything out from an audit perspective yet We would just kind of ask limited to the audit mailing list because we want to we don't want to get sucked into a bunch of different rat holes um So we want to make sure we we know all the requirements from an audit point of view first Before we start talking to the container people saying hey, here's what we're thinking of doing What what do you guys think about this? And part of this is also we're in the process of collecting requirements, too Uh, there's I mentioned in the beginning this little thing common criteria Which is very important to red hat and we need to make sure that we are doing the right thing from a common criteria perspective And we were hopefully going to have a discussion about that this past week, but um The person that was supposed to collect the requirements didn't get done time So we didn't have the discussion, but we will be having that shortly Um, so anyway, if you're interested there's a github issue you can subscribe to it you can subscribe to our mailing list and Hey patches are welcome You know so Anyway, um as far as personal contacts. I I already gave you my email address, but that's richards email address rgb Um at red hat. He'd love to hear from you too. Um But I think richard would probably tell you the same thing that i'm going to tell you and that send it to the mailing list Uh, because if it's something you're interested in odds are somebody else in this room is interested in it, too And it's better if we can get everybody talking together as opposed to having a lot of one-on-one conversations So, all right, so I think we've got 10 minutes left Um, I can go back over the conclusion slide if you'd like, but I'd rather answer any questions You guys may have other than why isn't it done yet? I'm sorry, please Okay, so the question and correct me if i'm wrong. You said that we have two Contradicting statements. I was actually kind of happy that was only two um One was that we don't want anyone to be able to escape audit logging And the other one was that you can only have one audit demon per username space You can't have one audit. Okay, so let me try and answer that if I think where you're getting at right now You can only have one audit demon So we know that that's going to be a problem later So right now we can only have one audit demon. So you can't really escape auditing right now Maybe i'm doing a poor job at explaining this but Yeah, yeah, it's a good thing. I mean eventually we would like to be able to support running One audit demon per username space But right now we don't support that and Why? All sorts of things break Um Basically one one of the issues is how do you start resolving? The uids and the gids and the fs Gid and the fs uid and there's about seven or eight different user identifiers What's that? Yes, but which one do we record? Yeah, yeah, see oh, i'm sorry repeat the question. Um Now I have to remember all the questions So I think basically the question was why can't you have multiple? Um audit demons right now running in different username spaces was that Okay, so So just a minute. I'll repeat it in just a minute. I'm just trying to make sure I understand it So you're basically wondering in the future when we can have multiple audit demons running How would we handle that is that? Okay, so you're you're never going to be able to escape lotting all right, so The concern For the recording is being able to escape audit logging right, okay, so And especially with username spaces because an unprivileged user can create a new username space um You're never going to be able to escape audit logging because the audit demon running in the initial namespace Will always be running and it will always have visibility of everything that happens on the system because there is no audit namespace This is one of the reasons we don't want an audit namespace So that even if you create a second username space and you do run an audit d in that second username space The kernel still has visibility into that second username space and you can still configure the kernel To report on events that happen in that username space to the audit d running in the initial username space keeping and So to repeat the question you're concerned about the the actual uid values and whatnot from there Okay, so you're you're concerned. I'm gonna answer this one question and somebody else had one more So I'm more than happy to talk to you after I just want to make sure we gather things answered So the question was how do we handle the different filtering? So the filtering in the username space adverse the filtering in the initial So And as we talked about we're going to have different rule sets You know, you can't have different rule sets for each username space. So The audit d running in the initial username space can be configured to watch everything if you want it or you can turn that off and User running an audit daemon in Their own created username space can have their own rule set So basically you can adjust auditing that for the username space that you create But you cannot escape the filtering and the logging that happens in the initial namespace So but i'm more than happy to talk about it later if you want last one Okay, so the last question the obvious question for security people is the audit daemon running in the username space will that have visibility to Audit events that happen outside of its own username space. No, we don't want to do that for the obvious reason The way we deal with it today right Merwin I said these are the problems we know about today We're going to have so his his question was about the other namespaces and my response is simply put We know there's some hurdles We have to overcome right now and I fully expect like I said as we get halfway down There's going to be other concerns as well This is this was a talk about presenting some of the problems and some of the solutions. This isn't a talk about saying Hey, here's our design. It's perfect Um, we don't have that yet, which is half the reason why we can't give it to you yet today So if we had all the answers we'd have us we'd have it working So anyway, if you're interested if you want to help answer those questions There's the audit tracker. There's the mailing list So all right. Thanks everybody. I appreciate it Hey, no good.  I'm a consultant for that customer They want to audit everything that happens in every container on the system and essentially that sounds like no Well, no, we can we can actually the funny the funny the funny part is um, the fact that we don't The fact that we oh, thank you Good thing I didn't start swearing