Malcolm X looked to Booker T. Washington, to Madam C.J. Walker, from every Buffalo soldier to every red-tailed Mustang. It's hard to choose, you know, and all the names in between because they're all inspirations. They serve as a reminder to myself and to every black person, living that we must always strive to be the best version of ourselves we can be. Work hard and enjoy life to the fullest extent. Each of them made decisions not for themselves, but for us, here and now. So that we may have the opportunity to vote, to earn an education, laugh, fall in love, be happy, live and breathe as free men and women. And I don't take that for granted. We have the ability and therefore the responsibility to ensure that their sacrifices were well worth it. That those of our ancestors who could not experience opportunity and rich fulfilling lives see their dreams culminated and come to fruition through our actions now. We honor them with positive impacts and by making this world a better place for everyone. And that's the beauty of this month. It's a celebration for black people and our story and our heritage and even our culture. But more than that, it's a recognition and a tribute to how we can grow and transcend as a human race. We can learn to show compassion and enrich each other's lives through peace and tranquility.


So let me get on the grandstand. Hi there. I'm Turk Peterson from Seattle Fred Hudson Cancer Research Center. I will introduce myself today. There's no person who introduced me. I will introduce Bobby Minier from Swiftstack. We're both going to give a presentation about how to bring an object storage system into a real life research environment. And Bobby plays a key role in one of the access methods we use to store this system. So let's just get started. So what we'll cover today is who we are, why we did what we do, what we did, what were the requirements for our project, what else did we try, what we deployed, how we manage, what it costs, and how does it really work in day-to-day life. And what was the timeline? We saw that in implementation projects a couple of times that I assisted. The timeline is really something that in cloud goes from sometimes it's a couple of weeks, sometimes it's a couple of months. But we are talking enterprise, and that is often measured in years. So let's go ahead. What is Fred Hutch? We are a cancer and HIV research place. We have about half a billion dollar budget. We are largely funded by the federal government. And for the purpose of our talk, we possess multiple data centers on one campus. That's really important here. And we also are subject to this explosion and genomic data sets. And we introduce charge bags. So the charge bag is a researcher has to pay. They get a certain level for free. And then they have to pay when it exceeds a certain threshold. And that's what we implemented in July 2014. So it's really important for this talk. So why did we do it? Our researchers were faced with charge bags. They said, OK, we made up NAS charge bags. We didn't make them up. But we came to the conclusion that NAS charge bag would have to be $40 a terabyte a month. And a little bit is free. But when federal grants are declining, you probably heard about sequestration and everything, then that's a really tough nut to swallow for researchers. We had a lot of discussions about it, whether that would be feasible or not. And at the end, we came to the conclusion that they said, OK, if you charge us, you need to give us something really cheap, really inexpensive, where we can store older and also bigger genomic data sets. And of course, there's always this misperception about the drive from Best Buy. I always wonder if there's another company that sells like cheap drives that Best Buy is often mentioned as an example. Finance, we maintain very good relations with our finance department. And when they see that they have to flip a bill for some storage system, they're not necessarily concerned about the cost per se. But they're really getting worried when you come up with like $500,000 or a million dollar at one year, and then a couple of years later, you buy another one of these things. And they say, OK, you have this thing here with lots of yellow lights. And now you bring on something with green lights. But in reality, they all live side by side for a long time until you have your migration down and everything. And so they don't like that. And they know how to use a web browser. They go to Amazon, and they click a couple of clicks, and they see what storage should cost. And if that is fantastic out of line with what you want to offer, then you have a problem. So then we had, of course, the usual stuff that enterprise have always problems with data protection and archiving. In many cases, data protection is an afterthought. Why do you buy a fancy new storage system?  And then the backup performance is not as you expected. So that we've lived through several cycles of that. We do have a very nice backup system, but still it runs into scalability issues. So we did look for a solution, but we were not necessarily looking for an object storage system. We're going to talk about object storage today. We're going to talk about Swift. But we were not necessarily looking for that per se. We didn't know about it, actually. So users wanted a mountable file system of any sort. They wanted to use their Unix tools. They wanted to continue to use their standard Unix tool.  They can use any storage system. But the 10%, they're really the data users. And you have to be careful that if you invest larger amounts of time and effort, that you not focus only on your 5%. So performance needs were not too high. We were very nimble about that. But we wanted to grow about it. And we had concerns about disaster recovery. Seattle is a place where earthquakes can go up to 9.0, like the one that we saw in Japan, potentially. And that gives us some worry. So we wanted at least the first step to replicate our storage to multiple buildings that are a little bit further away from each other. But in the next phase, we wanted to get out and maybe do a DR site at the other side of the Rockies or something like that. So a couple of key characteristics. It was like permanent static data. We wanted like POSIX permissions. It's supposed to be optimized for economy, supposed to be self-protecting. We didn't want to have a dedicated backup system for it. So a project, our architecture team typically like dissects every project in like must have, should have, nice to have. In our case, there were lots of like must have. There were some nice to have. But one of the nice to have was this thing that we wanted to do like a broader storage strategy. It wasn't necessarily a hard requirement, but we had really the idea that, OK, you do this for like only a subset of users who use like POSIX file systems. And I wanted to like jump the genomic data in there. But it would be really good if we could also like leverage it for other things in the future. So object storage, scaling, manageability, no rate systems that sounded like that was like kind of like music in our ears. The resiliency, but also predictability. We ran often into performance bottlenecks when there's a rate rebuild in our storage system. So the perceived predictability of an object storage system is lower performance in our requirement, but at least that low performance is very predictable. That was something that we really wanted. So then we heard about Swift and Swift Stack. And at first we said, oh, wow, it's proven. Some really big clouds use it. They have, I heard rumors that there's installations up to plus 50 petabytes. So that's more than we ever will need. But if we want to get there, that would be great. Manageability, low cost, longevity. We have some requirements where we store data like for 10 years and longer. And some researchers want to store it for 20 years and longer. And they make up, or they define those requirements. And we deliver then the technology to serve that. And if you have a proprietary piece of equipment, storage equipment, and you have to do these upgrades, then that's really not longevity. So with Swift, the promise could be it's open source, it runs on commodity hardware. You might be able to run it forever. Forever and ever. And then the durability part, data centers fail. We can have, we said we can have earthquake, we can have fires, we can have disasters. That was an important component there. So conceptual view of how this will work. We have like three, basically three storage buckets, or we call them zones, and each of them in a dedicated data center. And then on top of that, there's a SIFSA NFS gateway. So traditional file access to serve our standard community. The research community we saw that in the previous talk is rather conservative. And a lot of object storage systems are not like brought to the end user yet. So we need to have some bridge technology to serve them in the meantime. There were previous talks about Swift, so I may not have to explain how it worked, but in general, the Swift system has three replicas, or three replicas are recommended, and they're stored in a unique as possible way. So it can either be like a single node, it can be a small cluster of multiple nodes, or it can be like multiple data centers that are distributed in multi-region across the country or across the globe. And so looking further into this, we really felt like, ooh, look at the underlying stuff. What is Swift? Linux, Python, R-Sync, those are all tools that we were already using.  It looked really easy, it looked really great. And so we felt a lot of comfort in the resiliency here. Here's an architectural overview of where we see this tier will fit. So we have, typically we have a POSIX file system, this is an Isilon scale out NAS system, then we have some high security stuff. It's basically a fully audited NetApp system. We have some block storage here. Block storage is rather small, so in our case, it doesn't really matter what we use. You could actually buy anything there. And here's our economy filed tier with Swift. Swift stack, right? And so this is like $40 a month and this is $3 a month. So this is gonna be an enormous economic pressure to move as much data into this tier. So what were the other, I often, or you'll get this asked, what were the other options that were out there? We were a long-term user of Gluster FS for scratch spaces and high performance computing. We did really like it, but when we looked at this, there was kind of the different level of resiliency that we wanted and we didn't really have any proof of scalability. We also, the replication features that were in Gluster is probably now more upcoming, but at the time, they were not really widely in use and we couldn't really find any production deployments that did that. And still you have the rate that you have to manage underneath Gluster. I'm not sure if that's still the case. Probably it's still the case. We looked at these magic HSM solutions where basically you take a file away and replace it with a little stubby stub and then when you hit the stubby stub, nothing happens for like half a minute and then it tries to get it to the tape from the tape system and then move it back. And we've had that in the past and it really didn't work so well. And it also required a forklift upgrade at some point. We also looked at several commercial vendors with who were touting erasure coding. And erasure coding is really good at first, but if you have to sustain a complete data center drop out, then the erasure coding algorithm has to be more conservative. So you're losing some of this cost advantage that you initially had with it. I'm also often asked by folks from the open source community how does it compare? And why don't you use, Seth, we saw this talk before. It's very similar use case, high performance computing, some genomic data. But in details, perhaps a little bit different use case. And Seth was more targeted towards that use case. In general, if you just want an object storage system, you can, for example, look at the lines of code that each of these projects has. And you can go to OLO, which is a website that analyzes lines of code of each open source project. And you see that Gloucester is really 10 times as heavy. And Seth is still heavier than Swift and Swift is a really, really, really lightweight piece of code. And there are some people, I'm not saying that, but there are some people who say that the numbers of errors per line of code is always the same, no matter what you do. And if that is true, then, you know, there would be not necessarily more bugs in a system that is like heavier, has more code. So it's just a, you know, that's just a developer view. And not everybody, I think that's plenty controversial and I'm happy to discuss this further. So what are the public cloud? We are about like a quarter mile from Amazon AWS headquarters. We know lots of people there.  And quite frankly, yeah, that's what we will probably end up doing at some point. Amazon is in use at the hutch today. And you know, people like it. There's still some fuzziness about government policies. You have to have business associate agreements. And so we are moving slowly towards that. And we will also be able to prepare for this day when it comes. At some point, we believe we will live in the public cloud. It may be like 10 years from now, 15 years from now, five years from now. But by actually using Swift, we can use the S3 API to adopt applications and programs to actually use S3 compatible ways of storing data. And that means that when the day comes, we will be able to switch over right away without any or without lots of hassles rather than having to migrate all like legacy POSIX file systems to the cloud. But there's still some truth about the cost here. So we've just seen recently some cost reductions at Google and Amazon. We see our NEST here is about like $40 a terabyte a month. And the public cloud is in the neighborhood of 25. That's without transfer cost, it's just storing it. And Swift, TCO, backup, everything included, except for the data center, which we already have, it's $13. Including energy cost and everything. So as long as we don't find it less expensive to move to the public cloud, there's no reason to do it. And maybe the day will never come and then we stay there. In some cases, we will need to storage Switzerland. That means people who wanna share data with us don't trust necessarily me to store that data when they're not in our institution. They want like kind of a Switzerland and then they will use the public cloud. But for many types of data, that's not true. How are we doing on time? Okay, we're good. So what we deploy, we deploy one zone per data center and we have the Swift management, the Swift stack management node in the cloud. And that is something, so here we go. So here's the bird eye view on Fred Hutch Google Earth. We see there's three zones, zone one in one building and they're about like 1,000, maybe 1,500 feet apart and zone two and zone three all in separate buildings. So if there's an earthquake and the highway falls on this building, we still have these two available. And that will be without any downtime. So we would be able to sustain a building failure without zero downtime. An architecture view, again, three storage zones. There's a low cost switch in each zone and then that switch is hooked up to the data center switch and it's hooked up to the building and the core router. So this is how it goes through the entire campus. Some building blocks. We have a super micro chassis. It uses desktop grade hard drives and for about 130 terabyte usable space, you pay about $13,000 plus tax. This is a discussion that for cloud people is, because why would you use anything else but consumer grade drives? Google published this paper in 2007 that said, those drives are just fine. They fail a little bit more often or they couldn't even prove that they would fail more often. They found them reliable. But for enterprise folks, for enterprise storage groups, that is a tough call to say that you're using some quote unquote junk drive. But the folks from Backplace, that's an online backup company, they really have collected some data over the last five years and they saw how actually does it look like when failure rates go up? So here's at year three, you see that these drives typically seem to fail more here. Five and six are projections. So they assume that it will go there, but at year three really see an increase. So enterprise storage, people say, oh my God, we need to replace all these drives and the sky is falling in year three. But cloud operators say, oh wow, after six years, you still might have 50% of the drives in the system. That's great. Continue to use them. They're not broken. They work. So that's a little bit of a mind-change in the enterprise. If drive failure rates are in the name of 8% per year, which is pretty high, we estimate that at the end of the build out at three petabyte system, we would have 20 drive failures per month. And the beauty about SWIFT is that you can just replace them all at once, like once a month. You don't have to run because it's not a RAID system. You'd have to run into your data center full of sweat to replace that one or two drives. You just wait. Over time, the storage capacity shrinks. And then when the day comes, like on the first Monday every month, you rip out the old drives and place them with new drives and you're good to go. And you could actually replace them with a bigger drive, which has an impact on your storage costs. So if you had like three, two terabyte drives before, you can suddenly use maybe four terabyte drives. This is the Netgear switch component that we use, $1,400 for a full 10G switch. It has about, we did some performance testing. It has about 75% of a Cisco switch. So it's pretty good. It took some time to have the Enterprise Networking Team getting familiar with it. At the end of the day we came to the conclusion it's just like an unmanned switch. It's a throw away piece. We put like one end storage and then if it breaks, you just throw it away and replace it with something else for $1,400. So basically, I think it's the price of a Cisco G-BIC. So that's not too bad. Architecture view. So here you see the super micro systems, the Cisco, the Netgear switches. And then you have the data center switch up here. And there's a NFS SIFS gateway. So the totals on storage is like 260 usable for $83,000 and the monthly cost is about $5. So how do we manage it? The initial deployment took about seven days. What's really important is that you have to make sure that you have a vendor like Silicon Mechanics, for example, who offers you these desktop grade drives and drive carriers. You can't ask your storage team to get out a screwdriver and put them into drive carriers. I think that's too much to ask. And then it's about 0.25 FTE. We calculated that. It's perhaps a little bit less. So open source Swift is functional. You can just deploy it. It sounds all very easy. But there are a couple of things that are really not so easy, like maintenance and upgrades and just the general monitoring. And we've seen people who have actually used like an FTE or one and a half FTEs to maintain a Swift environment. And perhaps it's even like large or maybe three or four FTEs in some cases. So that's why we came to Swift Stack. We said, we are not people who could do that. We want to keep the cost low. We can't just have this storage adventure at our place and then not pay anything for hardware and software, but then invest like two FTEs in maintaining it. That would be rather absurd. So we said, we need to have something that does it for us. And so then Swift Stack basically came to the rescue. Provides a cloud-based management console. You have enterprise features like delete protection. Really important in standard Swift. When you delete your file, it's gone. And if you give that to your end users, well, what happens if you don't have backup? You can't get the file back. So Swift Stack actually offers a middleware layer that puts files that are deleted in a trash can for 60 days or so, and then they're expunged afterwards. So that's a very critical enterprise feature. And then the Swift Stack controller lives in the cloud. So you don't have to upgrade that. You log in and sometimes it's there's new features. They're just popping up and you use them. And it's a Swift Stack is a real 24-7 global company, which I didn't know initially. I always wanted to, at some point, the staff was answering email at like 2 a.m. in the morning and I said, oh, you know, he probably doesn't have really good sleep. And then he continued doing that. And I just wonder what that was when he was actually in Taiwan. So Swift Stack provides all this control and visibility. You can completely monitor your cluster with almost every detail. And the rollout with Swift Stack is literally 10 minutes. You have your Linux boxes up.  You type a couple of commands. They are all there and it's like 10 minutes and you're done. Then the one feature that really made us like, we were just like speechless was the upgrade. So we had a full blow of like files right into the cluster, you know, multiple hundred megabytes per second to it. And then at some point, an upgrade button showed up. And you can just like this, nothing more. You just click this upgrade button and silently it upgrades the entire cluster to the latest version of Swift. You know, not a single ping missed. Nearly full performance. This thing is degrading like by 20 something percent. But that was really a spectacular experience for us. Upgrades otherwise eat like lots of time and we have like project managers involved and whatnot. It's really intensive normally for storage upgrades. So that was really good. Swift Stack Controller. I mentioned that. What's the difference between like Swift and Swift Stack? So the Swift Stack Controller provides these features on the left. There's also a runtime on your local node and you know, you just have open stack Swift. That's the standard stuff, Linux and your standard hardware. Pretty straightforward. So now I'm gonna talk about one of the most, Bobby's gonna talk about the most critical component in our case. We said we can't use today, the objects store directly most of our users are, you know, use legacy NFS and SIFS. And Bobby's gonna talk about this new nice technology that's available today. Thank you. And again, I'm Bobby Maneer, Director of Technology at Swift Stack, specifically focused on access methods to the object store that are not the API. And have been working with Fred Hutch, specifically on the SIFS and NFS gateway.  So if you're not familiar with a gateway that it's effectively a means to getting data into an object store other than using the native object store API. And its role in life is to effectively lower the barrier to entry to be able to get data into the object store. And use familiar IT workflows, paradigms that your operation staff, you know, sort of know and love or hate, depending on who they are. But just allow the object store to be utilized without having to take the time to immediately start recoding applications to use the native object store APIs. And in fact, in some cases, you know, we've specifically heard from customers that, you know, they have certain applications that they're never going to rewrite. At some point in the future, that, you know, that use case just goes away but the data persists. And if you look at the spectrum of gateways that are out there in the marketplace right now, I like to really kind of come up with a taxonomy of two types here. One is, how does the gateway effectively treat the data as it's being ingested into the object store? There's certainly a class of gateways where they're taking the data, the files that you're wanting to upload, chopping them up into, you know, very small segments, potentially performing some compression, some deduplication, and then shoving all of that, those little blobs of objects into the object store. And, you know, it's a very efficient means to do it.  But effectively, it's now taken your data, your objects, and has, you know, morphed them into something where you have no idea what form they're being stored in.  We have preserved the one-to-one mapping of the object as it's, you know, produced by the applications and how it's stored in the object store ultimately. Because that's just, in our view, hugely important for mixed use cases. So, you know, you don't want to be able to, you know, push something up in, you know, via one of these file system gateways, and then want to later access that data with a different application. And, you know, you don't want to have to backhaul that data through the gateway in order to then, you know, reconstitute the data and make it accessible. You just want your, these new applications to effectively be able to hit that data in the object store and utilize that without any other piece there. So, effectively, that is, you know, no technology lock-in. That the data that you put in the object store is exactly what you know and expect.  Another element of the gateway that, you know, it's an ideal point for, you know, possible data transformations or data extractions.  And they can perform these types of extractions. And I'll talk, you know, a little bit more about the metadata extraction in a minute. But basically give you that consistent view and enable different workflows here. You know, one that Dirk had mentioned or at least had on the slide, was this notion of write, want, read, mutt, many. So the gateway, again, is giving you a control point in your storage network, whereby you can ingest an object and then coordinate with the other gateways to make sure that that object is then protected from future modification. So I'm just gonna touch really briefly here because this is obviously a case study about Fred Hutch and not on, you know, gateway deep dive. But just some of the ideal use cases that, you know, we see for a, you know, non-blobifying gateway, I'm sure I'm gonna patent that at some point. So one is active data archiving. So when you're pushing tons and tons of data into your object store, one of the things the gateway can do is extract metadata that's inherently embedded into the files and associate that in the object's meta tags or even potentially vector that information off into an index server. So that you can form these catalogs of the data that's in the object store without having to go back and try to pull or, you know, search the object store in ways that are not currently exposed, you know, in the API. And, you know, I think that's something that, you know, we've started here a lot of interest in. And really the key is having an automated way of being able to pull that information out of the files and get it directly into these servers. Touched a little bit on this, but multi-site data sharing, whereby you can have these gateways deployed at all your remote sites. You have your one core centralized object store and you want to have a unified view, again, via SIFS and NFS, your standard file system protocols. And the gateways have the ability to, you know, coordinate their file system view of the information stored in the object store. And then anybody in a remote location can access these objects just like they were coming out of an AS file system and can do some specific things to improve performance there. And then finally, if this ever clicks, maybe. Data migration. So because the gateway is providing a SIFS and NFS mount point, it's just very easy if you have large amounts of data that you want to start pushing into the object store, just mount, you know, a container as a mount point via the gateway and just start a copy job or an r-sync job and it just gets all of your data up into the object store without having to, you know, fiddle around with, again, you know, tools or anything else. It's just, you know, emphasize that the IT workflows are preserved. The IT shops are very, very experienced in dealing with gateways or with file systems. And we'd like to just preserve that functionality. So where do gateways go? Ah. You know, right now the gateway itself is utilizing the object store APIs directly, again, saving, you know, the potential for having to rewrite the application right now. Ultimately, I think there's better performance that we can get out of the gateways with tighter integration with the cluster. It's something, again, we're working at, you know, very hard at Swiftstack and are going to continue to look at these ingest points for other types of data grooming that can be done on the objects to, you know, open up the potential of, you know, accessing the metadata and being able to, you know, work with the data directly. All right, thanks. Super. So let me just... Oh, if your phone was a pointer. What's that? Oh yeah, that would not be good. Five minutes. Okay, so then let's run through it. Oops. Okay, so one more. So with the assistant NFS gateway, I said initially that our performance requirements were not very high. We actually have achieved about 300 megabytes per second. And if we use, you know, the native Swift API, then we can actually get gigabytes per second throughput. So that is really good. We have enough performance to get started and we can grow later and we can have people who are in the same research group who use Swiftstack NFS. And then we have advanced programmers who can access the really same data. And that's kind of unique. If you buy like a standard file system gateway into an object storage system, then you really wanna drive that home, then you necessarily can't access it in different ways. You have to either choose Swiftstack NFS or you do object. And that's really the benefit, the long-term benefit that we see here from the system. We have a couple of limitations.  There are some limitations that you have to be aware of and you have to discuss with your stakeholders. And some links are not available. That's for some people, it's tough to swallow, but for most folks, it was not a problem. You can't easily rename directory structures. You need to copy and delete them. And the renaming of the directory structure hangs other IO in that certain directory. So for most use cases that we looked at was an elimination at all, if you look at it really. And the DU command doesn't work on Linux. So we wrote a replacement for that to make that work. Overall, we've really implemented with Bobby's help, basically everything that's needed for an enterprise. Like we have like esoteric POSIX permissions that we can do now. We have, Bobby mentioned that Active Directory integration has been worked on. It's improved, but it's production ready today. We use Active Directory integration today and it works. So a little bit about the timeline. We started two years ago. That's for an enterprise. It's probably appropriate for many other use cases. You will find that amusing. But two years is the time in that we took the initial evaluation of Swiss sector only two days and said, hey, it's a good tool that took us two days and implementing it two years. Lessons learned, we really need to communicate with everybody at all times about pros and cons and limitations and what the system does. And you need to be ultra responsive during your test period, ultra, ultra responsive. Then you get some buy-in from people. Prioritizing requirements, do you really need symbolic links? Is that a requirement? And if it is, we need to work on them. And if not, then we put that at the back of our priority list. Oops. So tape backup is a big step for people to give up. Very big step. And with SwissSec help, we really think we pull that off, that we can have data secure long term without any tapes. That's a big thing. So I mentioned that already, the POSIX gateway. And I have some things that I want to do in the future. Where do we really land? Or what's the next step? Once we have implemented this use case, I mentioned initially that we want to use it for other things as well. And we have an endpoint backup. There's a product called Druver. We use CloudBerry backup. We can use that. Then we have, there's a high performance POSIX file system front, and it's called Avere. That's available. CloudBerry drive, expand drive. There's all sorts of tools that we want to investigate in the next year. So can you fire it up again? I'm sorry. I just want to mention this as a final comment. I should probably not push too many buttons here. Shift F5. Oop, no, leave it there. No. We're also working with Seagate on their new Kinetic platform, which is an Ethernet-enabled drive that really allows you to use a hard drive as key value store. Basically, you can remove several layers from the Swift architecture and just store data directly in the drive. I mean, you saw initially before that I was really a big friend of Nimble systems, like 70,000 lines of code. Well, we've taken outlines of code. We reduce it even further. We simplify it even further with these new drives that come online from Seagate. So I don't know if Seagate is actually here, but it's a great project, and we hope that we can make some good inroads on that. OK, Doug, thank you very much. That's it, enough. If you have questions, two questions? There's a protocol that is being utilized by multiple gateways to communicate and synchronize their file system view, so that's how that's done. Well, the file system itself does not have snapshotting. You can keep versions within Swift. So Swift has the ability to keep snapshots, but it keeps the trash can that I mentioned before, and it also has versioning, which basically delivers you the full data protection. There's one more question here. We are currently in a project to use a Fraunhofer file system, FHGFS, which is the highest performing HPC file system that we currently know. And we use that for our Scratch file system. But it's not for this use case. This is really like archiving long-term, what we call an active archive. It's an archive that is not going away, that you can always get to immediately. All right, thank you. Thank you very much.


You will see in the morning you will wake up at 3.30, to every mouthful you choose twenty-four times.  If every time you sit for meditation, if it so happens that you're tending to fall asleep, I want you to understand what it is, first of all. Most people, not most people, lot of people in the world cannot sleep even when they want to sleep. Yes. Lot of people go to bed and desperately try to sleep and they cannot sleep. They have to pop a pill to sleep. This is the condition of millions and millions of people on the planet. So if sleep is invading your life just any time of the day, one thing is, fundamental thing is to check up your basic health. Just checking if there's something wrong with your system, physically wrong, that you're going through as a… you have noticed you're not well. When you're not well, you sleep much more than normal, isn't it? Because something is happening in the body which does not allow enough energy to do other things. It wants to rest. First thing is that, the second thing is the food that you eat.  I'm not saying this to everybody.  Why twenty-four? I will not go into that number business now, but there is a science behind it.  Mouthful does not mean…  And this is crucial if you want to meditate. Not doing fifteen minutes shown here. If you want to become meditative in your life process, this alertness, alertness not of the mind alone, alertness of your energy, that your energy is alert to keep it in an alert state, is why you're doing the sadhana. So to assist it, you eat only twenty-four morsels. Try from dinner or you want to try… do it tomorrow. Shall we try this dinner or tomorrow? Now if you do eat only twenty-four morsels, normal sized morsels, you will see in the morning you will wake up at three-thirty. Yes, in the Indian way of looking at time, there is something called as a nimisha which has something to do with twenty-four. That's why you eat twenty-four.  You must be sitting right there. Not simply sitting, you must chew. To every mouthful, you choose twenty-four times. One, two, three, four, chew it twenty-four times. Now you will see you will take more time than anybody else, but your food is pre-digested in your mouth before it goes in. It will not cause dullness. If you chew your food sufficiently, you will see it will not… food will not cause dullness if it is masticated in the mouth region. Then morning you will wake up at three-thirty. What do I do, Sadhguru? Just what you ask, merit it. You will feel hungry at three o'clock, three-thirty. You are up, drink one glass of water and sit. If you feel drowsy even with water, it's all right, still there's time for five-thirty. You can doze a bit.  You will survive till eight o'clock in the sadhana time without sleeping.  Chew it well. If you chew it well, the digestive process, one-and-a-half, two hours you will be hungry and that's a way to stay. Hunger does not mean you have to… if stomach is empty, does not mean you have to put food, you just have to put water. And you will stay alert, you will be very energetic, you will not lose any weight because your system will learn to use the food that you've eaten well rather than simply wasting it. It's both economically and ecologically very good for the world and very good for your health, probably you will never fall sick if you eat like that. So, twenty-four morsels of food, you must chew twenty-four times each morsel.  You will see you will do wonderfully well, you can meditate.


The Oregon State Police Command has assured residents of the state of a better working relationship and protection of lives and property. The Commissioner of Police, Frank Mba, made this known while inaugurating a police post at Cal Jolla Community along Shagamo Expressway. Frank Mba says that police would always be available for the people and ensure transparency and efficiency in the discharge of their constitutional duties. He also commanded the donor of the police post and appealed to people on organization to continue to support the police. I think that policemen are committed to making their leaving peaceful and the environment secured. Members of the community must understand that police stations are not just for fighting crimes. When there are fire incidents in your community, the first person most people will call will be the police because oftentimes you don't even have the number of the firefighters. It is the police that will help you to link up with the firefighters. Perhaps when a child has fallen into the well, God forbid, bad things, it is the police you will call. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


This time, I'm going to convert 6.25 into the floating point format. I have this helpful box here that's divided up into each of the three parts that I'm going to need to fill in. So I'm going to figure out what those parts are, and then I will just be able to fill them into the box, and there will be my floating point representation. So I will start by converting my number into binary. So 5.625 is 5 point, and this is a half plus a 1 eighth. So I've got a half bit, no quarter bit, but a 1 eighth bit. So there's 5.625 in binary. Next I need to represent this in normalized scientific notation. So I'm going to want to move my binary point two places to the left. This will give me 1.01101 times 2 squared. I moved it two places to the left, so my exponent is a 2. Now I get to fill in all of these fields. So 5.65 is a positive number, so my sine bit is a 0. Next I need an exponent. My exponent is 2, but I have a bias of 127. So 2 plus 127 is 129, and in binary that will look like 128 plus 1. So I will copy those 8 bits into my exponent. And last I need my mantissa. For my mantissa, I will just copy these 5 bits in. Again, I don't need my leading 1 because my number is in normalized scientific notation, so I know that I always have a leading 1. Since I already know what this number is, there's no reason to write it down. Now I need a 32 bit number, so I'm going to need to fill in the right-hand side with 0s until I've got 32 bits. I have 23 bits for my mantissa. I've used 5 of them so far, which means I have 18 more to go. So I will just add 18 0s to the right-hand side. And there is 5.65 in the floating point format. Now I might want to also convert this to a hexadecimal representation because that might make it easier to read, to talk about, in which case I'm going to look for blocks of 4 and just kind of read those off. So here I have 4, 0, 8 plus 3 is 11, which is B, and then 4, and then 4 blocks of 0s. And I'll put a 0x at the beginning to indicate that this is a hexadecimal representation. So either of these would be an acceptable form of the floating point representation for 5.625.


Adam Cuppey, I am a zealot, let it be known. I am a part of a company called Zeal. We're actually located in Southern Oregon, but we're actually, come January, we're actually opening up an office down here in San Diego, Southern California area, so if any of you Rubyists are in the area, please talk to me. So you can find us on our website, codingzeal.com, and of course my Twitter handle is Adam Cuppey as well, please follow. Just heckle from there, like not here, heckle from that point, you know what I'm saying. So I don't encourage you entirely to go get this now, but all of the content, because there's actually a lot of code that I'm gonna be presenting today, is going to be located at this URL on this GitHub. You can get it now. There are some modifications, they are intentional, so just be kind of smart, intelligent people and kind of acknowledge that there's a little bit of a difference. I subscribe to the Sandy Metz mentality of really small classes, so in this talk I'm gonna be doing things that are more consolidated in one, a much larger class, but I break it apart into its more subsequent parts in this repo, so again, don't maim me. If you didn't hear me the first time, follow me on Twitter, so please by all means heckle me from there, it's perfectly acceptable. And so this talk, so what I'm gonna be going over, again, this is the system architecture track, so this talk is on rapidly mapping APIs. Now in the talk description, I mentioned JSON and XML, XML is kind of like Latin, so I'm gonna be really focusing on JSON, but at the same time, the reason why I put it in there is that kind of one of the core fundamental ideas is that you should be able to adapt any data structure into the appropriate Ruby objects and start working with them, so again, I'm gonna be focusing a lot on JSON in this talk, but just know that XML and YAML, at the end of the talk, I'm gonna give some great examples of some tools you can utilize to interchange that schema into whatever you want. Oh, I forgot something, oh I totally forgot something, this is really important, I need to do a presenter selfie. I need you guys to scoot in over here, do it, scoot in, okay, ready? Oh wait, I need to do this, okay. Presenter selfie, come on, give me some energy, woo! Okay, that's all I needed, thank you, okay, moving on. Okay, so a couple of assumptions, didn't wanna miss that, seriously, a couple of assumptions that I'm going to be making about your skill level as a Ruby developer is that you're gonna be familiar with a good amount of the standard lib, and more specifically like decorator, simple decorator, the decorator and forwardable. I'm not gonna be hitting on them too heavily, but just know that that is kind of a major construct. And similarly, the decorator pattern, so if you're not familiar with the decorator programming pattern, I recommend you look it up online, it's a pretty popular pattern where you're delegating or you're assigning additional attributes onto an object kind of at runtime. So take a look at that, I'm gonna hit on that today, and if it goes over your head, Wikipedia has answers, maybe wrong ones, but it does have answers. So those are some assumptions. Now, to start this off, I'm gonna first talk about the three Cs, and one of them is specifically when we're talking about mapping an API is that we have to build adapters. So the first step is that our goal needs to be that we're gonna consume the data set of some kind. So if you're familiar with a standard, like we're dealing with Postgres or something along those lines and some sort of database or relational database system, you'd have something like ActiveRecord, which is going to hit the endpoint, it's gonna pull it all in, it's gonna build it into objects, and then you interact with those objects. We rarely ever parse the actual return coming back, which would be the equivalent of a terminal from any of these databases. So similar sort of things. So in this talk, we're gonna be going through the process of building out our own very small adapter to be able to connect to the resource itself. But the goal of that is to consume it. So the first thing we're gonna talk about is the connection itself, right? So this, the way to think about the connection is that it is just simply the relationship manager. So its intent is to, it's kind of like a lock and key, right? So it knows we're gonna have a real strong separation between our resource and then our application. And the furthest point to the resource is gonna be our connection. And again, this is our relationship manager. And what it holds is it holds the translation table, essentially, right? So the rest of our application doesn't need to know a single thing about how it gets the data, where it's getting it from, or any of that. If it's using URLs or internal connections, it is totally separated and it's totally isolated. This is a really important concept. So in the case of our application, we're gonna start with this very simple diagram, is we have a resource down below, and at the top we have our connection, maybe just connection RB or something along those lines, but we have a connection module. And its simple function is to handle the communication between the two. And in our case, we're gonna talk about HTTP and specifically get and post requests, okay? But again, this connection should be, hold that translation table. It should know all of those various components, how it connects, what it does when it connects, and all of that. The rest of your application should have no concern or awareness towards it, okay? This is a really kind of core component to it. So I'm gonna start with a use case here. And the use case is the Google Civic API, okay? So here's a very simple endpoint, okay? So in our endpoint, you'll notice that the first, the opening component is that we have just, you know, we have a scheme of some kind, right? So our protocol here is gonna be secure connection, right? HTTPS. We're gonna hit the Google APIs domain and specifically nested amongst that, we're gonna hit the Civic Info, right? Very simple, very basic. Now I also wanna point out that this is very common with APIs. When our consultancy builds out an API, we very much encourage the versioning that API for kind of obvious reasons. Again, with our module with our connection, our connection needs to be aware of that so that it can route to the appropriate places, right? It can handle that translation table appropriately, right? So I just wanna point out here that we're currently working with version one, all right? So building out the connection, we're just gonna start with a very, very simple connection class, right? So in this case, I would recommend, you're gonna notice right there, we're including HTT Party, okay? And this is a very simple library that will handle the basics of the HTTP connection. Very, very simple. You can use anything you like really. There's many fair days, very, very popular. I've used that a lot. A little bit more configurations required for it. That's the only reason why I'm not using it here. But in this situation, we're gonna have a basic connection class. We're gonna mix in this library. Now the other thing we're gonna do is, remember what I was talking about is, our connection object, its primary scope of awareness is towards those first two components, right? It's gonna concern itself with the version that it's on, and it's gonna concern itself with the route, right? Where am I gonna get the data and what version am I gonna be working with, right? The rest of our application reality doesn't care, right? It's like, I'm gonna call on this information, I want you to give it to me and I now wanna work with it. It's really that simple, right? So we're gonna isolate all of that into our connection object. Now, just for the sake of HTTP Party, you're gonna see that a base URL, we're gonna be passing that in. This is kind of a practice that I very much encourage, and that is that you utilize constants to set default values that you want, and specifically, and this is generally the case anyway, but you load those towards the top of the file. Part of that is, again, we're talking developer happiness here with Ruby and the ability for another engineer to jump in and jump on the project. So with that in mind, this gives a really strong sense of if I do nothing, then this is what I'll get, okay? If I modify nothing. Now, in our initializer, we're gonna set this API version into an instance variable. Now, this is gonna provide a couple of different things which are gonna be really helpful. One of them is we, at runtime, we can redefine this if we want to. So if we're talking about a lock and key, think of our connection sitting over here as the lock and key. That lock and key is gonna contain that routing. So let's say, for example, you have one connection that manages users, you have another connection that manages companies, and at some point, you're gonna consolidate that information down. Well, you, in theory, could have two different connection objects that are sitting on even different versions entirely that are pulling from different resources, and as a result, but at the end of the day, when we look at our application, our application doesn't know that and frankly doesn't care, okay? So it can be collecting all that data and working with it in one single container. Now, the only other thing I wanted to point out on this endpoint is that for this endpoint specifically, the authentication key or the API key is being done as a query param. That's not always the case. Many APIs will utilize headers. In either situation, the, these default values, specifically if it's headers, or if it's cookies or anything along those lines, they would also stay inside the connection manager as well. So those values would be set on that object and it would be handled that way, okay? So extending out our connection class a little bit more, you're gonna see that we've added in a default query and again, I'm a big fan of just default values to set expectation. So we're gonna say it's just an empty hash of parameters. We're gonna add in another instance variable into our initializer for query and similar to the API version, all we're gonna do is we're gonna say, okay, I want, if there is a query being passed into the initializer, use it, if not, use my defaults. That's the case with both. And then we're gonna add just a, it's not a full setter, but a method for query where we can bind in additional attributes into that query, okay? So it's going to persist our API key as an example or other default values that we may pass in and then we can just bind into that and they will persist through the entire connection, okay? So our implementation of this, our initialization and implementation will look something like this, right? So we have a connection object that we're gonna instantiate. We're gonna pass in an API version of two. Coincidentally, our API version is defaulted into two anyway, so it wouldn't really matter, but just for the sake of example. But again, because we have defaults, we could exclude that entirely and that's okay. It's gonna default. Then we've got a query that we're going to set in where we're gonna pass in our API key, right? And again, our connection object now, it's going to have all those values that are gonna persist and it's gonna be able to handle those through entirety, right? Now let's talk about the next component to this, right? So we have our connection that sits out here that's gonna call on our resource. It's gonna pull information via an HTTP request, but now we have to have some sort of routing and translation table between our application and the data that's retrieved, right? So the client object is often mistaken as the connection object where they are generally combined. It's my feeling that that's a bad practice because the lock and key is a really important concept. So as an example, or actually I'll come to that here in just a second. So our client object should represent the, as Gary Bernhard puts it, the imperative shell, right? So what it has is it contains all of the logic, effectively a controller, that knows how to take the request, make the request of the connection, retrieve the results, parse those results and hand it back to the application, right? It's the intermediary, right? So if you're familiar with Rails, you're talking about models and you're talking about views, right, and your controller sits in the middle. The client does the same, okay? So routing and collection is its primary function. If we just have a really simple diagram, it's something along those lines, right? So its function, again, is to stand between the application and the connection. So our application is gonna make a request to the client, client's gonna reference out to the connection and it's gonna come, the communication's gonna come back. So to bind these together, we're gonna set up a basic client object, right? So we've got two that are gonna be passed in. The first is the connection itself and the second is a routing table, okay? So we're just gonna have two instance variables that are sitting in there, right? Now, here's when we're talking about a routing table, this is what I'm talking about is we're talking about the scoping of the endpoint, right? Again, this may not be in the URL if we're talking about like HTTP bindings and so forth. This may be something that's a parameter that's passed through a form, just an additional value of some kind, but the net result is that we have a route here that is going to combine the two. Now, our connection doesn't care about what you wanna do with the data, it doesn't care at all. Our client doesn't necessarily care what you wanna do with the data either, but it understands that you want this data and I know how to get it for you, okay? That's the relationship. So if we were to take this and we were just to represent it here as just a straight hash, it may look something like this, right? So this is kind of the basics of our routing table, per se, is we have an elections key. This is the route that we're gonna define and we've got the HTTP method on that as a get request and then the path more specifically is elections, right? So this is our routing table. So if we were to put all of this together, it would look something like this, right? So we've established our connection, we've defined a bunch of routes and then we're going to pass those into our client. Now we've endowed our client with the ability to understand the correlation between our application and the endpoint, right? Now all we have to do is we have to tell our client what we want and how we want that information returned back to us, okay? So this is all the very basics of it, but you can expand this concept kind of to fit the needs, but again, I just wanna point out that there is clear separation between the two components. So we may expect something like this of our client, right? We would like to be able to say, okay, of the client, I wanna call the method elections and I want it to return back, something, right? Exactly that, right? So how do we get there, right? Now the simple answer would be to just say, okay, well, we're gonna define an elections object on it, but the problem is we don't run into a lot of flexibility, that's kind of the challenge, right? So a quick and rapid way of getting there is that we can take a look again at our route, we've got this key in there that's elections, we're gonna keep it dry, right? That's in there, and we've got our routing map. So what we could do is we could start by doing something like this. So we're gonna define method missing where the method is going to be the key, right? Elections, for example, okay? Now our routing map, which was passed in, which was passed in during initialization, and route map, you'll see there is the routes, and we're just gonna fetch that method, elections. Now we have a route map, and from that map we're gonna extract out the HTTP method in the path, and again we're gonna call on our connection object, and we're gonna pass those in. So again, so what's gonna happen is it would be the equivalent of saying connection.get, let's say we had a post, we could do connection.post, and then our relative path. Now because our connection object contains all the base properties, the domain and the version, none of that has to be passed in, okay? So again when we're calling on the client from the context of our application, it doesn't ever need to know any of that stuff. It doesn't need to know that we're on version two or version three. Now when you're talking about very large applications, so when we'll work on really large apps that have 1,000 classes in them and so forth, the amount of intertwining that you can find yourself into is substantial, right? Especially when it comes to versioning.  If you have to go and you have to touch find and replace and parse through 30 files, you got a problem. That's a smell that there's an issue, right? You should be able to isolate this thing down to one or two instances really, no more than four or five different instances that you have to adjust to make this all of a sudden work. So again when we're talking about versioning, which can be a real serious problem, we can handle that in a really beautiful way by separating the concern and isolating it down so that our client doesn't care, okay? Now from our, now that we've run this method, so again we're gonna run this guy, right? Client elections. Now when we get back because it's gonna hit our end point is we're gonna get a JSON payload, okay? Now it could be an XML payload or whatever, but in our example it's a JSON payload. An HTTP party, luckily, is gonna actually parse this immediately into a Ruby hash, the primitive, and this is actually what it will come back as, right? So we have some simple attributes here and then we have another collection that's in there called elections, and that collection is an array with more objects in it. In this case it only has one, but it could come back with 30, doesn't really matter, okay? And we've got three different attributes that are in there, an ID, a name, and an election day, okay? So we've got our data back, it's in a Ruby construct, it's in a hash, now we can start to work with it, right? Now the problem is this, that inner application by living in the world of a hash, and this is the case with most of the apps that I've seen, is that we'll keep it in the form of a hash, and that's how we'll interact with this payload, right? In a Ruby hash, right? Well here's the problem, right? The problem is that you lose a lot of support for dynamically adjusting for that data, right? So as an example, if I'm using brackets per se to access my hash, what does Ruby return if I use brackets and there is no key for those brackets? What's a return? Nill, right? There's few things I hate more than nil, and Kesha is only one of them, okay?  Well, so the goal should be, I wanna get it out of that construct, right? I wanna pull it into something that I can use, that I can add additional methods to, maybe some validations, you name it, right? But I can't easily do that. If I think that I'm gonna, if else my way through this whole thing, ain't gonna happen, okay? So some of the things that we could do, and I've seen this way too many times, is monkey patch hash, don't do that, please, please don't do that, right? Monkey patch hash, that's one. Or I've also seen decorating a hash, right? Where they take the hash, they throw it in, they decorate it, they wrap it, and they give it more functionality, which is okay, and I'm gonna do something similar to that, but there are better ways, people, okay? So the next of the Cs is to coerce, right? So we're gonna coerce this into another format that our application understands, right? So again, when we're talking about the separation of concerns here, right? We have a resource over here, which is just the raw dataset. We have our connection layer, which understands, it has awareness and understanding of that resource. We have a client that knows nothing about the resource, but it understands the connection and how to interact with the connection. Now we have our application that knows about the client and understands that it can make requests of the client. I want this information, please return it back to me, but the expectation should be that I want the same type of data, and so I have a phrase that I'd like to use and it's instance-oriented architecture. IOA, isn't that great? Thank you, trademarked it, people. Okay, so instance-oriented architecture, which it gets it out of primitives. Yes, Ruby deals, everything's a class, right? But it pulls it out of these primitive constructs like a hash and it moves it into more of like the straight dot, straight class instance architecture, right, so that we can make requests on it, we can add more methods to it, and we can decorate the object itself in ways that are more fitting to our application, okay? So I'm gonna introduce this idea and that is that you have a representation, right, and a representation is an entity mapping, okay? So again, we have this hash, we have this hash, right? So we have our application, oh, sorry, we have our application. There are a collection of representations that represent hash instances, and those hash instances, and the representations can communicate with the client as well, right? So if we're talking about our JavaScript, oh, not JavaScript, not JavaScript. Ooh, if we're talking about our JSON, excuse me, right, we would hope to be able to do something like this, that our representation can interchange between a Ruby instance and its JSON counterpart, right? Can anyone name the part of standard lib that has this fairly interchangeably? Open struct, right? So it's a very, very simple library, right, for the most part, but it gives this kind of baseline functionality, right? Now, you can use open struct, and it's not a bad one, it actually does really, really well. It tends to be a little slower than just working with a hash, and that tends to be a big criticism of it. However, if we're talking about our application architecture and specifically Ruby 3.0 that's coming out, right? All these performance issues are gonna go to the wayside, that's great, right? Okay, so, but I'm a big fan, one of the things I love about Ruby is just the fact that it's so easy to work with, right? I love the fact that it's not Java, that it's not C, right? All of these other languages have a ton of benefits, but one of the great benefits that Ruby has is the interchangeability between data types and structures. So I'm a big fan of getting it into the instance structure as quickly as possible, getting it out of its primitives for the most part, and getting into something that I can work with. Now, in addition to that, our application, the separation here is that our application should not have to concern itself with what the data looks like at the resource level. Again, we have the separation of these concerns. So our application should not have to go, oh yeah, that's right, I remember. The data structure changed totally. I gotta fix that, I'm gonna if else my way to solutions, okay? Should never have to do that, right? I should be able to call the same method on the object and get a result and get a common expectation. So we're gonna set up our representation. Right now, all we're gonna do just for naming, we're gonna say we're gonna require OpenStruct, we're gonna create a representation that simply inherits from OpenStruct, okay? Now our next chunk here, and I'll come back to the representation here in a second, but our next chunk here is that our goal needs to be to consolidate this data, right? So going back to our payload, our payload is gonna come back with all of this data. It's gonna come back with attributes and collections, and we need to be able to consolidate that data and we need to course that data into formats that our application can understand and work with, and at the same time as we wanna be able to consolidate that data into a data set that we can work with more easily and that our application expects. This is a part of good architecture, is the separation of these big, large components, right? So as your apps get bigger and bigger, or you need to be able to split these apart, the separate to segment these components even greater is not that big of a deal anymore because they're so separated already, right? So separation of concerns is really great at this point. So going back to the representation class, we're gonna introduce a few items here. So you'll notice that in our initializer, we're gonna pass two arguments in. One of them is going to be the representation and it's gonna default to just a blank hash, and we're also gonna pass an apparent and the default is nil. In other words, it could be the grandfather of all things, right? Now we're gonna run super on it, we're gonna pass in a representation. This is obviously gonna defer itself to open struct, which only takes one argument. So it's gonna build out the open struct representation, and then down below we've got four different declarations in here. Two of them are we're gonna define the attributes of the parent and the attributes of the child, which of course, we're gonna assign the parent to parent and for children, we're just gonna say that it's a blank array, okay? Pretty simple. Then we've got two methods, it's very large ice. We've got two methods in here. One of them is that we're gonna bind the parent, which I could show you the implementation, you could see it online, but it's very basic. Just essentially, it's going to check if the parent exists and if so, bind it in. And more specifically, make this representation the child of the parent. That's very basic, but again, you can see it, you can see it through that repo I showed you. Now here's the real hook in here is the method represent children and we're gonna pass in our hash representation, okay? Now this is where the rapidly mapping part, this is it. This is it, people, right now, right here, right now. Rapidly mapping, this is why it's rapid-ish, okay? So looking at these, a few of these methods, okay? The core of these methods are two different components. One of them is the represent method, the other one is the definition class for type method. Now walking through these separately and explaining them a little bit, the first thing we have, let's look at represent, and represent is simply going to take a type. In our case, it will be elections, okay? It's going to take a collection and it's going to take a parent for a later assignment. Now we're going to coerce that collection in the event that it's a single entity and that first line there into an array. So if you pass in a single entity, it'll just be an array of one. Then we're gonna map through the collection and for each item, we're gonna determine the definition, right? So more specifically, this is the kind of hook that we've got going right now, okay? So we're gonna pull a definition class and how we determine that is in our definition class for type method, which basically is going to convert the key to a string, a class string. So elections lowercase to election singular with an uppercase, okay? And we're going to look, our application is going to look for that entity type, okay? If it's initialized. Now if it's not initialized, or not, excuse me, not initialized, but defined, if it's not defined, then it's gonna throw a name error which we will catch and we are then going to create a new entity under that name type that is going to be representation, okay? So you see that down there, it says object const set, definition class, which would be election. We're gonna say we want it to model representation, okay? So now what we have is we have structures in place that are gonna map our hash, our JSON models, which are now hashes, and it's going to create entities that map to those. And it's all gonna do this very, very rapidly, okay? Now, I wanna point out a couple of things that when I wrote this, I was like, I'm gonna shoot myself a little bit, and that is one of the dangers in constant get and constant set is that you may overwrite things, okay? Or not overwrite things, excuse me, that you may define a class in a namespace that you don't want, that's okay. Well, it's not okay, but a way to get around that is in the definition class declaration there, you could specify a namescape, like a scope, or a subset, like you could say representations and then a class of representations is election or whatever it happens to be, okay? So again, it's going to create the class if it doesn't exist. So now what we can do, and this is really cool, this is where our separation is really strong, is now we can declare an election class that inherits from representation. We can do that now. And once we've done that, when we call representation.new and we pass in our hash, okay? It's gonna convert this, okay? Our raw hash to that, right? So now what we have is a representation. We have an attribute of kind. We have an elections collection, which also has collections inside of it that are elections that also have attributes of ID, name, and election day, right? So all of that's happening in that code that I wrote, okay? So it's a very, very rapid thing. And again, what we have the ability to do is we have the ability to define these classes and give them more functionality, right? So we're just simply mapping these together. So when we're talking about mutability, when we're talking about the ability to mutate over time, right, and the support of that, because again, our API may change. So in this use case, we're talking about the Google Civic API, for example. Well, I don't have control over that. And last time I checked, I can't really just get Google on the phone and say, hey, can you change that back, right? So when we're dealing with a lot of third party APIs and things that are outside of our control our code needs to be able to support this mutation and it needs to be able to do it in a really rapid manner. If you've ever dealt with a production application where you've had a bug show up, like it becomes like, I gotta solve this, right? Well, if you have to go through 30 files to make these adjustments to change it, it may literally equate to time and money, okay? And that may not be good. But by simply supporting this mutability throughout the application, we can very, very quickly deal with these changes. So as an example, let's say our data set for election changes slightly and now it adds in locations, okay? We have a new attribute called locations. Well, in our representation, we simply declare it. Now we can add address. And this will work in the reverse. So let's say for example, our application starts with an attribute called address, okay? And it's just a concatenation of these different values. But then the API changes, the payload changes. We no longer have address. Our application squawks and goes nuts and we now have to deal with that well. That now can be quickly addressed inside the representation. Nothing else has to change. It's gonna handle it, okay? So this is the rapid part. We like rapid, rapid be good, okay? Now here's another really cool thing, and I love this, is that now that we have classes and we have objects we can interact with, we can mix things into them, right? So what about something like that, right? We have an object now. So now with our, instead of dealing with a hash, where we've got a bunch of brackets and we're like, okay, now, okay, if election day is greater than tomorrow than this or okay, well now I need to build a JSON schema validator to make sure that the payload comes back right and it parses in the right things. Yeah, that's someone else's problem, right? Now what we can do is we can mix in the validations or any other libraries if we want and we can deal with that here, okay? And our objects are all the same. Now this was kind of the chunk that, regarding the JSON or the XML or YAML or whatever, I mean there's gonna be something else maybe, we'll find out, right? And so to speak to that, and it looks like I'm running quite early, which is very good for me, I guess. But with that, I wanna introduce a gem that I have found very, very helpful. I could go through the details of this, but it would take a lot of time. Representable is a really fantastic gem. Big fan of this one. There's a few others that are like it. One of the things that Representable does is it allows for, and you can read from the description, is it allows you to decorate representor modules, okay? To render to and from JSON, XML, or YAML, and you can change that parsing strategy however you like. So it's very interchangeable. So as an example, you could mix in, you could take Representable, you could mix it into our election model, and now we can, now that model is going to handle the various properties and benefits of Representable, okay, and those various representations, and the XML or whatever, right? So that really heavily addresses our flexibility. So, I wanna talk about just a couple, briefly a couple of points that if you have to make changes today, right? Things that you can do to really save yourself, save yourself a ton and catch a lot of errors, right? So what I wanna talk about is just very briefly, catch and release, right? So, be able to catch changes and release patches very, very quickly, okay? So again, just very briefly, here's our data set, our payload that comes back, and again, we've got the location model right there, okay? And you'll notice that locations, and specifically the address, only exists on one of the two records, okay? So that when we run something like this, okay, I want elections, the first one in the array, I want elections and address, it will come back with a value, but when we try and run it on the second one, I don't know if any of you have ever experienced this, you'll get this really happy thing, right? Raise your hand if you've ever seen this beautiful, beautiful, yes, yes, my people, yes, right? It's the biggest pain in the world, and the reason, as we know, for those of you who don't know, is because that is an issue, it does not exist, right? So, this is fairly basic, I highly recommend you look at it, but the very simple adjustment you can make, right? And I recommend everybody do it, and there's no functional change, is that, right? The reason is because now when you run Fetch on this, it's gonna do the same thing, it's gonna ask for that key, but the difference is going to be, you will get that back. So we'll tell you that there is a change, effectively a change in your data structure. And instead of the error presenting itself down the pike, down the pike, where you don't have the ability to catch those things, or who knows? I mean, if you've ever dealt with that last, the brackets method doesn't exist for nil, it's way down in the app, and it takes a lot of debugging. Well, this is now gonna pop up right when it's asked on. Now here's some of the things that people don't really commonly use, but Fetch can take arguments, right? So it was an example, let's say we're, again, we're looking at location, right? But what we're really wanting, is we're wanting address, right? Well, again, when it runs the second one, location doesn't exist. So address won't exist, right? So we have to, now again, we're gonna like, if else are way down to that, right? No good. So what we can do, so we're gonna do something like that, right? So now we can tell Fetch, I want you to now, when locations is called, if it doesn't exist, I want you to return back a new hash, and the hash is gonna have an address in it that says not determined. So now when I call Fetch address, right? From before, it's gonna get this value back. Now this is not a super long-term solution, okay? What this is, is catch and release. I'm gonna catch an error quickly, I'm gonna make a quick patch, get it into production so that I can solve my problem, then I can go back, I can work through my representations, I can make my modifications, and get those out the door. Okay, does that make sense? So, that ladies and germs is the end of this bad boy. I wanna thank you guys very much again, feel free to go back. I've got all of this plus the slide notes, I'll put that slide back. There's probably a faster way to get there. Okay, there it is, okay. So feel free to go here, all of the code I wrote plus some, that gives a lot more detail is there. I really appreciate it, again, my name's Adam from Zeal, love to stay in touch with you. If you see anybody that guys in, or people in red shirts, come up, have a good time, let's chat, it'd be great. I really do appreciate it. Thank you guys, have a really awesome, really cool.


QuickBooks Desktop 2024. Receive inventory bill, invoice customer, and receive payment. Get ready and some coffee because we're locking into some non-stop QuickBooks Desktop 2024. First, a word from our sponsor. Yeah, actually, we're sponsoring ourselves on this one because apparently the merchandisers, they don't want to be seen with us.  Like our, trust me, I'm an accountant product line. Yeah, it's paramount that you let people know that you're an accountant because apparently we're among the only ones equipped with the number crunching skills to answer society's current deep complex and nuanced questions. If you would like a commercial free experience, consider subscribing to our website at accountinginstruction.com or accountinginstruction.thinkific.com. Here we are in our QuickBooks Desktop sample company file. We set up in a prior presentation using the enterprise version of QuickBooks Desktop so we can practice the new unearned revenue feature within it. For the settings under the view tab, we currently have the hide icon bars selected and the open windows open. Home page open within the open windows, which you can do by going to the company drop down selecting the home page. We're going to open our reports as we do every time going to the reports drop down company and financial. Let's take a look at that balance sheet reports to start out with changing the range from 1231 to 7. And then I'm going to increase the size of the font by customizing it fonts and numbers and then I'm going to change the font.  OK, yes and OK. We do this every time. So I'm doing it fairly fast reports drop down company and financial. This time the P&L the profit and loss the income statement change in the range from 010127 to 123127 customizing the report fonts and the numbers. Change in the size of that font bringing it on up to 14. OK, yes and OK. Back to the home page. Now in prior presentations we discussed the issue with the unearned revenue where we basically get paid before we do the work to compare the new process to the older methods. We first want to look at the normal flow process, which is what we are doing at this point in time in the prior presentation. We entered an estimate and then we entered a sales order remembering that the estimate doesn't actually record anything. It's just basically saying to the customer this is what it would cost if we're going to pick up this job in our case. We were imagining that we're going to have a custom surfboard that we're going to order that has a crazy crazy psychedelic airbrush that they want or something. And then they had a sales order which kind of locks in the estimate, but we haven't completed it at that point. Then if we don't have the inventory on hand, then we might go to the purchase order, meaning we're going to go to our vendor to buy the custom surfboard that we're going to turn around and sell to the customer. And then we're going to have to enter the bill. And then also from the sales order, then we're going to create the invoice once we get the actual goods that we can sell, then we can receive the payment. Now we've been tracking this in terms of journal entries in our little worksheet over here as well. We did the estimate. We did the sales order. We did the purchase order. We had no journal entries related to them. Those are all internal documents. We can track those internal documents by going to the customer drop down and the customer center. And we set up our customer number one customer center. I have all my dates viewing over here, even though I'm working in the future and I could see my sales order and my estimate here for that customer. So they had an estimate and then the sales order under the vendor side vendor drop down vendor center. We had then our normal vendor over here. There's the purchase order. So now what we're going to do is we're going to do the next step. We're going to say, okay, purchase order happened. Now I'm going to pretend that we received the psychedelic surfboard. So we're going to hit the drop down over here and say we got a box with the board in it and it came with a bill. So the bill then if I enter the vendor, which was one normal and tab, it's going to say QuickBooks says open purchase order exists for this vendor. Do you want to receive against one or more of these orders? We're going to say, yes, we do. Thank you, QuickBooks. That's exactly what we want. And then I'll say this happens as of 01, let's say like 0527 because we're working in 2027. The amount is $100. That's the cost. That's not what we're going to sell them for. That doesn't include sales tax. It's just the cost side of stuff. So there we have it. And then down here, we can see that it populates on the items tab instead of the expenses tab because this is going to be an item that we purchased. There's the name looks good. There's the cost. Okay, what's this going to do when we record it? I can't quite see the buttons at the bottom. So I'm going to make this a little bit smaller so we could see that. When we record it, what will it do? It's going to record a bill. A bill is going to increase the accounts payable. So the AP is going to go up and then the other side is going to go to the inventory because we're purchasing inventory. Now I'm going to uncheck this billable item. I don't want to make it billable, meaning that would tie it out to the invoice because I could tie the invoice out to the sales order. And that's the process that I'm going to be using here. Otherwise, I'll have two things that I could make the invoice from. So I'm not going to use this billable component. I'll tie it out basically to the sales order. Now the other issue, of course, that we have is the sub ledgers and the sub ledger will be tracking by vendor, which will tie out to the accounts payable. We also have the sub ledger for inventory, which is going to track the fact that we have this one psychedelic surfboard, which we called the product number one on hand. So let's save it and close it and check that out. So we'll say save and close. And then I'm going to go into my balance sheet and I'm going to scroll down and say, okay, we should have an accounts payable down here somewhere in the liabilities. There's the accounts payable, double clicking on that, changing the range from 0101 to 7. There's our $100 bills. That makes sense. And then the other side of it should have going up to inventory. So if I go up to my inventory account, where is, there it is, inventory, double clicking on that, changing the first day, 0101 to 7. There's our bill increasing the inventory. I should have a sub ledger for inventory, reports, drop down, and then go into my inventory, inventory, inventory evaluation summary. Let's just do that. And this is going to be as of 1231 to 7. So there's our inventory that is on hand now. Here's our quantity and the total asset value is at 30,783.38. If I go to the balance sheet, we're at 30,783.38. Those sub ledgers are often the issues that get messed up. The sub ledgers for inventory, the sub ledgers for vendors are the things that we often forget about when we're thinking about it in terms of simply debits and credits, right? And then if I go back on over here and I track it from my vendor center, on the vendor center, now we have the purchase order. If I go into the purchase order, it has now received. We've received the purchase order. And then we have our bill. And of course we now need to pay the bill at some point. If I do that with debits and credits over here, let's just say this is going to be the bill for the inventory, what happened? Inventory is going to go up. I'm going to say equals the inventory from a debit and credit standpoint. And the other side is going to be going to accounts payable. And it was for $100, 100. I'm going to represent the credit with a negative 100. So debits are positive, credits are negative. If I record that over here, then we have inventory going up, inventory goes up, boom. And then accounts payable goes up in the credit direction. So that's what we have thus far. Debits and credits still in balance. And the next step would be we'd have to pay off the accounts payable with cash. We don't have any right now because I didn't put any on the books, but and this is just showing this one transaction doesn't have all the other detail that was in our balance sheet over here just to give an idea. So then if I go to the home page and we're going to say, so now I've got my inventory, I made my sales order. So now I want to, I have my psychedelic surfboard. I can create an invoice from it now. The way that would probably happen is we would go to the customer center and this person that ordered the crazy psychedelic surfboard would be like, I'm ready to get the surfboard. And so then we're going to go in here and go to the sales order. And I'll create now notice that you could create the invoice from either of these documents and estimate a sales order or even the bill if we custom ordered the bill that we created. That's why I made it unbillable. I'm going to make it here from the sales order, which might be a kind of normal type of process to complete the order from the sales order because maybe we don't have that inventory situation that we had to purchase the inventory for that customer. So I'm going to go here and say that we're going to create an invoice from it from the sales order. It says specify what includes on the sales order. We're going to order invoice for all the sales order. So everything on the sales order. So we'll say, okay, pulls it in. And so the date is once again going to be sometime in 01. Let's say 09 this time to seven. Boom, boom, boom, boom, boom. I should put terms of like net 30 or something. And then so there's our item pulls in and our detail down below. So what's this going to do? The invoice is actually fairly complex. And what it's actually recording here, we can see that it's got 1-888-58. That's the total amount that's going to be increasing the accounts receivable. And then the other side is going to go to sales, but the sales only going to be for the amount that we charged the 1-75, the 13-58 difference is going to be then the sales tax payable, which is a liability account. And we're going to have the cost of goods sold is going to go up an expense account as well as the inventory going down and the sub ledger for inventory is going to be going down and the sub ledger for the customers is going to go up is going to go up. So let's do that with a journal entry first. So let's say we have the invoice and voice. So what accounts are affected? Well, we know that accounts receivable is going to go up. The account receivable is going to go up for the amount plus the sales tax. Let's do that. I'll do that in a second. And then the sales or income is going to go up. Now the income we would know because we charged 1-75. So I'm going to say that's negative 1-75. It's a credit. Negative 1-75. And then we had sales tax payable. The government wants their piece, their protection money. Sales tax pay. You want to work around here? You want to be in business around here? Well, it's going to cost you. This is for your own good. So we'll say this is going to be equal to this times 0.75. Is that what they did? That times 0.075. So we come out to about 1356. Let's add some decimals here. Adding a couple decimals. We'll add some decimals now that they've made it complicated with decimals. Okay, you had to do that. And then the negative sum of this is the receivable amount. So that's one half of the journal entry. And then if we have inventory, we've got to deal with that whole thing, a whole nother thing to deal with. So now we're going to have the cost of good sold. Let's do it this way. Cost of good sold is going to go up. And then inventory is going to go down. And that was for $100, I believe, and a mountain that's not on this form. But we noted it was there and will be recorded on a perpetual basis because we turned on the inventory. So there's 100, negative 100. So let's record this now. Accounts receivable goes up. This equals that AR is going to go up. And then the income is going to go up. So on the income line, income equals that. It goes up in the credit direction. Sales tax payable, I have to add that. It's going to be a liability. I'll put it under the accounts payable. Select these. I'll just drag this down and say this is going to be formatted like that. And I'll call this sales tax payable. Starts with a zero copying down this formula. And that one is going to go up in the credit direction. And then we've got the that's done. And then the cost of good sold is going to go up down here. And inventory is going to go back down back down to zero on the inventory. Hopefully I got that that everything correct there. So so then I don't know why that one was showing as red. So I made it unread but this is so you if you can see now we now we have the accounts receivable that went up. Now the accounts receivable is part of the issue because I would also have to track this not just by debits and credits, but by the sub ledger who is those who owes us the money. The inventory also needed to be tracked up and down with sub ledgers as well as just these debits and credits. And and then we also have the net income down here. Income is a the credit is good minus the debits are the expenses. So the net of those is seventy five on the net income, which is of course the sales price that we sold the thing for less the cost that we that we spent for it. Right. So that's going to be the journal entry. Let's see if we can record that over here. So we'll save it and close it. You you've changed normal process terms. Yes, that's OK. And then we're going to go OK. So on the balance sheet, the accounts receivable should have gone up. So double click in the A.R. Change in the start date. Oh one oh one two seven to do to do. There it is one eight eight. Boom. Looks good. Mui B to the end being closing that out. Other sides on the income statement profit and loss P and L one seventy five. There's the one seventy five Mui B to the end. And then we said the sales tax balance sheet. There should be some kind of liability account might not be called sales tax. Payable. But something like that is something something like that. Yeah, sales tax payable is called that. Exactly. Oh one oh one two seven to do to do. And so that's going to be the 1356 1356 Mui B to the end. And then we've got the inventory should have gone down. Inventor E asset. Boom. Going in that from oh one oh one two seven to do to do. So it went up and then it went back down. So it's back down to zero or it's zero for us. It's back down to what it was for them. So it went back down back down to zero. Okay. And then we had the other side on the profit and loss. The P and L cost a good sold 100. And so there's the 100 net income 75 net income 75. Those are some cool beans. Cool beans. The beans are cool enough. You can take them out of the fridge now. Cool beans. Okay. So then of course the next step would be that we're going to receive a payment on it. So the next step would be and we probably would do that in the customer center. And we would say okay so there's the invoice now they're going to pay us. So so this is where we could we could go into the invoice here double click on it and say that I would like to get paid if if we would and we could go to the receive payment from here receive payment from this particular invoice normal process it automatically ticks it off here ties it out there to the invoice. This is the invoice. This is the payment. I'm going to imagine it's let's just say cash just for the sake of our practice problem from it's going to happen. Oh one let's say 1027 boom what's this going to do. It's going to then increase. It's going to go into unearned revenue. It's not going to get into the whole unearned revenue thing but basically a cash type of account. And then the other side is going to decrease the accounts receivable right. So if I did that with a journal entry over here we could say okay then this is the receive payment and what's going to happen. I'm just going to call it cash is going up even though it's going into unearned revenue because I'm just I'm not going to deal with that right now that's not where our point of focus is and then the other side is going to go and decrease the A to the R AR and so they and the amount is going to be equal to the amount from the invoice. And so now cash is going to go up.  So then so that's the so that's going to be the normal process but again tracking this sub ledger for AR becomes the issue. It's easy with debits and credits because you don't have the sub ledger. So then let's go ahead and record this. So we'll just save it and close it. And then we can check it out. So if I go to it's back to the invoice I'll close the invoice to the invoice shows is paid now. But if I go back to the balance sheet then it didn't go into my checking account but it should have gone into undeposited funds. Again that's a whole nother issue in of itself 010127 that I'm not going to get into right now. But the point is it's kind of like a cash type of account went up for the 188 right 188 boom. And then the other side closing that out went to the P went not to the P to the L went to the A to the R A to the R R 010127. And so there's the payment. So it went up and then it went back down. And of course we can track that on the sub ledgers with reports. So if I go to the reports drop down and we go to the customer receivables. Let's go to the customer balance detail report. And so now you've got the normal balance. It's going up. It's going down. We can see all the activity. It's very clear. It's very nice. Very tidy. If I go to the customer center over here we can see in the normal process that the whole we can see the whole thing that happened right. We've got the estimate. We got the sales order and and we've got the the invoice and the payment. We can then we could select and sort these by just invoices for example if I wanted to sort by invoices we could do that. I have to change the year to the whole year again. So for the whole year and then we could sort by open invoices right. That would be a common thing. There are none. And then we could say overdue invoices are all invoices. Okay. We get the picture. So that's going to be. That's the normal process. So that's going to be the normal process. Now we'll throw the wrench into the process where we get the payment before we do the invoice and we'll start off with the with the old way we used to do it which still might be useful in some cases and some scenarios and some bookkeeping systems and look at the pros and cons to the new case and the problem is going to be when we have the sub ledgers for this accounts receivable account. How do we track that if I then include another account down here which is a liability account this this customer balance detail report which ties out to the accounts receivable is going to be what are you going to do with that right. It's going to be messy. So which means we're going to get another report and it gets a little bit more complicated but makes more sense from a financial reporting purpose situation. So we'll get into that in the future presentations.