Good hello everybody Thank you for joining us this afternoon My name is Kyle Forster. I'm the founder of big switch networks The three of us are on stage here to talk about some of the lessons that we learned in a very large NFE deployment altogether Yeah, we've this was a this was a heck of a project joined by By Chris Emmons from Verizon and Redesh Balakrishnan from Red Hat We're gonna go through one by one and talk a bit about the lessons that we learned over the course of the the nine-month sprint For for the first phase of the project Thanks for joining us Chris, let me turn over to you to kick us off Hi everybody Welcome to the third day of the conference. I guess We'll we'll jump right in here. I'll start out by just you know, there's a lot of buzz words on this slide I Think what the point of this slide is really what we set out to do and that was to build the foundation of the next gen Network for Verizon so we wanted to have a Network that are to build a network that we could use across all of our locations all of our business units So not just wireless not just telecom not just our enterprise business But across everything to let to leverage that the economies of scale and the operational efficiencies that we can gain by doing that And we wanted it to evolve in a modern way with the pace of software and Everything that we're here at this conference to talk about so how can we leverage the the most modern technology to deliver the same Quality network the same reliability network, but in a more agile cheaper more efficient way So we have a multi-year plan in place kind of the second bullet here is about the multi-year plan We have to build out the network across Verizon and to then move all of our workloads We have a plan set up to move all of our workloads on to the infrastructure So that in the not too distant future will have a fully programmable network that will enable us to innovate and experiment and and heal the network and grow the network and all those kind of things in a much more In as much faster in a much more programmable way So this slide's kind of found the foundation for what we did So if you start at the bottom We say we want to we want to embark here on this on this project starting with commodity hardware and open source software So the benefits here are obvious right the commodity hardware We can drive some of the cost out of the network of the equipment that we purchase and install The open source software we also see an opportunity for cost there But more importantly we see an opportunity there to enable Competition and an innovation in a way that we never have in our network before so these were key concepts to everything We wanted to do Embarking on this project so and then the pillars on top of that foundation that we're that we're taking here is Starting on the right software defined networking virtualization and automation so my partners on the stage here that that helped us with the first princess project as Kyle explained Big switch helped us out with the the software defined networking component And then the red hat component for virtualization and then on the automation side Obviously a lot of work there going on internally as we automate our processes and our procedures and and get things more to that That software based model that we're looking for Importantly on the commodity based hardware piece. It was important again that we use that hardware and that We picked our vendors in such a way to maximize that so big switch Was one of the key vendors we had who was capable of running on the white box hardware that we selected for our switching Fabric and it was it was an immediate plug-and-play there They integrated great with the with the hardware and then also with the open stack software that we selected from red hat My other partner on the stage here We'll go through a little bit more of the details here and hopefully hopefully we'll have some some time for questions at the end so I thought I'd jump to some lessons learned before we talk about the kind of the timeline of the of the of the project so The left side of the slide here. We have a funny-looking airplane with the wings backwards and all that kind of stuff That's that's meant to symbolize kind of oops. What happened? There we go The the funny-looking airplane here is meant to symbolize kind of a different way of doing something right? So we're used to landing airplanes with wings that look normal and all that kind of stuff But when you start talking about moving our infrastructure to a cloud-based to software programmable to a self-healing all those things that We want to accomplish that looks very different than what we're used to doing in the network So so you think about that airplane and building that airplane is one challenge. Why does it keep doing that? Michelle is that you? So anyway funny-looking airplane trying to land it at an air at an airport that's not used to seeing that funny-looking airplane So we have a network that's tried and true We have we have processes and procedures and things we've done over the years that have that have built the most reliable network And we can't sacrifice that but we do want to try new things So the point of this slide is kind of you know prepare the runway for things that even you're not used to seeing funny-looking airplanes funny-looking Technologies different processes and procedures different ways of thinking You know you really got to adjust yourself to be prepared to take something like this on and that's kind of what the right Side of the slide indicates the people and that make all this happen. They have to be prepared for this as well They have to learn the new technologies have to be willing to challenge the status quo They have to be willing to look at things differently like yes We don't want to sacrifice who we are or or how we got where we are or or the services that we provide to our customers But maybe there's better ways to do that and you have to be willing to kind of step outside of your comfort zone and and take that on So now I really want to move on to this slide as opposed to whatever was going on here earlier. It must be this Mac Anyway, so from project inception this slide's a little bit deceiving because it kind of indicates that we started in March of 2015 March of 2015 was when we kicked off in earnest with this particular deployment there was a lot of foundational and and other work that went on well before this lots of trials lots of work lots of Lab work lots of POCs that kind of stuff just like a lot of you in the room have probably done But by March of last year. We said look we think this technology is really really getting to the point where we can deploy it And really start to use it in production. So that's kind of the March date is alright guys We've done enough playing around. Let's go figure out what we're going to build and deploy it So roll through a couple of months of kind of final technology evaluations looking at different vendors looking at different options You know doing some final POCs that kind of thing just to make sure that we were sure certain of our selections and how we wanted to move forward Took us into the you know the June-July time frame where we we actually locked down the design built it in the lab And then did all of our testing so this is our reliability testing our destructive testing our load testing all those kind of Things that we did in the lab. I think Kyle's going to talk a little bit later About some chaos monkey type stuff I mean we really went to great extents to abuse this thing and make sure that it was going to work the way that we wanted it to work When we got it deployed in production So then August to November August to November we actually deployed the hardware to the five data centers as was discussed in the press release read we released this week and And we had those five data centers ready by the end of 2015. So we're now full-on into You know getting workloads migrated into those data centers And actually transforming our network so kind of a parallel line to all this that you don't see is the work We were doing on the applications in parallel. So working on virtualizing the applications getting them to run in our environment making sure the network Requirements were satisfied all those kind of things. So All together, you know a very aggressive timeline a great team of people we put together our partners and The teams within Verizon many of you sitting in the room here. Thank you for your contributions But you know we moved at light speed here, and then we're really moving the network along. So with that I think I'll turn it over to Thanks, Chris. Good afternoon everybody I want to do three things or touch upon three things touch a little bit about what's red ads Approach when it comes to NFE and what's our offering and then quickly go to talk about the nine month of frenetic pace That Chris Simmons and his world-class team set on ourselves and what pay the learnings at a high level and then Last but not the least from a forward-looking perspective What are the areas that we are working on to make sure that we continue to delight Chris and his team as well as have an impact together so our approach is Upstream by default or upstream first Now if you had to pull a random red-ad engineer and ask them why are you doing that? They might say that's the only thing we know or that's the only way if you have ever done Software it's important to ask and understand why we do what we do Our approach as a product company is to make sure that we don't create any kind of lock-in Which some of the services organizations tend to create? Where in you know some of the customers have told me a six by six by six model six consultants for six months and Rinse and repeat the same thing six months later Whereas our approach is to make sure that every bit of code that we write is available upstream The second Thing to ask is okay. What is the value of doing that the value of doing that is making sure that there's no vendor locking because at Any point in time the source code is available in fact We do our best to make sure that it's documented exactly where it's available as well But most importantly as a product company it helps to make sure that maintenance And being able to support Implementations across the globe is can be easily done because if every customer is going to have a different forked implementation Clearly we can't scale to be able to address that because we're not in the services business The flip side of the question is okay. What's the risk of? Forking probably the biggest thing is the fact that you end up having technical debt because every customer was stuck in an island of Proprietary fork is going to have a technical debt Especially in a place like OpenStack where the pace of innovation thanks to every one of you who's in the part of the community Helping us move the needle on a daily basis. You don't want to get stuck in that island So that in a nutshell is our approach in general for any Problem space when it comes, you know from a product perspective particularly so when it comes to NFV and OpenStack as well What is our offering in this space? There's a lot of chatter about OpenStack being an integration engine We you know if you think of it as an engine we believe it as more as an engine plus steering wheel and the you know The wheels that are required to actually run a car if you will just in my attempt at real-time to give an analogy, but The reality to realize the important thing to realize is that we are fundamentally Co-engineering with Renat Enterprise Linux KVM as well as OpenStack a solution that a customer can bring in into their environment Continue to live with it as long as a lifecycle of the product currently at three years With a piece of mind that you know Somebody is there to watch out for Upgrades and patches along the way now The important thing to realize is that from an NFV perspective the innovation is not just at the OpenStack layer In fact the interdependency of innovation across all layers from Linux to KVM as well as in partnership with Intel DPDK LibWord from a KVM virtualization perspective all these are moving at different paces at any given point in time Our job as a product company is to make sure that we are taking the customer input driving the innovation at the same time Stabilizing and giving an experience that a customer such as Verizon can deploy and live with on an ongoing basis The second aspect I wanted to touch upon was the lessons learned from the project itself I You know I'm a big fan of number three so three areas that we got feedback around one is a set of core capabilities that we wanted to make sure are Available from a deployment perspective so clearly there were Tens of features and I think Kyle will talk a little bit about the kind of the Increase in the number of features and expectation that were placed on us as we went through the journey The attempt here is to at least characterize the top three areas where we had to actually do something To make sure that we are meeting the project needs so IPv6 support Exists from a you know open-stack Perspective, but we had to make sure that from a deployment and management perspective its surface in a way That's easy to deploy and manage the second area was around SSL support from endpoint SSL endpoint support again Making sure that director which is our deployment technology can make it easy to roll out SSL Implementation was another area that we had to work on in a truly upstream aligned manner and make sure it's in the product as well High availability. This is a critical area because you know in this NFV space. There is a lot of foot around Quote-unquote carrier grade without even a standard definition of what that actually means The good news is that you know between Verizon OS and big switch we agreed on the fact that it's not about some carrier grade is auto return but it's all about Reliability availability and service ability of the infrastructure. That's important, right? So that being the focus availability clearly became a key area focus for us. Our Implementation is fairly straightforward with pacemaker Corosink HAProxy implementation and we are to also make sure that you know we can stand up a three-note cluster With high availability from a controller perspective We also have the option to turn on Instance HAPut, you know from the deployment perspective. We're not there yet, but just to highlight the fact that The feedback that we got during the project actually helped shape the product the fact that we are also Leading contributors be it at Corosink level HAProxy level As well as pacemaker does help the fact that we can react to the demand from a customer perspective And make sure that we're doing it in an upstream aligned manner. So we feel very good about having met the needs here Two more areas of Feedback is one is around partner integration itself. So, you know big switch was you know after a lot of trial and Successful proof of concept was chosen as the SDN Provider of choice now the reality is that this opportunity was the one actually made us work closely together at two levels initially It was about how do we certify and make sure that big switch solution is Coexisting peacefully with our open stack platform right at open stack platform offering The next step that you know, we were challenged to was how do can you make sure that the? serviceability of the implementation Is at a level where the deployment of our open stack the deployment tool director can actually take care of Integration at a much tighter level so that it's not a two parallel implementations, but one seamless implementation experience as you scale So definitely this meant that in addition to engaging with Verizon We also work closely with big switch and you know Kailin as a world-class team where super energetic in terms of engaging with our engineering team as well as product management to not just make sure that We have a solution for Verizon, but also think through What it takes to sustain this roadmap alignment on an ongoing basis, so that was another Exciting outcome out of this process and last but not the least in this third area is scale This is an area where we're actually as we speak We are spending some cycles at a Dell lab to start You know testing scales of you know hundreds of nodes not that every implementation is going to hit, you know hundreds or thousands of nodes, but You know in an environment where if it's a conceptual maximum that we are talking about open stack scale You know that it doesn't actually behave very well for the kind of the pushing the limit on where technology needs to be So what we want to focus on is not only just test and prove that we can get to You know multiple hundreds of nodes, but also the serviceability as well as the performance that you get from multi hundred node Implementation it continues to be meeting the SLA requirements as well as customer requirements So this would be an ongoing area that we would be investing as well Now I kind of touched upon our Learnings as we were in the project the third aspect. I want to touch upon is forward-looking. What are the areas that we are focused on I Could synthesize this into maybe three points first one is There's the classical put the genie back in the bottle Requirement around hyperconverged infrastructure with Open stack and Ceph by that I mean today. We are at a point where you know seven nodes is the minimum footprint required to have a You know well architected implementation for Ceph and open stack How do we make sure that we you know shrink it down because the reality from a Verizon perspective the requirement is that They you know, it's not just the data center. There are you know multiple, you know, there is Corona corona light and also light in terms of scope of and size of deployments that are being considered So how do we make sure that? technically we can bring these elements together as an area that we are invested on and Especially an area that we are focused on from a Newton perspective now that metaka is Out the door already The other areas around split stack or a composable role here you know the the driver is to make sure that there is you know isolation from a role perspective so that maintenance dr Can be accomplished successfully So think of it as you want to scale out the controllers by a given specific role or a service if you will so that's the second direction that we are focused on as well and a third area has to do with the fact that there are Different kinds of workloads that are going to be placed Which require different kind of underlying infrastructure with bare metal on one extreme to you know super small lightweight implementation with a minimal little or easier to implement high availability without pacemaker dependency etc. So In a nutshell these are all areas that we would not have normally thought about if we weren't engaged in this project and In line with our philosophy of stream aligned innovation We have taken all this feedback and we're working through the road map to make sure that come Newton time frame We are knocking these off the park as well as continuing to innovate on our NFE solution here And like I'd mentioned part of the solution or critical element of the solution was partnership with big switch So tell you more about that. Let me invite Kyle over Thanks, Redesh So I think to me one of the One of the big things that worked here is if you see the context and I think this was especially like last summer was really when this was to me most most visible You have a team with a big vision, right? I mean Chris laid out for his team a big challenge and Said, you know, it's time to rise up and meet it with a whole series of new technologies and a new kind of cultural approach So it in that context being able to grasp on to a few familiar things Makes a big difference It's my perception that one of the things that really worked over the course this engagement We have this metaphor called one big switch if you're a networking professional and you're familiar with Logging into a big chassis Well, everything that you used to do on a soup in this model you do on the STN controllers Whether it's CLI in or BGP out. This is a very comfortable model Everything that you're accustomed to doing on the chassis backplane ie nothing That's what happens on the spine switches. You don't configure a spine switch You don't add a protocol on a spine switch in this model. It acts like a chassis backplane Everything that you're familiar with on a line card in this model. That's how the leaf switches act You don't have separate software images on lease switches you plug them in and they update from the soup You don't have to configure protocols so that line cards can talk to each other So being able to draw on this very familiar and comfortable metaphor in my view is a lot of Explains a lot of how we were able to accelerate very rapidly with the team make it very intuitive and easy for everybody to understand When they're swallowing, you know STN software open stack NFV Instances and planning coming down the pipe and all of this going on on bare metal hardware I think one of the neat things that's very specific to NFV. NFV stresses network designs like you wouldn't believe It puts pressure on L2 L3 designs in ways that it's our belief that only SDN can solve It puts pressure on network designs That make this type of model very very interesting so when we have In this model and we have an integrated v-switch when the v-switch plugs in the v-switch acts like the line card and The leaf switches to which it's attached begin acting like the chassis backplane this we call P plus V integration for NFV simplifies so many of the really really hard routing and switching problems that show up in NFV That this to me was one of the big reasons why we were able to accelerate through the project the way that we did Chris mentioned resiliency one of the things we did in in preparation was this test this chaos monkey test barred from the Netflix team and The I would say that several of the teams on your end put us through tests that were even more extreme than the chaos monkey testing We could do in our labs So this we load up on a full pod with about 48,000 VMs a mix of Hadoop workers and traffic jam Shoot a controller every 30 seconds to shoot a random switch every 8 seconds shoot a random link every 4 seconds and Every software image that ever went into the Verizon lab would pass this test for half an hour before we would send the software image over I Mean 640 component failures in about half an hour There is no spanning tree network in the world that can withstand this kind of beating and It's our belief for NFV This is the new bar. I think when we talk about a culture chain there's this culture change of the Yeah, kind of the I'd say the the old approach to HA In the communication provider world And I think under under this team's leadership. This was a very very different approach This was a much more Google style approach. This was an approach that says hey a component can fail any time And it doesn't matter that much The only thing that matters is that the system as a whole stays up and it's a different cultural way of thinking and it's incredibly important and It was so much fun going through this cultural transition with this team that I was a at least for me one of the really exciting parts of the project You know I Think as Redesh mentioned we started with kind of five really key criteria that were really good fit and literally by three months in had a Had a long list of the requirements, you know, this was I think a snapshot from June of last year of Here are the set of requirements that we need in order to really deliver NFV into full production I think somebody might have gotten a photo of my calendar there I'm not gonna go through all this but I think one one that I would highlight This was a problem that we had solved before the engagement, but it became a really really interesting one How do you how do you integrate the the v-switch and the physical switch? right there's a huge amount of operational and Frankly packet performance optimizations you can get there do and hardware what hardware is really good at and doing v-switch software Would v-switch software is really good at? And what we found over the course of really loading this thing up and doing tests for NFV traffic mixes and NFV operational models Was that this integration goes from being you know an interesting thing to an absolute must have It's the only way that you can get both the performance and the operational flex that you need for NFV You know So that I think that we've made sort of progress on before we started the before we started the engagement It was one of the hypotheses that we had come in and improved it over the course of engagement But one new thing that we really discovered let me just point out the second bullet This was a really interesting one. You know when you sit there in the lab and you're trying to design product on paper You make all these assumptions about NFV workloads That like hey, there's one connection model and that one will always work Right and or there's like some best way to connect an NFV workload into a NFV infrastructure And that turns out to really be not true and frankly that's not really in anybody's control There are a lot of different NFV vendors. They're extremely heterogeneous they have very heterogeneous views on what their own performance requirements are and very heterogeneous legacy code bases That they're building on and we have to you know the law physics of engineering here apply, you know people are On the NFV side for the most part building from legacy code bases. So just using open v-switch is Great for some terrible for others For us our switch light vx our switch light v-switch just using that great for some workloads not enough for others Just using single attach sri ov Worked for a very small number of workloads. It didn't work for a very very large number dual attach sri ov That creates a very interesting and hairy set of problems on the physical network But it's the only option for some of the NFV's out. So something the VNF's out there And then last a lot of DP DPK work that we're doing, you know that's a very very interesting one, but it's certainly not going to cover the entire galaxy of NFV is here in the short run so if I'd kind of point out two on the slide, I'd say the you know The technology for the the P and the V is is an absolute must-have and that creates an operational Silo challenge on a silo busting challenge, which is a management challenge. It's really hard And the one size fits all it was a for us a big lesson learned that was not obvious on paper not obvious Not obvious day one until we really started doing a lot of the heavier VNF work So me and there and I think we still have a tiny bit of time for questions. Thank you Could you comment on your date to monitoring? Logging system and what what do you plan to do when you have to do upgrade to new turn or whatnot? What's your plan? So We've got our initial set you ask about day two so an initial set of tools kind of kind of very basic rudimentary to get us up and running and then You know Keeping with the open stack or the open source Themes here we're looking at what are the best and brightest opportunities to do the monitoring going forward the monitoring the whole service Assurance package is a challenge and how that we get the information out of the infrastructure and process it and react to Invents in infrastructural. So I would say from a day to perspective. We're still Evaluating and picking those tools and same goes for update and upgrades Update update and upgrades to which part open stack services. Let's see the open stack services Yeah, so we've been through in our lab environments now. We've been through many upgrades The production part. We're still working through our first set of upgrades there. Thank you If I could add to that from an open stack perspective We just released a open style platform version eight last week And that's the first version that has got in place upgrade version version to version So seven to eight and then eight to nine when nine becomes available The challenge there is that we even gotten confidence that there'll be API stability beyond one release So this is an area that you know, we're looking very actively to make sure that At some point in time we can get to end to end plus two as well. We're not there yet technology wise Thank you. I think on the dead on the fabric side We put in something very early I mean this is almost three years ago now that you can actually upgrade the whole fabric and about the same amount of time as it Takes to upgrade an iPhone and The downside it's like a really cool thing the downside of that is people actually use it a lot So you wind up getting these like very the speed of upgrade and I think this is going to be true across these infrastructures On the in the vendor community. We have to plan for speed of upgrade being much much much faster I'm incredibly excited about the the ROS bait side. I mean, that's We're creating the capability to for for people to be much more agile in this area Thank you It's really interesting case study and I could not imagine in just six to seven months time You guys have achieved so much one quick question DPI and GTP related scenarios is I think pretty complex to handle in virtualized scenario. So what was your learning on that? I'm gonna have to pass on that one. I don't know honestly I'm not the network guy. I'm the guy that runs the team with all the smart people on it So I don't have the answer to that off top my head Thank you, sir. One thing that so across some of the early performance tests You find a lot of the default open-stack stuff and kind of default L2L3 networking is very tied to TCP So we actually had to do GTP optimizations at the spine for GTP hashing at the leaf and even down at the V Switch to VNIC GTP hash Thank you. It's a very good presentation It's wondering which are the specific NFE use cases which are being deployed in production Is it when you're talking GTP? Is it really the EPC use case or are you looking at something else too? So I'm not going to comment specifically on what we're deploying right now That's that's not part of what we're ready to announce at this point But I will tell you that we are looking at our entire directional network. So that includes EPC it includes IMS it includes Volte includes all of those things So we will be solving all of those workloads in the coming weeks and months and years Thank you. I Think there are several sites for the open-stack site and how many controller sets a control of those sites I mean there Sure, I actually meant to comment that on in my section. So thank you for bringing that up we At each site right now, we have one control infrastructure We've deployed a pod approach So we're deploying individual pods at each location and then connecting that into the data center network So our intention is to deploy pods at at the scale. We can support today, which is about 14 racks and one pod That'll evolve to larger pods as we go But then we'll just replicate that pod infrastructure and each pod infrastructure will have its own open-stack control plane So in other words, it's each port kind of independent Independent at the controller level. Yes, but then we're using higher level Orchestration and coordination techniques to work across those pods. Yeah Okay. Thank you very much You mentioned that The data center is setting up us for your whole company Are you plan on is Verizon plan on moving in of its telecom services to this network? Telecom services, you mean the wire line component or you mean just in general are Yeah, so so again, we built one infrastructure here for Verizon as one company So we're gonna serve workloads for the entire company out of these locations. So those five locations are just the first five There's multiple locations that are coming after that and each location Each application will be evaluated for where it needs to be based on latency characteristics things like that We'll place those loads that where they best will function in the infrastructure and but we will serve all of the all of the business Units within Verizon from from this infrastructure Thanks everyone


Now, we'll do that more when we get into the section or course on bank feeds. We'll basically build our books in essence from the transactions from the bank. Although it's not a full-service accounting system, it totally makes sense to do that because it's easy to do in certain situations. So if you have all of your transactions that are just electronic transfers that you're paying for for the expense side, and if your deposit side or electronic or transactions from like gig work or something like that, you might be able to do your entire bookkeeping system just from getting the information directly from the bank. But there's going to be a lot of wrenches that get thrown into that system. Anything that's accrual based will kind of mess things up. So if I go back to the balance sheet, if you have to deal with accounts receivable, that's going to mess things up. If you have to deal with inventory, that's going to mess up your system. Prepaid insurance is going to mess up your system. The furniture and fixture often will mess up the system if you do accounts payable. It might mess things up. Sales tax also complicates the things as well. And the type of income that you're generating will also make it more difficult on the revenue side of things. We'll talk more about that when we get to the bank feed course or section. The other misconception I think people often have is thinking that the reconciling process is the bank reconciliation. So in other words, when I reconcile what happens is I go over here and then if we have the bank transactions on and we're pulling the information in from the bank, notice that that we can either be building our books from the bank transactions or we might be doing the entries on our side and then using the bank transactions to match to what we've already done, in which case the bank transactions would basically be helping us with the bank reconciliation process because now the bank transactions would just be a double check that we have entered the information into the system. Now when we go into the reconciliation here, what we would do and we'll do this in a future presentation where we would get started with the bank reconciliation and the reconciliation will allow us to basically tick and tie what's on our bank statement to what is on the books. And once we get the reconciliation down to zero, meaning we've tick and tied everything on the bank statement onto our books, then we're done with the bank reconciliation process. But oftentimes people kind of leave it there. They forget that the end result is actually a bank reconciliation report which will take the ending balance as of the point in time, in this case the end of January, and compare it to our book balance and show us the difference, which would be the outstanding checks and the outstanding deposits. So the general idea when we do the bank reconciliation process is we're just going to take each transaction that's on our bank statement and then tie it out to each transaction that is on our books. And if it's on the bank statement, it should be in our books unless there's an error on the bank side, which isn't typically the case. And if it's not, then we would add it to our books. And then if it's on our books, but it's not on the bank statement, it might not be an error because it might be an outstanding check or deposit that we know about, the bank doesn't know about. So that means that if we tick and tie everything on the bank statement onto our books, we might still have some things that are unticked that are on our books that aren't on the bank, and we can still reconcile because the reconciliation process will say there's a zero difference between the cleared balances. But the report that will be generated will show those unchecked amounts as the difference, the reconciling items. And so that's what we'll do basically in future presentations. We'll go through the bank reconciliation for the first two months. And we'll see that the first month in January is often a little bit more complex. And then once you have the first month done in February, it should be a little bit easier and we can do it more smoothly. So the bottom line is this, the bank reconciliations are huge. It's a huge internal control. Every company, big, small, should be doing the bank reconciliations. The bank feeds can help with the bank reconciliations. And if you're building your books entirely from the bank feeds, the bank feeds will make the reconciliation. The reconciliation will be easy, really easy to do. But the bank feeds don't exactly just remove the need to do reconciliations, unless maybe again, if you're doing the reports, the books directly all completely from the bank feeds. But even then, I would still do the reconciliation because it's just two clicks to do it. You're basically double checking that you haven't entered things twice or that you haven't entered something. But in any case, the ending balance should tie out exactly to what's on your books at any given time if that were the case. So the bank feeds, you'd still want to do the bank reconciliations. And when you're done with the bank reconciling process, note that there will be reports, bank reconciliation reports, which are really what the end result is. Everything that you didn't check off will generate a report showing the difference between what's on our books as of a point in time, the end of the month, typically, and what's on the bank statement as of a point in time. And if we get that done exactly, then we have confidence not only about the big ending balance, but about the entire, all the transactions in the system. And that's important to be able to explain to people because again, a lot of people just think you're just trying to reconcile cash because cash is important in and of itself. And it could be people could steal cash and whatnot. But like I said, you want to be able to say, hey, look, no, this is about the whole, the whole accounting system and giving a double check and internal control on the whole thing, the whole income statement. Basically, we're giving some assurance by doing this reconciliation process. If somebody has their books in QuickBooks and they've done proper bank reconciliation, meaning they haven't forced an adjustment in the bank reconciliation process, a plug, a journal entry, then my confidence on the bookkeeping has gone up hugely. Like I have way more confidence in the financial statements. You could still have errors. Things might not be posted to the right place. Could be transactions that are wrong, but you are way, way better than if you didn't do bank reconciliation and if you didn't use accounting software at all.


Everybody forced here ripping 2022 Donner's Optic baseball hobby 12 box full case number two pick your team style Sorry, I'm still waiting on the list of names and teams to load up. It's taking a minute here today almost got him She get the blue. All right. Here we go Adam L. Astros Braves Alexandros Guardians Nationals Andrew M. Dodgers Marlins Brian F. Orioles Cody G. D. Backs Dustin with the Royals Ed B Phillies Jason H. Cubs John V. White Sox Jonathan Q. Has the Red Sox Jorge pirates Ken S. Yankees Mark R. Reds Matthew S. Mariners Matthew B. Angels Cardinals Michael O Twins Patrick D. Rangers Roger P. Padres Giants Russell G. Brewers Scott T. Rays Scott W. Tigers Seth K. Blue Jays Timothy M. Rockies William Q. Has the Mets and the A's Thanks everybody for filling this one up Tend to go the next one. Let's get it filled Let's see some more gold vinyl Let's keep it going with the gold vinyls Get a Tony Pollard jersey. No, I don't think they Never get a running bag jersey for no Hello Red, white, blue, Jimenez White Sox to $1.99 You guys gonna have to go faster The pace I was going towards the end of last case the pace we're gonna have to go to D. Backs Otto Tyler Gilbert forever Not numbered a black stars Narvaez for the Brewers numbered to 150 black stars There's there's four hot boxes a case. There's two auto boxes in two parallel I didn't know I forgot about them. Tell me this doesn't look like blue Bieber. It's a 75 Yeah paint velocity stanton to 249 Yankees Hey, what's up, Brian Blue velocity yellow for the Brewers Orange Mookie number to 125 sounds like you're born to be on the gaming channel Tyler Gilbert a red Acunia Oh Yeah pink velocity strider for Atlanta to 249 Hello, Kyle Lewis red, white, blue, Jake Meyers to $1.99 Astros They were also a few games out of the playoffs so Muller for Atlanta on the auto there and Muller black stars that landed a 150 Like the twins were like just a couple games worse than the Red Sox or something and they got the fifth big the Red Sox stated 14 Got a hollow rated Ricky Curtis Terry twins Black this time Kyle Tucker to 25 for the Astros Dude, that's like that's what they keep signing all these pink velocities to hook you for the Mets to 249 And the Pirates got the first pick so it's basically mythical. Hello. You're done Astros. This is a hop house is Black stars Gonzales to 149 white Sox Basically two through four is gonna Orange Friedel Reds to 125 yeah, the pirates always take a top like ten guy but never never the Marcello wire Yeah, like last year they passed on lighter and Meyer Stanton a 76 Yankees Yeah, I think velocity black man to 249 Iraqis Bomb squad hollow Vladdie jr. A red-white blue diamond Kings Alcantra for the Marlins to 199 A photon Andre Jackson Dodgers a blue velocity Louise he'll Yankees to 99 playoffs Hollow Torkelson rated rookie a light blue or Carolina blue they might call it Xander Bogart's to 50 It's light blue and like select and stuff, but a pink velocity Eli Morgan Guardians to 249 You under couldn't care less there Which which team oh Oh, the pirates probably oh, yeah. Yeah. You don't even care the pirate owner refuses to spend one Lights out hollow who's rejected it like an hour later Like every media person was just like well Oh Instantly, but yeah the initial offer was like eight or nine years Dylan Coleman Royals optographs hollow Has to be the worst ownership in the league tied with that not even the A's because the A's make the playoffs every few years But like the pirates have to be the worst Yeah, they are Well, they develop pitching well everything else Yeah, exactly that Zach Gallin orange to 125 d-backs I mean, it was like one or two hitters that has like a great year Hollow rated rookie gray or rated prospect gray rod Orioles It seems to be paid. No, we'll send him to the mat. Dude. I remember when he was hollow Kevin Smith. No, he was traded to the Red Sox They traded Lester for a cesspit is Here's the Red Sox for half a season And yes, I believe he went straight to the men's That was that's such a bizarre trade from like 2014 or something the stranger trade was hollow India Reds Queens last year Soto hollow Red-white blue Harper to 199 Phillies Sabin or Perez bomb squad hollow. I want to see one of those Fort Knox answers this this case Which are gold hollow mythical out to they No way dude Oh my god, what if I don't know what I'm like, well guys, I don't know who's correct Nobody gets great credit Radar rookie auto Andre Jackson Dodgers If I don't whenever it gets mad ready rookie hollow great Deakman everyone's gonna get mad either way It's not hard to just know the exact number of hits itchero has in his entire career. No Japan Japan too. Yeah, it's like like over 4,000. It's like over it's almost 5,000 Yeah, it's outrageous hollow diamond King Cedric Mullins Or at least just like I Mean I'll probably know they're not gonna be like incredibly hard, but Are they Then it's like whoever has best connection The 76 Charlie Blackwood Rockies I got I got that $200 Royce Lewis to 125 twins Dari Maretta auto for the Reds elite series there, why is Carolina blue Detmer's angels To 50 And then pink velocity Merrifield Royals to one are to 249 I Didn't know I didn't try color to ask her Hernandez blue jays to 199 He got drafted by the mythical hollow Soto black stars will cicatreras cubs Number to 149 Yeah, another hot box here by the way guys, so we should have another parallel every pack Pink velocity AJ Alexi to 249 Rangers Strap by the sixes where his mom worked and I'm getting treated like an hour later to the sun I didn't know about that red bucks than number 260. It's kind of mess twins Yeah, that stinks That's the business hollow Scherzer Red white blue Marta to 249 Dbex Seth beer on the hollow auto Dbex I'm googling it guys. I don't know hang on Black stars vado to 149 Reds They're going quick they're gonna be gone by the end of the day I would think yeah Blue velocity Alcantara to 99 one of the Marlins Cy Young when I And Eli Morgan to 76 guardians A red white blue you stremskieted to 199 Giants Orange Jose Ramirez guardians diamond Kings to 125 a Black Connor Wong to 25 Red Sox one of 25 Pink velocity to 249 Devers Red Sox More Red Sox announced the the Bogart signing in the Devers extension Now it's it already I'll buy both our jerseys immediately Dustin Harris Rated Prospect hollow no longer not a black stars to 149 Cardinals He's not gonna he's got not gonna accept a one-year deal unless it was for like 60 million or something Photon Walker Bueller Short print not numbered a blue Barlow for the Rangers To 75 MLB PA would never agree to that hollow your non pink velocity Santheon for the Reds Football owners have a way too much power Red white blue Acevedo number to 199 But in baseball it's all guaranteed money, so it's like Yeah, but the guys want the longevity yeah injury protection Percentage wise it might be a percentage wise it might be Right but percentage wise it might Rainy rookie hollow hunter green 76 Cal Ripken Orioles, that's a cool one For you to be Chris sale just Constantly be heard from something Yeah, but hockey they play through everything yeah, they'll just like find ways like yeah, he has had a broken arm for two months Years later, they'll probably have to amputate it But he's he's good to play in the the semi-finals got read Denmark's auto angels Stitches he's he's back next period though hollow Cabrion Hayes Dude you think Tyler Huntley will beat the Steelers defense He did good in relief of an injured Lamar Jackson Joe Ryan rated rookie hollow. Yo is Brock perty the truth I have no idea. I do not know much about Brock perty I literally had never heard the guy's name of my life. I'm gonna have to now because everybody's gonna be going crazy for him It's he have stuff and products a few things. Yeah rated rookie hollow MJ Melendez rated prospect hollow and Jay Melendez I didn't know he was mr. Irrelevant You know that I did not know that rated rookie red white blue O'Neill cruise pirates to 199 I'm the only one that didn't apparently Garrett Cole Logan Gilbert Mariners auto here not numbered if you were to rank See bold rated rookie hollow As far as like hardest to play or just like toughest players. Yeah, I would say hockey is toughest. Yeah, mythical. Hello, Tatis It's hockey Rugby I would throw in there other sports. I don't know North American. It's got to be hockey football And then between baseball Baseball is tougher then Hello, Nolan, I were not a sucker at the bottom. No Pink velocity of Lattie jr. Number to 249. I mean if we count you see you could yeah UFC would be there Hockey's hockey over football then football second the rookies hollow wonder Franco But I mean skating and like playing on a field are different though. Yeah, I Feel like falling on ice hurts. I think it's easier to play with an injury and hockey. I'll say like Football yeah, so I'm not saying like football player sits out. They're not as tough. I just still think hockey players are I don't know they they gut it. They have to gut it out more. I think a black Luke Williams To 25 I wasn't saying tough isn't like difficult. I was saying tough is Jason Dominguez rated prospect hollow I was saying tough isn't like the players have to play through injury and not Kevin Smith to 49 a rated rookie auto blue jays Anyway, Jaren Duran lime green Andre Jackson rated rookie hollow bomb squad hollow Shohei Otani And O'Neill cruise it's 25 teal velocity auto that is sick That's nice. There you go pirates Four of 25. We're gonna mag that one. It's a nice card Hollow Detmer's rated rookie. I'll still I'll still go hockey Blocking blocking the puck and Just constantly I don't know there's just a different level of like their speed on the hits The rookies hollow Brandon Marsh. They play through more injuries and hockey not more but I I'll stay with hockey not that I Not that my opinion matters One Miller out for the year Torn ACL hollow Correa It's all an opinion on that subject. Oh, yeah, I'm Alina. I'll say the NFL is more dangerous Rated Rookie Pink Vlasi Tony Santee on Number 159 to 249. There's much more collisions in that sport, obviously mythical hollow setting Mullins Orioles They did not evaluate to They sent him into She got notification Boston Bogarts talks heating up. Let's go. Let's go. Come on Bogarts Come on Red Sox.  Yeah, because of the dolphins now they over corrected everything and now everyone comes out for everything Which I don't disagree with the dolphins really messed up, but you got Chris Bryant to 50 for the Rockies Andre Jackson lights out. Yeah, that's just a false statement Pink velocity Bregman numbered to 249 bold strategy. Come see how it works out Why would anybody sign Odell right now? Why but now Bruhan the rookies hollow for the rest especially the Cowboys The Cowboys don't need Odell They're fine Lars Newport of 49 on the Carolina blue velocity Cardinals were rated Rookie Otto They're not even saying he's fully healthy like Maybe someone could use him as like depth, but the Cowboys Yeah, pass could use Anything on offense, honestly Something right in rookie hollow Jake McCarthy So we had a receiver who could like get open and make patches Like come on man, you know what him and as mythical hollow Look at the best league. Look at the best teams in league. They all have a serious number neon for the Rangers Corey Seeger Short print parallel yeah, I don't think the coordinator would be really really great Even worse then like if we just got rid of Patricia didn't replace him We'd be off we'd be better off than having him at all That's how you know it's bad is like you could just leave the position void and you're probably better off That's how you know it's just terrible Greater prospect hollow Battenfield Odell the Philly Why would they do why would they do that? When Goddard comes back, I think they're their team is fine hollow Bueller Dodgers Yeah, I'm not saying I'm I agree the rumor might be there. I just don't know why teams would sign him a Votto Diamond Kings the 76 Like if he's not gonna be a hundred percent, then I don't know why go get him splash hollow Machado Maybe if you want like a number four receiver, I guess The patch won't go get him. So I'm not like saying that but I don't think the Eagles need I don't think the Cowboys are the eagle name But those teams need to stay healthy and get healthy honestly like that's what would be best for those two teams Five to go in the next case everybody Those five teams are the White Sox guardians Rockies Marlins and Rangers lights out Verlander First take her take just the L Duncan Charlie are all the podcast I mean the less I have to listen to Stephen a and Russo. I love Stephen a I do like Stephen a just because he's a meme But to 199 Shane boss raise I think he he's entertaining. I wouldn't go to him for like about advice outside of it Out of sight of NBA But he's he's hilarious Right a prospect hollow Royce Lewis twins. I think he does his job very well. Yeah He's on every ESPN show how much he needs to make more money. I don't know how much you're paying him But double it. Yeah, honestly with how much he does. He's on every Love Stephen a Stay off. I don't want to say it, but yeah Him in the cowboy hat do it all so good bomb squad hollow Mike Cowboy had the cigar Bobby Whit jr. Elite series out of Royals His sleeves he probably sleeps at that probably sleeps in Bristol You go Royals Bobby quit non-numbered Yeah, we're just talking about that T hop that definitely hurts them. I don't think it Don't know how much it really affects them. Honestly Hollow rated rookie Mike Bowman. I guess when you're competing with the the chiefs in Bengals for like top AFC spot I guess you kind of want everybody on deck Rookie's hollow Durand Who was on That one's out of pocket Yeah When Merrifield Pink lost you to He just said with a he does he's basically a Twitter commenter for that moment. Yeah Mythical hollow Mike trout Matt Manning's Huyger's rated rookie auto. He kind of was T hop That's a good way to put it You get a lead and then you just kind of have von Miller just do his edge rush thing and you know Get that fumble recovery rated rookie hollow boss You know their hot box here Come on hot box You still have at least one more auto hot box coming. I'm not sure about the parallels Ravage definitely be concerned about Lamar They need him So bad Matt brash Mariners on the rated rookie on card auto Hollow Brandon Lau Carolina blue boss another one of those to 50 for Tampa Bay No, there's schedule. I mean, it's not difficult. It's not super difficult, but it's not super easy. It's a lot of division games. I Guess no game for the Ravens is even easy with Lamar Jackson without him. It's gonna Hollow new bar rated rookie Hollow rated prospect Taj Bradley Jose Ramirez mythical hollow a tricky thing though, which is faster than you $5 to twitch and YouTube That's actually not a bad idea hollow bomb squad Aaron judge No Contreras Yeah, Ali Dela Cruz is in this he has autos and rated prospect inserts and he pages rated rookie hollow He's also brother, he's he's also in it. Yes to both the Dela Cruz is being in this There's another day the cruise in the posey series rated rookie auto astros Free us rated rookie hollow Brian Brian. Yeah, and then there's a there's one that's younger than Brian Exactly you want to make it to well be you can have our last name Bobby with junior black stars Number 98 of 149 Royals doing well man Dustin you had the the auto and now the black stars That was the case There's just not many people that that one word even applies to Mythical hollow Jose I know for a fact everybody in our chat would know that Sam probably talks about that every day. I think our chat might be pink velocity Brian Dela Cruz 1099 auto Marlins Yeah, our chat is smart like a like they don't miss anything either if I if I say the wrong team or like something like that Land based Pink velocity of Dulles cars 249 I think this is a hot box here It's not a hot box Rated rookie base auto seabold for the Red Sox Hey, there's a great prospect. I think he has ready rookie too. I'm not sure go with rated rookie. Hello. Yeah We just check that real quick He does have rated rookie and rated prospect interesting The rated prospect is an insert so it's not part of the base set. I think that's why I think Royce Lewis has it too I think he has both Yeah, the rated prospect parallels there's only gold black finite and gold vinyl other than that It's just this is hollow and base Gabriel Moreno He's overpaid for judge Comparatively, no, I think it's an overpay for for judging about her who signed up at that deal got pink velocity delayed to 49 Rockies auto just for someone that's You know the other side of 30. I think it's an overpay But I don't I don't think it's like crazy because in a few years like the Ford the 40 some odd million year is gonna be You know fine. Hello rated rookie Lidolo for the Reds. It'll be 50 mil before you know it Who's gonna sign the next monster contract probably Salvador Perez red white blue to 199 I would think depending on how the next year goes. So do yeah, I was just gonna ask one soda for agent Splash Durand. I mean the Vlad he's gonna get to that point kind of soon Hopefully Debra Mike Bowman hollow rated rookie Otto for the Orioles I was gonna be monster. It's gonna be a lot of money It'd be like trade Turner ish or like Machado ish rated rookie hollow Sun T on I think a little bit over overpay But if you want him, you kind of have to at this point baseball, right? You kind of have to overpay a bit mythical hollow Boba shed blue jays You just have to overpay for players in baseball now Connor Wong rated rookie out of Red Sox a lot of people I know it was it was a meme last night, but a lot of people generally thought he would silent Giants Like that was the belief a lot of places apparently So they were they were for real Yankees had the Riley Adams black stars number to 149 nationals Yeah, exactly they and like I Mean the Yankees fans love so much they they had to boo him in the playoffs so super marketable That's 62 home run season people get like the gavels and Give a whole section of your stadium devoted to him. Yep. Yeah, I mean I Think it's it's honestly more so like what happens if you don't sign him is much worse than like the benefits of signing him for the Yankees like Yeah Like you go into the offseason getting swept by the astros in the ALCS your best player leaves like It would be like bad times around there, man Think a lot. I think that had a lot to do with it Yeah, who knows if they have a successful year and like when the world series or something Oh, let's go Kyle. Yeah, they were the World Series way he signs his his price tag is higher. He probably didn't go back Then the Yankees probably feel like that. Okay, we did it like Or they just do what the Red Sox is so I give it everybody contracts everything when the World Series and it Ends up hurting them a lot rated prospect hollow Robert plays on a's Yeah, they got a produce in the playoffs now. They got to get some bats around him 49 Aaron Ashby ready to rookie out of Brewers I Can't just be judge and standing glaver in the lineup. It has to be like other people Getting on base and paint your ready to rookie hollow for the Astros Black stars Trevor story Red Sox to 149 Hollow glaver Torres, you might have you might have to You could put like a hint or something in it to like differentiate it, but I don't know what you would put you know what I mean First one then like sweet otherwise, like, you'll know the second try color max shirts or the 199 mats Bombs quite a hollow of laddie, you know There's lanes boy now, too You want an Alvarez mythical hollow flipping through cards fast today Andre Jackson to 150 rated rookie purple lotto Dodgers Hollow Roman Gonzalez What's up, Hammy, hello, Hamby time. Oh, it sounds like an injury Contra, let's see that next case Fortigo White Sox guardians Rockies and Rangers You know what it is, what is it? It's just like his face Super All I see is hello Let me see. Let me actually go over to YouTube real quick. Oh It hated because of language Yeah, top sedemption status pending after two months is normal. Yeah, I still have one from like five months Sometimes it takes a while and you can request like an alternate Or whatever, but I hear sometimes the alternates they send you aren't like as good So yeah, you just I'd wait a little while honestly Nick Allen rated a rookie red for the A's Number to 60. Yeah, I have one that I redeemed in Like May or June and nothing's happened with it, so I'm just gonna wait. It's not it's not a massive card But it's one that I want pink velocity Josh low rated rookie auto for Tampa Bay at a 49 Sometimes you can take while you know my redemption is this is Andrew Bogart's auto. So maybe once he signs, you'll sign Yeah, I definitely wait on the only especially since he's he was rookie this year. I'd wait on that one for sure Hollow Luis on hell rated prospect Purple rated rookie auto on hell Zerpa for the Royals Numbered to 100 Yeah, that's not out of the ordinary. Unfortunately, that's just how the redemption game is Hello, Brian Reynolds Since he's is cruise Dominican. I don't want to assume his Nationality Duranda to 49 red Sox He's also probably playing fall or winter ball somewhere Nick Prato Thank you, Hammy. He is playing winter ball actually I saw a video hit a ball like 500 feet It was so fun press the auto rated prospect auto for the Cubs to 25 nice hit Cubs Reggie press the auto guys. So tops calls him Reginald and Penini calls him Reggie Wonder where he goes by probably Reggie. I would assume Reggie. Yeah Orange Diamond Kings Giants Brandon Crawford to 125 Penini is a lot more involved with like the players Really? Yeah tops is like all the way he's all of these heel.  I bet it says You're gonna sign Blake. No, I actually signed my I signed Steven Right ready rookie auto Luis Frias. I guess it's because it's illegal. Yeah, if I'm signing a document I'm not gonna sign Blake. So it's just kind of what it's hollow Mike Shrout So I guess you could sign Blake if it's your signature Is there a log in I have no idea there is there a law against like what you go by if you sign that About what I go by is my legal name mythical. Hello, Nolan. I ran out. Oh, I'm sorry going by night. Oh And purple rated Ricky Otto's to pookie for the mess I think it was the 100 we've got a conversation going got cards flying Yeah, how box is coming to go and there's also a lot of hotboxes to be fair And it just blends together with a whole case of optic You just flip it through flip it through flipping through Sam's live at 5 p.m. Eastern guys lime green read Detmerce Got more group breaks tomorrow. We got a case of eminence going tomorrow as well. Oh Random hit style. Hello Mike Bowman. I just discovered that You know it does Yep black stars mani machado, I think penny confirmed it to 149 Padres should be sometime separate January Eminence soccer dude My opinion I like it more than the football one. Yeah Mythical Kyle Lewis. Hello. Oh, yeah, it's coming For ones you can try us to 150 rated Ricky purple auto pirates pirates hitting a lot today There's a dual auto booklet of Neymar and Pelle You know, it's gonna be on card. It's not actually Ricky orange 125 Luke Williams. I'm not too surprised at that. I just think that that card Splash Louise Robert, especially if Neymar passes in this tournament for Brazilian goals all time He's what one shy of tying the record Look at three matches do that's not easy rated Ricky auto AJ Alexi, especially since he's not like a big goal scorer Red white and blue Tyler O'Neill the 199 cardinals. Oh my god, I hope we get some If you want some squad hollow Vladdie redemption Say a Suzuki elite series signatures. Hello cubs There you go cubs blue velocity Bryce Harper number two ninety nine for the Phillies Poison rated prospect Tray Turner sign it was kind of quiet Arias a checklist called studs and it's It's like good players with a stud of their cleat. Oh my god pink velocity Corbin burns a 249 That's tell me that one is it's okay right there Relics then yeah, they have some of them. Some of them are relics. It's out to Grom. They're just only relics He's playing he might be the best fortune. He's player. He's so good He's so young and he looks like a kid out there. He's well at 22. Yeah Gorgeous ease man. They're not going back to run although they can't start him All right Redemption cubs say a Suzuki. I'm just gonna call out names and teams guys not the whole parallel numbering Pirates contrae us Mets up hooky D-backs Frias cubs presciato Royals Zerpa raise low Dodgers Jackson Brewers Ashby Red Sox Wong Orioles Bowman Rockies Velaid Red Sox seabold Marlins Dela Cruz a Bobby Whit black stars parallel that was number to 149 recap worthy for sure Astros Siri brash mariners Manning Tigers neon thing is your case. Hey Corey seager Rangers You got new bar cardinals sheets white socks blue jays Smith mariners Gilbert angels Detmer's D-backs beer Reds more at the Dodgers Jackson Royals Coleman Braves Muller D-backs Gilbert Elite series auto Bobby Whit jr. Not numbered for Kansas City and a steel velocity O'Neill cruise on-card auto to 25 that throwback design Thanks again, everybody. We've got another case coming up