Good morning, good evening, good day, wherever you happen to be when you're viewing this. I'm Greg Tucker, I'm the executive director of CSDMS and thrilled to welcome you to this CSDMS webinar. Today we are delighted to have Dr. Davide Vanzo from ETH Zurich in the computational fluid and morphodynamics group. I'm going to go ahead and just hand over the floor to Davide to present to you basement, a not so basic simulation environment for river process modeling one quick logistical note. I'm going to hold questions to the end. And then when we get to the question and answer phase will ask you either to use the raise hand feature and zoom or to type your question into the chat. So without further ado, Davide, the floor is yours. Thank you very much, Greg. And thank you for the CSDMS community for inviting me, given this webinar. And yes, in this hour or less than one hour, I'm happy and excited to describe and introduce a bit to you what is basement about. And I would like to start resolving the title directly. So this is the agenda for the meeting I would simply go through setting the tone and explains the type of physical processes we can simulate with basement. Present some interesting feature that sometimes are overlooked by user that we have this field feeling a set of basement tools that comes together with the software. I would like to present a bit what we are doing in terms of research with the software and finalize with some dissemination aspect of this project. Again, to resolve the title. Why is I call the presentation and not so basic simulation environment because actually basement, then the name was born as basic simulation environment in the years it grows and it got way more complicated than what we expect. So, as I mentioned, it started as let me find the is a pointer. It started as a numerical software for simulation of either more for dynamic processes in reverse and with the years and the development we add way more element. The initial idea and that it's still the main idea of the project is to provide powerful yet free and user friendly tool to facilitate practitioners students, as well as scientists in setting up and performing river modeling analysis. The project started in. 2002 at the laboratory of hydraulics hydrology hydrology and glaciology for what they at each series, and it was always develop in this Institute. What is today basement. It's more than just either more for dynamics. So we, it's a tool that can support research and application in different domains and here I highlight three. Three main components so either dynamics sediment transport and more for dynamics but also application towards what we call equidro leaks application and research so habitat and vegetation dynamics, just to make an example. Rolling back, I would like to present ourself we are the fall of the Institute the laboratory of hydraulics hydrology and glaciology, very old and historical Institute of each. We do research in several domains spanning from really hydraulic research and on air and flow. Air and water flows. We have a strong know how on hydropower, but also we, we, we do research in the domain of medical modeling for in different subject, but also we do experiment for example in eto hydraulics with a live fish in our lab. We conduct research with different infrastructure and methods, and I think one of our also recognized skill is the capability of also a group and join and link different approach and methods in our research. If we narrow down to the computational fluid and morphodynamic groups. We conducted based on the ply research in hydraulic and environmental engineer, and one of the main scope of the group it's also the development and maintenance of the basement project, but we do way more we have a we have a quite large group. funded, often by a research project, but we have some stable support in terms of software and research engineer. And the project itself is mainly funded by the Swiss federal office of environment and maintain and develop by our group, if you want to know a bit more. We have all this information regarding the development also in our web page. The project as I mentioned started in 2002 and it grows with the years we have, there were several milestones along this path. But as you see, we have modules and components appearing through the years. Today I would like to present you an overview of the version for that is the current version and presenting some of the more relevant modules of this version. Just to list some main features if you are interested in installing, let's say the software. Right now we have a version for both windows and Linux based operating system with the GUI but also command line interface. This is the general framework or structure of the code so we have a unified software and that is divided into main sub module. I want to highlight that I will during the presentation I will present some features of the base MD the multi domain version that host several sub modules, but also some of some features of the base. The HPC high performance performance computing version where that it's able enable to run on GPU cards. And of course, we are still developing several features of this modules. And it's not open source right now. So the software is a freeware is so you can download and use for free also for commercial purposes, but we have some some other tools connected to the project that are open source. This is for project and research activities and it's receiving growing attention in the last years, especially for from private companies but also public research institutes. If we now have a look to who is using basement of course our main, our main stake of users are from Switzerland but we have several users in the German speaking part of Europe, Austrian Germany and also in Italy. So, going to the more interesting part that just to set the tone which are the physical processes and scales we are investigating, we are sitting in the at the spatial and temporal scale of 2D and 1D more for dynamic process. What does it mean that we can we simulate processes on the temporal scale of some decades and some kilometers with, especially when we use the 1D version of the model up to event scale. So flood flood event scale processes that occur within days, hours or weeks in a more confined or limited spatial scale of some hundreds meter or some kilometers where we can use a way more accurate and detail approach to simulate river more for dynamic processes. When we talk about 1D version of basement, we have a cross section average. We use a cross section average approach so similar to a cross that it's a well known hydrodynamic model either more for dynamic model in the in the community. You can run of course extensive in terms of spatial extension flood risk analysis. And if we are talking about more for dynamic processes is suitable to to simulate and and evaluate slope adjustment or for example the response of river system to sediment injection or sediment deficit in terms of adjustment of the slope or modification of the crane size composition. So this is suitable for more large scale long scale application. There is an entire sector of application that is more suitable to be investigated with a 2D depth average approach. And here it's just to plot some some example of Swiss river where we have such morphological variability or the formation of some interesting morphological structures such as here with alternate bar where you need at least a 2D depth average approach to to to to investigate in an accurate way either more for dynamic processes that scale. And this is also the natural scale for many other processes spanning from more for dynamic one, but also to eco hydraulic processes and focusing on habitat dynamics for example or transport and fate of contaminants. And this is also often the scale where river restoration project take place. So, rich scale of some 100 meters. To understand a bit, I promise these are the only two slides with some some equations, but at the end of today basement solves a set of partial differential equation what we are solving is initial and boundary values problem where we evaluate in time conservation of some quantities. And then we have a set of multiple experimental closure relation to solve our problem in the easiest way I present here the one D some benante question these are equation for the conservation of the liquid phase and the momentum so to simulate the behavior of water in our in our physical domain. We solve this set of partial differential equation equation provided input boundary condition and some internal boundary conditions. We do the same for other quantities of interest so when we want to solve the evolution of the bottom so for the morphodynamic model, we make use of the external equation that simply is a partial differential equation, stating the conservation of the surface domain, and so on and so forth so at the end of the day, all the modules rely on on the solution of partial differential equation system, mostly of hyperbolic type so and then we have other module for the resolution of sediment sorting so multi get a multi grain simulation. We have a module for subsurface surface flow where we numerically integrate Richard's equation and so on and so forth. Why I presented this this brief overview of the governing equation just to say that this is the type of governing processes are linked to how we numerically solve them. And we solve this governing equation with a finite volume in a finite volume framework with explicit integration in time. This is, of course, pros and cons, and I have I add here other features that are relevant when simulating river processes, especially in alpine context. Why this it's important to remark because using explicit integration we are we are bounded by the choice of the integration time step. So, this is just to remark that when solving one the one the equation or one the approach we can have time step integration of scaling with with seconds, so we can easily run simulation for multiple years in with standard workstation. But bear in mind that when when we go with the 2d as as some rule we have integration time step of smaller than one second that it's it's can create a it's surely a barrier for long term simulation in 2d in a 2d framework. That also what motivated us to spend some quite some effort in design a GPU ready version of basement for for some application. So that's it with with the equations was just I think an important remark and and now I would like to give a brief overview of some interesting features that we have in basement that we feel sometimes are not well. Use or exploited by by user and one one feature is the possibility to coupling different type of domain. So, with basement. You can design and couple one d domain in in in a different type. So with one way coupling two way coupling and also one the one d domain with 2d domains and and so on, you can create actually and reproduce and simulate big network or entire basing scheme. basing networks. Having the advantage that whenever you have a simple channel that you can simulate with the one the approach you can can actually use the one diversion and save a lot of computational time and we have different strategies to coupling. Or you can also coupling as in this picture simulate the the main channel of your compound channel or confined channel with the one the approach and couple just the flood plain in case of your you want to simulate the natural hazard or or the or in case of flooding. Second feature is what we call base external that it's a data exchange protocol during runtime that allows for coupling between basement and an external program, for example, it was tested and use for the base sub so the simulation of sub flow. And you can, there are different ways you can exploit this this feature in one way so one way coupling so you provide input or you derived output from a basements for from a running basements simulation, or two way coupling, where you can during runtime, provide and fetch data to the basement simulation, even though I have to mention that it's this is experimental. So it requires still some coding. Then we have also a flow controller where you can control some variables. So you can manipulate some variables. And this is useful to simulate, for example, hydraulic structures are along your simulated river. And the last relevant feature is that we have a dedicated module for that runs on GPU cards. So graphical process unit cards, where we were able to obtain very high speed up values up to of of order of magnitude, higher than serial simulation. And what does it mean or what does it allows, for example, to account for uncertainty in flood modeling or other type of application so in this type of study we we use. We make use of that this accelerated version to run thousands of simulation of breaching hydrograph. And so, so we didn't simulate just one evaluate just one result but 3000 simulation simulation result. We build a probabilistic flow map so you can work towards the generation of probabilistic floods map directly by by by generating thousands of simulation output. And in this case we had the single simulation on a standard workstation with the GPU card. The single simulation competition time was of 36 seconds, and we run 3000 runs, and we evaluate that with the standard non GPU version of basement, the total computational time would have been 200 days, and it was at the end only 30 hours. So it really opens the door for for application that are not possible with standard machines a bit an overview. Now, I would like to mention not the basement software but some tools that we develop in order to create a user friendly workflow for for for the usage of the of the soft of basement. And the workflow for the usage of the software it's quite standard for for this type of environmental software, let's say, so you create. So you have a pre processing step where you have to prepare the numerical the computational mesh so having a topography data transforming some way in your computational mesh and then prepare some time series, you run your medical simulation, and then we have a post processing phase to help with that we have several tools. So for the pre processing, we have an in house develop plugin that works on QGIS it's an open source project I will mention briefly how it show briefly how it works we call it base mesh and help supporting the creation of the computational mesh. You can also run as a command line interface tool. It's the Python module you can work either way with it. And for the post processing. Again, you can visualize the results in QGIS via the mesh layer module that now it's in fully implemented in QGIS and I think it was originally and support develop and supported by crayfish company, but you can also use part of you, or some Python script that we provide to post process your data. When we talk about one day, either more for dynamic model model, we have one day grid editor, directly available from the GUI of the software that it's in some way similar to in a very basic form to to to to across geometrical handling of the cross section, let's say, or we have a script to generate some simple one day geometry from common by a common line interface that we call it base chain, or we have also a converter from as geometry file to basement geometry file with regards to 2D mesh to 2D computational mesh we have as anticipated base mesh that it's an open source plugin, and you can you can generate with a nice with a nice workflow and visual so it's, it's a via graphical user interface you can generate your computational mesh. It works well with relatively small meshes so let's say 100k smaller than 100k computational cell, if you have, if you have a really large application millions of cell. It's way more efficient to use the common line interface tool, but the under the hood, the, the approach for the generation of the computational mesh is actually the same. Just to mention some, I think, interesting features of base mesh, you can generate your mesh by interpolating elevation from multiple sources so if you have river cross section but also a DTM you can put together everything and use different elevation information to build your to build up your computation numerical computational mesh. Now it's not the base mesh feature but now there is a QGIS there is a native 3D visualization so you can build up your computational mesh and visualize in the 3D fashion straight after and check for quality, let's say. And also visualize the math ID so the different material index you assign to your mesh that are linked then to either more for dynamic parameters later on in the model setup so you can also color with identify with different color this material index. If we move to the post processing so once you have run your simulation you can easily visualize the results in a very, very convenient way with that back again with QGIS. There is this crayfish plug in that it's very convenient to visualize to the information but also trace to define cross section so you can visualize your result for example here we have this cross sectional profile of the velocity. And export the different type of map for visualization and presentation. I mentioned also part of you it's a more powerful tool let's say dedicated in general to computational fluid dynamics. Solvers, you can also directly read basement results there and manipulate a lot of variable do a lot of post processing and and so on and so forth or we provide also some. Script to help the user extracting for example data over time at provided coordinates. We are also happy to of course. Welcome other suggestion and script from from experienced user. Another module that we released just some months ago we had a publication so it's fully available also the Viki and and the tutorials for this module is based back it's a Python package that it's coupled with basement that help. To model vegetation that repair and vegetation dynamics coupled with the river more for dynamics so we try to model the effect that vegetation repair and vegetation as on the flow, but also on the sediment transport. And also accounting not only for the effect of above ground vegetation so the standing canopy here in the picture but also the role of fruit on the cohesion and the resistance if you want of the of the sediment. Just an overview how it works. The Python module take care of all these workflow, where you see from this simple sketch we have some steps that are handled via Python script in module and then automatically some basement simulation are called and executed. And with this module, you can simulate cycles and number and of cycles of vegetation growth phase where we where we assume that we have low flow condition. The perium vegetation can grow on the bear soil, and then we have flood phase when we where we have natural or artificial floods that cover the vegetated part we have the interaction between vegetation and morphodynamic processes. And then we can simulate evolution in terms of and cycles and here is just an example of a test we have done on the alpine Rhine. So you see the, the, here it's the delineation of the vegetation patch at the end of the investigated period. Here in with the with the color map. You see the simulated vegetation at the beginning of the simulated period that at the end after the flood, and we match fairly well. The, the observed distribution of the vegetation. So these were some some tools that comes out that that we develop and provide within the basement project. And then in the few next few slides I would like to, to, to share some highlight of which of some type of research we are doing currently and in the last years with basement just to give an example. So in general, as I mentioned we have three pillars or three main topics hydrodynamics sediment transport, but also a bit equidrolyx let's say habitat and vegetation dynamics. And, and we work continuously through this research project also in continuous on a continuous development of the of the software. For example, we, we use also the one the approach so what basement one diversion for, for research purposes here for example, we have one PhD student that is working on reservoir sedimentation management using the one the model to simulate the advancing the position of front on reservoir and to understand the impact of reservoir sedimentation on the energy store, but also, for example, the potential benefits of using or not using we use sediment bypass tunnel, and also evaluating the dynamics of sediment or reservoir sedimentation under climate change condition. For example, very, if you want a small scale application, or local scale application is the failure of embankment dams. Here we use of course the 2d version. You can see the, how we are here this PhD student is conducting a composite if you want analysis or investigation coupling experimental simulation that you can see here in the first video and numerical simulation. Let me okay. So here you see the lab experiment of this dam failure on the one side and on the bottom. You can see the numerical simulation conducted with basement on the same setup. Another example again here now at a larger scale, a rich, rich scale of some kilometers. It's the role or the dynamics of fine sediment on flood plains. And the interaction of fine said fine sediment with vegetation in flood plains. So in case of river restoration, we can imagine that we might have different vegetation configuration in our restored flood plains, and we wanted to investigate how these are affected by our sediment fine sediment transport is affected by different vegetation configuration on the flood plains. And to do this, we had to implement, of course, a suspended sediment transport module but also turbulence module to catch this shear layer that it's present at the interface between the main channel and the flood plain. So again, a lot of development to reach this research application. And here just a nice video to where we were testing our turbulence model module with with a fairly easy application. At the end, the goal as I was anticipating. It's to understand here we see the simulation of fine sediment deposition along this flood plain. So here we have the aerial images and here we overlay the distribution of the sedimentation of fine sediment with the with the simulation. Another type of application or research. It's, this is an example of another large scale application in the sense that we have a computational mesh of 3 million cells. Here we were investigating trying to reproduce tsunami lake generated by an earthquake in Lake Lucerne. And we simulate the landslide that generate this this tsunami by instantaneous removal of sediment in different region of the lakes. And we were a we were interesting understanding how this wave propagate in the in the on the shoreline and in the towns and villages that are close to the lake shore. And so to do this, you need a certain type of algorithm for wet and dry and and and a wave advancing in the dry soil. And of course, this was this type of application again was possible because of the GPU version otherwise it would have been computationally too demanding to simulate this type of problem. Moving to the last type of application. So if we talk about more equidrolyx, here we have an example where we simulated surface temperature dynamics at fine scale so here we are in a small flood plain in a in a Swiss river. And you see your fluctuation of discharge in blue due to hydropower production. And you see in red, the fluctuation of temperature where we have natural fluctuation plus this sharp spike due to hydropower production. So we implemented the surface heat exchange module where we can simulate the dynamics of water temperature in our reach so in this map here you see water depth, how it evolves during the simulated weeks, and here the water temperature, how we change and the variability of water temperature, comparing the main channel and the secondary channel of this small flood plain and this is of interest for for many ecological processes that can occur in this type of flood plains. Another equidrolic application habitat, the evolution of habitat dynamics, it's, we started some years ago investigating developing a mesoscale patch delineation algorithm, we call base meso, it's still under development but basically the idea is to identify mesoscale patches with some uniform characteristic starting from the simulation hydrodynamic simulation results. So with, with some basically clustering algorithm to identify this mesoscale patches. And I want to mention also that basement it's basement outputs can be read and directly use with hubby that it's a habitat modeling tool that was developed by colleagues and scientists in France from in right and with be an adf companies and institute. And we were we used also for this coupling with this tool for some habitat dynamics evaluations. So last last few slides. So I hope it was short but fast but yet clear overview of some components of basement, how we spread the word and disseminate our product. We have some events and some tools to do that. In terms of connection with the users we have an annual user meeting. Since COVID it turned to be in a hybrid mode so you're welcome also to participate in the next one generally it's in around February, January, February, every year. We have a user forum where the user can exchange or pose some question exchange experiences and post and report bugs and suggest feature development. So we organize a doc workshop, and we have a quite, we put quite some effort in education so we have two courses at each Zurich where we introduce the software but also in general, the approaches for river more for dynamic modeling. And we put also a lot of effort in to generate a very extensive set of manuals test cases and tutorials that are all available of course for free in the in our website. And to conclude, what's next in basement project, we are continuously working to integrate new features. We're now trying to expand all the the capability of GPU accelerated modules so to add more features in that modules because we think it's really relevant to break down this computational barrier for to the applications. One of the things we are working and planning is expanding or porting the documentation as a V key page to make it more even more user friendly the interaction, the interaction with the documentation for the users. We are considering also creating some libraries API interface to run also the base GPU version with with external and to couple with external programs and so we are also interested in the solution proposed by this community. And we don't have a date but but it's it's it's there on the table also sooner or later the discussion. If, if to move the entire project as open source. In this link, you can find all what I presented and more and more information. So, with this I thank you very much for your attention, and I'm very happy to take your question if you have. Thank you so much David a that was really fascinating. It's, it's clearly a package with a lot of capabilities. We're open now for questions you can either raise your hand in the, you go reactions on the button on the bottom of your screen and click raise hand, or you can post a question in the chat and I'll keep an eye on that as well. And I guess, while we're waiting for people to collect their thoughts I'll ask a quick one. I'm. This is a little bit in the weeds but I understand that one of the challenges in fluvial morphodynamic modeling is how you handle the lateral component of sediment transport that is lateral relative to the main flow direction that's driven by gravitational forces say down, you know down a bank or something like that how was that handled in basement. Okay, yes, it's a relevant process we have to correction for the for the sediment transport direction, the one as you, as you I think you were you were mentioning to sorry for the background noise there is a truck passing by. And one is the correction for the transversal slope so gravitational component. We use an implementation that it's, it was proposed by a kid and Talmond and there are several several approaches. We have this, we account for this correction in the external equation that it's proportional to the to the to the slope to the local transversal slope and to the intensity of the sediment transport. And then we have also the second correction that actually counter counter counteract this that it's the spiral flow correction so in case of a band or or a strong curvature of the flow, we have this spiral flow correction that scales with the radius of the, you can nicely simulate evolution of meanders, for example, or point bar with the position forming at the inner band inner part of the of the band. Thank you. We have a question from on the chat from I mean us Karina job. Thanks for the presentation I wonder if the soil mechanics processes during breaching, for example, unsaturated slope instability due to local tow erosion or simulated. I know right now. I hope I don't provide the wrong answer but right now we have some gravitational instability driven by some slope angle. So we have a slope angle of failure for dry condition and and wet conditions so we have a certain number of of different threshold for for the slope collapse. And this is how we evaluate the slope instability. The local tow erosion explicitly no, but I don't, maybe I'm not sure I got the, the correct question, but I hope it answered otherwise I'm, I'm happy to have a follow up question on this. So, you see there's a question and from Luis Lopez. Thanks for the presentation is it possible to model an estuary. Let's say that the basement was developed for alpine conditions so I'm sure there are project and software that were developing lowland rivers and estuarine context that are more advanced in that regards. So, not out of the box in the sense that we don't have a dedicated module for wave or or oscillating boundary condition that you need actually to simulate estuarine so I think we are missing some dedicated boundary condition for that you can in some way. You can force the model to do it but I would say that it was not designed for for for that context. And it. Yeah, Switzerland is also a lock land country so we, we have less concern with estuarine. Let's see. See level would have to rise quite a lot. Yeah. A question from Daniela Tonina. Very nice presentation for dissemination have you thought about video tutorials. Yes, we thought and discuss a lot but the bottleneck is the manpower in the sense that the project. It's limited funding and this funding it does not cover. Most of the efforts for dissemination so we don't have the capacity at the moment to produce nice and good video tutorials but it's there. It's a recurrent topic. Let's see so from Maureen Gretner. Thanks for the interesting presentation would you recommend using this tool to analyze the influence of bigger boulders on the water flow and rivers. There, it's a yes and no. It depends, but I would say that if you can. If you have. This is depending on your of the quality of your topographical data. If you have a high quality resolution scan where you can identify then define your big boulders. I would say, yes, it's fairly, you can use a very fine computational cells so you can really identify the boulders in your in your computational mesh and yes, but I would say it's more related to the quality of your of your input topography. And then from little Vico Agostini. Thanks for the nice presentation what are the main upstream boundary conditions for sediment transport. Oh, this is, it looks like a test, but but so we have equilibrium conditions so basically you you you ensure you ensure that you don't have morphological changes at the upstream boundary. You can have upstream boundary condition. You can have sediment transport capacity condition or you can provide a sediment value sediment hydro or sedimentograph. So you provide a certain amount of transport rate at your upstream boundary condition so I think we have. Well, the more common conditions for upstream boundary condition in terms of sediment transport. Okay, so I'm not seeing any more questions in the chat. A reminder to everyone that this webinar recording will be posted on the CSDMS website under education webinars so if you want to watch again you can if your friends and colleagues missed it and want to see it they can look it up there should be posted in a few days. And let's thank Davide again for a fantastic presentation. Thank you very much for inviting. Thank you.


The sessions with Jim, Chuck, Scott, Ovidio, I don't know what they did or how they did it, but they pulled out the one I wanted to be and that I wasn't sure that I can be. Now I am braver than I was three days ago and I'm willing to take more risk.


Crop improvement is essential for achieving sustainable food production systems. Stakeholders from different levels of the agri-food chain have agreed upon for main priorities for crop improvement, water, nitrogen and phosphorous efficiency, heat stress and consideration of alternative methods such as management strategies. Research is needed to assess the impacts of these priority options on sustainability, with particular attention paid to downstream effects. This will help to identify concrete targets for plant breeding innovations as a food system solution. This article was authored by Estetckiewicz, Jay Mennery, Annette and others.


Okay, so we're going to talk about troubleshooting networks with PF Sense and this also kind of leads into the question people ask a lot. Tom, why do you like PF Sense so much? Why do you like to put it at the head of your network? Well, that's what we're going to dive into a little bit here. So first, let's start with the network layout that I have, which is my computer is at 3.9. The PF Sense box has a WAN side of 3.150 and a LAN side of 40.1. That makes this the a 40 slash 24 network over here and we have one Linux box running and one Windows box running virtually. Everything in here, just so you know, is virtual. These are not physical boxes except for my computer here. So this is an idea of the network so you can kind of see how things are connected and kind of visualize it here. So I have this opened up on the WAN side so I can administer it from here. So anything you may see in the title bar if you're looking at the IP addresses is going to be from externals. I'm not doing it from inside. Also doing it that way so I can show you cleanly the data that's going through each of these machines and where they're connecting to. I've seen a real network and there's going to be a whole lot more of this and a little bit harder to sort through but I want to start with the basics so you can kind of get understanding of where these tools are in PF Sense and get an idea of how to start looking at the network connections. All right. It is May 2nd, 2018. We're running the 2.43 release of PF Sense. I also have in a package manager here. I'm going to go to install the packages. I have Entop 08.12 in here and I have that set up. I've done a video on Entop. It's pretty straightforward to set up, install the package, enable it, set a password. Do that just save you the trouble. You do that in the Entop PNG settings. After you install it, you check this box, enable it, come up with a password, bind it to an interface, hit save. Now, I went a step further and created a firewall rule. This is not recommended necessarily but it is for the demo here. I opened up port 3000 that it runs on and you can see 3.150 and 3000 is the port. I opened it up externally. Normal situation, you want to have that internal facing and you either VPN and remotely and look at it if you want to connect to Entop or it's running in your network and you can run Entop there. Now, I will mention with Entop, it is a little bit processor intensive so you got it. If you're running a really old slow machine for PF Sense, you may have some trouble with Entop keeping up. So just throwing that out there real quick. All right, let's take a look at the first tools. So we have the network layout. We have a couple of machines running. I have the Windows 10 booted and I have the Debian machine booted. We're going to go look at with diagnostics PF top. This is just a great go-to tool to get you started. I figured out where all the data is going. So there's just a whole bunch of established connections in here. And actually it's kind of funny if we filter them. The two IP addresses we have running here. So let's go ahead and oops, I'll put it in for typed wrong. This does it in real time. And by the way, don't press enter. So if you press enter, that will do that. So let's filter it to one host. So filter it to just this host here. You put host, then the name, wait a second, don't press enter and it updates. And now we can see every connection for this host. And if you can guess it, this is our Windows host, which has lots of connection. This is our Debian host. I booted it. Debian does not go out by default and make a bunch of connections. The one exception to that connection that you may find on a Debian box is if you have enabled the package survey manager that sends periodically packages or an automated updater, it will contact whatever server you told in there. Other than that, Debian by default is really quiet. And Windows, it connects to everything. It's got a lot of connections established just because we boot it. So let's go back over here to this host. Let's start our first connections. So I'm on my computer, as you can tell by it says Tom Spooter. My really less than clever naming, naming scream, and we're going to go ahead and SSH root 3.150. That is the firewall. But we have a rule that passes it through to the Debian box. So many SSH in. There's the Debian box. And there's the IP address of 40.50. And now we see a established TCP connection. So pftop in real time. Now you can run this from the command line as well. But they've made a really nice job of doing it in here. So when you got to start that tracing and going, all right, where is data going? What things are using what? This is what helps you a lot to do that. So you can string the commands together. So we found the host and we see the connections. And let's make some noise. So we actually have a lot of connections. I have DNS perf test on here because it will do a DNS test. So it's going to test DNS. And each DNS it does, obviously, fills up more and more state tables. Now you have a mess and you're going, how do I read this mess? So we're going to say, and port. Give me a second. There. I filtered it only for port 22. I could also filter it for port 53 to only find the DNS lookups. Now this is kind of that Swiss Army knife of tools that I like to use to get me started on why something isn't connecting, why something isn't working. If you know where it's supposed to be going, you can put host and host in there and start figuring out, okay, where's it connecting to? What's it connecting to? And why isn't it getting there? So this is one of the first tools is PF top to be able to start sorting out what's going on inside of PF sense on terms of network. Now you notice they all died. That's because the test is over. And if you're familiar with how UDP testing and that works UDP, it's short lived. So it sends out the packet, it got the responses, the state tables die. So if we go back over here, port 22, we can still see that I have an established connection here. Now, next thing I'll show you is this is just some rata here for understanding how it works. We're going to exit enter again, exit enter again. We're exiting and connecting exiting connecting. And now what's happening over here. Here's the established ones. Here's the ones that are closing, but they're waiting and then they're going to die off over time. So you're watching the countdown of them expiring. This tells you when they're going to expire in a minute. So that'll count down and then those connections will expire. Now, you can also, I believe there's even a filter for like active TCP. You can go to right here and a convenient click. And it gives you all the little options. So you can actually sort by when they're expiring. But this is one of those tools that is a great way that you can just start poking away and going, okay, let me filter. Let me find and let me sort out what is in the network, what's going on and what's doing that.  So you can start really drilling down into here for this. Also, and I don't have this set up on here. If you have queues set up, and I don't like, so I don't have any on here, it'll list all the queues and the queue types. Let me see if I can pull up to my system. All right, and now here's all the queues and you can see all the data going through there. And I think we can have fun with it by doing this. I'm going to, I got my phone in my hand here. I'm going to call my office and you'll watch the queues change. And we just seen more activity in the VoIP queue. So once again, it's just an absolute Swiss army knife of tools to start organizing things. And I have queues and HFC traffic shaping turned on for all of my ports. So it's really nice because I can see in real time each of those calls and you're going to watch in seconds can refresh and these numbers are going to go back down because there's nothing in the VoIP queue. I have it set up to prioritize the VoIP queue in terms of bandwidth. So let me close that. Other things you can do when you want to really dig in and do some diagnostics is we're going to go ahead and go over to packet capture. And we don't want to just do well, we'll do the Debian host. So that was at 50. We're going to filter it like this. Enable Persecus mode interface LAN. So I wanted to make sure anything going to this address here gets captured. And we're going to go ahead and start that capture. And we're going to go in exit enter. So we now we established another connection that the packet capture can see. Then we'll do that DNS perf test. So we just filled up and now we're creating a bunch of stuff going on in the packet capture logs. Sorry, we did some stuff. Stop, download capture. And of course everyone's favorite tool, wire shark file. And now we can see data going back and forth and start tracing out packet capture logs. It's nice to have this built in. So a lot of times this is the trouble is getting the interfaces to go into promiscuous mode to do this. PF Sense makes this really easy. So if you want to dig into packet capture to really understand what's going on inside of a network between public, private interfaces, whichever interface you want, because PF Sense runs at the head end of your network and you have designed a network to pass all the traffic through it or at least I think that's the way you should do it. It now understands and has all those details. So I can do packet capture. I can do filtering. I can start narrowing down data very, very quickly with PF Sense. It's an outstanding tool for doing this. And it also just in the capture here. This is nice when you're working remotely and just logging into the client's system. You can do a basic packet capture and start going looking at the back and forth. And this is just the log level detail that was going to show up here without actually doing it. And then you can just view the capture whenever you want. Now it only remembers the last capture. It doesn't have like a system that organizes all the files. So you got to download them out. But it's an amazing tool to be able to go in here and just say packet capture that network. Let me sort out what it's doing. Let me dig into it a little bit. And it's also handy for things like when we're talking about the Windows machine being right here. I did this earlier. It's amazing how many connections the Windows machine does just on a reboot basis of all the different outgoing. So that's another tool that is also really handy. Those are kind of your tool things that I use a lot. The other thing we do is trace route and test port. Now this is great because you can do host name lookup, port lookup, source lookup, where's the source address, WAN, LAN. And if they have multiple external IPs they'll all show up in the list here. And this is nice because we get into the machine often externally and they say I can't get to this website or I can't get to this. We can start here sometimes not even looking at their computer and go can they see this? Can they see this network? And you're going from that PF sense box to wherever the destination is. So being able to test the port and of course you can do this internally. Do the host name lookup, test the port internally. You can even set a source port for where it comes from.  It's actually a really handy tool to be able to do this. We sometimes when we're setting up remote ports we need to make sure a port's open. It's a quick way to go in our own PF sense system and go all right can it see it? Okay I can see a response from this. Log into another box and see how it sees the response from it. So another great diagnostic tool just right there. And just standard DNS lookup, you can do this here. So if you just want to look up LawrenceSystems.com it tells you based on all the DNS servers that are available to it, its own DNS, the local DNS and external DNS that I have programmed in there and away you go. Then you can right away jump to a tracer out to it or add an alias and start building something off of that information if you wanted to create aliases. Handy having this here, I mean it's great to, I'm used to doing it from a command line, but having a web interface that you do it then you can just start building on the rules from there if you have to do something really easy to do, especially if you're using aliases to block things. You can go in here, do the lookups, do the add aliases and start building lookup tables. Let me do a video on that sometimes. A lot of people ask about blocking things. There's also diagnostics for sockets, routes and all kinds of other little details in here, but the one ones I'd use are the ones I mentioned right there. Those are the absolute really handy for starting to drill into some of the network stuff. You can do the track graphs and package logs and some of that, but where we really like is the end top PNG. I load this on there. I don't always leave it running because depending on how much horsepower the machine has, but when you got to dig into some of the network diagnostics, this tool is wonderful for that. So I've done like it's an entire video on it, but we're just going to jump into just showing you a couple of the host options and how they work in here. So here's my Windows host and right now there's probably nothing going on. Nothing really going on here. It's got one connection to there, which I don't really know what it is. I don't know where that's going. Let's take a look. This is kind of neat. So it creates one connection to there. All right. I know where that one is. I know what it's going to. That's part of ScreenConnect hosting. So it's hosted. It is connected to ScreenConnect. It's quiet. So what we're actually going to do here, and I'm bringing it over so you guys can see what's going on. Here's our Windows machine. We add it to here. It's in a VM. I'm just going to restart it real quick. Oh boy. Now it's decided an update. Of course it's got an update. So I guess what the pause this video while it does an update. All right. Now Windows is booting and what we're watching here, this is the network. So we're going to watch for when the network peaks up up here. Hopefully it doesn't take too long when it wants to update. And once we see it do the peak here, that's when we know it's starting to establish network connections and we'll jump back over to end top and see where Windows wants to call home to. Because really, there's nothing installed in this Windows other than the one screen connect session that we have. Getting ready. I'll speed this up. All right. Windows is booting. It's pulling in some network traffic. So let's go to end top and see what it's talking to. And we'll start with the flows. So here's all the places it's starting to talk to. Here's the flow talkers as they call it. And this is kind of the cool feature too. You click this and it gives you an idea of everything it's phoning home to. Let's go back over to flows. Then go back over to talkers. And then over here. And it just doesn't auto refresh. It does play, but it does not auto refresh as they come. So you're starting to see it's phoning home to Redmond, phoning home to Menlo Park. Apparently it thinks this is in Chicago. I'm sorry. Oh, that is ours. That's in Detroit. So my bad. This is the one I was looking at. It's Iowa and City of Cheyenne. This one in Kansas is ours. Our G.O.I.P. is not always the most accurate. It insists we're in Kansas on that particular IP address. It's not. It's located in Detroit. So these are all the things that Windows does. Go back over to it. Sometimes it makes a connection overseas to Dublin. Now it's actually not made. Two of them over here. So now it has more data going over here. So we got this one and this one. So San Jose. Thanks, Windows. You're just going all over the place. Cheyenne, that's us. And there's the machine itself. So here's Windows sending all the data. Now this is how you can start digging into where your data is going. You can look at the protocols. You can look at the traffic. You can dig into it a little bit deeper. Now I'm going to go ahead and open up ours. Because I wanted to show you all the details of what we have running on our network. So all the different places that we're doing data. And I chose my computer. This is the 3.9. Like I said, this is mine. So you get a better idea of it. It's also fun to see just how many connections and where they're all going. I don't know. What is going to Spain? Some website I have apparently is open in Spain. So this gives you a better idea of the traffic. And let me show you what it looks like with more data. So you can see the TCP. Since I started it, this is how much data I've sent across the network. Package distribution. The current flows of data going back and forth. Protocol breakdowns. Top peers that I'm connected with. And if you didn't notice 2.24 happens to be this. There's a lot of data because I'm running a screen session in here with Windows running. So it's actually sending quite a bit of data right now. So like I said, between all these tools, starting with pftop, just looking at the basics and end top being the ultimate, we really need to dig into what this thing is doing. These are great tools you can use to try to understand where all the data on the network is going and start breaking it down. This also is handy as well. I went over here to the time slice and you can start saying 1 hour, 10 minute and start breaking. Okay, where is the data going in terms of like putting it on a graph? And because I haven't been running this for very long, I don't have it running all the time because it just is a little bit resource intensive. So you but you can start to look in here and go, okay, let's look at the amount of data. You can also use this to look at the overall aggregate data going across the network. So whoops, I actually went to the wrong one. So this would be my data slice. There we go. So this one's on ours. This is my computer. A little more data. I've been running it for, I think I turned it on yesterday. Yeah. So you can see that when I'm downloading stuff. And if I upload a YouTube video, I'd see that part two on the cent. I see peaks in the cent when I upload a YouTube video. So the last thing I'll show you is the traffic graph that I usually have put on here on the front page. This also kind of gets you started. Is there a traffic problem? And you just go here. And it's just the traffic graph is a little service status, which I have here. And then you have the you can put the firewall logs, interface statistics, some of that stuff in here. And then of course, the traffic graph. And then just move it around, put it where you want. Save the positions. Whoops, how have I closed it? But that gets you an idea too of just how much bandwidth the system's using before you start drilling in. So let me add it back. There we go. Save. All right. Hopefully, this is helpful. And we'll start your troubleshooting in PF sense. It's also fun for a lab environment. If you want to play around with it a lot and kind of get a better idea of how the connections are going in your network. But like I said, it's an amazing tool. They have just even more diagnostics. But like I said, the ones you can start with are the PF top and top PNG for your real in depth look. But just starting with PF top and just start looking at it or looking at the cues and how things are going really are the most of the time all I need to troubleshoot a network issue that's going on. So I can kind of like, you know, start and go, oh, this is where the connections are going. All right. Hopefully, it's helpful. Leave your comments and feedback below or in our forums. Thank you for watching. If you like this video, go ahead and give us a thumbs up. If you have some feedback, leave it in the comments below. If you'd like to subscribe to our channel, hit that subscribe button, hit the bell icon to let YouTube know you'd like to know about new releases on videos. You can also find new releases on our website, lauranceystems.com slash blog, where every video automatically gets posted. So you can always find our videos, whether YouTube notified you or not. Also, if you'd like to hire us for consulting services, go ahead and hit lauranceystems.com, fill out the contact form, tell us about the project you would like us to help you with. We work with a lot of businesses. We work with a lot of other IT people who need services done. Also, if you want to help the channel out in other ways, we have a Patreon. We have right below me a Amazon store where you can check out some of the products we've reviewed and as many of them as available on Amazon. You can also check out the things we love. And that's an ever-evolving list of discount codes, offer codes, and different software and affiliates. You can find that on our website as well, or just follow it in the link below, right to the Things We Love landing page, including the hot sauces we recommend, which is always changing. All right, once again, thank you for watching and see you in the next video.