Okay and I will talk about, it's time for my talk. So I will talk about Norikra. Norikra is an open source software and I just I love and to process and data streams and the open source software written in Ruby. So this and these are topics of today's talk. And at first I will talk about and why and I love Norikra and it is very important to understand what Norikra is and how Norikra works. So my name is Satoshi Tagomori, also known as Tagomoris and that is my account name over on Twitter and GitHub and many others. And I'm from Tokyo, Japan and I'm working in Ryan Corporation. Ryan Corporation is an internet service company and serving as a message application, Ryan. And Ryan is a message application just like on WhatsApp or Facebook Messenger or any other. And Ryan have about 130 million users in worldwide and mainly in Asia and South America and a part of Europe. And moreover, we have many services online platform like in Japanese manga, electric publishments and then camera and news or Q&A service or and weather news and many games. So we must handle a huge amount of logs and metrics and at the same time, and we must handle a various kind of metrics and logs. So and I am working about our data and analytics platforms and this is a very simple monitoring and data analytics platform overview. And at first, and we must and collect many data from many servers and pass in this data and clean up this data and store in distributed storages like in Hadoop HDFest and then and process this data and visualize it in graphs or chats on many others. So and that is why I am a committer of an FluentD project. FluentD project is Kiyoto Tamura talks about FluentD project in two days ago, the first day of this Rubicon. And please check the slides and about FluentD but in roughly saying FluentD is a log management system and to aggregate to correct many logs and to aggregate these logs and to put this data and result data into any storages or remote systems. So and we are using FluentD in our data platform systems and to deliver and many data and to control data flows. On the other hand, and we are using Hadoop and Hype to process our storage and stored data and Hadoop and Hype is a very famous open source software and then many internet service companies and use these softwares. And then and we must and process this stream data as soon as possible to find any troubles or any surprising changes of our internet service traffic and like HTTP response code and percentages or HTTP request rates per seconds or an HTTP response times graph. And these and these graph is generated by FluentD plugins. FluentD have a very... FluentD have an extension and features and the plugin features and we can write and use any plugins to aggregate and stream data and make a percentages or many values and we can put this data on graph tools or any other visualization tools. And so FluentD is very good for simple data or simple calculations but our services and there are many more and more different services and there are many changes in your day including in logging and then there are many kinds of blogs for each services and there are many different metrics for each services. So and FluentD, FluentD requires configuration changes and restart and to change what to do. So and FluentD is not so good for processing about complex data or and fragile environment like in data schema changes or changes what we want to do. So that we want to add or remove and queries anytime we want. So and we want what we want, we will be changing we will be changes and very frequently. So we want to add or remove and queries anytime that we want and we want to write many queries for service logs, log streams or we want to ignore events without data we want. So and data schema will be changes and very frequently. So and but the application engineers cannot know what requires, what data process platform requires. So and they, so we should create a system and that the application and application engineers can change their log schemas and their logs and meanings anytime. So and data analytics platform can or should be able to ignore events without data we want. And there in my company, there are many service directors and growth hackers, they're growth hackers. So they are not software engineers but they know what is important for growth of our services. So we want to make our service directors and growth hackers to write their own queries for what they want. That is why I wrote in Norikra. Norikra is a data processing platform and middleware and to realize these requirements. Okay and Norikra. Norikra is a schema stream processing as middleware and with SQL. That is the open source server software and written in JRuby and runs on JVM. Norikra is distributed in RubyGems.org. So we can install Norikra, just do a gem install Norikra and then and we can launch the server by a Norikra stat. And Norikra having some interfaces like an CLI client or client libraries named as and rubyclient.gem. So we can operate Norikra with these and CLI commands or we can also and control the software and over web UI and over HTTP API with Json and message pack. So let me show some demo of Norikra. Okay and we can install Norikra by this command and already installed. Okay and Norikra is written in JVM. So this and launch requires and very long seconds. So and here there, this is a sample. Okay and this is the event example and the Json objects and within two fields, name and quantity. So we can and feed these data into Norikra over Norikra, client and event send or my service sales. Okay, but by this command and nothing happens. So because Norikra requires okay, this isn't web UI and Norikra requires and target definition. So at first we should define the targets, target opens and now I'm cheating. So and in this shell I'm using and she'll be. So and she'll be does not require and several seconds to launch the commands. And okay, now I will feed log events and continuously into Norikra to Norikra. Okay, so Norikra can find and field names from these event streams, like a name and quantity. So now we want to select these fields from this and event stream, like by this and very simple SQL, my service filter and to specify to print and query results into console. Okay, successfully added and we can get, now we got the results data from results data with a name and quantity from this and event stream. So anytime we can change the input data schema. So now this is the another event example with another additional schema and now I will feed this data to Norikra but the previous query and works well and the previous events. But and schema is already changed. Okay, and drunk Boolean optional fields detected automatically. And now and we can write any SQL to, for example, we can count and any input events by this where drunk. Drunk is and Boolean, so this expression is correct. But this SQL is not correct for Norikra. Norikra requires a range and how long as a Norikra requires an aggregation range and what and okay, with this specification, Norikra can and count their events and for every five events and Norikra can and count the drunk and events with and drunk is true and in every five events.  We can get an output data, oh, oh, I missed text at, okay. And also we can count and any and quant and sum up quantities by this and very simple query name and summarization of a quantity from my service sales and group by name, order by summarization, descendant. And for and the, get and summarization for every five seconds. Okay, and we can get an event aggregated results by an SQL query for this and input stream. That is how Norikra works. And we can, so with Norikra handles and schemarize events stream and we can add or remove and data fields and whenever we want. And Norikra uses SQL and Norikra requires and normally starts to add or remove queries. And Norikra's SQL with Norikra SQL and joins and sub queries are available and we can add in user defined functions and in written in Ruby or Java or any other JVM languages. And we can publish that UDF as Ruby Jam. And Norikra can handle nested hash and arrays and these values are accessible directly from SQL like this. So user attribute and have, user attribute have a json object, nested json object and attend. Attend is a nested json array, but Norikra's query is extended so we can access user.age or an attend.data zero or any other specifications. Okay, and we are now using Norikra in our production environment and the first end use case is the error log summarizations. And we have an API, web API for our partners. That's over that API and any user sends a message to our partner's official accounts. And then our server sends these messages to our partner's server. And that's in written in business connect server the right side. And then our partners respond their own messages to our API server. And then we bring that response to our users on our application, in our application line. But these and our partner's server, if our partner's server goes down, so if our error messages, if all of our error messages and brings to our partners and that is really and flooding. So to avoid the flooding of error logs or error messages and we are now summarizing these error logs and error messages by this SQL in Norikra. And then Norikra puts the summarized and output and different D sends an email to our partners. And then at the same time, Norikra and saves these results of summarized drugs into MySQL. And then our administration console shows just the summarized drugs for two of our partners. And error log summarization is in very major and use case for Norikra. On the other hand, Norikra, we are using Hadoop and at the same time, in our data processing platform and we can process the same data with Hadoop and with Norikra and our service programmers can write queries on Hadoop and on Norikra for the same and just the same data. So and these features are used to generate and prompt reports and daily fix reports. And for ad services, our programmers and our lighting and these queries on Hive for and fixed reports in part each day and also writes and queries for Norikra for our two producer prompt reports. Prompt reports and generated every one hours or several minutes or several seconds. These and prompt reports and show in administration console for our customers of ad services. And this is a use case by a Google engineer and Google Cloud Platform Service Architect of Google Cloud Platform. And he uses and Norikra with Google BigQuery and to and count the web service requests and web service responses and to show these results on dashboard on Google spreadsheet and Google Apps script. And he uses and the web server engine X and engine X and writes its access log into disk. So friendly these are executed on each server and each reads an access log and sends it to Norikra. And Norikra summarizes these access logs per server and the summarized records are sent to BigQuery directory and another aggregation node. And this is a total overview of the system and the summarized data are collected into an aggregation node and then aggregation node can count the whole status of these events within another Norikra aggregation node. And then and grant D writes these results into Google spreadsheet and Google spreadsheet shows and graphs and summaries and by Google Apps script. If any, if users wants to metrics and not already defined and users can and throw queries into BigQuery. And the Norikra and BigQuery can process at just the same data set. And this architecture is called as an Lambda architecture and called and named by engineer of Twitter. So and the Lambda architecture and handles and batch processing and stream processing. So and we can use an SQL SQL like DSL and both of both in BigQuery and Norikra. So and we can build an Lambda architecture platform with Norikra. Okay. Okay. And the Y and Norikra is written in JLV. So and this is the main the big two are there are big two factor and one is Esper. Esper is an set and complex event processing library written in JLV. And Esper provides an SQL like DSL and that is the base of Norikra query. And Esper has been very and well and written library. So and we can process very huge amount of data with Esper. And rubygem.org and we rubygem.org is and of course an open library repository for public UDF plugins user defined function plugins of Norikra and provided at Rubygem. Okay. And JLV and I don't there before before Norikra I do not use JLV but and these two factors makes me to use JLV. So JLV is JLV for me is just Ruby and this is brought by great JLV developer team. And JLV makes and developing Norikra dramatically faster with an Esper and well-known rubies and positive points. And JLV with and rubygems.rubygems.org for an easy deployment and installation. And JLV and with JLV we can use in Java libraries and like JT or Esper or many other libraries. I am using many other Java libraries in Norikra. That is a very good point to build data processing middleware. But then minus point isn't there are not so many JLV users in especially in Tokyo. So of course and we can find and many Sylvie committers in Tokyo or in Japan but then not so many JLV programmers. We cannot find not so many JLV programmers in Tokyo. So when I got and confused confused confused to and call any Java method or any other processing but then I cannot ask these questions for not so many people. But then this isn't not so big minus point that JLV is very great software I think. So and this is a wrap up and I believe that if you are interested in Norikra and please check your software's documentation site or software you are GitHub. And but then I believe that Norikra brings and our data platform, Norikra brings and more queries and more simplicity and less latency to our data processing platforms. So if you are interested in Norikra please try it. Okay, thank you.


The enemy struck at Pearl Harbor, the whole of America's mighty productive machinery was thrown into high gear to place the country on a full war footing. The Congress quickly enacted legislation providing for the greatest construction program in the history of the world. Almost overnight, the Bureau of Yards and Docks of the Navy Department became a billion dollar concern. Recognizing the need for an organization of skilled workers to build advanced bases in active theaters of war, the Bureau of Yards and Docks obtained authority for the establishment of the first naval construction battalion in charge of officers of the Civil Engineer Corps of the Navy. From this first battalion of 1,100 officers and men, the personnel of the CBs increased with the demands of war to 3,300, 10,000, 30,000, until at the present time the total authorized strength is 222 active battalions with approximately 290,000 officers and men. It is difficult to put into words a full account of their services to the fighting forces. But I feel sure that this tour of CB activities, as seen through the eyes of the motion picture camera, will justify our confidence in this youngest member of the great Navy family whose motto, Construimus battuimus, we build, we fight, is translated into action in the film you are about to see. These are Japanese planes, their destination, Pearl Harbor, Midway, Cavite, Wake, their purpose, treachery, destruction, war. These are civilian American workmen. Japan is at war with them, but they are not yet at war with Japan. All they had come for was to build things, put a little door aside for the kids and hope to get home for Christmas with the family. No, they were not at war. But here of Hitoh was, he had other plans for their Christmas. He was delivering his Christmas surprises early this season. All of these men wore navy blue and some wore overalls, but through the enemy bomb sites they all looked the same, just Americans. We learned many lessons that morning, one of them was the need for giving workmen at advanced bases, military training, civilians for all their courage, lack weapons or the ability to use them. If they were wounded, there was no government compensation. If they were captured, they were not entitled to treatment as prisoners of war. A new war presented new problems and a new solution for out of the very fire that the Japs left behind them was forged a unique type of service organization, the Navy's construction battalion, or CB's as they came to be called, America's answer to the need for skilled craftsmen ready to work under fire and to defend their work and themselves against enemy attacks. All over the country, American workmen met this challenge, machinists, carpenters, pipe fitters, steel workers, truck drivers, rigors, electricians, haters, welders, divers. From 17-year-old boys just out of trade school to grey-haired veterans of World War I. Swedes from Minnesota, Poles from the coal mines of Pennsylvania, Irish boys from Brooklyn, Scotch, English, Jews, Italians, Negroes, all races, colors, creeds, but all Americans. The skill that helped to build our country goes to war. Not in seven easy lessons, but in eight tough weeks, the Navy teaches the civilian craftsmen to be a fighting builder. Thousands of individual workers now forged into one great military organization, passed in review with their motorized weapons, infused with the CB spirit, we build, we fight. Ready to build and defend advanced bases from the Arctic to the tropics, a new battalion leaves the training camp at the amazing rate of one every other day. American workmen, defenseless no longer, shoving off for battle stations. Destination for this CB battalion is Strategic Dutch Harbor, the first big stepping stone on the road to Edak, Kiska, Attu, and American reoccupation of the Aleutian chain, so vital to our offensive strategy in the North Pacific. Towering snow-capped mountains, rising from the sea, make this great arsenal of the North one of the most impressive sights in the Western Hemisphere. But the CBs haven't come to admire the scenery. They've come to change it. The iron jaws of the steam shovels bite into the fallen mountain sides, bulldozers, heavy tanks of construction, level roads where only walls of earth had stood before. In weather so cold that concrete has to be heated before it can be poured, the CBs are on the job, laying foundations for gun emplacements. Oil tanks, door houses, concrete foundations on which victory will be built. The fine art of logistics, a big word for the big job of getting the right thing to the right place at the right time, made it necessary for the CBs to bring their own lumber yard with them. And lumber plus CBs quickly adds up to an imposing total of barracks, hospitals, shipyards, warehouses. A modern advanced base built to withstand the most savage attacks of nature as well as the enemy, a naval base thoroughly equipped to supply and serve the planes and ships that were gathering for a great offensive in the North. No longer would warships damaged in action have to creep back to Seattle for necessary repairs. When the CBs promised to have this destroyer out to sea in 36 hours, the skipper said it couldn't be done. But the CBs' answer became a famous slogan, hand-do. 36 hours later, the destroyer was returning her task force and a jubilant skipper was expressing his gratitude to the CBs in an official commendation. In all my experience with shipyards, I have never seen such tireless energy. In spite of suggestions that the CBs be relieved for rest, many of them worked 36 hours straight until the job was done. That's how the CBs became a legend in the Aleutians. That's how victory was made inevitable at Atu. If you think of the Aleutians as an arm reaching out into the Pacific, the tip of the index finger pointing toward Japan is Atu. Bleak and forbidding, Atu had been a forgotten and unwanted pilot until it suddenly sprang into the headlines as a Japanese beach head threatening the security of the Western Hemisphere. While our heavy artillery pound Jap positions within earshot of the beach, the CBs go into action on the construction front. Today's beach head becomes tomorrow's naval base, and coal is the sustenance upon which a naval base feeds and grows strong. Whether it's hand-to-hand fighting or hand-to-hand unloading, CB stevedores know their job. Like a line of ants and justice tireless, they carry their heavy load across the barren slopes and valleys close behind the battle line.  But the CBs are practical guys who like their warming chow along with their warming thoughts. And hot beans and a cup of Java hit the spot on these sub-zero mornings when there's a day's work ahead to be finished by noon. With rifles on their backs or close at hand in case of enemy attack, they throw up temporary barrens. But materials for permanent quarters are on their way, 50 miles of tarpaper. And buildings shipped in prefabricated sections that fit together like parts of a jigsaw puzzle. But a puzzle that CBs solve with lightning speed. For working in perfect teamwork, they shave a few precious minutes off their own record for setting up this unit. A good-sized modern and substantial hospital in five hours. In the States, gas stations may be opening late and closing early. But the Attu service station does business 24 hours a day. Transforming this barren little island into a flourishing base was a 24-hour-a-day job. Or as the CBs like to put it, a 36-hour-a-day job. But a job that was done and finished in less time than the most optimistic American had hoped while the most pessimistic jab had feared. With the Aleutian chain anchored at one end by Dutch harbor and on the other by Attu, the enemy holding out on Kiska found himself hopelessly cut off from his source of supplies. For thoughtful Americans who like to read behind the headlines, the headline Yanks Retake Attu had become CBs Consolidate Attu. With the Aleutians back in American hands, hands that not only fired accurately but built swiftly, the finger-pointing toward Japan clenched into a fist that could strike a direct and punishing blow at the enemy's heart. From other points in the Pacific, the left hook was being planned. The CBs put midway back on the map. At Pearl Harbor, officers of the Civil Engineer Corps assisted in the salvaging of ships that had been irreparably lost according to the Japanese press. A strange name the far-off Solomon's became a household word. Guadalcanal. Guadalcanal, the Marines, had a new verse to a famous song. The Marines sing a fighting song, but their most extravagant words were an understatement at Guadalcanal. Yes, when you say Guadalcanal, you say Marines, but those Marines will be the first to tell you that if they made history at Guadal, those CBs working and fighting shoulder to shoulder with them made something more tangible than history. They made a thriving base out of the flaming wreckage that was left after Johnny Marine had proved his point to Mr. Tojo that this island was too small for the two of them. With possession of Henderson Field, conclusion jumpers were congratulating each other on our domination of the South Pacific. Well, after all, wasn't Henderson Field the key to the Solomon's? Yes, we had rested a key from the Japs that was going to open some mighty important doors, but before that key could turn a single lock, it was badly in need of repair. The CBs digging into that airfield with the same fighting spirit as the Marines had hit the beach repaired 53 shell and bomb craters in 48 hours. That would be a record in any league, but the CBs hung up their record under fire. But this one was a ship-fitter first class, but he seems to be a first-class gunner too. When Congressman Moss of Minnesota saw the CBs in action at Guadal, he reported, when Jap attacks were made, I saw CBs working on Henderson Field, drop their tools, pick up their rifles, and fight side by side with the Marines. When the attack was over, the CBs put on their guns, picked up their tools, and calmly went back to work again. Just in time to fight off the enemy's most desperate counterattack, our fighter planes were able to use the runway. But the CBs didn't stop for any bowels. They knew it was their job to put throttle on the offensive, widen the field, lengthen the runway, place the field, grade the surface, level it off, make it navigable, not only for fighter planes, but for flying fortresses. In so short a time that even the slant eyes of the enemy opened wide, the field is ready. And as the first bomber takes off on its mission of softening up enemy bases for the landing operations ahead, the CBs win a two-fisted tribute from a two-fisted Marine. Lieutenant General van der Veen, who said, I don't know how we would have gotten into the air without the CBs. The spectacular speed with which bombing operations got underway from Henderson Field played havoc with the Japanese timetable of conquest in the Solomon Islands. Bases from which the enemy planned great offensives found themselves desperately on the defensive. Every day our bombers were carrying the mail to Munda, Kolombangara, and Bougainville, a female that contained nothing but bad news for Hirohito. If the Japs were to regain their initiative in this theater, their rising sun had to go up over Henderson Field again. This was the aim of the powerful enemy fleet which sought to pierce our naval defense of Guadalcanal. These were making routine repairs on this carrier when she received orders to put the sea immediately to check the enemy counterattack. How those CBs stayed on the job through the critical sea battle could follow is a story that construction men will still be talking about and this war is safely bound between the covers of history. This becomes synonymous with battle stations. As the CB motto, we build, we fight, is written in the blood and fire of an unforgettable moment at sea. The last tongue of flame smothered out, she was a battered ship, a badly crippled ship, but thanks to our fighting builders, a ship that lived to fight and strike again. From skipper to ordinary seamen, every man aboard had played over his head and for those 75 CBs, Admiral William F. Paul Z. Jr., commander of the South Pacific area, had a special word of praise. Your commander wishes to express to the men of the construction battalion his appreciation for the services rendered by you in affecting emergency repairs during action against the enemy. I hereby commend them for their willingness, zeal and capability. The CBs can do. There in sea approaches to Brattle Canal, our forces were ready to fan out through the Solomon's inquest of new bases, from which warships could be supplied, from which land-based fighters and bombers could carry the battle to Salamaua, Rabau, and the other strategic names that headline today's reports and will headline tomorrow's victories. Halfway between Brattle Canal and the enemy stronghold at Bougainville, Rendova was just another South Pacific jungle until the Japs made it an important stopping-off place on our itinerary. D-Day, H-Hour, at Rendova. The same clammy mist that concealed our landing operations from the enemy might also be concealing Japs snipers and machine gun nests waiting for us in the car. Army, Marine, and CB teamwork take another beachhead. Banned scouts move inland, hunting the big game that stalks this jungle. Hunting Japs, flying low, and the jungle fighting are better for the newsreels. But the backbone of invasion is supplies and throwing up an air umbrella to cover those supplies before enemy planes can blast them from the beach. CBs look at a fire. They see more than smoke and plane. They see construction to be restored. Defense is replaced. Metal to be salvaged. Dressing stations set up. But not all the CBs saw these things in the fire on the beach at Rendova. Not all the CBs will come back from Rendova as promised them a warrior's heaven. And our boys would rather have them there than on the island of Rendova. Now that the beachhead has been secured, player like LSTs slide their bowels up on the beach and begin to pour forth from their cavernous valleys the heavy equipment with which the CBs will transform this primitive outpost into another powerful bait. Heavy trucks churn their way through the mud and undergrowth. And if they bog down, there's always a good old bulldozer as versatile as a CB himself to come to the rescue. Another favorite with the newsreels is the big gun blasting enemy positions. Less familiar is the backbreaking job that must be done before that gun is ready to go to work. One giant prehistoric beast. The 155 lumbers through the jungle. Guided by CBs to the place where it can do the most good. Shelling the vital Japanese stronghold on nearby Munda Point. Once a tropical paradise or a blue lagoon, then a threatening enemy base.  Even the Munda landing strip is dead. With gaping shell holes in its side. But wherever the CBs land, the order of death is quickly overcome. But they work a new kind of magic in the tropics that brings dead island bases back to life again. But their magic has a simple formula. One part swept and one part know-how. The CB word for getting materials where you can find them. In this case, tropical coral to surface the shell-cocked landing strip. Imperative that landing strip be ready as soon as possible. That was the message from headquarters. At Munda, the impossible or even Air Corps authorities considered impossible. The CBs accomplished in five days. The CBs receive an unexpected bonus for their overtime. A fighter pilot contemplating a crashed landing for his disabled corsair spots their handiwork in the nick of time and sets her down hard but safely on the coral strip. A generous but eloquent tribute to the CBs. Incorporates Munda into the intricate network of alive bases in the south Pacific. Here launches her own offensive in the Solomonis. Rainwater becomes runaway rivers turning quonset huts into flooded islands. Roads which the CBs clear in surface are swallowed in mud. Breathing ground for a malignant enemy even more deadly and toucherous than the jet. This enemy, sighted in the microscope, the Anopheles mosquito strikes silently behind the lines with a devastating weapon. Malaria. The one fear in the minds of brave men for it has decimated combat forces and postponed military offenses. A tragic toll of American lives has been taken in the struggle against this insect army in the tropics. But the fight for life goes on. The shock troops of insecticides slog into muddy action. With the supervision of the medical high command, the CBs launch a new offensive against the hidden enemy. Vigilant sentinels of the water supplies, the CBs install filtering systems to prevent the enemy from sneaking into camp. While the streams are tropical sirens beautiful but full of death. But like our enemies, malaria fights a losing battle. The water runs pure and the men grow strong. Vitality surges through the base again. American laughter. Healthy appetite. Spiritual strength. Another triumph in the CBs constructive conquest of the tropics as they go blasting their way through the South Pacific. A new road in 72 hours. Another airfield nine days from the falling of the first tree and so the work goes on. Three by three and base by base, the CBs blaze the path that leads to Tokyo and final victory. How many headline readers appreciate the far-sighted construction program that lies behind those banner lines? Not in 1943, but in 1941, not in southern Europe, but far off in an inconspicuous Scottish hamlet, began the great offensive on Fort Christ Europe that is so rapidly gathering momentum today. Here the CBs, under the very nose of the Luftwaffe, completed the base from which Anglo-American combined operations launched its initial drive into European theater.  As the rhythm of invasion quickened, the tempo of construction kept pace and around the causeways that would have played so decisive a part of the invasion of Sicily were assembled and tested. Before Berset could play the vital role the high command had assigned to it, every dock had to be restored, every scuttled access ship clogging the harbor had to be razed. Now the curtain was rising on the big show known for Sicily, the largest armada the world had ever seen. While the democratic people of the world held their breath, more than 2,000 vessels labored through angry seas to bring the war to Mussolini's doorstep. Invasion styles, ships that carry their own portable docks, and innovation in landing operations that was originated and developed by the Civil Engineer Corps before the outbreak of the war received its baptism in fire. Not even the sailors' bridle, the blue jackets manual, can tell them how to secure the heavy pontoons that formed the causeway from which the LST's equipment is unloaded when the shore is too shallow for beach landing. For these sea bees are adding a new page to the handbook of seamanship. As coolly and efficiently as if this were just another practice landing at Norfolk Bay, our amphibious forces barge in on Signora Mussolini. The operation that had become routine in cryo landings is a severe test of courage and skill as the causeway cut loose from the LST in a choppy surf sweeps in to shore on its own momentum. The conqueror of arctic wastes and Solomon jungles finds itself another job. The bridge from ship to shore stands firm. The first cat rolls down the runway. The beach has been occupied without incident, except for an unexpected offer from a group of Italian defenders who wish to help with the unloading. The Herman Göring's dive bombers prove less hospitable. Devastating cargo and the rumbling of heavy equipment across the causeways is the thunder before the storm that is soon to sweep brushes them out of Italy. 40 minutes to completely unload the staggering amount of supplies and equipment packed into every LST. Another record for the CBs. Another reason why our invasion technique electrified the world. The CBs are the cleanup men who salvage direct ships and splintered causeways that are left in the wake of every invasion. This is beach combing on a grand scale. For putting twisted steel and charred wood back to work again, saves millions of dollars for Uncle Sam. Our modernized divisions roll along the main streets of Sicilian towns lately as they cross the pontoon causeways. With Sicily secured as an alive base, a new D-Day looms up, the most daring and significant D-Day of World War II, the first Allied invasion of fortress Europe. Now plans so carefully laid at the Casa Blanca Conference months before are ready to be translated into violent and victorious reality. As they approach the coast of Italy, they were constantly meeting up with other convoys until the sea was completely covered with ships, converging through the smoke screen on the Bay of Solano. LSTs were loaded with bulldozers, tractors, and other CB as well as infantry equipment. The sea was calm and usable. But that was only the lull before the storm, the most terrible man-made storm that was ever seen. Those rangers and CBs in the first wave were a perfect target for Nazi artillery, firing on them from the hills. But they held on. Sudden death was in the air and hidden in the sea. One of the first pontoons dropped over the side, hit a mine before it could leave the LST. There were 35 CBs on that pontoon. A bathing beach where fun-loving Italians used to sun themselves in peaceful days was soaked in blood. When you're in the spot our boys were in that night, it helps to feel that gun in your hand and to know you can return blow for blow. But those CBs responsible for setting up the beach heads, marking off the spaces for the LSTs and LCIs to come in and directing the traffic that had to drive onto the beach in the face of that terrible enemy fire, they just had to stand there and take it. The CBs fought every chance they got. But there were thousands to do the fighting and not enough to do those dirty jobs on the beach without which our forces would never get ashore. Cutting through the wreckage of trucks and tanks that had been crushed by German firepower into a wall of twisted molten metal, the bulldozers cleared the way for powerful General Sherman. While the battle raged, the CBs threw up in plump two storage houses for ammunition and equipment and dressing stations for the wounded where men who would have fled to death could be pleaded in time to bring them back to life and to their homes again. First inch by inch, then foot by foot, and finally yard by yard, our troops fought their way up the beach, now littered with broken machines and broken men.  There was seldom been so much courage in one place as on that beach at Solano, from the lowest but private to General Clark. But the CBs were the unsung heroes of Solano. And as the armies of democracy drive north on the road to Naples, Rome and further on, Berlin, every man who took his chances on the shore that night could tell you, if we ever go into action as hot as those bloody days of Solano, I hope to God the CBs are there with us. There are great works ahead for the CBs. The nation is building for victory. And no matter what the difficulties involved, we must and will obtain it. There is a place in the CBs for every fighting and working American, regardless of race, creed or color, who has a stout heart, able hands and the will to win. We have shown you in part some of the things accomplished by the CBs. We are proud of them. We look to the future with complete confidence in our ability to perform all those tasks which may be imposed upon us by our commander in chief. With that spirit, we will win.


So thank you and welcome to this talk about introduction to Kubernetes My name is Bastian and I work at a company called research gate in Berlin research gate is a social network for scientists where scientists can sign up and collaborate with other scientists upload the data and so on and we have started using Kubernetes about Roughly like one-half years ago Started with the staging environment for our developers and are planning also to use it in production and with these experiences from the last One-half years. I kind of want also share a bit of information about Kubernetes with you who's been playing around with Kubernetes so far Who's been using that already somewhere in so one two brave souls So maybe if you want to we can also chat about it afterwards because I'm also very interested in your experiences using Kubernetes Because it is a very interesting and hot topic for those of you who have maybe no idea What Kubernetes is or just read a few blog posts about this Kubernetes is a container orchestration platform That means it helps you to run and scale your services in isolated containers That means with a container. I mean basically docker images It is very powerful in that and it helps to automate a lot of the things like deployment that you usually have to care about And basically helps with a lot of these things. There's a large community behind this with lots of contributors and developers There are also lots of large companies backing that For example Google Red Hat who are contributing a lot to Kubernetes Or it originated also as a Google project and they push this open source and the nice thing about this that In comparing to a lot of other cloud solutions where you can run so somewhere on a cloud hardware like AWS Azure With run using Kubernetes. There is no real vendor lock in there because Kubernetes runs perfectly fine on AWS Azure Google Cloud Platform on your business bare metal service, of course, you have to set it up yourself then but it also runs on your laptop and There are tools like mini cube or even if you just install the newest Docker desktop client on your Mac or Windows or Linux It will have in the Edge version already Kubernetes included so All these different ways to run it make this also very easy and if you want to switch at some point from bare metal to AWS or back or wanted to see what this it's very easy to say Okay, just use your whole Kubernetes configuration put it somewhere else deploy somewhere and it will work the same way On the other hand while it is very flexible and very powerful. It has also some learning curve there So in this talk, I want to help you with getting started with Kubernetes getting introduced to the core concepts So I'm going to explain some of the basics some of the things like concepts and how things are called in Kubernetes I'm also going to start deploying like a simple PHP web application to Kubernetes with like a bit of live demo And I'm already fairly scared about that But hopefully it will work on my machine and then afterwards We're going to look into some more ways how to use Kubernetes what you can do with this with this like introducing two more Advanced topics there and also cover things like logging monitoring security additional things You always have to think about when running something on deaf on production So that you also can I can give you pointers where to look it and how to get started there But first let's quickly revisit why we actually want to run things into things like containers Why we want to use Docker images also in production? The good thing about running things into containers compared to just having a server provisioning them with puppet And then are thinking your PHP files there and then kind of reloading Apache is that it decouples the operational The ops team a bit from the developer team. So that means your ops team can basically care about Running the Kubernetes cluster scaling that and the development team can care about building the docker images building the code for a service and deploying that and A lot of things like dependencies that your software application has for example a specific PHP version specific web server configuration in Apache or FPM with site grieve write rules some extensions that you need you as a developer can just put all of that into your docker image and You can upgrade this all yourself. Whereas the operation teams don't have to care about that so it makes things a lot easier to Trip things and software in docker images and the containers Deployment is a lot easier because you just say all the dependencies you need are inside of the docker images All the libraries you need the web server or the configuration you need as well as a code And then you can just put the docker image on any kind of server that just has Docker installed and you can run it there That also means it's very easier to upgrade system dependencies like the PHP version if you think back or if you think about the most normal setup where you have Your servers provisioned by puppet or something else or chef or Ansible Then you have to sync the upgrade of the PHP version that is usually done by the assist operation tools With the upgrades that you and changes you have to do in code to fix deprecation notices And whatever kind of things that happen when you upgrade the PHP version My favorite one for PHP 7 2 is for example that account on another value You know that has a warning and since we transform all warnings into exceptions. We have lots of exception I had lots of exceptions when operating the PHP 7 2 so having this like at one process of what the problem process is very nice Also, it makes it easier to scale applications because if I just have a Kubernetes cluster was 1020 whatever service all looks the same And I have my web application my service in PHP and I want to scale you need to scale it out to more Machines more instances. It's just like putting the Docker image on additional instances without having to provision anything with additionally This puppet and having to wait there Also, it makes it easier to develop applications because if I already have a Docker images image with all the Dependencies that I need for my service. I can also run it locally on my machine That means I don't have to install Like specific PHP extensions compile things on my laptop anymore And maybe also have different versions of dependencies between different developers And I don't have for the care about all the bucks that arise because of that And of course you can say yeah, I can also do this all with virtual machines and vagrant and virtual box The thing is Docker images are just way more performer because the processes run natively on the system where it's running on instead of like virtual machines that run somewhere in virtual box Okay, so you're thinking sold. It's very nice Let's then talk a bit more about details and Kubernetes first I want to define some color concepts of Kubernetes and tell you a bit about the verbs and About the resources that they are so we know all what we're gonna talking about And also if you're reading about Kubernetes a lot of these things and concept I'm going to talk about you will find in blog posts. So it's good to know what it is First of all, you have the Kubernetes cluster basically Kubernetes clusters all Notes all systems or servers where Kubernetes running on and where Kubernetes can run Docker images on it's called a big cluster On the small level you have an image an image is just a Docker image that contains all Things that is built from a Docker file and that contains all things that are necessary for a service to run all dependencies PHP Apache all the libraries you need all the app packages you want to install as best as well as the source code this image when you run it it runs into a container process on the on the server When you run an image you can only run one process inside of the container Of course, this process can then also run sub processes But it's not like a virtual machine where you can start different servers in parallel You may need to have an orchestration source for that It's also a very good practice to only have one process running inside of container Instead of like having one container that contains Apache and PHP FPM and a memcached server and whatever So you want to distribute this away if more concrete examples in a few seconds Because you oftentimes have the need that you not own for a service that you don't only have need like one process But multiple processes you may want to bundle them together and Kubernetes. That's called a pot A pot is a group of one or more containers that share the same port space So that means everything behaves like it's kind of a local host if I have one here One container that runs for example peach pfbm and one that runs engine x and you next can call call Peach pfbm or like it is on local host on the other hand both containers don't share a file system So they are very isolated. They don't share a process space. It's just a port space that are shared Then in Kubernetes Oh, yeah, and it's also important any every port that process is open here by default are not accessible from the outside of this pot You can think a bit of a pot like it's like to like an additional system It would be a server like everything that you have run locally on one server for your application would be every process Would be a container and every the whole server would be then a pot Then in Kubernetes you have the concept of a replica set Now replica set you can that defines and manages how many instances of a pot should be available So in this example have three pots here in one replica set and the replica set decides then okay on which server this pot should be These different pots should be a start and executed Around that in Kubernetes you have a deployment the deployment manages updates of replica sets So if you want to put a new version the deployment managed in a rolling way that the replica says updated Pots get all ports with the old software version get shut down and new pots get started up And it also ensures that your whole service is available through the whole point in time So it doesn't shut down all services instance at one point. I Said a few minutes ago that By default every port that is open in one of these containers is only available within this pot and not to the outside To any other services running there in order to make ports available to something else running in Kubernetes You have to define a service and that makes ports of pots accessible to other services running in Kubernetes But that's only within Kubernetes if you want to make it accessible to something outside of Kubernetes for example to a web browser or some services are not running inside of Kubernetes you have to define an ingress Which basically makes a port of a service accessible with a URL to the outside I will show you examples with concrete code and afterwards and then also this hopefully becomes clearer to finish this up Server a physical server is called a note But usually you don't care about notes because Kubernetes all does this for you to get automatically it decides on its own Where a pot should be deployed and where containers should be running Then you have config maps and secrets to store configuration in key value store These config maps in a key value store can be mounted as volumes inside of the containers where it's needed Also more examples for the later. Also, you can use volumes to get persistent storage and You also have a concept of namespaces which has an additional horizontal layer of encapsulation there We can build up features like every developer has its own namespace Can deploy their own versions of the service they are working on without affecting other services running in other namespaces So it's a way to encapsulate things So let's look at a more concrete example. We have a PHP application pot in The PHP application pot in this example We have a PHP FPM container running for serving PHP files on executing our PHP application We have an nginx container running as a web server to accept HTTP traffic and to serve static files We may have a memcached container running here for some local caching that we want to run in there and some MongoDB router for accessing MongoDB databases in the MongoDB cluster we're running as that's the demon for Monitoring and also some linker the demon that I'm going to talk about later So this would be like everything that's traditionally would run on one server And we have every process in its own container and this makes one pot One instance of our service and in a replica set we can say okay of this pot We want to have two instances. So if we deploy something that we can take this instance down and This is those serving traffic we build up a new instance with a new version and then take this down and do the same thing and The deployment instead of communities does this automatically for you without having you having to do anything Then we can store some configuration files in a config map or some configuration values in a config map because it is good practice if you have configuration that is especially environment specific to not put it into a docker image with your source code because you want to use the same docker image with the same source code in the service version in your depth environment as well as in your staging environment as well in production You don't want to have different docker images for different environments But on the other hand you want to put the configuration into a config map and mount this for example in the FPM container Also, we can define a service here that exposes port 80 from under necks to other Services running Kubernetes so that they can access our pot and then At last we can also define an ingress that then provides a URL like PHP app something So that our application is accessible from the outside To interact with Kubernetes and You need also a bit of tooling and what you first see is a tool CLI tool called Qubes ETL You can just install it with homebrew or whatever a package manager you want to use and if you use docker For Mac or docker for Windows or something also it comes Qubes ETL comes directly with it. So it's already installed Just an example if you run something like Qubes ETL get pots It will give you a list of pots that are available in your cluster and You can also get all kinds of other information about the cluster as well as modify resources create new deployments And it's a very powerful tool to do everything there in the end Qubes ETL Just uses a rest API Which also of course can use on your own to build your own tooling around this if you want to an example here to use The rest API you can also just call it and then if in this call we from the default namespace We also list all pots and we get the same results just in JSON format There's also a nice project called Kubernetes dashboard. This is URL down here Which this is a screenshot. I'll show also show it to your life in a nicer bigger font afterwards That also gives you the same information as Qubes ETL the same possibility is just in a web interface so you can click around there and Last but not least there is also there also Several projects around Kubernetes that try to solve common problems for example one I really like is harm as a package manager for Kubernetes and there you can do some thing like installing Packages and solutions that you commonly install for example wordpress where you can just say harm install stable wordpress You may give it a few more configuration options like a title some database credentials on and then it will do everything for you With like commonly shared configurations But let's look at a practical example We want to deploy a symphony demo application for that We have to first prepare our Kubernetes cluster a tiny bit first. We have to install or Docker client on a machine But if you want to do this locally Docker client installing on a Mac would be blue cast install Docker and then we have to in The Docker client from preferences We have to activate our Kubernetes cluster because by default Kubernetes is not started up because it uses kind of a bit of resources and also I figured out while working for this talk It drains battery a lot if it's running That is maybe one of the downsides there So we enable that it takes a couple minutes for everything to start up So I already prepared that on my machine So then don't have to wait for that and then we may want to install helm as a package manager Which also you can just install with a homebrew or your package measure of your choice on your operating system We have to initialize harm once so that it installs and configures your Kubernetes cluster correctly It just run home in it and then it does it and then we can start installing things like the Kubernetes dashboard with with a bit of configuration here so configuration various would be that we just give the UL for a company is that what that we want to use later on and then we can say harm install the Kubernetes dashboard This is configuration and it will install the Kubernetes dashboard and then also we need to accept traffic from the outside and Ingress controller for that. I'm using engine X. I'm going to talk more about English controllers later on so just take it as a given now that you need it and Same thing Helm install an engine X. Angus controller and then we have it and can also accept traffic from the outside These are kind of the preparations we need for our Kubernetes cluster and then let's start deploying the symphony demo application The URL file URL is here. So it's really just a symphony demo application and Yeah, let's hope that it works alive So let's synchronize the screen and Here in PHP storm. I have the symphony demo application in here. So I just did Composer create project symphony demo application The all the sources are downloaded here and I added a couple files like a docker file and the Kubernetes deployment configuration Yes. Oh, yes, sorry How do you make it bigger at the presentation, but yeah, thank you So this is a bit bigger, but I would just want to show you that and let's go maybe to the The shell first because also I already just to show you This application running so I'm just starting it now locally on my machine and if I go here Okay, that works. So I have this demo application running locally. Yeah That's not very interesting because we want to deploy this to Kubernetes I've already deployed one version here in my local Kubernetes cluster so the same version is here and What I want to show you is just like how we can get a change to our source code through the pipeline with building a Docker image and getting it into Kubernetes so let's go back to simp to PHP storm and let's go to the template here and Add like some tech Hello peach Okay, and Then let's get back to the console the first thing we have to do is we have to build a docker image for that so Let's maybe bump the version here, so I'm just saying now a docker build. I'm saying the name is symphony demo and I'm giving it the version 1.2.0 So I'm gonna build this now this gonna take a couple seconds while This is running. I can also show you the docker file that basically defines all the dependencies in here that I wrote for it Okay That is not nice. Okay. Let's restart that and hope for the best It worked earlier promise, but let's just go to the docker file So for the symphony thing I'm actually using a feature called multi-staged build in Kubernetes in docker because what we need is we have to build all the JavaScript dependencies once for that we need Node.js But we don't need Node.js in area afterwards to run our PHP application because all the JavaScript is built We just need an Apache and PHP there. So in the first step here I am saying okay I'm depending on a base image that just has Node.js and yarn because the symphony demo application is not using npm But the yarn Is and I'm just basing on that. I'm saying my working directory is var 3w html I'm copying the package JSON and the yarn lock file into that because it's the only thing I need to build my Node.js to do a yarn install. I'm running the yarn install and that's it Then I'm saying okay for my actual package for my actual image I'm basing then on the PHP 7.2 Apache Image so an image that contains PHP 7.2 Apache and some default configuration already I'm saying also my working directory is var 3w html Then I'm installing a couple packages that I need for symphony to run So this is this part here basic Debian packages then I'm Installing a couple PHP extensions at symphonies like curl and SQLite JSON and so on I'm activating a couple Apache modules like headers and rewrite. I'm Adding the necessary Apache configuration to it Then I have to install composer here Then I copy the composer lock and composer JSON file to my Docker image and then I can run composer install there After that after a run composer install and then can copy from the other Node.js image the node modules Inside of my current image so it copies from the first Zero's Image all the node modules to the current image and then I copy all the other sources into it And then I have to basically say okay I changed the owner of all those sources to 3w dash data because that's what the patch is running under and So that all the files are accessible the reason why I do this is complicated and not just do like copy dash that and then run composer install is This way if you because of the caching machine mechanism that Docker has Own I will only run a composer install or yarn install if my package JSON or composer JSON or composer lock file changes If I do any other code change Like code changes to any code. This will all stay the same because the composer JSON didn't change it doesn't have to run composer install Again, it doesn't have to copy the node modules and it will just Copy all my new PHP files or my new template files into the clock image So it makes just like building new images and iterating of it a lot faster and then let's hope This is all up running and let's hope that this is working now It looks a bit better Okay While that is still building I can also show you the next step because this is how I build a docker image The docker image is now lying locally on my machine and I need to get this some more incubators for that I have to write a bit of deployment configuration So I added a bit of configuration here Let's get this all over again. So the first thing is I defined a deployment Configuration incubators is usually done in JSON or YAML files And you often usually say see YAML files for it because they're just a bit better readable and less characters You have to write it compared to JSON So here I'm defining a deployment like think back there was deployment and replica set So I'm defining a deployment. I'm giving it a name Symphony demo and then I'm specifying this deployment And I'm saying here Basically inside of that is like the replica set definition And this is the pot definition and the pot definition is a list of containers I'm run and in this example I just have one container which is which also names if we demo and which then is Running the docker image Symphony demo in this version. It's one. What's one dot one dot three. So let's change this to one dot two dot oh Also, I defined an image pool policy I said never because I'm all everything running normally Coordinators would say, okay, I'm he wants to get a symphony demo content image So I'm gonna go to docker hub or whatever docker registry that's on and trying to pull it there Since I have everything running locally and I didn't want to push it first to the docker Just and then pull it again to my local machine I said, okay never try to pull just to make it on locals further. Normally, you would not do that Then I'm saying, okay, this container opens port 80, which is the Apache port And then I'm defining down here Lifeness probe and a readiness probe for health checks. So I'm saying hey Check frequently here every second on port 80 do a request to slash and only if that works and returns to 100 This container is available for serving traffic because Apache needs a bit of a boot up time Next I'm defining here a service To expose the port 80 of the port above to the port 80 under the name symphony demo and Then after last I'm defining ingress while I'm exposing the port 80 of the service symphony demo up here under the host name symphony dash demo local dot KS and Also a bit to cheat and make it easier. I also adapted my ETC host file to just serve symphony demo Local KS on my local IP address Okay, so let's see Yeah, everything worked. So down here. It says successfully attacked the image Let's double check that I did everything correctly and updated this year to one to go and then We can say, okay We have the image now locally here. So let's just say cube CTL apply dash F and the YAML files in the deployment folder. So I want to apply all YAML files that are in the deployment folder to my Kubernetes cluster And then it says, okay The service is unchanged. So it Kubernetes apply didn't do anything the ingress is unchanged But the deployment changed because I bumped the version So communities did something and if we just look at The pots that are running inside of my cluster I see there are Two symphony demo clusters that just 17 seconds ago restarted and If we go to the browser and do a reload here. Yeah, it worked So This is like a bit. So it's actually fairly easy like the configuration you have to write for this small demo is not a lot Just a couple of lines Of course in a real world scenario you have only have a bit more things like you have a database and stuff like that So it gets a bit more complicated or it's get a bit more, but it's also Not that hard and there's good documentation for it as well So let's go back to the slides a bit So we had the Docker file This is just my backup file if the demo didn't work. So we had the Docker file was installing everything and Installing composer running couple install The command it actually runs is inside of a rocker file already inside of the base rocker file So the command runs just patchy to the foreground Then we build the image Then we don't need to push it because it's all local then we have to tell Kubernetes what to do with the image or So it's a yaml. So we have the deployment here with the version down here. We have the health checks Actually what you can do in these deployment configurations in the container configuration is a lot more So it's not just these complex settings. You can also define many more options For example, you can set environment variables You can mount volumes there to get persistent data or to put configuration into it You can request resources saying okay my pot or my container needs like 200 megabytes of memory So companies should ensure that wherever it schedules the pot to run 200 megabytes of memory should be available You can define rollout and deployment strategies the command you can override the commands that are executed there You can configure network settings Affinities where you can say okay on one physical note only one or two instance of a pot should run in parallel Just for scaling issues You can create like attach life cycle events so that you can execute certain things if the pot starts or stops Things like that and a lot more and We also defined a service to expose it to other things running in Kubernetes and we defined an ingress Great, and we created everything with cube CDL apply, and it looked like that Symphony demo. Yeah, and we did deployment Great. So these are the basics of getting a service into Kubernetes Let's talk a bit about more other interesting and important aspects of it because there are other types of things that you can deploy and Do in Kubernetes one of them is called a demon set So we have deployments which is usually a default way to get a service into Kubernetes a demon set is a special kind of deployment and A demon set ensures that a certain pot runs once on every physical note Usually you don't need that you don't care about physical notes But for some things like lock collection demons I'll come to that also later monitoring agents You want to have one instance on every physical note and it was a demon set you can do that They basically work and are looking like deployments Just the rollout strategy and deployment strategy is different and thus you also have to compare configure them annually Documentation on that is in the good Kubernetes documentation repository and this is like a bit of an example So the kind here is not deployment as before the kind is demon set I also I'm giving a name and labels to find it later on the spec would be the same for a deployment I would have my containers down here The important thing is that I have to define an update strategy how the demons that should be updated in what way For example here I couldn't do a rolling deployment or would I would just say I don't care if I lock demon For example, it goes down then it puts up with a new version and then it starts collecting the locks Where it stopped before so they wouldn't care for it for other things. I may want to do is enroll in real way also What you can deploy in or what's great supports are cron jobs So these are good for regular repeating jobs that you want to run like on a with a cron demon usually they have on your system And it looks also very similar. The kind is here cron job. So I'm not demons that are deployment I also can give it a name then I have to give it a schedule How often it's and when it's supposed to run basic cron tap syntax and then I have the same kind of template spec where I have in the end a list of containers and There it's a bit expected that these Commands that run into this containers don't run forever like Apache runs forever until you kill it or it breaks But they should finish at some point, of course So this is good for any kind of cron jobs you want to do and Kubernetes then cares on its own where it's going to be executed various their space You can also say I need resources for that and specify I need like for memory like one CPU for it or so on So this is a very powerful tool also to run the cron jobs in a smarter way than having like one machine Where all the cron jobs are defined if that dies you have a problem Let's talk a bit more about how Kubernetes works internally and One important thing there is that you have to be aware of a service discovery So I'm not going like very deep into like internals how the scheduling works and how Kubernetes decides where to run a pot or that but service discovery is very essential because usually A bit more complex system at least have a web service like in PHP and a database and they have to talk to each other somehow Within a pot as I said you share a pot namespace. So everything behaves like local host But it's get a bit more complicated between pots Because usually in the example, I have a PHP application. I have a database I don't want to put the database in the same pot as my HP application because I will need to scale the web service and the database differently Also, if I deploy a new version of the web service, I don't want to restart the database It doesn't make any sense. So usually I have one Deployment for the database with its own pot and one deployment for the web application with its own pot And they have to talk to each other for that. You have to expose the ports with services So to revisit this this is a configuration where I'm saying hey This is a service and I want to make port 80 of the pot available to something else Internally what Kubernetes does that for the service name symphony demo, it will assign a virtual API address And you actually can see this so if you run QPCTL get service and then the name of the service It will tell you okay There's a service symphony demo and this is the virtual IP address for the service and this is a post Ports that are exposed on this IP address this IP address is virtual and with IP table routing Kubernetes if you just call this IP address will automatically Forward the traffic to the IP address of one of the running ports that are available. So it does Kind of like round robin local and saying they are automatically for you. The next thing is now, you know Okay, there's a service under this IP address. Somehow you have to In the service that wants to talk to the symphony demo application You have to find out what the IP address is and since this is not can change at any point in time It's not stable. You shouldn't hard-code it But it's easily discoverable in other containers either by environment variables because every Container that is started will get a lot of environment variables automatically injected by Kubernetes And it will get one environment variable for every service that is in the current namespace So in this example, I would have a symphony demo service environment for the host and port That gives me the IP address and the port that I can connect to so in my application I could read all this and these environment variables and then just go to these IP addresses or even more convenient Kubernetes also comes with a DNS interface for that so I can also just use the name of my service To look up the IP address and get it. So in the end I can just do curl symphony demo or use guzzled and go to HTTP symphony demo and in another container and accesses to that Alternatively, I also can use service mesh tools A couple examples for that that are fairly popular are linker D STU or conduit and These service mesh tools run either as a demon set in the cluster So that I install an instance of linker D For example on every physical node that I have and then my pots can use this to connect to other services or As a sidecar container, which would be this example that you've seen before that in my pot I don't only have like PHP FPM and nginx But I also have an instance of a linker D service mesh container and what I can do then is if my PHP FPM application wants to talk to an instance of this other service that is running node.js It does this request first To on localhost to the local linker D instance that we're just running on port 41 40 or whatever So it's going to localhost 41 40 saying hey, please Do a request to this other service wherever it is the linker D instance will look then on the Kubernetes API up Where on which IP addresses this is running? Basically the same as when it is already doing and then basically forward this request to one of the pot instances It is very similar what Kubernetes is already doing Out of the box. So why would you use something like this? Why would you put up another process to do this which also needs resources or that it has the benefits that you Can do advanced routing for example, you couldn't do something like Prefer the service that is running in the current namespace for example if I have my developer Bastion namespace I can just say okay. I deploy one service there But the other service I'm not caring about because I'm not developing on that So if the service is running a current namespace I go to that Otherwise it can fall back to the default namespace So these advanced routing things you can't do with Kubernetes out of the box so that you need like some kind of Service mesh mesh technology, but more importantly it also makes it much easier to add monitoring to it because all these tools come With my monitoring directly built in so what I can see there Then I can have dashboards. This is not really nice Okay, believe me what's in there that you can then automatically monitor the success rate of these calls to one service The latency and the request volume so that you have this monitoring already built in otherwise This is a bit more of a black box same with profiling all these tools Also support profiling tools like zipkin for example where you can follow one request that comes in at the beginning of your community discuss That's all the calls that you're making to other services databases and you get like a big break Large breakdown of different service calls and can profile this the whole way through with all the whole so that you know What time amount of time you spent on what time? In which part of your infrastructure on which service and it's not a black box anymore why a request took like two seconds So this is why you may want to use one of these service mesh containers as a sidecar as a demon said even if it uses If you if some of the things committees can do out of the box Also, let's talk a bit more about accessing things in communities from the outside For development you can also just use port forwarding through Ctl So Ctl has a functionality where you can say okay We can call Ctl port forward and then just enter a pod name here Then we can say the port 80 in this part. We're just going forward to Port 88 on local host on my local machine and it doesn't matter where the Kubernetes Cluster is running if it is on my local machine or somewhere on cloud platform or wherever I can basically make any port of any port accessible so that which is nice for development Of course, it doesn't work for users They can't you can't tell them use Ctl to access the website For that we need Ingress define ingress and we need to deploy an ingress controller You saw in the beginning with the preparations that I did like with harm deploy an ingress controller an ingress controller Usually is either in gen X or h a proxy But there's also Istio that can be run as an ingress controller and also potentially any kind of other tools and the way it works That an ingress controller is also just a docker image where engine X is running and there is like a special Demon Process running also inside of the docker image and that listens on the Kubernetes API for all ingress resources And as soon as there's a new ingress resource it rephrase the engine X configuration and reloads engine X and then forwards the Traffic that comes from the outside another given host name to the correct service to the correct virtual IP address So That is like if I define an ingress resource like that Saying I want to go to the service name symphony demo port 80 if anyone comes in with this host name Then the ingress controller will pick up on that Listen on that and then will automatically rewrite the engine X configuration or the h a proxy configuration correctly to support that Let's also talk a bit about data and storage because we have been talking about so far We haven't seen anything there We have seen things like a web application that is usually stateless or all the data that is in there can also go away Any time at point in time because a problem problem with docker images and Kubernetes is that you can't ensure that the image is running infinitely At any point in time Kubernetes can decide okay. I have to re-square your things to make space for other things so I'm gonna Can't exit the port stop it and put it somewhere else And of course all data that was created inside of the port is in lost because the data is not stored anywhere In order to have persistent storage we need volumes Documentation more documentation there is on that And this is like a very simplified example We have our port definition here with like one container and we also define a volume We name this volume cash volume in this case is just an empty directory So this is good for caching things because this empty directory will also go away when the pod cancels again But just to give you a very small example. We can then can say we mount this Volume this cash volume inside the pass slash cash So all the data there is an in slash cash and it will be stored not into the container directly But inside of this volume, of course, this is like this example. You probably don't need that often with an empty caching There are also other types of volumes for example persistent storage volumes for that You have to define a persistent volume and there are tons of providers there that you can use for example It can be volume that is on some NFS share It can be on S3 or all kinds of other storage technologies that you may need and then the pot can Specify a persistent volume claim saying I need a volume with like five gigabytes of data Please Kubernetes ensure that I get that Kubernetes will ensure that and this persistent volume claim can then be mounted inside of the container the same way as we saw before More information on that on persistent volumes can be found here Of course, this is also a bit more to configure than because you have to configure NFS and the storage itself And make sure that this is available and depending on the storage solution also latency of course is different If I have an NFS share, it's probably way faster than saving things on S3 So it's not good some storage for storage solutions for example I'm not very good for persistent caches or something like that and if I have a MySQL database I should not use an S3 storage volume for that So you have to kind of like it depends if you if I have file uploads for example S3 may be a good solution Also I already said Configuration is one thing to keep in mind because we don't want to store configuration inside of a Docker image We want to have this injected from the outside into a Docker image Because it's not a lot of data in configuration size-wise We don't need persistent volumes for that We can just use something else called config maps which is built in key value store and the Configuration in a YAML file could look like this So I have the kind config map that I also can just apply with kubectl to Kubernetes I give it a name special config special config here and in the data I have two key value pairs a special key with a value value and a bool value key with a value true and This Config map and this data here can then be accessed in the port either through environment variables so in my Container definition I can say okay with this NFR these I want to define a environment variable variable called special key and The value from for that environment variable should come from the config map Special config and I want to get the value from the key special key So then I would have an environment variable here Also, if I need I want to create environment variables for all keys in my config map I can also do it like this and I'll say okay Take the environment from this configuration this config map and then it would automatically create all of them It easier maybe or I can access also config maps for volumes where I can say okay I want to create a volume and It's not an empty dear here or not a persistent volume claim, but it's actually a config map I want to reference this too So this is a company makes special config and then I can mount this config volume inside of my container here under the Pass EDC config and what this will do it will create a file for every key that is inside of my config map and It will the value or the content of the file will be the value of the key So these are the different ways to do configuration Also, I put a reference here to like more things to read up on that Also, there's a concept called secrets which behave like config maps But they are there for storing more sensitive information like passwords because they are additional access security concept around that Also, I can't go into much more detail there because it gets very complicated quickly with C But usage-wise they behave the same as config maps and if you want to read up on that this is a reference Additional thing with Kubernetes I want to talk about quickly because it is something that is very common for PHP applications is a complex in its in its processes For that Kubernetes supports in it containers, which allows us to run a process inside of a container before the rest of the pot starts So before I start my Apache or my memcache or whatever The use case for that could be for example database migrations Like running them before the pot starts could be interesting or if multiple this is a more common one if multiple containers need the same sources What do I mean with that if I if I'm not using Apache, but if I'm using engine X and FPM my PHP application would look like this I have PHP FM running in one container. I have engine X running another container engine X should serve all the static content Like CSS JavaScript files images because I if I'm getting want to get an image I don't want to serve this through PHP it would slow it down Unnecessarily and PHP of course should execute my PHP scripts and serve traffic for that Of course, that means both of these containers have their own file system, but need access to my source code for that so That means when I want to build that I actually not only need to build one Container for my PhD application But actually to like an PHP FM container with also source code and then engine X container with all the source code and PHP applications tend to grow quite large So several hundred megabytes sometimes is all the images and all the resources there That means I have two very big images that I have to store my registry and update and this is kind of like tedious What you can do there is instead of that you create an init container Which just is like a very basic image that contains your source code and maybe our sync as a tool And then you can define this in communities as an in container and the command for that is so this connects only the source code Basically, and what it does then in a container you mount an empty dear volume inside of the internet container Then you can copy with our thing for example all the sources from the internet container into this empty dear volume Which will also go away when the pod gets stopped what gets stopped afterward, and then you can mount this mount this volume Both inside of the fpm container and the engine X container which both contain only engine X and fpm and not the source code But then the source code through this volume is accessible Sounds complicated sounds slow It's actually not because it's all on the local file system and there is no virtualization between that It's all done with Linux kernel features, and so it's the same speed you get there But you don't have to build like two big images only one Let's talk a bit more To finish it up about figuring out what's going on inside of Kubernetes for that. We need one monitoring Kubernetes comes with an additional project called heaps there Officially maintained by the Kubernetes project and heaps that allows us to take Take metrics from Kubernetes that Kubernetes already exposes and store them into any kind of monitoring solution that I have Things like influx to be from ethos graphite Whatever and then I can tools use the monitoring tool I may already be using like Rafaana for displaying the data for example, this is already quite small again So believe me here would see the overall cluster CPU usage the CPU usage by note Individual CPU usage on like IP addresses memory usage and so on so I get all this information in a time series database And not only on the cluster or node level, but also on an individual pot level so that is like kind of very convenient to know and See what is going on into my cluster? I Also put like a bit reference to a blog post there that explains this in way more detail. The second thing I need to know is logging Kubernetes, that's what the Qubectl comes directly with a command Qubectl locks So I can run Qubectl locks and then any kind of pot name to get the locks of this container of this pot there Which only works if your service locks not to a file inside of a Docker image But if it locks to standard out or standard error Which is good practice for running Docker images anyways because the file system is going to go away as soon as the image restarts So any kind of log files are lost so instead of that I should log to standard out and send error and that Kubernetes the container orchestration platform care about persisting the locks somewhere and Kubernetes locks everything that a container locks out on standard out or standard error Automatically writes it to disk on the local node where the pot is running and then I can run a demon set lock collector Something like lock stash fluent D or file beat also inside of Kubernetes A demon set runs once on every host and this lock collector would then just mount the lock folder like while lock Docker or why lock containers where all the locks are in and then tail on these locks and forward them to your central log management solution for example elastic search with kibana and then you can have it also automatically enriches all the Information or the lock messages with additional information from Kubernetes like what pot this lock messages from what services this More lock message from on which IP address it was running so that you can nicely filter things here For example, I can't really see this. I hear filtered for all Containers that we have in the name switchboard service in the namespace default and I can Then accesses as long as it's data stored in elastic search. There's a good blog post that also explains is a bit further from elastic That is called shipping Kubernetes locks to elastic search was far beat also that can you can read up on that? Earlier I also talked a bit about like scaling is very easy with Docker containers And I want to show you how easy it is because you can do things like manual scaling very nicely and Let's for that do the last short demo So if I'm gonna go here and let's look again I have here two symphony demo pots, which I've been running for 25 minutes now, right? So Let's say cube CTL scale Demo situations There's replicas and I want to have three now for the deployment symphony demo So it said now Symphony demo scaled and if we look at the pots that are running We see no three and One is running since 11 seconds So that is how you scale in Kubernetes manually So that is makes it is very easy. You don't have to do anything any provisioning any puppet magic Just say cube CTL scale if you need more instances Also, you can auto scale in Kubernetes So there is a thing called horizontal pot auto scaler and what it does is it's basically automatically watches How many resources or CPU usage? For example, your individual pots have and you can configure thresholds and if it goes over certain threshold Kubernetes automatically spawn up a new pot and also stop pots again if it goes below a certain threshold Which is very nice if you have for example batch shops that hit some servers very hard during the night when they're running and Then communities can directly detect that up to a certain amount of pots put up Spawn up new pots dynamically and also stop them again Very similar to also what AWS can do of course You can also write cron job that do that beforehand if you know every time every night at 2 a.m I have this one thing running that needs like 500 percent more resources on one service. I can also automatically do this before So this all very powerful if you want to read up more on that there is this documentation about horizontal pot auto scaling and Lastly keeping services secure you should read up on Docker container security Because it's very nice that everything is encapsulated But you have to also configure it Docker especially running Docker images in a way that if someone goes gets into a document image Like an attacker you can't get out of that or can't get root access to anything More information here is on this documentation conferring a pot container Security contacts and also you should maybe think about depending on your Organization about role-based access control because by default all users have access to the whole Kubernetes API Which may be fine in a development environment, which may be fine your company completely But in some organizations you want to limit this further for example You can say if I have Dedicated namespaces for every user a developer the developer has full API access to his namespace But not API access to other people's namespace things like that You can do also configure in communities with role-based access control And you can do this for users as well as services that you can also allow certain service only to access certain resources More information there also in the documentation and I'm going to point on that And I guess if you could talk a whole hour about role-based access control in Kubernetes because it's very powerful You can do a lot of things there to sum it up Kubernetes is very powerful. It can help you greatly with And also very helpful so it can help you greatly with automating deployments making this all easier for you scaling things It has a learning curve and also it's very fast-paced in development There are four big Kubernetes releases every year and a lot of things are added there changed not really changed in a backwards Could be a VC breaking way so they're very good with that But there are always like new things where you can this can make your life easier So there's a bit of information here about the release cycle But that also means you have to stay up to date if you're using Kubernetes to Benefit from all these new things that are happening where you can do things better in a more performant way the good place for that is the Kubernetes documentation which is Fairly large, but also very good with lots of tutorials for things. So they're getting better and better also writing documentation But what also can recommend is Cube cons which are a dedicated conference for Kubernetes and the nice thing about this They record every talk and a few weeks later. They're on YouTube So even if you can't afford the ticket to go there It's also very good to just watch some of the talks to know what's going on there And especially there's the last fast pace also on Cuban is tooling or cooling around when it's like helm Or things like that. They also change rapidly and advance rapidly because it's still a fairly new technology and ecosystem Also, I'm gonna upload the slides for reference with all the links later on hopefully today latest tomorrow morning on speaker deck I'll link it on joined in and also tweet about this and speaking about joined in It's a great platform where you can give feedback for the talk and I would encourage you to do that Because feedback makes my life easier to improve this talk and also the life of the organ is easier because they also know What you liked and can make the conference better next year So go to the joint in joint in and trade all the talks you have been seeing so far on the conference And they're gonna see in the next few slots


Marine protected area, MPA, monitoring has become increasingly important in order to ensure that conservation goals are met. Non-extractive monitoring techniques offer a range of advantages over traditional extractive sampling methods, including reduced costs, minimal disturbance to the environment, and improved accuracy. This paper reviews the current state of these techniques, discussing their capabilities, applications, advantages, limitations, and potential future developments. This article was authored by Ryan McKeity, Ryan McKeity, Robert Mrenia, and others.


And it always seemed like grand mom's back in the day was old.