Okay. Good afternoon everybody. My name is David Ke and I'm from Shenzhen of China. I'm honored to be here. I will share something about building our financial cloud. I will focus on first three questions. For the fourth part, I will have two experts to introduce and explain the solutions provided by Mirantis. The first question is what kind of clouds are we building? Before I answer this question, I want to tell you the background about our company. Shenzhen Stock Exchange was established in 1990 and it's one of the two stock exchanges in mainland China. Shenzhen SE has 1,700 listed companies with a total market capitalization of 514 billion U.S. standard by June 2015. It consists of the main board, MSC board, Chilex market. SCSE also has a multi-tiered capital market system for issuers at different stages of growth. Its products include equities, mutual funds and bonds. SCSE, Shenzhen Securities Communication Corporation was established in 1993. Its sub-company of Shenzhen stock is changed. It provides communication services including market data, trading and caring for Shenzhen stock market. Meanwhile, it provides a series of IT services for financial institutions. Let's look at the picture on the left. From 26 to 2014, the size of Chinese asset management has increased from 3 trillion to 36 trillion. That's a 10-foot growth in 10 years. On the right picture, the monthly turnover of Shenzhen market has soared from 2.2 trillion in 2014 to 14.7 trillion in May of 2015. The highest one is about 6 times the lowest. As a result of the rapid growth, securities market in China required advanced IT systems with high capacity and resilience. In the past two years, the size of many assets has a constant expansion, while more and more financial licenses were issued. The number of financial institutions also increased a lot. By September of 2015, according to the registration information from the AMAC, Asset Management Association of China, the number of private equity funds has reached 20,000, increasing by 200% comparing to the early of 2015. They have managed the assets of 4.5 trillion. This small or medium-sized organization helps to focus on the business which can show their professional abilities and also the long-term IT to other professional institutions. We see the huge demand of IT systems on the above slide. However, we still have to face two major IT problems in current security market of China. First, the traditional IT framework has some problems such as tight coupling and vendor locking, high risks, low efficiency and slow response to the business. So the traditional IT framework needs urgent reform. Second, lacking preling and co-ordination scattered resources the industry's infrastructure lacks behind. We believe that cloud computing is the most effective technological means to meet the market demand and solve the business problems, especially for its features such as elastic expansion and quick response. However, the private cloud has some problems like small size, technology weakness and slow application migration. It is also difficult for the public cloud to fulfill the requirements of data security and regulation. For the time being, by combining both industry and private clouds, the hybrid cloud structure is relatively suitable for the financial industry solution in China. As one of the main IT provider in the financial industry, the SCCC is now constructing and operating financial cloud. This will be our main strategic transformation. The financial cloud that we are building is an open, cooperative and win-win ecosystem. Based on an open cloud platform, many as with independent software vendors provide a rich application software services, each financial institution can choose from two or more suppliers. This diagram shows the layer service architecture of our financial cloud. Our company mainly focuses on offering the basic services, like ICE, and we work together with our partners. Some ice weights to offer pass and sus, customers can use various service applications they need. This financial cloud has four features. First, open. We use the mainstream open source cloud platform, and our platform is open to all our partners. Second, neutral. When standard stock is changed as our shareholder, the SCCC has enough credibility and the independent third party on business. So there is no commercial conflict between our company and customers. Third, we have professional infrastructure, technology platform and application service. We present our expertise. Fourth, with service experience of more than 20 years, we can ensure the data security for our customers and regulators. Since 2014, we have launched our financial cloud project. We have established several data centers in China such as Shenzhen, Shanghai, Beijing, Dongguan, Hong Kong, Wuhan and some other cities. Our financial cloud has the ability to offer service to customers all around China. I will talk about our new data center under construction in Dongguan. This center covers an area of 130,000 square meters with a 3 billion RMB total investment. The first phase investment is 1 billion RMB with a total capacity of 20,000 racks. The first phase is 5,000. This is a core production center of our financial cloud. Then let's talk about the cloud management system. This system manages the infrastructure of our cloud including computing, storage and networks. It handles all the resource requests from application systems. This system is an essential software component of our financial cloud. Later, I will focus on discuss our choice. After the exploration and practice in the last two years, we have successfully run a series of industry applications on our financial cloud. The first one is a market data cloud. We have established ICE platform in four cities in China. And Securities Company set up their service on this platform. With this service, investors around China can get real-time market data information through internet. With this cloud, the securities company can reduce costs by 30% and this response time of system expansion shrinks 90%. Up to now, the maximum number of online users at this same time is more than 1.2 million and more than 12 million visitors daily. The second one is fund cloud. Our company offers basic ICE and ICE offers applications that mature funds need. Fund companies run the standardized SaaS service on demand. As a result, it can reduce operation costs and save the provision time. The third case is our business innovation for using face recognition. We hope the Securities Company to build an online open account system on the financial cloud. It saves time and reduce costs by using the face recognition technology. Up to now, thousands of financial software vendors have become cooperative partners with our financial cloud. And the ICEV community began to take shape. Here are the logos of some software vendors. They cover the areas of asset management, market data information service, and quantitative trading, and so on. They can provide financial customers with various application software services. We have a face development plan for the financial cloud. The first phase is exploration period. The cloud size reaches 600 nodes or 12,000 vm. The second phase is ecological cultivation reaching to 1200 nodes or 24,000 vm. The third phase is maturity reaching to 3,000 nodes or 60,000 vm. As I mentioned earlier, cloud management system is the core of our financial cloud. I will talk about why we choose OpenStat. We want our financial cloud ecological to be open, collaborative, and win-win. The technology roadmap is consistent with the same open spirit. We want to avoid single vendor locking. When we choose open source software, we could invest on R&D and innovation to optimize and design. Those that our cloud platform can be the requirement of financial customers. We could master the cloud platform technology and enhance our capability to compete in market. OpenStat currently has a largest ecosystem including hardware vendors and software vendors. Compared to other open source cloud platform software, OpenStat community is the most active. OpenStat enables new technologies such as Docker and SET to be integrated fast. Soon our new Dongguan data center will be ready for operation. Then we will have a large number of racks. Large scale, scalability, and high performance is the design goal of OpenStat. This is in line with our requirements. OpenStat has a good cloud platform API interface. Accounting for this, we can build a software-defined data center and create a personalized business support system. OpenStat has released every half a year. The function is more and more rich and perfect. The enterprise can choose the right components according to their own needs. We choose Mirantis to help build our cloud and why. There are a lot of big companies and startups focused on OpenStat private cloud. I would like to share some points. Open source and technology are two of the most important factors when choosing a service vendor for us. Mirantis is well-known in OpenStat community and enterprise cloud market. The design solutions are fully based on customer requirements. As a financial cloud provider, we have a lot of special feature demand. It can always satisfy through customer design. Another aspect that makes us choose Mirantis is that their solution is totally zero-luck. The product can be integrated with various open source and commercial hypervisors, storages and SDN solutions. This gives us the flexibility to build cloud for complex use cases. Well, as the time is limited, I have to stop here. Let's welcome the CTO of Mirantis China, Mr. Zotin, to share his overall solution. Hello. Thank you, David. My name is Tien Zao and I'm the CTO for Mirantis China. This is my second time giving the presentation in the OpenStack Summit, but it is my first time to represent Mirantis to talk about some successful story for our customer. Well, I have been living in the Silicon Valley for 16 years before I took Mirantis offer to relocate back to Beijing, China to help set up the Mirantis Chinese entity and also ramp up a local Chinese technical team to better serve our local Chinese fast-growing OpenStack market. Ever since I came back to China, I have been talking to so many large enterprise customers and been listening to their requirement for building private or like this case, industry cloud platform using OpenStack. I think we face a challenge for most of the foreign company because the OpenStack in China basically the business is going very fast, but all the local Chinese customers facing a big top concern. They are concerned about the local support. So I have been talking to the customer, Chinese customer who not only require a stable enterprise OpenStack software district, they also demand a lot of localized special features and also industry related features just like this case. As we just heard about from Shenzhen Stock Change, actually Shenzhen Stock Change is planning to build a pretty large scalable financial industry cloud. And we have been working with them to make sure the customer success is always our first priority. And we usually help our customer to achieve their business goals through three stages in our project. Merantis usually starts engaging with the customer in a very early stage to make sure that the cloud, their building actually can meet their business needs instead of just blindly following the latest technology trend. And once the project gets kicked off, we usually sit down with the customer face to face and it's senior technical team three to five days at workshop, intensively discuss with the customer about to help them analyze their business needs and also try to understand the design of the customer's existing or future workload. Clear any doubts with the customer about the requirement in security, networking, storage or reliability requirements. Once we kick off the project, we work with the customer to create an architecture blueprint. We call it architecture blueprint with detailed cloud architecture design, including recommended underline network topology, storage design, SDN controller design, or even hardware recommendation list after. And the second phase we call customization and deployment based on the previous architecture blueprint. In case customer needs any customized integration, we help customer to fully automate this work through our few plug-in because we believe the automation is must have for building a large enterprise ready cloud. And once the deployment is done, we also perform a full blown health check automated test script to make sure the cloud is functioning correctly. The final stage, in the final stage for the most mission critical workload, it's very important the cloud is always available. So Mirantis provide the customer different level of technical support, 8x5, 24x7 or even proactive support. Proactive support includes a customer successful manager to make sure customer's goal can always be achieved at any time. So at Sentinel Stock Change, Mirantis was asked to deliver a secure stable cloud platform. This picture basically tells about very high level architecture design in our phase 1 projects. We are using KVN as our hypervisor for the Mirantis OpenStack because we believe KVN is best tested and supported in OpenStack community. Both distributed storage solution staff and traditional send storage are supported to provide a multiple choice for sender backend. Juniper OpenControl is chosen because we believe it's an open source SDN solution ready for large scale secure deployment with some extra features such as service chain. And it is also very well integrated with the Mirantis OpenStack field to provide automatic deployment for SDN solution. We implement high availability in almost in all components in this cloud including but not limited to OpenStack API, MySQL, RabbitMQ, SAF, SDN components or send storage array. We use our two deployment tool field to deploy and manage all components in the cloud itself. The end-to-end process is fully automated. The LMA tool chains in Mirantis OpenStack 7.0 provide the next generation login monitoring and alerting solution for the cloud. And we are going to use Marano to deploy and provide the financial workloads if needed and also using Sahara to quickly spin up Big Data Hadoop cluster if needed. So now I'm going to ask our solution architect Yaguantang to spend a little bit more time to do a quick deep dive in each area. Thank you. Hello everyone, I'm Tang Yaguang. I've joined Mirantis China for almost a year and before that I'm working for Kinanico and also for the OpenStack area. Today I would like to share some details of our solution for the SACC. The first is the Junifor control as the selected SDN solution. Together with SACC we have chosen Junifor control as SDN controller to provide a stable, scalable and secure virtual networking solution. And the Junifor SDN controller is deployed in HRE mode, combining the different controller components on a single node. Also the data plan and the north source traffic in HRE mode are controlled by lowering the scale of your north and south gateways. And the horizontally scalable is one of the major benefits of Junifor control is the independence of the central L3 agent. And running the work we wrote on each compute node. Each compute house is responsible for its own instances, distributed load and solutions to failure domain. So this is much different from the community open source neutron solution. We know that one of the major issues in the neutron so far is that the central L3 agent and we already have a solution called DVR. We have distributed virtual routing, but it requires almost every node to be a gateway. It's very flat. Junifor gateway is scalable through the hardware. And Junifor SDN also provides service training for advanced security services such as the WIT DPI and the virtual firewall. And another benefit is the Junifor SDN solution is also no hardware login. Contrary works with any underlist switches and multiple hardware gateways. This is also one of the requirements of SSCC. They don't want any hardware locked in the solution, including the storage and the networking and also the hypervisor. And Junifor also provides the advanced networking analysis of the whole networking, including traffic inspection and topology visualization. So this is why we are useful for the cloud administrators. And the next is that we not only build the HA solution for the software networking layer and we also build the cloud solution for the hardware. So this is a total solution, not only OpenStack. We have helped SSCC to design the hardware and the rack. So the solution is like that. And each rack we have the TOR switches and the two switches are bonding through LACP protocol and all the TOR to span all the level three. So that we can see from the picture that all the networking traffic is reductant. As a result, so we not only build the HA in the software side and also the hardware side to make sure that the mission critical applications are always useful. And the next is our storage design for the requirements. As we know that for the financial industry, they have some mission critical applications with guaranteed high performance and reliability requirements. For these workloads, we want dedicated sense storage with available multi-links for the optical reliability and performance. The downside of sense storage is cost inefficiency and scalability. Sense storage is posed as a standard backend. But we know that luckily most of the workloads only need high reliability and medium to high performance. So for those workloads, we utilize the SAP distributed storage software solution that provides both reliability and scalability cost efficiently. We utilize two different SAP posts. One is the SAP SAS pull, which is posed as a standard backend and also as a functional storage for NOAA. And in this way we can provide low migration functionality and provide quick backup and recovery of compute nodes. And for the SAP SSD pull, which is only posed as a standard backend, users can create volumes for high performance workloads from this SAP SSD pull. Depending on the workload, users are afraid to choose which SAP backend to use. For example, we could use SAP SAS pull for the operating system disk and SAP SSD pull for the master code application. And the other is the big data on the big cloud. And the established data processing on cloud is 1.0 as a success use case. The one-on-demand automated adobe cluster for the small jobs. We are using OpenStats CoreProject Sahara combining with Cinder's feature. Cinder has a block device driver which can use the physical house disk provided to the virtual house, the VM. So a group of servers are assigned to be a house aggregate and it's posed physical disk to the HDFS VMs. So in this way we make it possible to run adobe cluster on OpenStats with no performance issue. And the next for Morano. Morano is also one of the core project of OpenStats is for the application catalog. For we use Morano to enable cloud administrators to publish various cloud-ready financial applications in a portable cataloged, catalyzed catalog. It can also be integrated with the enterprise PSS system. With Morano we can make cloud in the users to self-service application provisioning, repeatable deployment for port and staging environments, and the standard building blocks for different application stacks. And the last is the LMA. Morano provides a solution for logging, mortgating, and alerting of the OpenStats cloud. As we know that we build cloud not only we finished, we have to do a lot of work to make the cloud work successful. To make the cloud applications works and we have to design a mortgating solution. The LMA is one of the multi solution. This two chain is service-originated, mortgating solution. It's comprised a collection of open source tools to help you monitor and diagnose service availability problems in your OpenStats environments at the infrastructure level. By combining the established search with Kibana, the LMA two chain provides an effective way to search and correlate all service-affaging events that occur in the system for the root cause. And alerting, including logs and notifications from the OpenStats ceramic messaging bus. Likewise, by combining the inflex DB with Grafana, the LMA two chain also brings insightful metrics analysis to visualize how OpenStats behave over time. With these two, we can do log analysis and troubleshooting, and we can also monitor all OpenStats services health status. We can also use this to do more further work, because it formulas all the data and in your standard format you can use the data for other use cases. So this solution provides a way that we can monitor the cloud in a wash-less way. So this is some details of our design solution for SSCC. And for any questions? Firstly, are you posting a copy of this presentation anywhere? And secondly, if you could do this again, what are the one or two lessons that you've learned from developing this FSI cloud on OpenStack? Sure. Let me take this question. Definitely, we will post this presentation also. I think I don't know whether this time we have video shooting and it will be posted online. And definitely, this presentation will be posted on the OpenStack.org, I guess. If not, you can talk to me offline. I can send you a presentation. And I want to emphasize basically this is in the early stage for our phase one project. Right now we finished our architecture design and we actually in the middle of deployment. So actually we are planning to do another sharing for the next year to talk about the lessons we learned and the scalability. Even I think until next April we will have this financial cloud online already. We target to bring it online early next year. So at that time definitely we can share a lot more about the lessons we learned in the deployment stage and also the lessons we learned after we bring it online. So stay tuned. We will share later. Actually I have a question about your LIMA. So you just mentioned you can capture the errors from the logs. I'm curious that you will get error information from the OpenStack log file. For example, the NOVA log file. And show it in the GUI. Yes. So we are collecting all the log files from the computer news and the controller news. And send them to the established search for the virtualized showing. You can select filtering the error messages. It's a way to help the cloud administrator to troubleshooting and to check the cloud health status. Another question is that do you have any additional tools to do that or are you just building your package? We are all using the OpenSource tools. So we are using a collect in the computer news to configure a plugin to collect the files and sending them to an LIMA collector centrally. And then put them to the established search. Okay. Thank you. Always OpenSource. You can also use it with just install the LIMA plugin. I just want to add one more. Basically LIMA is a new feature or new tool set we released in Mirantis OpenStack 7.0. And actually it's based on the OpenSource tools. And backhand actually based on other NOVA and send backhand log database as well. Yeah. Thank you. Yeah. I have a question. Yeah. Since this is a financial application, so what are the performance comparison you are planning to make like before moving to cloud and now, as you said, you are in the middle of the deployment. Sure. So what are the performance indicators you are going to measure in terms of concurrent API or the transactions per minute or the number of devices, you know? Sure. Let me take half of your question and I'll leave another half question to Ms. David. And actually from what I know, actually, Shenzhen Stock Exchange or SSCC already have a small environment. It's around a dozen of nodes based on the OpenStack environment. They are already online. They already take the customer request and they have real customers running on this OpenStack environment already. And I'll leave the second question about the performance. I can translate for you. Okay. So as I mentioned, they already have an OpenStack environment up online and have the real customer running their application on top of that. So right now the customer on the OpenStack cloud is a small business financial institute. And actually David's organization already did some performance comparison and even showed the results to the customer. There's not too much difference. So customer is okay with that. Okay. Thank you. So my question is if SSCC has a special requirement on Marinete's distribution, how do you fulfill their requirement? You want to pull into OpenStack community or you want to pull your own version? Yes. That's a very good question. Actually, after I talked to those so many enterprise customers, actually, SSCC is definitely the pioneer of those customers who want to try OpenStack. And I think I'm proud actually we figured out a way to deliver their request using the Marinete's global OpenStack engineering talent pool along with our Chinese local young engineer talents to combine together to deliver their request. And we actually this morning we even talked to another big player in OpenStack Foundation. We want to, Miss David actually would like to contribute a lot of work they will do in the OpenStack back to the community, especially in the security area actually. Security is a very important concern for financial institute and the system stock changes will do a lot of work in the security to help Marinete's OpenStack or even OpenStack community to enhance the security features and already set some effort not only along with Marinete's even with other platinum number in the OpenStack Foundation we want to contribute back to the OpenStack community. Thank you. How is a financial cloud different from a common cloud? Okay, I think I should be able to provide this. You can add more. Thank you. Actually this is financial, we call it financial industry cloud or we can call it financial community cloud. It's kind of public cloud but it's not open to the common consumer. It's only open to the set of the customer from Sundance Stock Exchange. Those customers mostly are small business financial institute, some small banks maybe brokers, traders, so it's not common consumer, individual consumer. How is the cloud different to satisfy those financial ones? Yeah, how should the cloud be different? There should be some difference in the financial cloud from a common cloud. How is the cloud different to satisfy those different needs? I think actually David mentioned in his presentation secret stable and also the security of the data, right? And also the stable, how stable the cloud it is, it's most important. Actually David even mentioned to me the cost, the price is not his top concern. That's probably somehow a little bit different with the common public cloud. David probably can add more. Yes, I forgot to add that. There are two target customer pools. Right now actually this financial cloud is helping those small financial institute in China because those small financial institute don't have their private cloud. Their IT is very weak, right? So if they want to leverage the benefits of cloud, actually they can use the financial cloud provided by SSCC. And later on once we have this system up and running, we will learn all the lessons, we enhance this system, we think it's ready. Or even talk to those large financial institute to integrate with their private cloud to become somehow the hybrid cloud with them so that they can migrate their workload in the peak time to this public financial cloud. This is actually our goal for this whole project. Thank you. Any more questions? Alright, thank you for coming. Thanks.


Hello, everyone. My name is Betul Durak from Microsoft Research and today I'm going to talk about the construction of format preserving encryption. We named it as FAST. It's a joint work with Henning and Michael Horst from Comfort in Germany and Serge Wodenin from ETFL. Format preserving encryption or FPE in short is essentially a deterministic block cipher. It encrypts the message from domain D to the same domain and the domain is defined with the format and in this slide I'm showing you an example of four digits and the format is defined with the length of the message. In this case it's four and with an alphabet in this case it's a set of digits so it is Z10. But there could be some other formats that we may see that we may observe in real databases such as social security numbers or credit card numbers or even smaller domains or a bit larger domains than this. So the idea in general is that when we would like to lift the security of a database by encrypting it we would like to keep the format of the data which is encrypted because of transparency reasons. So when there is an encryption which keeps the format there will be no need for applications to be rewritten or no schema changes will require when the security is updated lifted. So one way to think of the format preserving encryption is to use the traditional block ciphers like AES because AES is strong. AES has been analyzed for a long time it's very fast it has made very fast and it has everything we want. But then the problem would be the keeping the format in the suffer text because if you want to keep the format in the suffer text we need to truncate it and then then we cannot decrypt it back. Another idea could be to use the counter mode because we won't need to decrypt back if we can truncate it as we want but then it would require the encryption and decryption to sync and maintain a set of counters which is which is not so pleasing. And instead a big line of work big line of research has been proposing lots of different FPE constructions. The most notable ones are from NIST which are standards now they are called FF1 and FF3 and they are based on Feistel networks and if you may remember the Feistel networks are consisting of some rounds and in the specific case of FF1 and FF3 the round functions are defined with AES. And since the number of rounds are quite small for both cases for both standards it's practical and the security has been already analyzed by many cryptographers. In our work we designed a new form of conservative encryption which has a new taste, a different taste. It still consists of rounds but then the rounds are not traditionally defined with the AES round function instead it is defined with several layers of permutation boxes as we will detail in a short while. And where the secret keys are used is then to in the setup secure setup phase we generate a pool of S-boxes that will be used in those layers and these S-boxes will be generated with AES. And our other contribution is that we analyze the security of this such a construction and we formally proved that we can reduce the security from strong security to weak security and then it allowed us to analyze the crypt analyze the scheme only in the weak model with several different attacks that we instantiated. And then finally we adjusted our parameters for the construction such as number of layers, number of rounds etc with a security margin to protect, to provide good security. So before getting into the details of our construction let me say a few things in general about the FBE. FBEs are generally are tweakable in the sense that because of the domain our domains are too small there's a big probability that the ciphertext will for the same message it there's a big probability that two different entries, two different records in the database for corresponding to two different users will be having the same plain text and they will encrypt they will encrypt to the same ciphertext in which it leaks and therefore we introduce a tweak and this tweak essentially is very important for a domain separation. And on top of that which I don't show here is that the FBE takes the format as part of the input as well and it is a desire to have it in the input. So I would like to show you here the two different instances, two different styles of format presuming encryption. On the left we had the FISTL based FBE. These are only toy examples I didn't draw the entire FF1, FF3 or neither our construction it is just for illustrating the differences. On the left we have a message, we have a FISTL based FBE and a message is divided into two halves and these two halves are entering into the branches and there is a round function and each round is defined with the input with a key, a tweak and tweak changes for each round. And on the other hand we have a substitution network where we have the plain text which has length L and each character in the message is going to be a registers and each of them will run by one entering a substitution boxes. It's essentially a substitution network. And in here essentially the rounds on the FISTL based constructions are defined with the secret key, AES with the secret key and the tweak and in our case there will be no secret keys and no AES computations it will be only the substitution boxes of several layers and AES will be used in the secure setup for key derivation. So the parameters that we have that I will need for the rest of this talk are as follows. The format I already mentioned is defined with the size of the alphabet it is A and the length of the messages and in this case we can imagine that alphabet is ZA. It's a set of integers from 0 to A minus 1 and we also have another parameter called M which is the total number of S boxes to choose. And essentially the S pool is generating M many S boxes and in practice typically we can think of this M as 256 because we will need the index of each of these S boxes when we want to use it in the layers and it would be nice if we can represent them all with one byte. So this is what M is. R is the number of rounds and N is the total number of layers and N is set to L times R because each round will consist of the same number of layers and because the length of the message is L then we see that N is equal to L times R. So what is important here is that how to pick the R, what would be the R for security that we will be analyzing the paper and with crypt analysis results. So the one instance that I can show you here is that L is equal to 4 meaning that the message has length 4 and it has three rounds and 12 layers. So one example of round is this. It consists of four layers because the length of the message is L and there is layer 10 which is essentially only the round operation and the substitution box that we show here. This probably explains what we mean with layer and round better. So let me get back to the setup phase where we use the secret key. The secret key master secret key is used for key derivation and this key derivation works with the set of parameters, two different set of parameters. We separated these parameters because we wanted some flexibility as I will describe shortly that in the first set of parameters we have only the length of the as a size of the alphabet and M, the number of S boxes in the pool that we generate. In the second set of parameters we have the L, N and W1 and W2. I will not explain W1 and W2 yet, just yet. I will describe it with a picture in the following slides. I just assumed that those are two different numbers of parameters. So what happens is that then the key S, there is we drive a key key S from secret from master secret key K an instance one and only instance one with a constant and this means that there will be this key S will be drive the differently whenever the instance changes, instance one changes. It means that whenever the alphabet size changes it will be drive, it will be computed, recomputed but as long as there is no changes in the size of the pool and the size of the alphabet this won't be this won't need to be computed again. And key sec is going to be generated with the both instance one instance two as well as the tweak and let me explain a little more where they are used. These derived keys KS is going to be used to generate the pool of permutations and key sec is going to be used to generate the index sequence to use to which indices should be used for layers for for S boxes for each layer. So that means that when the tweak changes or the size of the method changes in instance two then only the key sec part needs to be regenerated only the index sequence needs to be regenerated. Other than that we don't need to keep creating the pool of S boxes from scratch as long as there is no changes in A and M so this was the property that you wanted therefore I put the thumbs up here and this is essentially our way of doing the domain separation. In the S box generation more in a bit more detail we have the input secret key KS and this key S is used in the AES for PRNG and this PRNG is instantiated with AES encounter mode it's parallelizable it's nice and the permutations are generated with the Fisher Yates shuffling and Fisher Yates is essentially some sort of rejection sampling it's simple enough and S box index sequence generation is again taking an input secret key sec and using the PRNG as well as some domain parameters and then we instantiate this PRNG again in the implementation with AES in encounter mode but these two PRNGs in the S box generation and the index sequence generation could be completely different it's up to the implementer but we chose AES encounter mode for parallelizable features. The core design that we have is that it is going to I show you first with the one round with four layers though it is four layers because we have four substitution boxes here but I was I have been seeing this since the beginning but let me correct myself let me make it more precise more correct it is not essentially one S box that I'm using here it is two S boxes two same S boxes that we are using here and there are two arrows as you can see here one is entering to the round operation it looks like the round operation but essentially entering input to the first S box S1 and the second one is entering to the S box permutation it's essentially again entering to the S box but the second S1 so in here what is what we define this blue arrow and orange arrows are that they need to come from some registers here and we want to define the distance of these registers to the S boxes so in the first case the input to the first S box is called W1 and the distance in this example is one and the second distance to the second S box is called W2 and it is shown with with an orange arrow here the distance to the S box over here is two and these these parameters W1 and W2 are the same for all layers as you can see here except that sometimes they rotate because there's nothing on the left side of the X1 register so it rotates and also it keeps shifting so if it starts from X4 but then it comes to X1 and X2 and X3 and so on so this is the design for one round at a high level if if you wonder why we choose it this way why we define it this way it would be difficult for me to explain the reasoning in a few sentences instead what I could see is that we essentially didn't think start our design with just one S we with two S boxes like this one but instead we define something more like this without these W1 and W2 and we designed it in a way that the diffusion property will be will be sufficient for both encryption and decryption but unfortunately when we design it with only one S box and in a very simple manner it would be it would require a lot of number of rounds for diffusion to happen well for the decryption decryption everything was fine for encryption but for decryption it was difficult and therefore we tried several different versions of this construction and we're convinced that this is this is resisting to our attacks and this makes a number of rounds smaller and more practical and yet secure enough so this is the high level intuition I could give you and for the three rounds with four layers again we have the same round repeating three times except that none of the S boxes that we use in the first round will be same in the second round they won't be same they will be different and they will be picked from the with AS computation in the index sequence generation as of security let me start with the assumption we have we have the assumption that the adversary has access to the S pool all the S boxes are known to the adversary because it's it's a reason of assumption because it may leak from side channels and you want to turn we didn't want to assume any such any such leakage but instead you want to be safely say that pool can be leaked and then we define the strong security model with multi-target chosen formative capable PRP essentially the the idea here is that the there is an adversary playing with encryption decryption machine machines and each of these encryption decryption machines have different secret keys so therefore they are there it is called multi-target and and the adversary can also change the format or tweak when she's querying these oracles when she's playing with these machine names in the weak security notion the adversary is allowed to play again with the encryption decryption oracle but it is only for one key and then he the adversary must fix the format and tweak once for all queries that he's making to this encryption decryption machine so this is the notions that we define and we formally prove in the paper that there is a reduction from the strong security notion to the weak security and more a bit more formally essentially what we say is that there's a there's a there's a tau here the tau is the number of different keys in multi-target game and the q is the number of queries that that has been done and what we say here is that there are four adversaries a b c d and this we say that if the prf prng1 and prng2 are secure and if the advantages of b c and d are negligible in the games that they play then the strong security is going to reduce to the weak security it means that if someone can give an attack in the strong model then we can construct an attack in the in the weak model and this is essentially the high-level idea of our our proof and this led us to as I said earlier to to only focus on the weak weak model we we kryptonize the construction in this weak model meaning that by only by fixing the key tweak and and the the format and then according to our kryptonizes results whatever we found we doubled the number of rounds for security much so a little more details not too much we defined we designed many different types of attacks like distinguishes or linear potential or linear attacks and and so on all the details are in the paper and you're welcome to try even more or improve those attacks if if you want that would be nice for us but essentially the the outcome of all those attacks is that we were able to define what the w1 and w2 would be because intuitively when we first started the discussing the design then we would we would guess that this w1 and w2 would be naturally something like l over 2 the half of the length of the messages or l over 3 but we wouldn't guess that it would be a square root of l and then it can still be set to l over 2 w1 and w2 can still be set to l over 2 but then it would require a asymptotically l square number of layers that we need to use in the construction to be secure but if we pick w1 and w2 being around the square root of l then the total number of layers asymptotically can be picked as l to the 3 half and it would be enough asymptotically again there is a constant the term that we multiply it and in the small when the parameters are small it it does not reflect well but when the number of parameters get larger it will be more visible that we need n equals to order of l to the 3 3 and 3 half so in here I'll show you one set of parameters selection of parameters for a fixed sorry for a so for number of rounds and we have a different size of alphabets alphabets and we have different length of messages and l is the different length different lengths of messages from 2 to 100 and a is from 4 to 16 but you can find more parameters in the paper so just to give an example when we encrypt the full 16 digits of credit card numbers the l would be 16 because the message length will be 16 and the alphabet size will be 10 because there are digits we need 37 rounds of our construction and we did the experimented our construction and we compared it with existing ff1 and ff3 for both constant tweak and changing tweak the performance doesn't change for ff1 and ff3 if the tweak changes but as I explained in the setup phase if the tweak changes we need to we need to generate the sequence of indices again and that would affect the performance therefore we tested both and this we tested it for alphabet size being equal to equal to 10 and we observed that in all cases fast the construction our construction fast is faster than both ff1 and ff3 so maybe what is important here is that like probably it was known to many many people who are implementing that after the certain length of the messages suppose 32 the characters of the message the there is a big jump in terms of complexity for ff1 and ff3 and this seems to be known to the known to people who are implementing these designs but in our case there is no such blow up happens the problem the the actually what happens here with ff1 and ff3 is that because of the modular reduction they have in the round functions if you remember the round functions are round operations are modular operations and the messages are divided by half and therefore the modular reduction happens with modular a to the l over 2 and if l becomes larger and larger there would there would be a significant performance problem with with the big integer libraries that we that we are using to implement them and that creates a problem for both ff1 and ff3 and ff1 is essentially ff3 is essentially does not recommend us to encrypt any messages larger than 50 54 or 56 i believe but we just abused it and we encrypted up to 64 but ff3 does not actually provide larger larger messages messages so this concludes my talk thank you very much for for your attention and for listening to my talk and we would like to welcome everyone every cryptographer everyone who is working on format preserving encryption are open to different designs to crypt analyze our our construction the code is provided in this github link we also linked it in the in our paper please join us to crypt analyze it further


Welcome everybody just before lunch Here we're gonna head to talk about Drupal and the open web and the New Zealand government and the keen-eyed They went to Drupal South Brisbane would have seen the similar talk there with the Australians This is the New Zealand flavour. It's a little bit more slides and a little bit less time. So just bear with me For those that don't know me my name's Sean and I help lead a team of talented people that look after enterprise customers I work for a MZIO and being in the Drupal Sphere for Too long Maybe a dozen or more years So the problem statement I'm here to try to answer is how popular is Drupal and and the open web in the New Zealand? And if you have a bit of money, you can actually probably pay to achieve this Following services but Me a real staunch open-source advocate. So I needed a more transparent way of doing so if you plug in wapalizer and Wapalizer under the hood uses puppeteer, which is a chrome orchestration engine You can achieve various The next statement is how do I get a list of all the New Zealand government? And just like in Australia. There's just too many sources There's gov.nz or slash organizations if you want a to z you can do some snooping on the transparency logs and the trails as well But the main problem is is that there's no ranking or importance associated with those and Here I've picked on the inland revenue who I used to work for and the Wara Council where I used to live under so But one of these sites is going to be a lot more sort of important to the country of New Zealand. So we need a way to quantify this and Toby actually let me Into the secret about this site. So there's actually a cool little site called dom cop And they have a giant list of 10 million Domains on the internet sorted by popularity And you can download it. So a plus And it's got this thing called page rank attached What the heck is page rank for the people that have been in the web for 1996 you would probably Remember there used to be a page rank toolbar you could add to your browser and it would tell you the score And it was really important to get a big score, you know Because that's what Google used back in the day solely to determine how likely you would appear on the search results For the purposes of this talk The score was zero basically means that their site is low quality or just brand new A score of 10 would be like Wikipedia. So if that's how you Think about it page rank is logarithmic with a base of five Similar to the rectus scale a bit different So this basically means a page rank of four is five times more authoritative The site with page rank and you can convert it to decimal for the purposes of math. I like then we can add stuff and Just like in Australia, New Zealand is also victim of the mogs and link rot and Where I've got a pulled out a couple, but big cities has moved We won't tell you where And the Auckland City Council They can't be found especially when it's flooding and Lins, I think you go you get bonus points for this era page. It's just Crawling its scale and let's just say a lot of New Zealand government domains have encapsular sitting in front and a Capsular is not a my home might be in a capsule are not friends anymore. So we had to change it a few times luckily dynamic ideas exist and For the people they just want to see some graphs Here we go. And this is based on March 5th data. It is not perfect underscore that it's the best that we have available at the moment and If you have a brand new site, they've just launched. Yeah, don't be sad if it doesn't list just yet But this is the one chart. I can't actually break it down by state in New Zealand because we don't have any Not like Australia, but what you can probably see at a nutshell here is that silver stripe is actually Dominating with over one-third of the sites that you likely visit on a day-to-day power by silver Unknown is only 22% was actually quite low It's actually higher in Australia and Drupal 11.6 coming and Then it's actually quite hard to see but there's actually extremely long tail of CMS's and I'd say just a lot more in general and I just pulled up the Australian one just so you guys can compare and contrast so So Drupal over there about 27.7 and the next top is actually squids So very different kind of composition sites even though, you know, roughly the same Demographic the top 10 sites We've got health coming in page ranked 5.8 7 which if you convert to You know normal base 10 about 12 and a half thousand te Papa doc stats the COVID site immigration, you know The Christchurch City Council Sure Now here's a part. I actually had a lot of fun with was actually pulling out all the weird and wonderful thing as a course of doing this presentation and First thing I'd highlight is that Drupal powers roughly, you know, 11% of all the digital So that helps to like when you're at the code sprint on Friday, hopefully that will help you You know realize where the efforts going Silver stripe is definitely the lead and one one sort of Observation I had from this is that if you do supply a common web platform Like in Australia, it happens to be based on Drupal in New Zealand happens in the base based on silver stripe it's hugely influential and Government department loved this so I Think that's a real big win for the teams involved And squizz is less popular in New Zealand, which I found weird They've got a lot of headquarters and people here Drupal by version I don't know if you know this, but you don't have much time to hang around on Drupal 17 So I think it's November so No pressure No Drupal 10s either, which I found kind of interesting Small smattering of But this is by far and away my favorite chart out of this exercise is that roughly three-quarters of all the powered by So this I think is absolutely really happy there's still a few sites that don't believe in certificates and Yeah, there's always a few And I find this one particularly I don't know like a little bit irony maybe like the sense of their site so much that they don't want to display any content Yeah, so it's just an Apache index page good stuff and Yeah, just like in Australia people kind of get sick of just dub dub dub, you know like what if I come up with a new site What now So enter dub dub one You know, it's the holy grail dub dub two that's for pros and Then they're like other dub dub one dot math, you know, like this isn't just a test page It's an enterprise page and then I thought these guys actually stepped the game up. They aren't dub dub 100 Like it's an actual site. You can visit this thing It's about the centenary of the First World War. So the domains a little bit of a play on words And I thought well, it's only a matter of time really so my challenge to you guys is on a dub dub 9000 like make it happen There's an awful lot of redirects happening in a non-sensical fashion This is extremely common in the New Zealand government and For the people they can pass HTML in the head. This actually does a redirect in your browser after zero seconds And it's an actually be 200. It's not a redirect response cozy and so very difficult to crawl this because You need an actual browser and here's a lens example same thing And this is actually a fully fleshed out CSS images. The full works just to let you know We're going to redirect you and this guy has actually Reimplemented a engine X rewrite map in Japsk Perfection like mwah dangling DNS This is potentially a security issue So I've redacted the domains out of it and I'll probably get in contact with the departments after this but Suffice to say if you point your DNS at Access provider, this is a zero and Then you forget about it. Maybe you decom the is your side, but you leave the domain pointing at it I wouldn't do that This one points the site host So, yeah, I'll probably reach out to those government department do And we're also seeing the sort of rise of engagement platform Also as well. So from consultations to Yeah, just generally getting feedback and from the constituents and I came across a few apps during My crawling here one from datacom could data scape Another one could sit in space and then the last one is bang the table But they've just rebranded to engagement HQ and then I thought I'd leave you with the highest quality fine So we've got the perfection of layout here one that one spacer gifts We've got tables because CSS were never kept it was perfected 20 years ago and has never been updated It's got support for modern browsers like IE for it's got mm preload If you recognize this function, you read the side is ill It's got dogs in the home page and instead of a search button. It's got a button called fetch It's got TLS though. Think about that for a second. Um, yeah, this is it. So if you ever wondering I go Visit them because I've got Google next. I'll see a spike in traffic and I go what's going on Dog safety dot gov dot NZ. Oh, I love it That dog's got a really long tongue. I don't know what's going on there. So going further I'm gonna also probably Make this something that will be trending over time because I realize is giving a snapshot of the data is only so useful I think actually having the trends over time and seeing if Drupal's increasing or decreasing or the relative Changes in the landscape would be more important I think also the US government would be super interesting as well because they've got 50 states be a bit hard to condense down but and Yeah, I still want to publish this data as well. So I still want accessible Yeah Again, there's been a bunch of Wapalizer changes and there'll be maybe a dozen more after this talk as well just for better SNFs for technology and stuff like that and That's thank you for coming. All right. Any questions? Tom Right noted. I could be a kid. Yeah, they might block me from entering the US. That's Yeah, I mean I mean often you Sorry. Yep. Yep Yeah, so the question was how aware was the New Zealand government of this information and would it be beneficial? So I think it would be of benefit to them to be more aware of this, especially things that security angles like DNS Stuff like that. I think it's super useful to to know about But also maybe if I've got sites that I just Need to be put to pasture like maybe just Been a bit proactive and maybe making it static and sticking them on you know Static hosting provider like GitHub pages National Library. Yeah, there's already a system for this like let's just let's just use it Having a lotus notes application Yes, I don't know the most secure thing the internet So the question was how do I like not there's a small fraction of Drupal sites I couldn't discern the major version from and like to answer that question. I'm not actually doing deep SNFs as to Going any further if your site identifies and you know the meta tag Then I will pick that up If you're obfuscating it because security and obfuscation is your thing then It's not it's not going to go any further. Obviously you still can discern it It does not end like I have used other tools in the past They're a bit more aggressive and they will empty five hash literally every static file that you present and then compare that with the no one hashes and But that's super aggressive. It has heaps of requests and I haven't I've tried to be nice to the site so Yeah, and like I think just in general like if you are if you do have a website I think it's quite important that you do present the major version. I think that's quite useful I don't want to know your minor and your patch care about that but at least Sure major is quite you and that goes for all CMSs as well like Silver straight four versus five like I think that'll be really cool


Good morning. So this is Sandeep and we'll be looking at trans stats now to track translation string change So basically we will be covering the tracking Translation string change part today and we'll look at jobs how to run jobs in trans stats and how they contribute to different levels of Processes in Elton and Iotinian stuffs. So let's get started So these are the topics we are looking at first is why trans stats and the challenges it Face and challenges it tries to solve second our second is what are the touch points through which users can get some interest using this and how to use jobs to actually Actually tracks something and actually create diff to use it at server and how to use it by client and and what is next coming in trans stats at the level of developments and integrations to different systems and How we can actually get involved at two levels maybe to just try and test and maybe to start Contributing into the project So first is why trans stats what is the need of trans stats and why it is required? Elton in process I'm in a localization process as a whole in the life cycle. There are some challenges which We are facing and these challenges are first is how to track Strings of packages have been duly pushed pushed and pulled from respective translation platform Second is latest strings are being packaged and built in Koji. So these are the two major bottlenecks where Like how actually we can track this so that packages could be in sync So trans just actually try solving this by comparing translation statistics derived from different places The places are up a stream repository translation platform and well system Second challenge is what is the volume of translation required for next Fedora release? This is actually required by Elton and team at very early stage to have an estimate like what What would do what would be the volume and in which languages? So answer to this is Transstats release summary can tell us volume of translation required per languages The third point is can be fixed major translation bugs reported after translation deadline and before final phrase this is this is related to QE point of view and Answer to this is the why Melviz jobs can pass packaged srpm to check criticality of translation bugs Actually everything will be very clear in the demo. So I'm just I'm just Explaining the answers to what are the challenges are and how we are getting these answers in trans stats The fourth is which set of languages need attention and the fifth is is there any way to automate few steps? So answer to this is in trans stats We have the system where why we can find the packages which are out of sync which has has differences from translation platform versus what What is being built into well system and then? Yes, we can actually have some automation by adding few tasks in why Melviz jobs and those automation can be achieved So these are the challenges and these are the respective answers to these challenges now We'll see how we can actually get I mean get to these answers What are the touch points in trans stats so left-hand side we have the Actually, this is the release milestones For translations point of view there are two major milestones the first is software string freeze and Second is translation deadline so software string freeze is the point where The developer has to stop doing all the changes in string and then Translator has to start their translation work and translation deadline is where Translators are supposed to finish all the tasks and then QE people has to test everything so that things could be packaged and well-shipped In trans stats we have some features like translation of date volume and coverage the packages which are out of sync and Two features are linked to out of sync packages first is string change and second is languages which need attention So these are the features which are available in trans stats and which can actually solve many of the challenges as we have discussed in previous slide So now let's look at how to use jobs In trans stats why jobs so jobs is Really very flexible in the terms of creation We can run them we can say we can schedule and we can share schedule is in progress It is not available yet, but others we can do The predefined job types are like there are two basic types first is predefined jobs again is why well-based jobs so in predefined everything is defined we can't change them and They are basically to sync with different parts so that we can fetch out different data and Stow them in trans stats in predefined jobs We have sync build tax just to have all the Build tax which are available in Koji to get into trans stats Second is sync with translation platform to have all the information from translation platform regarding projects their details and High-level overview meterator third is sync with the release calendar This is why this is important because Actually, there are certain dates which keeps sleeping a week or so So we should have all the latest milestones in trans stats. So this jobs handles that Now next is why I'm a bit job. These are very interesting part in trans stats These are vinyl based and we can actually go and define them so the first is sync with up a stream that is Get repositories second is down a stream that is Koji build system Third is a string change fourth is verify translations verify translation is in progress I mean development is in progress. It is not yet available and The last one is inspect jobs. There is a page or there is a section Whereby we can inspect all the jobs how they are like what has happened running them So this is actually log view. So it has list and detailed view So we are going to see each and everything in so much of detail now Sync why I'm a job template. So this is actually job templates. So you can see that in sync This is up a stream sync job Template where what it does it close the git repository filter all PO files and calculate the statistics on the basis of those PO files This is sync downstream Job template where it takes the build system. It supports to build system first is Koji and second is brew and It takes package name build tag. We are running this job for What it will do it will get the latest build info download as rpm unpack that as rpm load a spec file and then unpack Turbo and apply patch which are available in a spec file then filter PO files and calculate the statistics So the beauty of this is we can actually go and remove these a step or we can add an additional step So this is a lot more flexible Now let's move to another templates. These are actually templates to track a string change So this is the first template Like for a string change it requires two things first is package name second is release lug Why release lug is required because suppose you you are tracking for Fedora 29 So we are actually keeping branch mapping for each and every package. I'll explain what is the branch mapping and Through branch mapping. We are getting to the point like a suppose it is for Fedora 29 So through branch mapping we are calculating like which is the project version of translation platform for the package where translations for Fedora 29 are Kept or are being done. So that calculation we are that mapping we are getting From release lug now for task thing. We are cloning git repository and branches master It is well configurable like it suppose git repository as of now SVN and other things are not yet supported But you can mention branch here. Currently it is master. You can mention whatever you want And then it then actually after cloning it will generate the part file from the source itself Now why domain is required. This is the command you have to give to generate that particular part file So by default it will show Intel tool command which is being used by almost every GNOME packages And few of the packages are slowly moving into get text Get text based system where this command will not work like anaconda. So for those systems I'll show like which command will work And this is the command here we have to give the command to generate part file And domain is required because some of the packages do not have the same name for their part file So we actually need to give domain so that it can generate part file of that particular domain And to download part file from translation platform of the of the same release we are running this job for And then it will calculate the differences between part file which are at translation platform And part file we are generating from source repository. So if we create diff from these two We'll be getting what are the strings which has been changed Now this is like upstream like git repository versus translation platform And this is like translation platform versus what is being built picture So this is under development, but the steps are written here So the first step is download and unpack let us let us srpm from build system Create fresh translation template like part file from the source tab all so we'll be repeating this step here itself Then pull translation from translation platform. This is very tricky because we have to pull each and every translation for all the locals from translation platform Then we actually we are actually merging those translations To newly created part file, which is in set one And then we are applying patch and filtering PO files So we are actually having two set first is set one which we have generated right now And second is the set which we have found in the srpm So if we create diff between these two then we'll find diff for each locale And what has been packaged versus what is being translated So this will solve most of the problems we are facing in localization cycle And if we see the results of these two Then actually we can answer many of the questions like What what was breaking and what was not in sync So now let's see them in action So this is an important slide like if we find some differences So this is like string changes detected and if it is after string freeze date Then it should be a breakage. So we should send a warning to maintainer And if it is after translation deadline, so it could be an error So we should alert a quality engineer that it is an error error so actually Once it is integrated to different systems, we can we can actually redirect those messages to bugzilla and then we can handle several errors as well And there are just four steps at server through which we can run these jobs The first step is to we have to pick the template The second is we have to define task in the terms of yml file Then we have to set values in yml file. We have some variables like package name release lug and those all things So we have to set those values and then we can trigger the job So now it's time for demo Let's uh So I have recorded everything and I'll be keep explaining So this is the landing page of transters in which we have just five packages Where uh, these are the messages and what has been translated not translated the packages in packages section We can see each and every package in so much of detail So uh, we have selected one package the first one that is abert, I think So this is the page for abert and These two this is coming from upstream and the one having yellow background Going from downstream now see in anaconda In anaconda everything is coming from translation platform because they their background color is just normal So these are the statistics which we are getting from translation platform This is the branch mapping I was talking about for anaconda Now since we don't have Statistics from downstream we are going and selecting the downstream package job and this is the yml file for downstream sync Now we are going to run this job So, uh, it will do everything which has been written in yml file. First it will go and see uh, which which is the latest build Which uh, which has been Into koji and then which is out those details Download the srpm file from the koji build system unpack that srpm load spec file And this is actually loading spec file into a python python object. So it is very flexible Now it is unpacking table And then applying patch no patches has been found Filtered all the po files and then at last it will calculate statistics. Why it is flexible because It is loading each and every po file in python objects So if we are getting some error like syntax error or something in po file So here itself it will show like we have found one error or whatsoever So it is very important and it is very interesting tool to inspect into srpm. So this is how We are getting translation statistics of anaconda the build system packaged level So now let's move back to anaconda detail pitch. Yeah, this is the job success url Here you have the yml file Which has been run and this is the log off output of each and every step What has been performed at the server level And the good point is we can actually share this url So this is what I was talking about. You can just copy the that url and share across so that everybody can know that What is the current statistics of anaconda package at branch map at build level Now let's move back to anaconda Here we are here. We are having statistics of Anaconda from koji build system for fedora 29 and now Generative statistics diff once we are generating it will show that these languages are Not complete at koji level. So these languages need attention for anaconda package for fedora 29 Now let's move to gnom initial setup in initial setup. Everything has been synced. We have statistics from koji build system We have statistics from translation platform. So we can directly go and generate diff So once we generate diff it is in red color. That is chinese traditional And it is showing one person that means There are some strings translated at translation platform, but they are not being pulled and packaged So that is why that is it's red and it is showing some differences So this is for gnom shell And for gnom shell, we just have japanese. That is not complete at koji So let's move to python may For python may statistics has been generated and no languages found as need attention. So it is well and good For python may for fedora 29 as of now Let's move to Settings part like how trans stats is keeping things in his database. So this is all languages These are the translation platforms currently trans stats has in products we have fedora koji and this is the release schedule of fedora 29 So actually we can keep multiple products like we can keep rel and fedora at the same time This is uh the packages and their list Let's go and add one package password d So for adding a package we need upstream url Copy upstream url. That is pager And we need to give The name password d password d the name has to be matched with whatsoever we have package name at translation platform So validation is being done over there and we select as fedora then add package So now package d is being tracked for every release in ith of fedora So now you can see that everything is empty here Like it is newly added package. So we don't have any statistics from any of the sources Go and sync with translation platform. It will sync with fedora zanata currently Trans start suppose to translation platform damd license zanata So Now it is fetching all the statistics available from zanata And there are the statistics we have two project was in master and password Now it's go and sync with upstream repository So what it will actually do it will go and clone the git repository It will filter pio files and calculate the statistics So it is cloning uh buggyu repo And this is the statistics we are getting from the repository And this is the job success url Once we land to pass the details page we can see that we have statistics from upstream And this is the branch mapping Go and sync with downstream as well So we have statistics from translation platform upstream and we are getting statistics from build system So mostly for koji build system. We don't Have any patches for a brew build system. We do have I mean various back patches which has to be applied This is job success url for password d downstream sync Now let's get back and calculate differences for password d So, uh, this is all red because for password d things has been translated at zanata But they have not been pulled and packaged. So we are getting differences here So this solves some of our problem like Actually add to for package maintainer level or something Once we refresh this we can have statistics like how much of packages need attention that is for currently for this instance Once we land to federal 29 translation statistics, we can see the password d has been added And different statistics has been refreshed as well So this is update volume thing like what volume has been updated and in which languages we need I mean most attention in the terms of translation Once we refresh this we can have statistics for each and every language Like What is the number of untranslated messages? This is shortcut to land to details page of each and every language For translation part. So these are the pre-defined jobs. I was talking about these are three number build tags translation platform and relay schedule And this is yml based job. It has three templates as of now There are four steps pick template define test set values and trigger job And let's see like how things are being stored. So this is the yml for upstream sync as we have done for so many packages This is the yml for build system and this is the yml for track string change Now we will be running a couple of string change job to see actually how it works These are inspect logs. This is the list view of logs And like this is forcing downstream and password d what we have ran just now These are the job IDs here And this is the type this is the remarks in remarks mostly we'll be having package name and this is Here like this is success or fail whatsoever The time taken and this is the job details page URL So here by you can land to job details page It has most of the details like the time the output the input yml and everything Yeah, here I was talking about this is the command for anaconda So it is different from intel tool. So this command is required for anaconda port file And this is the steps which occurred When we fired that command to generate port Zero message default has been found for anaconda and in full diff you can actually see The entire line to line diff of anaconda port file Like this from the source To like what is what has been created just now to what we have found a translation platform So this is what I was talking about as the message id difference is zero So we are safe at anaconda Now let's run it for some other package So we are keeping the command as intel tool itself And we are just looking into abert So we are just going and see like if do we have any branch for fedora 29 As of now, we don't have any branch for fedora 29 in abert So we are just going ahead with master branch itself and So here master is branch is master recursive is true And command would be intel tool now let's select package abert And then release slug that is fedora 29 And run this job So first it will clone the repository Entire repository and then It will go and generate port file So port file has been generated and then we can see the differences which we are getting So these are the differences which we are getting in abert So for current master branch if we'll consider it with fedora sanata abert There are 14 Messages which differ So we have either we have 14 new messages or we have some new messages and some has been updated So we can have answer to them as well These are the output these are the 14 messages which has been differed with their line numbers like At which line they are differing And once we click full diff we'll be having line by line diff diff of a port file On line number 176 We are having this line added. I mean we are having this difference found So now let's go and see so here in full diff you can see that this has been added to the port file And this is how we can actually go and see Like which string has been added here Or which string has been updated and where Now let's go ahead and run one more job with different command So now we'll be looking at python may in python may in po. We don't have any po. We have one port file So And we don't have port file. So intel tool update will not work here We need to figure out like which command we have to give there to make it work. So it is make port file simply For python may so it is just make port file And here we are keeping branch as master The interesting part is we are adding one thing like Overwrite as true because we have port file in repository And we don't want to use that port file. We want to generate our own. So for that you have to add override true Once it's get added for python may and release later on 29 Run the job So what it will do it will clone the repository and it will delete that port file And then it will create the new port file and then go and see the differences So here we are safe because there is no new messages found for python may So job success URL is this and you can see here that no messages differ So This is actually message id diff and full diff is line by line diff. So everything Is here on this page and this is the URL which you can copy and share basically In archive, I don't think we don't have anything because Once we have more than 15 or 20 jobs run then it comes to archive And this is how it is being navigated Like from list view you can go and go to details page We have a quick start page where you can go and read like how to get started with trans stats Some of the FAQs are here itself, which will answer to most of the questions most of the common questions basically So this was the demo Next is trans stats cli. So what I was explaining in the server parag is going to explain everything with cli as well So over to parag So I'm going to talk about the trans stats cli so basically why trans stats cli is needed because It is needed by the developers if they want to quickly track their translation status Or even by the translators if they want to know what is the current status of their package to which they want to contribute their translations So it looks better now Okay So basically in trans stats cli there are five commands have been developed The first command that have been developed is the ping server to test whether the trans stats server is alive or not Okay So this is the Ping server. So basically trans stats is using a rest api. So This trans stats cli is just Using this rest apis. So this is the gate kind of api The ping It's a simple command second Trans stats cli I think is nothing but getting a package status So package status is nothing but returning the translation stats of a package for enabled languages Uh, I will show you the demo as well. So just let's go through these commands and their descriptions So, uh, the package is one of the cards for trans stats cli Uh, third one is a graph rule coverage. So you can create your own rule And you can get the Stats from the For that package. So we have developed this command called as coverage So it returns the translation coverage according to the graph rule. So you can create your own custom graph rule there It's also a gate kind of command Uh, then the fourth one is a release status It's returns the translation stats of a package which are being tracked for a given release. So suppose you now want to track us Whole whole whole suppose for any branch. So release branch. So let's say the current release branch is federal 29 So, uh, you can use this release command Uh, the recent one that we have developed in the current trans stats cli release is a job command And this was one of the biggest command, uh, that has been developed into the trans stats cli So job detail. So we have two sub commands that have been added here Job run and job log. So with the job run, you can run the jobs of three different types That is nothing but a sync upstream job type sync downstream job type and string change job type and, uh, when you run these Through trans stats cli, you will get a url and in order to view the log from that url You need to use another sub command that is trans stats Uh job log and you need to give the log id that is written from the run command So like, uh The example has been added. It's this big job ID has been generated for the each job that you you are going to submit And you can just substitute that job id, uh, in that, uh, instead of that job ID field and you will get the log so, uh So these are the commands that have been developed into the trans stats cli Yeah, so like I said before trans stats cli is written in python. It uses rest api and the code is hosted on the github So we have a trans stats organization on github where you can find a code for the trans stats server as well as the trans stats cli Uh, it uh, this also provides output in json format as well as the norwix format. So basically, uh, The the the output that is coming from the trans stats server is in the json format, but in order to Make sure that the end user will read it properly We have made a default output format as a normal text format But to all this command we have, uh, added a hyphen hyphen json option So if you add a hyphen hyphen json option output in json format only So so there are two types of, uh, output format available through trans stats cli Yeah, in summary, uh, coverage job package release and version are the five commands that are available So I can show you So, uh, we do have a man page Uh, in the trans stats cli package trans stats cli package is available in fedora as well as for epel 7 as well And this is the man page So, uh The same information is available here like commands coverage package version release job And we do have examples also for end users how to run it So suppose Trans stat release you want for fedora 27 then if you want in detail you can use a detail option Version I think we better so, uh Just if you want to test What version trans stats is currently using so you can just use a command trans stat version So it will just return a client version But if you want to know what trans stat server version is, uh, Then you can just add a hyphen hyphen server option and you will get to know so currently Trans stat client is at 0 to 0 version and trans stat server at 0 1 6 hyphen rc 2 version So this is one of the Big output commands So if you want to know about the status for the any package, uh, say anaconda So I mean I have already run this command here. Oh, it's already written the output. So, uh, see So for every branch that is available Like rail seven is there rail six is there so It will give you the output in this format like, uh, for African language say the completion status is 1.09% only for Albanian It's a 79.92 So it's it's a bit easy, but it's uh, but uh, yes, I can understand it's lengthy So we are just adding a sub options for easy to get the information in the output like Like if there are you can see if there are five or six branches It will give you the all the output along with all the enable languages for that package So the package command can be used to get the status for all the available branches for all the available languages Yeah, so I I think uh, I will cover this because this is very important thing a transfer job. So trans stats job So, uh, how to run this command? So trans stats job run And then which kind of a job type you want to choose say suppose string change You want to test whether there has been any string changes happen for the gnomeshell package And then that too into the fedora 29 branch stream Then this is the command. So once I run this command, I got a job log and In order to see the log Yep, so this is the command Transstat job log a job ID So it will just return in a normal text format like what is the job ID? What was the job type for this job? What is the start time end time and the YAML template that just this Previously we have seen different different YAML templates that one can use to submit the job. So it also returns what YAML YAML text has been used for this job. So this is that So it just clone and generate a part file and it use this command Downloaded the part file So yeah, so this is how we are just trying to make it as as pretty as possible So that the end user like developer and translator can understand or can pass it At the cli level only instead of going to the web and look into the log files I mean server just returns it server generates some job ID once you submit the job And as I'm using a rest API. I'm just getting that thing. So Even even like I explained you let's See how much time it gets So JSON outputs are also available Or maybe I can show you this quick If you want output in JSON format You can get it It takes some time Something going on wrong see Here we got the output in JSON format It's success job created and log and we got it Job URL job ID So, yeah, I mean This is the way. I mean you can use the transfer at cli and It will be easy for developers and translators to quickly interact with the transfer server and to get the status Yes We have a release command so 29 and Yeah, let's see.  Ha ha. It broke. So, uh So this is the way release and release look and then we have added a local sub option to get only Local information particular local information Do you want to address? Sure. So, uh, actually when we when we'll go to server, uh, then we can actually see that can you navigate to server? I mean Maybe transaction dot xyz Yeah, uh, so click on dashboard basically So here we have Here we have federal 28 and 29 statistics Just scroll down. Yeah And scroll this bar. So here we have all the languages which are Have have some untranslated messages and once we click on any of the languages click on any of the languages Here we can see that, uh, what are the packages? What is the total translated? Package, uh, I mean messages and what are not so this is the navigation for entire release to languages to messages kind of thing And click on dashboard once So basically we have two tabs first is releases and second is packages So if you want to get into packages navigation, just select second tab And then you can select any of the packages and then you can see all the packages stuffs So, uh, basically it is categorized with release and packages and similarly we have two commands release and packages at sly also So Hence categorization is being done So, uh, should we move further in the Uh, thank you para for covering cli part Now what's next? So, uh, the next in development plans are this First and most priority thing is to deploy transters in federal infrastructure Second is to have some more tablets in ymelvis jobs like verify trans which is still in progress Third is better UI to eliminate navigation confusions and for this we are developing new dashboard Fourth is transters job scheduling feature To avoid all manual Go and run jobs thing Fifth is notification about string change breakage and translation deadlines to their counterparts and better user guide We have very short demo for I mean screen recording for new dashboard as well So this is how new dashboard will look like This is created using pattern fly and here we have all the navigations like dashboard packages jobs languages platform and custom graphs So by default we'll land at release page And here here the stuffs are similar to what we have currently So we have planned the migration from old to new dashboard somewhere around in september And this is everything which you are having currently in the old UI Uh, this is locale to details page navigation And these are just the tool tool tips for different Fields This is release schedule page Where you can see the release schedule of fedora I mean which are very much linked to translations This is summary page of packages and this is the list view of all the packages Once once you'll select any of the packages you will land to their details page Like we are going to aberts detail page and this is the details page for abert The internal the internal looked like is very much similar to what we have currently but the outer and Outer skin is being changed So it will avoid all the navigation confusions and it will give us opportunity to actually go and Include all the things Which which which is actually required at UI level So this is languages page Where we can actually filter everything using shortcuts and see the language list Similarly for platform so Uh So these are the platforms which actually we can support any number of engine Like if we have four instances of zanata, we can support all four if we support Damdolize and for the some parts we support trans effects as well. So There are several translation platforms with These are the custom graph rules on the basis of which coverage graphs are being generated Like and these are the filters Which is being added so in the coverage we can add different filters and filter those rules on the basis of that And this is the coverage graph So we are in the process to modify this graph to be more meaningful currently, which is probably not Let's select one and then Yeah, this is the about dialogue of trans charts. So This this has incorporated And this is a still into progress. I think So Let's move to the next slide, which is for integrations So we have some planned integration for trans charts with bodhi so that we can have some automated test plugin We should be updating us like if there is any string breakage or not And if there is a string breakage, then it should flag maintainer Second is message queues like fit msg It is very important to have in sync with if we found any translation error in any of the packages Then we can respond to that error if we have an answer in trans charts Third is chart process like isc bot. This is interesting like in even in isc bot If we can link that to trans charts, then we can have some commands through which we can actually go and see What is happening behind the scenes and why things are breaking through isc itself In get involved we can get involved with trans charts By first test and try deployments So for deployments, you can just pull docker image using this command docker pull docker iotrans charts trans charts And run that image. So application will be available at port 8080 the server with some of the sample data And this is the steps for open ship deployment. You can go to github.com as trans charts For all these things and if you want to contribute You can get the development environment in just four commands These are the technical specs of the project Project url is trans charts.org docs is at docs.trans charts and source is at github So anytime if you would be interested in looking into it You can go to these urls and you can find details there So if you like the idea, you can go and submit one feedback feedback.trans charts.xyz Why I'm looking forward this because I have some questions to have for the development in trans charts So that we can decide on priority like which things we should first And we are available at federal globalization and at trans chart channel In irc free notes. So now I'm open for questions If somebody wants to discuss We can integrate it to the documentation, but we actually go what is the actual use case For that for example Probably most probably you will be having some projects at zanata Where you are pushing all docs file to get them translated and pull them back So we can add those repository into trans charts and then we can have tracking there So this is one use case. Are you talking about this use case or do you have any Currently it is not but we are looking forward it Thanks. Thanks. Thanks. Thanks. Thank you so much. So any other questions So, uh, basically I would say this is ready So you can have these things in trans charts as of now So we can have translation update volume and we can go and see for out of sync packages String change languages which need attention and coverage Yes Yes at trans chart.xyz it is available and it is available for almost 90 packages and for 70 languages I'm thankful to Rafa. He has submitted one bug for portuguese. So I'll fix that. I'll fix that. Yeah I'll fix that. I've seen that and it needs some back end changes. So I I'll keep and hold I'll fix that So most probably I'll do that today Sure See for that actually you need to get into integrities like Which which projects you are handling underneath of what? So, uh, suppose you have 10 projects. So you need to create 10 projects at zanata Where or whatsoever your translation platform is and then you need to add those projects in trans charts So that tracking could be enabled for those packages So whatsoever packages are being added to trans charts. They are being tracked So actually we need to get into that like, uh, I mean Uh, do you have anything to add in a question like? Basically for this system On the basis of that, uh, we don't have any plans as of now We are just creating some instance for fedora rel and then we'll be looking forward for other platforms as well Like if we are looking out to weblet, so First we need to enable all the weblet counterparts in trans charts as we have enabled it for zanata So if we have some apis in weblet, we have to consume those apis and then we can add projects which are in weblet to trans charts So this could be one use case, but it will require a small study before we go ahead simply so, uh Anyone has further questions Or we should break out for lunch Okay, thank you. Thank you for joining