Custom UDP protocol encryption without DTLS
<p><strong>Background Info</strong></p>

<p>I'm writing a custom UDP protocol which is targeted for general-purpose use, but will probably be used in a game development setting. My protocol handles all the issues of UDP, reliability, ordering and fragmentation are all handled. I'm using UDP because of the flexibility as I can send some packets unreliable and others reliable.</p>

<p><strong>The Problem</strong></p>

<p>I want my protocol to be encrypted, and I'm also very concerned about MITM. I've read several questions of people who also want to encrypt their UDP protocols and most people recommend DTLS. However the problem with DTLS is that it seems that no one uses it. I can't find decent guides or documentation on how to set up a client/server program for my chosen language (Java). It looks like the only option is <a href="https://github.com/bcgit/bc-java" rel="nofollow noreferrer">bouncycastle</a>, however <a href="https://github.com/bcgit/bc-java/issues/170" rel="nofollow noreferrer">considering the fact that their client/server test programs won't work with each other</a>, it's probably not a good idea.</p>

<p>I then decided to write the low-level packet receiving and sending code in C and use OpenSSL for the DTLS implementation. I would then call my C code using JNA. However, once again I could not find decent guides or tutorials on how to do DTLS. I could only find two ones which were somewhat helpful, the first one just went over the general C functions to call in which order. I got the impression that your application had to do the client verification yourself and since I have no idea how to do that it wasn't very helpful. The second one was just a raw client/server program which worked when ran but with closer inspection it seems to disable client verification.</p>

<p><strong>The actual question</strong></p>

<p>How would I go about creating my own encrypted transport system over UDP? I read a little about DHKE, but I don't know how to write a good implementation of it in Java using UDP and closer inspection seems that it doesn't prevent MITM. For my specific application I actually probably don't need a key exchange, the clients can actually have the key pair with the server installed beforehand. Would going this route work well? (I would probably just encrypt each packet body with the pair and send it to the server/client)</p>