C# - Is it possible to Sign hash using RSA SHA512?
<p>I have created a selfsign certificate using OpenSSL as follows:</p>

<p><code>openssl req -x509 -sha512 -new keyrsa:2048 -keyout key2.pem -out cert2.pem -days 100</code></p>

<p><code>openssl pkcs12 -export -out pkcs12_cert_test2.pfx -inkey key2.pem -in cert2.pem</code></p>

<p>I have installed <code>pkcs12_cert_test2.pfx</code> on Windows and the certificate sign algorithm value is <code>sha512RSA</code></p>

<p>Then, I have coded the following in C# .NET 4.0:</p>

<pre><code>public static bool DSHandler(string operation, string path, string devicePath)
{
    bool result = false;
        CryptoConfig.AddAlgorithm(typeof(USBSaferAppEFB.RsaPkCs1Sha512SignatureDescription),
            "http://www.w3.org/2001/04/xmldsig-more#rsa-sha512");
        string password = "xxxx";
        bool validSignature = false;
        byte[] hashchain = generateHash(path);
        string digSignFile = Utils.RegVarValue(DIGSIGNFILE);
        string subjectDN = Utils.RegVarValue(SUBJECTDN);

        if (operation == "sign")
        {
            byte[] signature = SignFromContainer(hashchain, subjectDN);
            if (signature != null)
            {
                string signaturestring = Convert.ToBase64String(signature);

                File.WriteAllBytes(devicePath + digSignFile, signature);
                result = true;
            }
            else
                result = false;
        }
}

static byte[] SignFromContainer(byte[] hashchain, string certSubject)
{
    try
    {
        X509Store my = new X509Store(StoreName.My, StoreLocation.CurrentUser);
        my.Open(OpenFlags.ReadOnly);
        RSACryptoServiceProvider csp = null;
        foreach (X509Certificate2 cert in my.Certificates)
        {
            if (cert.Subject.Contains(certSubject))
            {
                // We found it.
                // Get its associated CSP and private key
                csp = (RSACryptoServiceProvider)cert.PrivateKey;
            }
        }

        if (csp == null)
        {
            return null;
        }

        byte[] myHash = { 59,4,248,102,77,97,142,201,
        210,12,224,93,25,41,100,197,
        210,12,224,93};

        // Sign the hash
        return csp.SignHash(hashchain, CryptoConfig.MapNameToOID("SHA-512"));
    }
    catch (Exception e)
    {
        log.Logger("Exception: "+e.Message, "debug");
        log.Logger(e.StackTrace, "debug");
        return null;
    }
}
</code></pre>

<p>Using solutions from some articles and answers, I have <code>RsaPkCs1Sha512SignatureDescription</code> class where I try to implement and register the signature description:</p>

<pre><code>public class RsaPkCs1Sha512SignatureDescription : SignatureDescription
{
    public RsaPkCs1Sha512SignatureDescription()
    {
        KeyAlgorithm = typeof(RSACryptoServiceProvider).FullName;
        DigestAlgorithm = typeof(SHA512Managed).FullName;
        FormatterAlgorithm = typeof(RSAPKCS1SignatureFormatter).FullName;
        DeformatterAlgorithm = typeof(RSAPKCS1SignatureDeformatter).FullName;
    }

    public override AsymmetricSignatureDeformatter CreateDeformatter(AsymmetricAlgorithm key)
    {
        var sigProcessor = (AsymmetricSignatureDeformatter)CryptoConfig.CreateFromName(DeformatterAlgorithm);
        sigProcessor.SetKey(key);
        sigProcessor.SetHashAlgorithm("SHA-512");
        return sigProcessor;
    }

    public override AsymmetricSignatureFormatter CreateFormatter(AsymmetricAlgorithm key)
    {
        var sigProcessor =
            (AsymmetricSignatureFormatter)CryptoConfig.CreateFromName(FormatterAlgorithm);
        sigProcessor.SetKey(key);
        sigProcessor.SetHashAlgorithm("SHA-512");
        return sigProcessor;
    }
}
</code></pre>

<p>I have verified that hash is <code>64</code> bytes long and it returns a <code>CryptographicException: Bad Hash</code>. But if I use <code>myHash</code> var (20 bytes long), although algorithm specified in <code>SignHash</code> is <code>SHA512</code>, it works, and sign the hash using <code>SHA1</code> algorithm.</p>

<p>Also, if I print <code>csp.SignatureAlgorithm</code>, its value is <a href="http://www.w3.org/2000/09/xmldsig#rsa-sha1" rel="nofollow">http://www.w3.org/2000/09/xmldsig#rsa-sha1</a>. Why is signature algorithm equals to <a href="http://www.w3.org/2000/09/xmldsig#rsa-sha1" rel="nofollow">http://www.w3.org/2000/09/xmldsig#rsa-sha1</a> if the detail of the installed certificate shows <code>sha512RSA</code> as sign algorithm? Could be this point the real problem? If so, how could I create the certificate properly? Thanks in advance!</p>