iText signing PDF using external signature with sm

I've been toying with iTextSharp 5.5.7 for a while and can't find the right way to make a valid digital signature for PDF from Smart Card - Adobe Reader always says its signed by and unknown and can't decode signatures' DER data.

I've looked at MakeSignature.cs code for reference and what is does:

        Stream data = signatureAppearance.GetRangeStream(); 
        // gets the first hash
        byte[] hash = DigestAlgorithms.Digest(data, hashAlgorithm);
        // gets the second hash or is it not a hash at all ?
        byte[] sh = sgn.getAuthenticatedAttributeBytes(hash, ocsp, crlBytes, sigtype);

then, according to "sign" method in IExternalSignature.cs

"@param message the message you want to be hashed and signed"

        // looks like externalSignature.Sign() should make another hash out of "sh"
        // and use this hash to compute a signature
        byte[] extSignature = externalSignature.Sign(sh);

so I understood the procedure of signing as the following:

source PDF is loaded
new PDF with empty signature field is created
Byte range of that field is hashed (by default produces 20 bytes for sha-1, tried 32 bytes with sha-256 too)
that Hash + some other properties are hashed again (number of bytes varies, why? might not be a hash after all?)
that second hash is hashed again inside of external signature object
that third hash is finally sent to a Smart Card to compute a signature
Signature is inserted into new PDF

When i sign PDF with Adobe Reader, at step 6, the third hash is 32 bytes long. From Smart Card's perspective i do the same steps with both Acrobat and iText, but with iText the signature is invalid, what could be wrong ?

the code i use:

public void StartTest(){
        X509Certificate2 cert = new X509Certificate2();
        cert.Import("cert.cer"); // certificate obtained from smart card

        X509CertificateParser certParse = new Org.BouncyCastle.X509.X509CertificateParser();

        Org.BouncyCastle.X509.X509Certificate[] chain = new Org.BouncyCastle.X509.X509Certificate[] { certParse.ReadCertificate(cert.RawData) };

        // Reader and stamper
        PdfReader pdfReader = new PdfReader("original.pdf");
        Stream signedPdf = new FileStream("signed.pdf", FileMode.Create);
        PdfStamper stamper = PdfStamper.CreateSignature(pdfReader, signedPdf, '\0', null, false);

        // Appearance
        PdfSignatureAppearance appearance = stamper.SignatureAppearance;
        appearance.SignatureCreator = "Me";
        appearance.Reason = "Testing iText";
        appearance.Location = "On my Laptop";
        appearance.SignatureGraphic = Image.GetInstance("img.png"); // visual image
        appearance.SetVisibleSignature(new Rectangle(50, 50, 250, 100), pdfReader.NumberOfPages, "Signature");
        appearance.SignatureRenderingMode = PdfSignatureAppearance.RenderingMode.GRAPHIC_AND_DESCRIPTION;

        // Timestamp
        TSAClientBouncyCastle tsc = new TSAClientBouncyCastle("http://ts.cartaodecidadao.pt/tsa/server", "", "");

        // Digital signature
        IExternalSignature externalSignature = new MyExternalSignature2("SHA-1");
        MyMakeSignature.SignDetached(appearance, externalSignature, chain, null, null, tsc, 0, CryptoStandard.CADES);

        stamper.Close();
}

external signature implementation (class MyExternalSignature2):

    class MyExternalSignature2 : IExternalSignature
{
    private String hashAlgorithm;
    private String encryptionAlgorithm;

    public MyExternalSignature2(String hashAlgorithm)
    {
        this.encryptionAlgorithm = "RSA";
        this.hashAlgorithm = DigestAlgorithms.GetDigest(DigestAlgorithms.GetAllowedDigests(hashAlgorithm));
    }

    public virtual byte[] Sign(byte[] message) {

        byte[] hash = null;
        using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
        {
            hash = sha1.ComputeHash(message);
        }

        byte[] sig = MySC.GetSignature(hash);

        return sig;
    }

    public virtual String GetHashAlgorithm() {
        return hashAlgorithm;
    }

    public virtual String GetEncryptionAlgorithm() {
        return encryptionAlgorithm;
    }
}

The OP supplied a number of sample signed documents, and analyzing them it became obvious that my initial answer was based on a misunderstanding.

Analysis of supplied signed PDFs

The OP supplied three PDFs signed by his code, at first these two:

ex_signed.pdf (signed with sha-1 and CryptoStandard.CADES)
ex_signed_2.pdf (signed with sha-1 and CryptoStandard.CADES)

Inspecting them a peculiarity of iText CAdES signing with SHA1 became appearant: for CryptoStandard.CADES iText uses the SigningCertificateV2 attribute even for SHA1 but specifications recommend using the SigningCertificate attribute there instead. To prevent this peculiarity from interfering, the OP supplied the third file

ex_signed_3.pdf (signed with sha-256 and CryptoStandard.CMS)

It turned out, though, that this quirk was not the cause of the OP's observations, Adobe Reader still reports cryptography library errors.

Thus, back to analysis.

As the signature algorithms were SHA1withRSA/2048 and SHA256withRSA/2048, one can simply decrypt the inner signature value using the public key from the respective certificate.

This succeeded for ex_signed.pdf but not for ex_signed_2.pdf or ex_signed_3.pdf.

The OP meanwhile indicated in a comment:

the difference between those files are certificates used for signatures, in the first i used a certificate for authentication, which is (i think is) acceptable but i have another certificate specifically for digital signatures, which i used in 2nd and 3rd file.

So I tried to decode the signature value in the second and third file using the certificate from the first file, and indeed, that worked! So the second and third file claim to be signed by the private key associated with that alternative certificate but actually are signed using the private key associated with the former certificate.

Thus: Problem 1: Signatures in files 2 and 3 are signed with the wrong private key / the wrong application on the smart card.

To further analyze the issue, I looked at the signature values successfully decoded with the authentication certificate:

ex_signed.pdf:

2a8945abe450b2c1cd232249b8f811d352ad0d29

ex_signed_2.pdf

cc24acc848002df63733941e34437f8aef1c746c

ex_signed_3.pdf

45f8e451f8b9f39f0c1f59eea8b6308fba22176ac62ebd14bbf07e5407aed7e8

So for SHA1 there are 20 bytes and for SHA-256 there are 32 bytes. This is exactly the size of the hash values, so most likely these simply are the naked hash values.

This is wrong, though, XXXwithRSA signatures are expected to contain an encrypted structure which contains an OID of the hashing algorithm and the hash, in ASN.1 notation:

DigestInfo ::= SEQUENCE {
  digestAlgorithm DigestAlgorithmIdentifier, 
  digest Digest
  }

DigestAlgorithmIdentifier ::= AlgorithmIdentifier

Digest ::= OCTET STRING

For backgrounds cf. RFC 3447.

This explains the OP's observation:

the error in the first is described as "BER decoding error",

The verifier tries to interpret the naked hash as an ASN.1 sequence encoded using the BER which obviously fails.

Thus: Problem 2: The smart card encrypts the naked hash value but has to encrypt a DigestInfo object encapsulating that hash.

Initial, obsolete answer

If your MySC.GetSignature call hashes while signing the data (and does not expect data to be already hashed before), you should replace

public virtual byte[] Sign(byte[] message)
{
    byte[] hash = null;
    using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
    {
        hash = sha1.ComputeHash(message);
    }

    byte[] sig = MySC.GetSignature(hash);

    return sig;
}

by something like

public virtual byte[] Sign(byte[] message)
{
    byte[] sig = MySC.GetSignature(message);

    return sig;
}