Page 1 of 1

DKIM validation fails for facebook.com

Posted: 2013-02-10 00:53
by japi
Good evening everyone,

I just noticed that mails from facebook.com fail to validate their DKIM signature, although the mail is authentic.
For other domains like gmail.com the validation works.
I am using hMailServer 5.4 - 1946.

Signature:

Code: Select all

DKIM-Signature: v=1; a=rsa-sha256; d=facebookmail.com; s=s1024-2011-q2; c=relaxed/simple; q=dns/txt;
 i=@facebookmail.com; t=1360442533; h=From:Subject:Date:To:MIME-Version:Content-Type;
 bh=Jg[...]Cg=; b=Je[...]z0=;
Logs:

Code: Select all

"DEBUG"	8416	"2013-02-09 18:55:27.891"	"DKIM: Unable to base64 decode public key found in DNS record. Key: MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDLWnmo7aFBKfL4+mogTe/cXx6D4MUF7VUM9O+nmXAcUP6jJh1RDgZuSJ/KKxo+KMpDiF5xnawr4p3N4eFruSZWFB1vtHgDiy3iPke/u0lmXB2PDQphFRJU4Raghm9e2duPfuSExbvSu9COWIoaz1vH/T+8zc0vuonClGuPfxoqhQIDAQAB"
"DEBUG"	8416	"2013-02-09 18:55:27.891"	"Spam test: SpamTestDKIM, Score: 5"
"DEBUG"	8416	"2013-02-09 18:55:27.891"	"Total spam score: 5"
Nslookup:

Code: Select all

PS C:\WINDOWS\system32> .\nslookup.exe
> set querytype=TXT
> s1024-2011-q2._domainkey.facebookmail.com
s1024-2011-q2._domainkey.facebookmail.com       text =

        "k=rsa; t=s; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDLWnmo7aFBKfL4+mogTe/cXx6D4MUF7VUM9O+nmXAcUP6jJh1RDgZuSJ/KKxo+KMpDiF5xnawr4p3N4eFruSZWFB1vtHgDiy3iPke/u0lmXB2PDQphFRJU4Raghm9e2duPfuSExbvSu9COWIoaz1vH/T+8zc0vuonClGuPfxoqhQIDAQAB"
Is it just me or does anyone else have this error?

Best regards,
japi

Re: DKIM validation fails for facebook.com

Posted: 2013-02-10 01:11
by Bill48105
Looking at the 5.4 source it appears hmail has the right key since it matches you manual dns lookup but the decode error suggests there is a problem with the key itself. Decoding manually results in binary.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-11 21:28
by japi
Can someone please validate if this is a problem of my setup?
It seems to be nonexistent in 5.3.3. (at least i had no [SPAM] Tags in front of every FB mail before upgrading to 5.4)
I don't want to file another bug :mrgreen:

Re: DKIM validation fails for facebook.com

Posted: 2013-02-11 21:58
by Bill48105
I don't use DKIM in hmail because i use ASSP but I'll check another hmail server I run. They didn't have DKIM enabled either so I enabled & just need to wait for a facebook email to come in.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-11 22:07
by Bill48105
btw comparing facebookmail to gmail the only obvious difference is fb adds t=s; h=sha256; where those don't exist on gmail's record. Not sure if that's the cause or how to easily test unless we find another domain with those & compare.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 00:43
by japi
Bill48105 wrote:btw comparing facebookmail to gmail the only obvious difference is fb adds t=s; h=sha256; where those don't exist on gmail's record. Not sure if that's the cause or how to easily test unless we find another domain with those & compare.
Bill
Good idea, i skipped testing it, because the hmail output looks like the key has been detected properly and just failed to convert.
I will test it anyway :)

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 00:47
by Bill48105
i'm still waiting on a facebook email to come into their server to see the result since enabling dkim in hmail. Clearly there's something different so matter of figuring out what & going from there
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 00:57
by Bill48105
OK I sent a friend invite to a box I got there & get the same DKIM error. So not something on your end. Wonder if it's encrypted using a method hmail doesn't support. Would have to look at the code or check other domains for their DKIM to see if we can find another with t=s; h=sha256; as well to see.

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 00:59
by Bill48105
btw according to this site facebookmail.com dkim is OK:
http://dkimcore.org/c/keycheck

Code: Select all

DKIM Record

k=rsa; t=s; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDLWnmo7aFBKf
L4+mogTe/cXx6D4MUF7VUM9O+nmXAcUP6jJh1RDgZuSJ/KKxo+KMpDiF5xnawr4p3N4eFruSZW
FB1vtHgDiy3iPke/u0lmXB2PDQphFRJU4Raghm9e2duPfuSExbvSu9COWIoaz1vH/T+8zc0vuo
nClGuPfxoqhQIDAQAB

This is a valid DKIM key record

Key type
k=	rsa
Flags
t=	s
Hash algorithm
h=	sha256
Public key
p=	MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDLWnmo7aFBKfL4+mogTe/cXx6D4MUF7VUM9O+nmX
AcUP6jJh1RDgZuSJ/KKxo+KMpDiF5xnawr4p3N4eFruSZWFB1vtHgDiy3iPke/u0lmXB2PDQphFRJU4Raghm9e2d
uPfuSExbvSu9COWIoaz1vH/T+8zc0vuonClGuPfxoqhQIDAQAB

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 01:58
by japi
I set up an identical DKIM record and it worked... :x
Lengths of the public keys are identical.
Start and ending of the keys are identical.
Selector is identical.
Flags etc. in the TXT record are identical.
Am I missing something? :shock:
FB has to be exploiting a bug in hmails base64 decoder intentionally :wink:

Signature:

Code: Select all

dkim-signature: v=1; a=rsa-sha256; d=mydomain.com; s=s1024-2011-q2; c=relaxed/simple; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=vD[...]J8=; b=Ez[...]jk=
Logs:

Code: Select all

"DEBUG"	2764	"2013-02-12 00:41:01.224"	"DKIM: Message passed validation."
"DEBUG"	2764	"2013-02-12 00:41:01.224"	"Spam test: SpamTestDKIM, Score: 0"
"DEBUG"	2764	"2013-02-12 00:41:01.224"	"Total spam score: 0"
Nslookup:

Code: Select all

s1024-2011-q2._domainkey.mydomain.com        text =

        "k=rsa; t=s; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCeJi9bISaDoqFsWcerHhmU0ONKZStJ+UXh3/SD/D4rORPU6Dl
KBi0wIhz99+nSZc9j9hS+WV7u5aJ9Ou+VI/tkTmkiPDm6xXAB0BcDbG69G742+FmOcV1OpNhcL1fHksNeg+uXYuW5JPHqVCSh4wkFMrP7XnsHZOfX2a208dN
T5wIDAQAB"

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 02:42
by japi
I just had a look at http://hmailserver.com:60951/svn/hms/tr ... Base64.cpp

Stumbled upon the following Text:

Code: Select all

   AnsiString 
   Base64::Decode(const char *input, int inputLength)
   {
      // base64 encode the signature.
      MimeCodeBase64 decoder;
      decoder.SetInput(input, inputLength, false);

      AnsiString sEncodedValue;
      decoder.GetOutput(sEncodedValue);

      // the MIME encoder will insert newlines. We don't want this
      // here since this is a generic base64 encoder which may be
      // used in none-mime environments (key encoding anyone?)
      sEncodedValue.Replace("\r\n", "");

      return sEncodedValue;
   }
// the MIME encoder will insert newlines. We don't want this
// here since this is a generic base64 encoder which may be
// used in none-mime environments (key encoding anyone?)
sEncodedValue.Replace("\r\n", "");


Yes, the encoder will insert newlines.
But this is the decoder!
We just decoded base64 into binary and then removed "\r\n" ^^
That could be the error, if FB has a \r\n in the public keys binary stream.
Because after that, the decoded (and \r\n free) data is given to openssl, which notices the corruption and thus the Line

Code: Select all

EVP_PKEY *publicKey = d2i_PUBKEY(NULL, &publicKeyDataPointer, publicKeyData.GetLength());
in http://hmailserver.com:60951/svn/hms/tr ... M/DKIM.cpp
will fail, which leads to the key not being saved.

The code

Code: Select all

      EVP_PKEY *publicKey = _GetPublicKey(publicKeyString);
      if (!publicKey)
      {
         // unable to extract public key from record. broken?
         LOG_DEBUG("DKIM: Unable to base64 decode public key found in DNS record. Key: " + publicKeyString);
         return result;
      }
finally throws the Debug info into the logfile.

Could this be the error?
I'm not good at C/C++ but I learned a bit java at school :mrgreen:

Best regards,
japi

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 02:52
by Bill48105
I checked the hex of the decoded binary there is indeed a 0D 0A sequence in there! Wow great find there. I had looked at the dkim code but did't get back far enough to look at the b64 decode call. I could do a test build if you wanted to try it out, I'd just remove that replace as a quick test. Maybe a noreplace argument or 2nd 'binary friendly' function is needed if that's the case.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 03:10
by Bill48105
OK here's a special build if you want to test it:
http://www.mediafire.com/file/zqr76f2vb ... Replace.7z (1.8M)
MD5: 1ebb504a4ab7a436276cec7958c17e29 SHA1: 3f9c1f7eb34749b8062520bec39a423c12102655

Since you're on 5.4 B1946 this should be direct drop-in per the experimental builds instructions:
http://www.hmailserver.com/forum/viewto ... 10&t=21420

As a minimum stop hmailserver.exe service & backup BIN folder files then drop in all the files in 7z into BIN folder & start hmail. If you have issues you can just restore your BIN folder.

For this build I added a 2nd decode function Base64::DecodeNoReplace(const char *input, int inputLength) that is called instead of the original in DKIM which does not have the sEncodedValue.Replace("\r\n", ""); line. If your theory is correct this should solve your facebook issue & not break anything else since the original decode is still used everywhere else. (If the original should even have the replace or not is for another day. For now tracking down the issue you ran into.) There is a risk that we WANT the replace for some DKIM but I highly doubt that. Either way doing it this way should have the least risk of collateral issues.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 03:23
by japi
It works! :D

Is the decode function used anywhere else?
If not, the /r/n removal could be removed without any substitute imho :)

@ martin & Bill:
Thank you for your great support!
I have never experienced such a good and fast support in any other open or closed source product! :)

Best regards,
japi

Log of success:

Code: Select all

"SMTPD"	1500	9	"2013-02-12 02:21:42.521"	"69.171.232.149"	"SENT: 354 OK, send."
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Execute"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Found URL: facebook.com"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Found URL: fbcdn.net"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: 2 unique addresses found."
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Lookup: facebook.com.multi.surbl.org"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Lookup: fbcdn.net.multi.surbl.org"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"SURBL: Match not found"
"DEBUG"	3108	"2013-02-12 02:21:42.859"	"Spam test: SpamTestSURBL, Score: 0"
"DEBUG"	3108	"2013-02-12 02:21:42.874"	"DKIM: Message passed validation."
"DEBUG"	3108	"2013-02-12 02:21:42.874"	"Spam test: SpamTestDKIM, Score: 0"
"DEBUG"	3108	"2013-02-12 02:21:42.874"	"Total spam score: 0"
DEBUG

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 03:34
by Bill48105
WOO cool beans! Sure no problem I'm happy to help do test builds especially when someone helps out tracking it down & testing. But yeah we strive to please. :)

Well I didn't search the code but I'll imagine that base64 decode function is called MANY MANY places so not likely safe to remove completely or at least without careful thought.. As I said I did i the way I did it as what I'd consider a pretty safe method since I essentially gave DKIM it's own function just in case to minimize risk of breaking anything else in the meantime. If later it's determined the original one can safely have that removed we can do that & remove the new function I made but for what seems we got a winner.
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 03:45
by Bill48105
OK quick scan of the code seems this particular decode is only used for DKIM so there's no need to add the 2nd function so I'll fix that. I confirmed by renaming the original function & rebuilding with only errors coming from the regression tests that call it. Renaming those as well results in clean build. (Btw there are other base64 decode functions that handle other areas of code)
Bill

Re: DKIM validation fails for facebook.com

Posted: 2013-02-12 03:55
by japi
Great work! :D

Thanks again,
glad I could help :D

Re: DKIM validation fails for facebook.com

Posted: 2013-05-12 18:31
by kmwade
Bill, is this fix in subsequent experimental builds? Or are we frozen at this point if we need the DKIM fix?

Re: DKIM validation fails for facebook.com

Posted: 2013-05-13 00:40
by Bill48105
kmwade wrote:Bill, is this fix in subsequent experimental builds? Or are we frozen at this point if we need the DKIM fix?
Hey kmwade. Yeah this change has been in my experimentals since I posted the special build in Feb & I did commit to official code since Feb 12, 2013 so B1947+ have it.
Bill