Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-06 05:50

Just upgraded to 5.4 B1950. Everything appears to have upgraded correctly EXCEPT my entries for whitelisting have all been changed from *@somedomain.com to %@somedomain.com (wildcard characters changed from * to %).

Do I need to manually change the % characters back to * characters, or will this continue to work the same way as before?

Thanks in advance!

percepts
Senior user
Senior user
Posts: 5282
Joined: 2009-10-20 16:33
Location: Sceptred Isle

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by percepts » 2013-07-06 06:46

from the mysql manual:
SQL pattern matching enables you to use “_” to match any single character and “%” to match an arbitrary number of characters (including zero characters). In MySQL, SQL patterns are case-insensitive by default.
So I think it will be fine the way it is assuming it is being used directly in an SQL query within hmail. MySQL also includes some extended pattern matching using regex expressions. I believe some regex pattern matching changes were made to hmail recently and this may be why the upgrade has converted * to %.

Hopefully one of the developers can confirm if this is correct. i.e. don't assume I'm correct on this one.

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-06 16:18

As a matter of fact the wildcards will be changed again if you try to change them. :D
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-11 03:43

Thanks for the quick replies! I'll leave everything as-is.

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-16 03:47

Hate to revisit this issue, but I've just spent the last few hours trying to figure out why someone who was on our whitelist had their e-mail rejected because of being on a DNS blacklist.

%@linfinancial.com is an entry in our whitelist (and has been for some time)

I thought adding the address to our whitelist allowed the e-mail to pass through WITHOUT checking it against the blacklists???

Here's the relevant log entries (our server name and the sending e-mail address changed to protect the innocent):

"SMTPD" 66872 100250 "2013-07-12 12:01:47.415" "65.74.153.195" "SENT: 220 Welcome to the SMTP server."
"SMTPD" 64612 100250 "2013-07-12 12:01:47.461" "65.74.153.195" "RECEIVED: EHLO smtp.redtailtechnology.com"
"SMTPD" 64612 100250 "2013-07-12 12:01:47.462" "65.74.153.195" "SENT: 250-ourserver.arvixevps.com[nl]250-SIZE 100000000[nl]250 AUTH LOGIN"
"SMTPD" 65804 100250 "2013-07-12 12:01:47.508" "65.74.153.195" "RECEIVED: MAIL FROM:<admin@linfinancial.com>"
"SMTPD" 65804 0 "2013-07-12 12:01:47.551" "TCP" "DNS lookup: 195.153.74.65.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.601" "TCP" "DNS lookup: 195.153.74.65.b.barracudacentral.org, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.639" "TCP" "DNS lookup: 195.153.74.65.psbl.surriel.com, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.675" "TCP" "DNS lookup: 195.153.74.65.bl.spamcop.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.810" "TCP" "DNS lookup: 195.153.74.65.ix.dnsbl.manitu.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.816" "TCP" "DNS lookup: 195.153.74.65.dnsbl-1.uceprotect.net, 1 addresses found: 127.0.0.2, Match: True"
"SMTPD" 65804 0 "2013-07-12 12:01:47.831" "TCP" "DNS lookup: 195.153.74.65.dnsbl.sorbs.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.956" "TCP" "DNS lookup: 195.153.74.65.bl.mailspike.net, 1 addresses found: 127.0.0.12, Match: True"
"SMTPD" 65804 0 "2013-07-12 12:01:47.961" "TCP" "DNS lookup: 195.153.74.65.dyna.spamrats.com, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:47.994" "TCP" "DNS lookup: 195.153.74.65.dnsbl-2.uceprotect.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:48.046" "TCP" "DNS lookup: 195.153.74.65.hostkarma.junkemailfilter.com, 2 addresses found: 127.0.0.1, 127.0.1.1, Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:48.080" "TCP" "DNS lookup: 195.153.74.65.dnsbl-3.uceprotect.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:48.119" "TCP" "DNS lookup: 195.153.74.65.l2.apews.org, 0 addresses found: (none), Match: False"
"SMTPD" 65804 0 "2013-07-12 12:01:48.143" "TCP" "DNS lookup: 195.153.74.65.bl.spameatingmonkey.net, 0 addresses found: (none), Match: False"
"SMTPD" 65804 100250 "2013-07-12 12:01:48.144" "65.74.153.195" "SENT: 550 Rejected by Mailspike-BL."
"SMTPD" 65804 100250 "2013-07-12 12:01:48.211" "65.74.153.195" "RECEIVED: RSET"
"SMTPD" 65804 100250 "2013-07-12 12:01:48.212" "65.74.153.195" "SENT: 250 OK"
"SMTPD" 15836 100250 "2013-07-12 12:01:48.261" "65.74.153.195" "RECEIVED: QUIT"
"SMTPD" 15836 100250 "2013-07-12 12:01:48.261" "65.74.153.195" "SENT: 221 goodbye"


Much appreciation to anyone who can shed some light on why this e-mail was rejected.

Thanks,
Warren

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-16 05:26

Hey Warren,
Can you post a screenshot of your whitlist shown in hmail? Obfuscate as needed if you want. Or please answer this.. You say changed from * to %. Do you mean in the database or in the hmail admin GUI? Did you set IP's when creating the whitelist? What database do you use? I ask because it should show * in the admin but be % in the database. IP's should be 0.0.0.0 255.255.255.255 at least for testing then later you can shrink the window down to their actual IP's once you know it's working.
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-16 15:03

@Bill:

I'll try to get a screenshot later in the day when I'm back in the office. For the sake of speed, here's the answers to your other questions:

The *'s changed to %'s in the hMailServer admin GUI during the upgrade. They ALSO show as % in the database table.

The whitelist is composed entirely of individual e-mail addresses (someone@somewhere.com) and domains with wildcards (%@somewhere.com) with IPs specified as 0.0.0.0 to 255.255.255.255.

Using MS SQL Server Express 2008 R2 running on Windows Server 2008.

Appreciate any help you can provide - thanks!
Warren

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-16 22:00

Ok yeah that's a problem. I've tested & they should be *'s in the admin not %. If you add a new whitelist entry with * does it remain a * shown in admin?
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-17 00:34

It appears that new entries retain the * in the admin GUI - so do edited entries that were previously changed to %.

So I'm guessing I'll have to manually correct all of the entries?

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-17 05:46

WHairstonLOI wrote:It appears that new entries retain the * in the admin GUI - so do edited entries that were previously changed to %.

So I'm guessing I'll have to manually correct all of the entries?
Yeah maybe there is an issue with the upgrade scripts. What database do you use?
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
mattg
Moderator
Moderator
Posts: 21114
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by mattg » 2013-07-17 06:42

WHairstonLOI wrote:Using MS SQL Server Express 2008 R2
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-17 16:04

I just changed the % to a * in the hMailServer admin GUI in one whitelist record as a test. It appears to keep the * in both the GUI and in the database.

Is it supposed to stay as a * in the database as well?

I'd hate to manually update all of these entries if it's going to potentially cause other issues. Not quite sure how to proceed...

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-17 23:05

OK - now I'm starting to wonder if whitelisting is working at all in this latest upgrade.

I've had a couple of calls from people who have been rejected by the Nix Spam DNS blacklist, but their full e-mail addresses are in our whitelist (no wildcards, i.e.: actualname@frontiernet.net).

Can someone in development look at the actual code for 5.4 B1950 and confirm that the whitelist is actually being checked prior to the DNS blacklists?

Thanks,
Warren

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-18 00:51

it should be % in database & * in GUI, at least when using mysql. I'd have to look if it's done differently with mssql. nothing has been changed with antispam or whitelisting in awhile.
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

WHairstonLOI
New user
New user
Posts: 16
Joined: 2013-07-06 05:46

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by WHairstonLOI » 2013-07-18 03:35

I've manually gone through the hMailServer admin GUI and updated all of our wildcard entries from % to *. I can confirm that all of them now show * in both the admin GUI and the database.

At this point, I can only wait to see if some of my clients report DNS blacklist rejections from addresses that are in the whitelist. It's really been a hassle for one client in particular, especially considering that they knew certain addresses had already been whitelisted.

I REALLY don't want to relive this experience if/when we do another upgrade of hMailServer - can this issue be addressed?

Thanks in advance,
Warren

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2013-07-18 22:10

WHairstonLOI wrote:I've manually gone through the hMailServer admin GUI and updated all of our wildcard entries from % to *. I can confirm that all of them now show * in both the admin GUI and the database.

At this point, I can only wait to see if some of my clients report DNS blacklist rejections from addresses that are in the whitelist. It's really been a hassle for one client in particular, especially considering that they knew certain addresses had already been whitelisted.

I REALLY don't want to relive this experience if/when we do another upgrade of hMailServer - can this issue be addressed?

Thanks in advance,
Warren
Obviously it's frustrating to run into problems but considering this is the 1st time we've heard of this can't say it's a wide-spread common issue.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rob2k2
New user
New user
Posts: 28
Joined: 2013-12-12 03:07

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by rob2k2 » 2014-01-16 20:54

I just created a copy of my production environment in which i'm upgrading from the exact same version 5.3.3 b1879 to 5.4.. and sure enough i see the % signs in the white listing..

I havent actually done the upgrade yet.. as i'm just testing things out..

First off i want to thank you for the post.. as i wouldn't have even check if i didn't read this post.. but did editing and getting rid of the % though the gui solve the issue as far as you know???

i have about 15 or so white list entries.. that i need to keep.. or edit.. and not really able to test to see if they work... that i can think of..

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2014-01-16 23:31

rob2k2 wrote:I just created a copy of my production environment in which i'm upgrading from the exact same version 5.3.3 b1879 to 5.4.. and sure enough i see the % signs in the white listing..

I havent actually done the upgrade yet.. as i'm just testing things out..

First off i want to thank you for the post.. as i wouldn't have even check if i didn't read this post.. but did editing and getting rid of the % though the gui solve the issue as far as you know???

i have about 15 or so white list entries.. that i need to keep.. or edit.. and not really able to test to see if they work... that i can think of..
I'll have to look at the code to see what would cause that. What database are you using?

Based on OP's responses it sounded like it fixed the issue since the character was changed but hadn't heard back with confirmation.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rob2k2
New user
New user
Posts: 28
Joined: 2013-12-12 03:07

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by rob2k2 » 2014-01-17 02:32

Bill48105 wrote: I'll have to look at the code to see what would cause that. What database are you using?

Based on OP's responses it sounded like it fixed the issue since the character was changed but hadn't heard back with confirmation.
Bill
mysql 5.5.29

Bill48105
Developer
Developer
Posts: 6192
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by Bill48105 » 2014-01-17 05:44

rob2k2 wrote:
Bill48105 wrote: I'll have to look at the code to see what would cause that. What database are you using?

Based on OP's responses it sounded like it fixed the issue since the character was changed but hadn't heard back with confirmation.
Bill
mysql 5.5.29
Thx. I checked the 5.3 to 5.4 update script. It does:

Code: Select all

update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '/', '//');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '%', '/%');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '_', '/_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '?', '_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '*', '%');
That last one says "break me"
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

rob2k2
New user
New user
Posts: 28
Joined: 2013-12-12 03:07

Re: Upgrade from 5.3.3 B1879 to 5.4 B1950 changed wildcards?

Post by rob2k2 » 2014-01-21 00:38

Bill48105 wrote:
rob2k2 wrote:
Bill48105 wrote: I'll have to look at the code to see what would cause that. What database are you using?

Based on OP's responses it sounded like it fixed the issue since the character was changed but hadn't heard back with confirmation.
Bill
mysql 5.5.29
Thx. I checked the 5.3 to 5.4 update script. It does:

Code: Select all

update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '/', '//');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '%', '/%');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '_', '/_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '?', '_');
update hm_whitelist set whiteemailaddress = replace(whiteemailaddress, '*', '%');
That last one says "break me"
Nice Find.. Well I haven't had any complaints.. seems the fix is just to edit them.. I just replaced them all myself.. seems to be working all good..

getting off topic a little - it's been 4 days since i upgraded my production server.. and my error log has dropped to like one entry.. which has to do with spamassin timeing out sometime.. which is nothing to do with hmail server.. no more imap errors so far.. Upgrade totally worth it! To those who haven't from 5.3 cause there worried.. do it.. and it doesn't take long to manually edit your white list entries.. if you have them.. (would only be the wildcard entries anyway)..

Post Reply