Page 4 of 4

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-08 23:05
by percepts
Ah, just spotted following so I guess the answer is I'll wait until an experimental build based B1951 since I don't need/use SSL except for experimenting.

http://www.hmailserver.com/forum/viewto ... 2e#p160238

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-09 02:08
by Bill48105
percepts wrote:Bill,

Now martin has released B1951 production release where does that leave us with regard to experimental releases?
Will there need to be new experimental release based on B1951 or can put already released eperimental versions on top of B1951? Does 1951 include any/all of the experimental features?
mattg wrote:
percepts wrote: Does 1951 include any/all of the experimental features?
Or other fixes that you have developed...Like the UTF-8 Indexing, the IOCP fixes etc
martin did not say what was changed in B1951 but I assume it is likely B1950 with the openssl update. I have not had time to submit changes so the official releases don't have any post B1950 changes I've made in the experimental builds. No STARTTLS, no UTF-8 fixes, no IOCP race condition fixes, none of them. Although HE might have made changes post B1950. Won't know until he says or I review his change logs & compare.

Anyone who was using one of my experimental builds because of they wanted/needed changes I made should likely use the experimental I posted today. Anyone who is on an official build will likely do B1951 unless they want my changes.

btw in talking to martin today an automated build system is likely going to be up soon. I just don't know if it will only build official source once he accepts proposed changes or if it can build my fork whenever I make changes. I'm guessing the prior which means while it should make it easier to merge my changes with the official it may not make official releases with my changes come any quicker.
Thx
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 12:41
by mivimex
Hallo, dear developers.
Please tell me, if I want to avoid "LOT of extra debug logging by default" in alpha builds, should I just add

Code: Select all

[Settings]
LogLevel=10
to hMailServer.INI file?

Thank you.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 12:52
by percepts
mivimex wrote:Hallo, dear developers.
Please tell me, if I want to avoid "LOT of extra debug logging by default" in alpha builds, should I just add

Code: Select all

[Settings]
LogLevel=10
to hMailServer.INI file?

Thank you.
The lower the number the less debug lines you get. I think you can just comment it out by starting the line with a ;

Debug logging is only for when you need it. i.e. I wouldn't even have it switched on for everyday use so you can disbale it in hmail logging options unless you really need it.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 13:10
by mivimex
Thank you, percepts
just comment it out
I have no [Settings] section in my hMailServer.INI file.
My question is - should I add this section and text?

I still need logging, since I've just moved to hMailServer, and want to see some "internal" info

UPD: added section & loglevel=10, and now there is no unneeded logging (including message text) in pop3d log.
Service works without fails.

Thanks.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 13:25
by percepts
copy and paste following into your ini file taking care not to duplicate anything or overwrite anything.

These ini settings relate to current latest alpha build and maybe the previous one or two alpha builds. They won't harm anything if you have a slightly older version of hmail but with the bleeding heart fix you should be using absolute latest alpha build or latest production release now.

http://www.hmailserver.com/forum/viewto ... 38#p132238

and note that you can switch debug logging off but leave smtp, tcp, imap and pop switched on. So you can still see mail coming and going when debug is switched off.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 13:33
by mivimex
Just done.
Thank you for the link, maybe I should wear glasses ))

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 13:46
by percepts
mivimex wrote:Just done.
Thank you for the link, maybe I should wear glasses ))
quite possibly :mrgreen:

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-11 19:33
by Bill48105
mivimex wrote:Hallo, dear developers.
Please tell me, if I want to avoid "LOT of extra debug logging by default" in alpha builds, should I just add

Code: Select all

[Settings]
LogLevel=10
to hMailServer.INI file?

Thank you.
Hello. By default LogLevel is not defined & only "normal" debug logging will show. Setting it lower will show even less (turning off some normal debug logging in official builds), setting it higher (such as 100) will show INSANE debug level (NOT recommended for live server as you will get TONS & TONS of very detailed logging). At some point we'll get it in the GUI admin & it'll be a scroll bar like:

Code: Select all

Debug logging
[                          *        ]
Minimal        Normal        Maximum
1                 9            100
Where "Normal" is 9 which is the default if LogLevel is not defined.

The LogLevel stuff is a bit confusing because I changed how it works a few times but I think the way it is now will remain since once you understand how it works it makes sense.
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014040801

Posted: 2014-04-22 00:43
by Bill48105
Posted 2 new test builds. Chose not to post here like normal since they are indeed highly experimental & don't really want people accidentally using them in production:
http://www.hmailserver.com/forum/viewto ... 24#p160724

Here is a copy of the post:
I have a TEST BUILD if anyone wants to try it. Understand this is not tested beyond making sure the values set show up as expected in the logs as shown.
*** I would not recommend using this test build in production ***

hmailserver.ini

Code: Select all

[Settings]
SSLOptionList=default_workarounds,no_sslv2,no_tlsv1
;DO NOT USE this example SSLOptionList. It is just an example
;Default if not defined is default_workarounds,no_sslv2
SSLCipherList=ECDH:DH
;DO NOT USE this example SSLCipherList. It is just an example
;Default if not defined is OpenSSL default Ciphers
IMPORTANT!
* Note that SSLOptionList defaults to default_workarounds,no_sslv2 if not set (that is what hmailserver uses now) and that it is a COMMA DELIMITED LIST. I would not put spaces between I do not know what that would do.
* SSLCipherList defaults to NOTHING as is the case with hmailserver now and it is a COLON DELIMITED LIST as shown & from the available options shown on openssl site: https://www.openssl.org/docs/apps/ciphers.html# Using ! in front of a cipher disables it. Do not put spaces between each.
* For now ensure all OPTIONS set to lower case and CIPHERS set to UPPER CASE, unless confirmed to work otherwise. (Tempted to force case in the code but for now just match case of example INI above)
* These lists are used for both listening (incoming) and outgoing. If there becomes a need I can make 2 more INI settings but for now this allows testing.

* IMPORTANT: This build has a LOT of extra debug logging but NOT shown by default. [Settings]LogLevel=10 for some extra to 100 for extremely verbose (10 needed to show below log lines)

Code: Select all

"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - SSLOptionList Option: default_workarounds,no_sslv2,no_tlsv1"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - SSL Cipher Option: ECDH:DH"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - Found SSL Option: default_workarounds"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - SSL Option SET: default_workarounds"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - Found SSL Option: no_sslv2"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - SSL Option SET: no_sslv2"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - Found SSL Option: no_tlsv1"
"DEBUG"	3448	"2014-04-21 17:08:32.079"	"TCPServer::InitSSL() - SSL Option SET: no_tlsv1"
*** I would not recommend using this test build in production ***

Other changes in this build since 4/8 build:
* Fixed bug where built-in backup would abort if Data > 15GB despite BackupMessagesDBOnly=1 option. (Thx CU2U)
* Using martin's openssl 1.0.1g from official source vs my assembly built one. (Might as well stick with his now that it is available)

In addition I am posting up a 2nd test build which also changes:
* Used /LARGEADDRESSAWARE option during build which should allow hmailserver to use more memory if needed (such as heavy IMAP load)
* WARNING!!! - UNTESTED!!! - Could cause unknown problems due to memory space addressing.
* WARNING!!! - Should ONLY be tested on 64bit Windows Vista or later with AT LEAST 4GB MEMORY!! (It makes no sense to try this on 32bit or <2GB anyway but you should have at least 3GB before it could make a difference but really 4GB.)
* YOU'VE BEEN WARNED!

OK here are the downloads: (Install just like any other experimental build)

SSLOptions SSL Cipher TEST build: (See warnings above)
http://www.mediafire.com/download/dzel2 ... ns-TEST.7z
MD5: 531605ee555e7b2d175a2ebe31071e0b SHA1: c1209e9e666fe4ff3cfa4fd43b316015e3a02eb3

LargeMemory SSLPtions SSL Cipher TEST build: (See warnings above)
http://www.mediafire.com/download/nu0f7 ... ns-TEST.7z
MD5: e1fd3a6787e62297af607893d2d5fd4a SHA1: 8f50730f7783ee1dcec8631c0b28de757529e6ba

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050201

Posted: 2014-05-03 00:20
by Bill48105
2014-04-08 5.4-B2014050201
* IMPORTANT: This build has a LOT of extra debug logging but NOT shown by default. [Settings]LogLevel=10 for some extra to 100 for extremely verbose
* Added DisableAUTHList INI setting (See INI post below) - Appears to work
* Fixed bug where built-in backup would abort if Data > 15GB despite BackupMessagesDBOnly=1 option. (Thx CU2U)
* Using martin's openssl 1.0.1g from official source vs my assembly built one. (Might as well stick with his now that it is available)
* Includes SSLOptionList & SSLCipherList INI settings (See INI post below) - Appear to work
* DOES NOT include /LARGEADDRESSAWARE that was put in 1 test build

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-04 21:16
by Bill48105
2014-05-04 5.4-B2014050401
* IMPORTANT: This build has a LOT of extra debug logging but NOT shown by default. [Settings]LogLevel=10 for some extra to 100 for extremely verbose
* FIX: EHLO response now "250 AUTH" when DisableAUTHList set since gmail refused to send if no AUTH in EHLO response

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-26 22:25
by fearworksmedia
Hi,

I saw in another thread a post about keeping an eye on experimental builds as you were planning on adding new event calls, and I wondered if you were planning on adding one for when a message is moved from one folder to another, along the lines of onFolderMove, or something similar? This would be great to allow writing some code that would make SpamAssassin learn the message as spam when moved to a folder called, let's say, "Spam", and learn a message as ham when moved out of a folder called "Spam".

Would be a great event call if you could add it! :D

Thanks!

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-28 07:41
by Bill48105
fearworksmedia wrote:Hi,

I saw in another thread a post about keeping an eye on experimental builds as you were planning on adding new event calls, and I wondered if you were planning on adding one for when a message is moved from one folder to another, along the lines of onFolderMove, or something similar? This would be great to allow writing some code that would make SpamAssassin learn the message as spam when moved to a folder called, let's say, "Spam", and learn a message as ham when moved out of a folder called "Spam".

Would be a great event call if you could add it! :D

Thanks!
Hi. We've had a few people ask for that but I find it very unlikely it could happen. The issue is what happens when someone moves 10,000 emails? It's called 10k times. :o

So adding new event calls isn't really that bad but extreme care needs to be taken to avoid placing them in areas where they'd be called too frequently. For example I added OnSMTPData awhile back which is handy to act on email after SMTP conversation but before email itself. But can you imagine OnReceiveSMTPLine or something like that? On imap move wouldn't be that bad but could be awful. The closest that exists might me onconnect which could get ugly. Put some code in there & connection 10,000 times I tell me how your load is then get back with me. :D
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2011111901

Posted: 2014-05-31 02:28
by mattg
Bill48105 wrote:Latest list of experimental INI settings.

Code: Select all

hMailServer/bin/hMailServer.INI
[Settings]

;SPFPolicyOverride=v=spf1 -all
; Set SPF policy to override sender's policy such as block +all
; Default is do not override sender's policy
Bill, I'm not sure that I understand this one...

What do I need to do to override all +all spf records?

I currently have this

Code: Select all

SPFDefaultPolicy=v=spf1 MX A -all
; Set default SPF policy if sender does not have SPF
; Default is no default policy so don't use SPF

;SPFDefaultPolicyA=v=spf1 MX A ~all
; Set default SPF policy if sender does not have SPF but has A record
; Default is no default policy so don't use SPF

SPFPolicyOverride=v=spf1 -all
; Set SPF policy to override sender's policy such as block +all
; Default is do not override sender's policy
I want to apply a policy of 'v=spf1 MX A -all' to all that either have no spf, or those that have a '+all' at the end,
and replace '~all' with '-all' to make soft spf records into hard spf records

ie I want to enforce hard valid spf records for all domains

I expect that I can't currently make soft spf records into hard spf records, but replacing the +all ones is the one that I'm not sure that I have correct.

Matt

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-31 08:43
by Bill48105
mattg are you using a recent experimental build? these haven't made it into any official builds yet
This should work:

Code: Select all

SPFDefaultPolicy=v=spf1 -all
SPFDefaultPolicyA=v=spf1 -all
SPFPolicyOverride=v=spf1 -all
IOW if SPF doesn't exist use -all. (Although I'd highly suggest A MX too) If no A record use -all. (Same here MX A too) The Override one might be tricky as I forget it it replaces just the all with -all or if it replaces entire record. I think it just replaces +all & ~all with -all but i'd have to look back in the posts or test it. Btw LogLevel=99 should show you crazy verbose logging if i recall.
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-31 09:06
by mattg
Bill48105 wrote:mattg are you using a recent experimental build?
Yep, and the default v=spf1 MX A -all works well

Anyone without a spf gets that one added, and then scores a spam score if it fails (most mail actually pass my default)

I'm just not sure about the +all ones, I haven't seen where I get one, perhaps I'll have to set that for one of my domains to test

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-05-31 18:24
by Bill48105
mattg wrote:
Bill48105 wrote:mattg are you using a recent experimental build?
Yep, and the default v=spf1 MX A -all works well

Anyone without a spf gets that one added, and then scores a spam score if it fails (most mail actually pass my default)

I'm just not sure about the +all ones, I haven't seen where I get one, perhaps I'll have to set that for one of my domains to test
I looked at the code & default & defaultA are different than override. the defaults look if spf exists & if not use EXACTLY what you set in the INI. That's why A MX are generally good/safe. It'd be tough to do ip's or includes etc since need to be generic. For override whatever you put in the ini is passed to the SPF function from the SPF library. My understanding from reading the comments in that code is that whatever values you state replace the ones in their SPF. So -all should replace ~all or +all. If i recall I tested it & it worked as expected but spf is a tough thing to test without real-world use. (I use ASSP so I don't gave hmail doing SPF myself). Do you suspect it's not working as expected?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-06-01 00:56
by mattg
No I suspect that it is working as I intended, but I've got not way to tell without a lot of research
Perhaps I'll try some very verbose logging for a bit (or just leave it alone..)

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014050401

Posted: 2014-06-01 03:53
by Bill48105
mattg wrote:No I suspect that it is working as I intended, but I've got not way to tell without a lot of research
Perhaps I'll try some very verbose logging for a bit (or just leave it alone..)
Yeah it was a &^#% to test. I ended up using one of my unused idle domains. That way I could do various SPF DNS records & various INI settings to see how they worked. I'm quite certain they work as I expected but again please test & confirm then let me know :D
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-06-06 01:04
by Bill48105
Yes yet another OpenSSL vulnerability so I posted up a new build

2014-06-05 5.4-B2014060501
* IMPORTANT: This build has a LOT of extra debug logging but NOT shown by default. [Settings]LogLevel=10 for some extra to 100 for extremely verbose
* URGENT: Critical OpenSSL MitM vulnerability http://www.pcworld.com/article/2360560/ ... pying.html
* Upated hmailserver to openssl-1.0.1h
* FIX: Added new 250 Help as always last EHLO response to fix 250-STARTTLS gmail issue (Last in list MUST be space not dash)

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-06-24 12:19
by prisma
Possibly a stupid question: Why is RETURN-PATH used?

Code: Select all

[Settings]
AutoReplyReturnPath=noreply@my-internal-hmail-domain.net
Has it to do with anti-spam actions? Return-Path is not used by some email clients for answering. This leads to answers to FROM instead of answers to RETURN-PATH. What was the aim of introducing this setting?

Would it make sense to use instead or additionally:

Code: Select all

AutoReplyReplyTo=noreply@my-internal-hmail-domain.net
??

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-06-24 14:10
by prisma
Feature Request:
Remember already sent auto-reply recipients after restart.

Reason:
We use your experimental builds in semi-productive environment. That's our way to proof stability. Because of frequent SSL-testing and SSL-bugfixing we often had to restart the server lately and will have to do this in future...

Possible Solution:
It was perfect (and possibly would find it's way to a release) if you'd use an dedicated table to store you're structure.
An ugly fix with a dump to a file on graceful shut down and reload on start would help too...

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-06-29 19:26
by japi
Hi Bill,

I have tested the latest versions STARTTLS functionality and it works great! :D
It did work with every email provider I tested. I tested mostly German and international providers.
STARTTLS worked with incoming mails from: web.de, aim.com, gmx.de, gmail.com, hotmail.com, yahoo.com, t-online.de and posteo.de.
At the same time I was pleasantly surprised, how widespread usage of STARTTLS already is. Every tested provider already supported it.

One thing I noticed though:
Every time the socket is closed in STARTTLS mode there is a "read operation failed" message:

Example1:

Code: Select all

"DEBUG"	376	"2014-06-29 18:34:17.022"	"Closing TCP/IP socket"
"DEBUG"	3468	"2014-06-29 18:34:17.022"	"The read operation failed. Bytes transferred: 0 Remote IP: , Session: 535, Code: 995, Message: The I/O operation has been aborted because of either a thread exit or an application request"
"DEBUG"	376	"2014-06-29 18:34:17.022"	"Ending session 535"
Example2:

Code: Select all

"DEBUG"	2368	"2014-06-29 18:35:30.273"	"Closing TCP/IP socket"
"DEBUG"	1140	"2014-06-29 18:35:30.273"	"The read operation failed. Bytes transferred: 0 Remote IP: , Session: 536, Code: 995, Message: The I/O operation has been aborted because of either a thread exit or an application request"
"DEBUG"	1140	"2014-06-29 18:35:30.274"	"Ending session 536"
In non-SSL mode the error does not occur:

Code: Select all

"DEBUG"	1540	"2014-06-29 18:46:13.793"	"Closing TCP/IP socket"
"DEBUG"	1540	"2014-06-29 18:46:13.793"	"Ending session 537"
But it seems as if this error has no further side-effects.

Thank you for your hard work, can't wait to see outgoing STARTTLS working! :mrgreen:

If I can support your work in any way, I would be happy to help!

Best regards,
Jan

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-01 18:24
by Bill48105
prisma wrote:Possibly a stupid question: Why is RETURN-PATH used?

Code: Select all

[Settings]
AutoReplyReturnPath=noreply@my-internal-hmail-domain.net
Has it to do with anti-spam actions? Return-Path is not used by some email clients for answering. This leads to answers to FROM instead of answers to RETURN-PATH. What was the aim of introducing this setting?

Would it make sense to use instead or additionally:

Code: Select all

AutoReplyReplyTo=noreply@my-internal-hmail-domain.net
??
All I was doing is allow customization of what what being set by the already existing code. Return-Path header was being set upon generation of the auto response so I added key so you have option to change it vs it being set to the original recipient. In terms of why it's there, it's supposed to be there as message is handled by mail server. normally hmail sets it as messages pass thru but in this case hmail is generating it & won't be handled by SMTP so I figure martin wanted to take care of it at that point vs risk it not existing.

Is autoreplyto a standard header?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-01 18:28
by Bill48105
prisma wrote:Feature Request:
Remember already sent auto-reply recipients after restart.

Reason:
We use your experimental builds in semi-productive environment. That's our way to proof stability. Because of frequent SSL-testing and SSL-bugfixing we often had to restart the server lately and will have to do this in future...

Possible Solution:
It was perfect (and possibly would find it's way to a release) if you'd use an dedicated table to store you're structure.
An ugly fix with a dump to a file on graceful shut down and reload on start would help too...
Addresses that auto-replies have been sent to are stored in an array/vector so are volatile. I had talked to martin about making them non-volatile but there were concerns both with performance & confusion since people are told to restart hmail to clear the list. yeah ideally database table would just be used but at least dumps & imports of the array for a partial solution. I myself am annoyed by the fact the list is lost but I don't restart too often.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-01 18:33
by Bill48105
japi wrote:Hi Bill,

I have tested the latest versions STARTTLS functionality and it works great! :D
It did work with every email provider I tested. I tested mostly German and international providers.
STARTTLS worked with incoming mails from: web.de, aim.com, gmx.de, gmail.com, hotmail.com, yahoo.com, t-online.de and posteo.de.
At the same time I was pleasantly surprised, how widespread usage of STARTTLS already is. Every tested provider already supported it.

One thing I noticed though:
Every time the socket is closed in STARTTLS mode there is a "read operation failed" message:

Example1:

Code: Select all

"DEBUG"	376	"2014-06-29 18:34:17.022"	"Closing TCP/IP socket"
"DEBUG"	3468	"2014-06-29 18:34:17.022"	"The read operation failed. Bytes transferred: 0 Remote IP: , Session: 535, Code: 995, Message: The I/O operation has been aborted because of either a thread exit or an application request"
"DEBUG"	376	"2014-06-29 18:34:17.022"	"Ending session 535"
Example2:

Code: Select all

"DEBUG"	2368	"2014-06-29 18:35:30.273"	"Closing TCP/IP socket"
"DEBUG"	1140	"2014-06-29 18:35:30.273"	"The read operation failed. Bytes transferred: 0 Remote IP: , Session: 536, Code: 995, Message: The I/O operation has been aborted because of either a thread exit or an application request"
"DEBUG"	1140	"2014-06-29 18:35:30.274"	"Ending session 536"
In non-SSL mode the error does not occur:

Code: Select all

"DEBUG"	1540	"2014-06-29 18:46:13.793"	"Closing TCP/IP socket"
"DEBUG"	1540	"2014-06-29 18:46:13.793"	"Ending session 537"
But it seems as if this error has no further side-effects.

Thank you for your hard work, can't wait to see outgoing STARTTLS working! :mrgreen:

If I can support your work in any way, I would be happy to help!

Best regards,
Jan
Hey Jan. Ok great glad to hear it. Yeah STARTTLS was never completed & what you have is really just "something" to get started & luckily it works well enough to be useful. At the same time since it works so well there has been little interest in working out the last bits not only due to lack of time but also fear of breaking something. I'm not sure what the cause of that error is but doesn't seem to hurt anything. Odds are hmail thinks something else should be sent by the remote end that isn't or maybe the socket is supposed to be switched back to non-TLS for that read.

Btw I did post up a highly experimental build for testing outgoing SMTP STARTTLS if you have any interest in helping figure out what is wrong with it. It does not work so be warned of that but it SHOULD which is the problem. I've been unable to get back handshake errors and gave up for now. It could be something very trivial & maybe can get around it by forcing crypto options but i've not had time to mess with it further.
Thx
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-07 13:31
by prisma
Bill48105 wrote: Addresses that auto-replies have been sent to are stored in an array/vector so are volatile. I had talked to martin about making them non-volatile but there were concerns both with performance & confusion since people are told to restart hmail to clear the list. yeah ideally database table would just be used but at least dumps & imports of the array for a partial solution. I myself am annoyed by the fact the list is lost but I don't restart too often.
If Martin has concerns, I would really really appreciate the dumb "dump" solution and an INI switch :) If it's easy to do. If not, there are other more important things to do...

Was somebody able to bring al little bit light into your outgoing STARTTLS problems? I must confess, I did not have any time to take care about it.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-07 22:51
by Bill48105
prisma wrote:
Bill48105 wrote: Addresses that auto-replies have been sent to are stored in an array/vector so are volatile. I had talked to martin about making them non-volatile but there were concerns both with performance & confusion since people are told to restart hmail to clear the list. yeah ideally database table would just be used but at least dumps & imports of the array for a partial solution. I myself am annoyed by the fact the list is lost but I don't restart too often.
If Martin has concerns, I would really really appreciate the dumb "dump" solution and an INI switch :) If it's easy to do. If not, there are other more important things to do...

Was somebody able to bring al little bit light into your outgoing STARTTLS problems? I must confess, I did not have any time to take care about it.
I don't recall the specifics it's been awhile. I just know it was done as vector instead of db from the start so I figure a reason.

Actually yes in another post it was pointed out that the handshake might be out of sync due to hmail not waiting for the ok to handshake but not sure when i'll have time to look at it.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-09 22:12
by martin
Bill, anyway to share your "fork" of hMailServer with me so I can take a look at the STARTTLS implementation and integrate it into the main release? I understand it's not "done", but hopefully it can work as a startingpoint?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-10 15:05
by mattg
Don't forget the ICOP error fix...

That is the bigger fix for me.
Essentially, it was shown that the main build has an intermittent bug>> http://www.hmailserver.com/forum/viewto ... 04#p155304

There are some other nicities in Bill's builds. SPF defaults is great, as is the quick retries, and other INI settings

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-11 17:53
by Bill48105
martin wrote:Bill, anyway to share your "fork" of hMailServer with me so I can take a look at the STARTTLS implementation and integrate it into the main release? I understand it's not "done", but hopefully it can work as a startingpoint?
mattg wrote:Don't forget the ICOP error fix...

That is the bigger fix for me.
Essentially, it was shown that the main build has an intermittent bug>> http://www.hmailserver.com/forum/viewto ... 04#p155304

There are some other nicities in Bill's builds. SPF defaults is great, as is the quick retries, and other INI settings
I replied to your PM martin & yeah I'll try to get something to you. Life has been a bit hectic here with 5 year old special needs son with seizures & a new born baby who was premature & needed a lot of attention. 2014 has been hectic to say the least.

Indeed mattg there are MANY things besides STARTTLS that many people rely on & should be in official releases. The IOCP race fixes for sure. They not only seem to solve the crashes but seem to make hmail a lot faster too. (Btw greylock's find was just one piece the bigger one was found & fixed later on during debugging of the STARTTLS code). QuickRetries stuff has been in official hmail for awhile (unless it was removed which I doubt) but yeah the SPF tweaks among other things. Check out the experimental change logs to get an idea way too many to list here.

So yeah I'll try to get time soon. I think I could get it done in a few hours but the trick is getting enough free time without distractions.
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-14 22:18
by martin
Bill48105, can only imagine that that may be time consuming...

There's really no need to clean up the code or anything like that. I'm certain I'll be able to locate the relevant parts without a cleanup. Don't really care what the code looks like, just what it does. :) I won't pick any code as-is but will review every part before I included it anyway.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-23 08:50
by brashquido
First off, thanks a ton for the custom builds Bill et al. You efforts are appreciated.

Have a few queries around SMTP AUTH.
2014-05-04 5.4-B2014050401
* IMPORTANT: This build has a LOT of extra debug logging but NOT shown by default. [Settings]LogLevel=10 for some extra to 100 for extremely verbose
* FIX: EHLO response now "250 AUTH" when DisableAUTHList is set since gmail refused to send if no AUTH in EHLO response
Have set DisableAUTHList" to on for port 25, however this seems to be causing issues with outgoing email across a variety of email clients. Anyone else finding this?

Second question. When disabling plain text authentication in HMS it leaves AUTH LOGIN as the only authentication method. As far as I understand both AUTH PLAIN & LOGIN are just Base64 encoded, no encryption. Just wanting to confirm if this is the case in HMS?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-23 15:00
by brashquido
Sorry, meant to mention that I am not "250 AUTH" from a EHLO response when DisableAUTHList is enabled. Have verified that I am using build 5.4-B2014060501 which I had the impression should have this fix?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-23 16:11
by SorenR
brashquido wrote:Sorry, meant to mention that I am not "250 AUTH" from a EHLO response when DisableAUTHList is enabled. Have verified that I am using build 5.4-B2014060501 which I had the impression should have this fix?
2014-06-05 5.4-B2014060501 and later should be the one.

When you use this feature, you DISABLE login on the specified port(s). IF you specify DisableAUTHList=25 you MUST provide an alternative SMTP port for your clients to send on. A suggestion could be port 587 (aka - submission).

The option was implemented to only allow server <-> server communication on the specified port.

I believe primarely initiated by requests to help stop bots from brute-force atacking the server.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-07-24 05:26
by brashquido
Thanks for clearing that up Soren, makes sense. As to the lack of a "250 AUTH" response from the EHLO command, anyone else seeing this? Have since tried replacing the experimental build binaries, same thing.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-08-06 00:22
by Bill48105
brashquido wrote:Thanks for clearing that up Soren, makes sense. As to the lack of a "250 AUTH" response from the EHLO command, anyone else seeing this? Have since tried replacing the experimental build binaries, same thing.
The most recent experimental build should only not show the 250 AUTH if you have disabled auth for port 25 in the INI. I've been using it live on a few servers since it was posted & have not seen any other behavior. Are you sure you are on the newest experimental & don't have the INI setting?

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-08-28 21:47
by jdurkacs
Hello,

Our production mail server regularly gets the IOCP related freezes where the mail server stops responding to connections. In April I saw mention of fixes for this and updated to the latest, hoping that this issue would be resolved. Now I realize that these fixes were put in Bill's experiment builds and not the production release. I also see that there are later fixes, what Bill refers to as "IOCP race fixes".

I am wondering if the full list of IOCP related fixes have now finally made it into the production release 5.4.2 Build 1964 or even the latest Beta 5.5 Build 2058? I reviewed all the latest changelogs but I don't see a clear answer on this. I really need to fix our server soon or move to a different product, so if anyone can answer that, I would really appreciate it. Thank you!

Jody

PS - for clarity's sake, this is the error we get when hmailserver stops working, so I presume it's the same thing as the IOCP race condition:
"ERROR" 2772 "2014-08-26 00:01:12.430" "Severity: 2 (High), Code: HM4208, Source: IOCPQueueWorkerTask::DoWork, Description: An unknown error occured while handling asynchronous requests."

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-08-28 22:28
by percepts
The fix was only in Bills experimental build. It is definitely NOT in 5.4.2.

Whether it has made it into 5.5 beta I don't know. But Martin has said he has Bills experimental code so whether he's actually put changes into 5.5 is unknown until Martin says so.

So in the short term I would use Bills last experimetal build which you can simply put on top of 5.4.2. And then when/if Martin pipes in with info on IOCP errors/race condition you can upgrade to 5.5 beta or production if its gone live.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-06 20:36
by Bill48105
percepts wrote:The fix was only in Bills experimental build. It is definitely NOT in 5.4.2.

Whether it has made it into 5.5 beta I don't know. But Martin has said he has Bills experimental code so whether he's actually put changes into 5.5 is unknown until Martin says so.

So in the short term I would use Bills last experimetal build which you can simply put on top of 5.4.2. And then when/if Martin pipes in with info on IOCP errors/race condition you can upgrade to 5.5 beta or production if its gone live.
I posted my code to github but have not heard from martin on if he's looked at it let alone made use of my changes or if he plans to. I've sent him multiple messages but guess he's too busy to respond.

I told him many months ago what changes I made & why so he had the info on what to change even without my code so not sure why it hasn't been done yet considering we had been chasing the IOCP crashes for years.

As far as I am aware no one has experienced IOCP crashes with my recent builds so I assume it solves the problem but without martin looking at the changes & giving his input I can't say for sure if it really fixes them. I will say there have been numerous reports that I have seen myself that my builds seem MUCH faster than the official builds ever since the IOCP fixes were put in. I think the reason is for those who were not having IOCP CRASHES such myself the race conditions that cause the crashes for some were causing thrashing without actual crashes for others making hmail very slow in some circumstances especially on a busy server.
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-07 09:55
by martin
I posted my code to github but have not heard from martin on if he's looked at it let alone made use of my changes or if he plans to. I've sent him multiple messages but guess he's too busy to respond.
That's not entirely correct. I posted in the other thread here my stance on the IOCP issue and your other changes in this build.

As I pointed out in that thread, I don't include changes unless they are covered by documentation, available in the hMailServer Administrator and WebAdmin and covered by automatic tests. I have nowhere near the time to manually test *all* features and settings in hMailServer before every build, which would be required if there are zero tests. This is why automated tests are vital to me. Testing all features on all supported versions of Windows and on the supported databases would take days. Hence, things which are only in ini-files and not documented are not 'officially supported' and may break, since no-one is actually making sure they work every time on all supported versions of Windows and with all database types. In other words, I don't consider a feature really implemented until it's covered by doc,test,ui and I don't want to include a bunch of half-implemented features.

As for performance, it would be interesting to see some actual benchmarks made on this, using some bench marking software. Without that I'm not sure how you could claim that it's much faster now, since that could be extremely simple to misjudge. It sounds a bit strange that switching from multi-threading to single-threading will make things faster since most of the work hMailServer does is IO-bound, unless you've configured hMailServer to use like >100 threads.

Everyone which are happy with this experimental build can of course just continue to use it. If you want specific parts of it to be ported to hMailServer, my suggestion is to post that as an enhancement request to the Github tracker and I'll have a look at it.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-07 20:25
by Bill48105
martin wrote:
I posted my code to github but have not heard from martin on if he's looked at it let alone made use of my changes or if he plans to. I've sent him multiple messages but guess he's too busy to respond.
That's not entirely correct. I posted in the other thread here my stance on the IOCP issue and your other changes in this build.

As I pointed out in that thread, I don't include changes unless they are covered by documentation, available in the hMailServer Administrator and WebAdmin and covered by automatic tests. I have nowhere near the time to manually test *all* features and settings in hMailServer before every build, which would be required if there are zero tests. This is why automated tests are vital to me. Testing all features on all supported versions of Windows and on the supported databases would take days. Hence, things which are only in ini-files and not documented are not 'officially supported' and may break, since no-one is actually making sure they work every time on all supported versions of Windows and with all database types. In other words, I don't consider a feature really implemented until it's covered by doc,test,ui and I don't want to include a bunch of half-implemented features.

As for performance, it would be interesting to see some actual benchmarks made on this, using some bench marking software. Without that I'm not sure how you could claim that it's much faster now, since that could be extremely simple to misjudge. It sounds a bit strange that switching from multi-threading to single-threading will make things faster since most of the work hMailServer does is IO-bound, unless you've configured hMailServer to use like >100 threads.

Everyone which are happy with this experimental build can of course just continue to use it. If you want specific parts of it to be ported to hMailServer, my suggestion is to post that as an enhancement request to the Github tracker and I'll have a look at it.
glad to see your time is more valuable than ours, i'll get right on that. nice for you to refer me to some post I'd have little chance of seeing rather than respond to my numerous forum PM's, skype messages or emails regarding all this.

as far as docs, testing, admins, etc, that's been gone over numerous times & thought we had an agreement. NOTHING was happening in hmail in FOREVER. The feature requests were backed up with some waiting for YEARS. So JamesDR/Rolaids & I along with a TON of help & testing from others spend a lot of time on it. Believe it or not while you were busy with your non-hmail life the rest of us needed shit fixed & features added. Go back & read the last 3+ years of forum posts to catch up. It's cool you are back & have time to work on hmail again but not cool to act like you come back as if time didn't pass in your absence.

I get why the automated tests are important & understand the docs need updated (heck they are mostly 5.3 or earlier as it is) and would love features to be in the admin but there is only so much time and we did the best we could getting SOMETHING done and most people appreciated that. It was assumed you would help finish what was started instead of acting like what we did was worthless. As it stands in your absence many very helpful people in the hmail community have moved on or rarely around. Tired of waiting for your triumphant return to help finish things.

As it stands I have very little time these days so hmail is on back burner for me now but I'll have zero time to merge my changes with the official code or write up docs beyond the forum posts or help with writing automated tests. But i will try to be available in IRC or by skype if someone decides to work on merging the code I posted with the official in terms of understanding what was done or why etc. Until then my focus is on family & work unless something urgent comes up with the experimental build as it works great for my needs ATM.
Peace out
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-07 20:45
by martin
Oh come on Bill. You're placing words in my mouth which I have not spoken. Lots of the things you write that I'm saying/thinking I have not said and I have not thought, such as your changes being worthless or my time more worth than yours. Obviously this is not true and it's sad you feel that way. Not sure why you are getting those ideas.

People have already asked that a specific part of your code base should be merged to hMailServer source (changes to the Received header). That specific part lacked test and seemed a bit messy so I re-implemented that. If people feel that stuff in your port is required they are free to place a request on Github and I'll look into merging that - So I really don't see what the problem is. But again, for one thing to be integrated it has to have test coverage, exposed in the API, shown in the .NET/PHP UI and covered in documentation. This is a lot more work than adding an inifile and the code to the server. Of course the core code is a very good starting point, it's like 50% of the actual work.

Also, to be fair, I asked several times of you to provide your source code but that took quite some time as well for you (you wanted to cleanup etc). If you had shared it earlier, we would know the above earlier.

I feel that one of the benefits with hMailServer over setting up some Linux machine with a mail server is the ease-of-configuration. If you force users to edit text files without any kind of validation I feel that this is a big step backward. Considering what you write here, it seems like we disagree on this which is fine.

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-08 07:02
by Bill48105
martin wrote:Oh come on Bill. You're placing words in my mouth which I have not spoken. Lots of the things you write that I'm saying/thinking I have not said and I have not thought, such as your changes being worthless or my time more worth than yours. Obviously this is not true and it's sad you feel that way. Not sure why you are getting those ideas.

People have already asked that a specific part of your code base should be merged to hMailServer source (changes to the Received header). That specific part lacked test and seemed a bit messy so I re-implemented that. If people feel that stuff in your port is required they are free to place a request on Github and I'll look into merging that - So I really don't see what the problem is. But again, for one thing to be integrated it has to have test coverage, exposed in the API, shown in the .NET/PHP UI and covered in documentation. This is a lot more work than adding an inifile and the code to the server. Of course the core code is a very good starting point, it's like 50% of the actual work.

Also, to be fair, I asked several times of you to provide your source code but that took quite some time as well for you (you wanted to cleanup etc). If you had shared it earlier, we would know the above earlier.

I feel that one of the benefits with hMailServer over setting up some Linux machine with a mail server is the ease-of-configuration. If you force users to edit text files without any kind of validation I feel that this is a big step backward. Considering what you write here, it seems like we disagree on this which is fine.
It took some time because as I explained multiple times I was a bit busy. I was in the hospital with my daughter who was in the NICU because she was born a month early while dealing with my son's seizures & post brain surgery Dr appointments & therapy while trying to work to pay bills & deal with other family matters & personal obligations going on. I told you there'd be no time for me to work on it for a couple months which didn't seem like it'd be a big deal since you were barely around for the past 3 years. What the hell is a couple months waiting on me? Then you moved the code to github before I could get my changes merged on SVN repo then you made too many changes on github to easily merge them at that point. Yes you told me you planned to do the move but that notice was no good when given to me during a period where I had no time to work on it. To make matters worse I had no real way to cleanup my code without the SVN repo being up. Eventually it was suggested I try tortoise for githib which luckily worked thanks to my still having cached SVN status locally to see which files had been modified. It was a painstaking process over multiple days when I had virtually no sleep to get it done to have you tell me you had moved on.

Btw we discussed it numerous times & it has been mentioned MANY MANY times in the forums that INI's were not ideal but they were meant to be TEMPORARY to avoid database changes between builds plus make the experimental builds easy to drop-in. When you had time you were going to move the ones that would be commonly useful to database as a group, deprecate the INI settings & add the config to the GUI. The only reason why something would stay in the INI is if it was an obscure setting that'd be advanced or rarely used. It was never the plan to leave all those INI settings.

Anyway, I've posted my source as promised it just took longer than you wanted. It's a fork to the official code so use it how you will, or don't. Hopefully you choose to implement the changes people rely on as most were added to fulfill a need with the assumption they'd be put into official code as some point.

That being said, I've been away from hmail most of 2014 except for urgent matters & will continue to have other obligations into the foreseeable future so unlikely I'll be around although I can be found if needed.
Bill

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-09-08 17:15
by martin
I'm not blaming you for anything here, so you don't have to defend yourself.

I hope your son gets better soon. :-\

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-11-24 03:10
by ^DooM^
So are you actually back now Martin? we all appreciated your extended absence greatly... Bill, James, Mattg and Percepts have all done a great job picking up the slack in your absence. You should reward them with cookies or something.

hMail lives on yay \o/

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2014-11-26 18:56
by martin
I'm all out of cookies :(

Re: LATEST EXPERIMENTAL BUILD - 5.4-B2014060501

Posted: 2015-01-26 23:10
by Bill48105
Mmm cookies. FYI I'm not dead, just busy with other things like family & work ATM.
Bill