Realtime hMailServer Log viewer

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Realtime hMailServer Log viewer

Post by Hazark » 2014-05-08 23:47

Hi,

I was using the BareTail (*nix tail -f for Windows) for my casual log watching needs but due to multithreaded nature of hMailServer; log lines usually interleaved to another events.

For that reason I wrote a VERY basic log viewer. It will work on your browser, can directly use log file on filesystem, continuously (every 500ms) check for changes and reflect any changes on screen while its on.

Its primary reason to exist is being a offline realtime grouped SMTP session viewer. At the moment there are pretty much no other feature exist.

You can download/contribute/fork on https://github.com/hazarkarabay/hmailserver-logviewer or if you have a log file on your hands right now, go to http://jsfiddle.net/P2bu9/ and use it directly there.

For the getting most out it, use it on a "live" log file.


-English is not my primary language, so please excuse for my English. Thanks

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Realtime hMailServer Log viewer

Post by Bill48105 » 2014-05-18 08:30

Hey Hazark,
I checked it out & it's quite cool. Thanks for sharing.
Of course I had to dig in & see how it worked. :D I changed the smtpd smtpc case to default so I could use it for POP, IMAP & "networking" log files too. (I split my logs instead of them all being in 1 GIANT file. I also changed it to 100K instead of 10K to see more sessions.

You might be interested in some of the advanced logging changed I've made to hmailserver
http://www.hmailserver.com/forum/viewto ... 38#p132238
In particular:

Code: Select all

;SepSvcLogs=1
;Split logs so 1 per sevice (pop, smtp, imap)
; Default is to have all services logged together in 1 file.

;LogLevel=9
;1=lowest/least 9=highest/most
;NOTE: Not really functional at this point except a few specific places

;MaxLogLineLen=300
; This key causes log lines to be neatly truncated at specified # of characters to reduce log sizes.
; Example: "POP3D"   3952   2   "2010-10-24 15:18:59.924"   "127.0.0.1"   "SENT: 1 756[nl]2 ... 350[nl]1973 371[nl]."
; This defaults to disabled if key doesn't exist or LogLevel > 1 or if debug logging is enabled
If you are only concerned with SMTP setting SepSvcLogs=1 then restarting hmail should help you out a lot since you can monitor JUST SMTP log file without the script having to wade thru all the other logs.

Seems i never put description for MaxLogLineLen but that one is SUPER handy for out of control POP & IMAP logging. For example a POP UIDL or LIST might be 1000's of entries long causing the log line to wrap If you set it to something like 800 then I have hmail nicely shorten those crazy lines. I forget now how many exactly but it essentially truncates at MaxLogLineLen - 25 then tacks on the last 25 chars with " ... " in the removed part.

Code: Select all

Example: "POP3D"   3952   2   "2010-10-24 15:18:59.924"   "127.0.0.1"   "SENT: 1 756[nl]2 ... 350[nl]1973 371[nl]."
See how it took what would have been 1973 which would have been crazy lazy & condensed to a more reasonable log line. Granted you lose out on the middle info but generally speaking the other 1970 UID's are not going to be too useful & you can always adjust if debugging.

Btw:

Code: Select all

/* This is some internal ID or code I don't know yet, so it is hidden */
You must be talking about the ThreadID column. In the logs hmail tells which thread handled the task. One peculiar thing about how hmail's multi-threading works is that a session might be handled by multiple threads which can make for real fun following in logs.

What you've done here with this script is along the same lines as we've talked about for the admin GUI. As a matter of fact it might be it should be useful to actually doing it in the admin. :)
Thx
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Re: Realtime hMailServer Log viewer

Post by Hazark » 2014-05-18 10:40

Thanks for the detailed reply Bill.

You are right, my primary interest is SMTP logs. At this moment I'm using an outdated official build of Hmail so splitted logs is not available on my install.
My server generates only ~15MB of SMTP logs daily, not a very busy server so still manageable. (except the one very day when I'm started to take over the server from old owners, ~400MB. Sadly its caused by a brute force account hijack and you know the rest. Tens of thousands of spam, explaining customers to why their emails suddenly started rejecting at remote host, lost hairs. It is the primary reason that why I'm so obsessed with SMTP logs :| )

Anyway, I will definitely try the forum builds when I upgrade my hardware (which involves an OS upgrade and full rebuild).

I guessed that column must be thread ID but I was found it changes also middle of session, so I decide to not make assumptions and confess my illiteracy in the code comments :) .
Your comment about Thread ID explains it all.

Thanks again.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Realtime hMailServer Log viewer

Post by Bill48105 » 2014-05-19 02:32

Oh you are lucky my logs can be 1GB a day. :)

Wow how old of hmail are you on? 5.4 has had that logging stuff since the 2nd alpha. lol

Btw 5.4 should use no more resources than 5.3.x & unlikely any more than 4.x. Actually based on my testing & reports it seems newer 5.4 are actually lighter & faster than 5.3.x by a large margin in some cases. Plus all the new tweaks added like quickindex & queue retry stuff makes it much much more efficient.

Yeah the thread stuff can seem screwy. Sometimes the same thread handles a session start to finish & sometimes many threads handle.

Thx again it's a cool script
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Re: Realtime hMailServer Log viewer

Post by Hazark » 2014-05-19 10:12

I'm using 5.3 branch, cant RDP right now to see exact version.

The real problem is not hardware, currently it is running on Windows 2000. Blame the previous owners :shock:

I'm getting some 2012 R2 and since I already have a bunch of Linux boxes; going to beef up the hardware & virtualize it and while its going rebuild the Windows environment, finally this time "correct" way.

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Realtime hMailServer Log viewer

Post by Bill48105 » 2014-05-19 14:36

Hazark wrote:I'm using 5.3 branch, cant RDP right now to see exact version.

The real problem is not hardware, currently it is running on Windows 2000. Blame the previous owners :shock:

I'm getting some 2012 R2 and since I already have a bunch of Linux boxes; going to beef up the hardware & virtualize it and while its going rebuild the Windows environment, finally this time "correct" way.
Ahh gotcha. Well last I knew 2k wasn't mainly depreciated due to how old it was. The experimental QueueRandomness setting is not compatible and there might be an issue with QuickIndex but i'd have to look back. (We don't see much 2k these days) Otherwise 2k should be ok. BUt yeah if you got something in the works no sense messing with it & just wait unless you had urgent specific reason.

Btw even the latest official releases have known race conditions that can cause crashes especially on VM's (look at changelogs since Jan) so I'd get B1951 on then use one of the experimentals unless my changes are in the officials by then.
Bill
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Re: Realtime hMailServer Log viewer

Post by Hazark » 2014-07-27 23:55

I have got some time on my hands and improved this a bit.

Details are in my commit log, but notable things are:
* ~%200 performance improvement
* Can snip long lines
* Base64 decoding on AUTH login usernames (idea borrowed from Doom's script)
* Country flags for IP addresses

https://github.com/hazarkarabay/hmailserver-logviewer

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Realtime hMailServer Log viewer

Post by ^DooM^ » 2014-07-29 23:49

Nice work Hazark, Will certainly keep an eye on this one
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Realtime hMailServer Log viewer

Post by ^DooM^ » 2014-07-30 00:11

Some things i would like to see expanded on if you were of a mind to continue to develop it.

Have you select the log dir instead of a file and have the script read the latest one in there.

Colour code suspicious lines so it's easy to spot them.
(I can send you my parser if you want, may save you time on looking up all the errors that i have added to mine)

Have the ability to mute common lines that are pointless and just take up real estate, especially on a busy server.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

Hazark
New user
New user
Posts: 26
Joined: 2014-05-08 23:17

Re: Realtime hMailServer Log viewer

Post by Hazark » 2014-07-30 02:25

^DooM^ wrote:Some things i would like to see expanded on if you were of a mind to continue to develop it.
My intention is continue to develop it, but that will happen probably slowly.
^DooM^ wrote:Have you select the log dir instead of a file and have the script read the latest one in there.
Actually I want to do this at the beginning but unfortunately it is impossible. One can't monitor a directory with HTML5 File API hence the name. (It is possible with Local Storage API, but it will not solve anything because it works in a sandboxed environment and can't access to "real" filesystem.)
^DooM^ wrote:Colour code suspicious lines so it's easy to spot them.
(I can send you my parser if you want, may save you time on looking up all the errors that i have added to mine)
Nice idea, should be easy to implement if I have a lookup table. If you can share your code that would be great. You can send an email at hazar@[my github username].com.tr
^DooM^ wrote:Have the ability to mute common lines that are pointless and just take up real estate, especially on a busy server.
Probably this will need a "line type" knowledge for every event type. For SMTP I can use SMTP reply codes (codes bigger than 400 may be important), but I'm not sure about other events like IMAP, POP3 etc.


Thanks for feedback!

^DooM^
Site Admin
Posts: 13862
Joined: 2005-07-29 16:18
Location: UK

Re: Realtime hMailServer Log viewer

Post by ^DooM^ » 2014-07-31 23:08

Not sure how the sandbox works, If it works the way i think I am guessing it will need human interaction before the browser is allowed to access a file? If not then could you not just open todays date file worked out from the pc's date/time? Or at least fill in the name / path for them for them to click ok, or just leave it as is and i will stop being lazy :P

Email sent with code in. I am sure you can dig the stuff you need out of it.

Shout if you need anything else.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

kevin@wine.co.za
New user
New user
Posts: 21
Joined: 2007-04-26 12:19
Location: South Africa
Contact:

Re: Realtime hMailServer Log viewer

Post by kevin@wine.co.za » 2016-11-04 16:22

I realise this is an old thread but..
Bill48105 - do you still have the code changes you made to this great viewer to allow pop and imap
You said : "I changed the smtpd smtpc case to default so I could use it for POP, IMAP..."
I searched for a case statement but can't see where to make these changes
Any help very much appreciated

User avatar
mattg
Moderator
Moderator
Posts: 19810
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Realtime hMailServer Log viewer

Post by mattg » 2016-11-05 01:14

If Bill doesn't answer (he's not about much any more) check for him on the IRC in his signature
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Realtime hMailServer Log viewer

Post by Bill48105 » 2017-02-02 19:55

hey matt long time no see. :) I happened to stumble into the forum today looking for an old post of mine & realize I had pm. Yes I'm always on IRC either the official channel or the unofficial one on freenode. just be patient they are so quite I don't keep focus on them & might take few hours to see someone is alive in there lol
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

User avatar
jimimaseye
Moderator
Moderator
Posts: 7953
Joined: 2011-09-08 17:48

Re: Realtime hMailServer Log viewer

Post by jimimaseye » 2017-02-02 20:00

Bill48105 wrote:hey matt long time no see. :) I happened to stumble into the forum today looking for an old post of mine & realize I had pm. Yes I'm always on IRC either the official channel or the unofficial one on freenode. just be patient they are so quite I don't keep focus on them & might take few hours to see someone is alive in there lol
In fact I did catch you once in the past few years on IRC and actually spoke directly to you. (Shame we dont have the benefit of your input like we used to.)
HMS 5.6.6 B2383 on Win Server 2008 R2 Foundation, + 5.6.7-B2415 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

Bill48105
Developer
Developer
Posts: 6189
Joined: 2010-04-24 23:16
Location: Michigan, USA

Re: Realtime hMailServer Log viewer

Post by Bill48105 » 2017-02-03 07:22

jimimaseye wrote:
Bill48105 wrote:hey matt long time no see. :) I happened to stumble into the forum today looking for an old post of mine & realize I had pm. Yes I'm always on IRC either the official channel or the unofficial one on freenode. just be patient they are so quite I don't keep focus on them & might take few hours to see someone is alive in there lol
In fact I did catch you once in the past few years on IRC and actually spoke directly to you. (Shame we dont have the benefit of your input like we used to.)
Howdy. I'm always in both hmail channels. ALWAYS. I just don't stare at those tabs since there is literally nothing going on day in & day out. Today we had someone in BOTH channels needing help so it was a busy day lol So if you wanna talk gotta stick around awhile. :) I usually check the tabs a few times a day but sometimes it could be a day or 2. If someone says Bill48105 in either channel it will alert me & I'll see it quicker. Not sure how helpful I'll be besides general email knowledge or retro 5.4 & earlier help. I stopped hosting email for people & stopped supporting those who I had helped setup hmail on so I've eliminated all but 1 of my hmail servers & even it is almost just for nostalgia really & could move what's left elsewhere. Catch you in IRC sometime!
hMailServer build LIVE on my servers: 5.4-B2014050402
#hmailserver on FreeNode IRC https://webchat.freenode.net/?channels=#hmailserver
*** ABSENT FROM hMail! Those in IRC know how to find me if urgent. ***

Post Reply