Live Server Stats

sheffters · Post by **sheffters** » 2009-07-25 20:25

Hi,

Not sure where to post, but this seemed the most likely place ...

I'm starting to develop some live stats from the hmail database and thought I'd share.

The idea is a website than quieries the db in real time for stats. I've found ones that log deliveries and historical data (AW Stats being the most obvious), but nothing real-time to show the status of your server, thats just for your mail server.

Anwyay ... I've started to code one. It's in its first primative form ... http://maillive.ianbradshaw.co.uk/ ... I'm going to add some graphs with openflashchart and the like, do some more stats etc and generally clean it up.

I'm thinking a V2 for the processing of logs further down the road when the live stuff is finished.

Basically wondering if anyone is interested and if you are, what would you like adding?

When it's cleaned up I'll create a config file and offer it up for download for everyone to use as they wish if theres any interest. Code is in Perl so you'd need that installed on your server.

Any thoughts anyone?

Cheers

S.

^DooM^ · Post by **^DooM^** » 2009-07-25 22:42

Very nice,

You beat me to this script. I have been thinking about creating something similar for a while, just not had the time

Thats run as a CGI application? What would a user need to install to get this to work?

sheffters · Post by **sheffters** » 2009-07-25 23:12

Thanks

Yep ... it's standard IIS Perl CGI (although it should run as Perl ISAPI I've never had much luck with IIS and ISAPI working properly so it's as CGI on my server).

You'd need a copy of Perl (http://www.activestate.com/activeperl/) - using v5.8. Should work with 5.10 if you can find the modules for it, although no promises!

From there, get it running in IIS (simple helo world script to check) then use PPM (installed with perl) to get the following modules:

use CGI;
use DBI;
use Filesys::DfPortable;
use Time::HiRes qw(time);
use Date::Calc qw(Today_and_Now Add_Delta_DHMS);

There common modules so are available in the standard PPM repos installed with Perl.

Web server would have to be able to talk to your hmail db (obviously!).

When I get the graphs working you'll need to have openflashchart (http://teethgrinder.co.uk/open-flash-chart/) setup - I want to just bundle it in the zip file, but need to check the licence. It's not difficult to setup though.

At the moment I'm working on functionality rather than code tidyness - I'll have to go back and streamline the SQL for people with large hmail installs I think - although trying to keep it simple as simple usually = fast.

Have you any thoughts on what stats you'd like added Doom? Also, if possible, do you fancy testing some versions before I offer them for download - i.e. checking speed on a larger install, config options etc. I've not got a very big hmail install and theres bound to be some funnies that crop up with more data.

Cheers

S.

^DooM^ · Post by **^DooM^** » 2009-07-26 10:36

Thanks for the writeup but I won't be installing Perl on my production system just to test this out. My servers have been online for well over a year and are as solid as I can make them. I can install it on my test box though but that has 1 user and 1 account on the mail server so wouldn't help you.

As for adding stuff, Drop down list of domains that you can select the accounts after that. Someone with thousands of domains and millions of accounts would need a way to limit what they were seeing / searching for.

sheffters · Post by **sheffters** » 2009-07-26 13:02

hi,

Cool, I'll get that added.

Cheers

S.

p.s. wouldn't need to be on a production machine, just needs access to the hmail database, can be on any machine with access to the DB, or a backup of the DB on a different box.

^DooM^ · Post by **^DooM^** » 2009-07-26 14:19

Aye i might try it with a backup because (no offence) i didn't code it so i don't trust it to have access to my live DB

sheffters · Post by **sheffters** » 2009-07-26 15:37

Hi,

None taken - it's the sensible thing to do!

I've done a domain filter ... now ...

"select the accounts after that"

do you mean you'd want to be able to select an individual user specifically?

Cheers

Ian.

^DooM^ · Post by **^DooM^** » 2009-07-26 20:18

The more you could drill down the better an app it would be but whatever you are comfortable with coding, it is your application

sheffters · Post by **sheffters** » 2009-07-27 00:27

Hi,

All good, gives my wee cell something to think about!

Got the search working ... now just to get the results filtered! (JS / AJAX isn't my thing so thought I'd have a go at doing it like that to learn a bit more!)

Cheers

I.

sheffters · Post by **sheffters** » 2009-07-27 22:20

Hi Doom (& and anyone else reading this!)

Got the user thing working and added a quick MX checker on there as well.

Have a look if you get chance and let me know what you think / if you can think of anything else that would be handy to have on there.

I'm going to start packaging it up to read a config file when it's got all the basics covered with some install instructions.

Cheers

S.

^DooM^ · Post by **^DooM^** » 2009-07-28 00:15

Loving the realtime user search however it was a little confusing as all the other results were still on the main page. Perhaps make it a bit more obvious that you are looking at the user you filtered. Other than that, sweet bit of work.

sheffters · Post by **sheffters** » 2009-07-28 00:26

Hi,

Thanks

The only bits where its not filtered by user when it's applied are the MX Check (obviously as thats a domain thing), the server cpu / ram / storage stuff and the Top Users bit.

I'll delete the top users bit as that could get a bit misleading (the mailbox count and size) and replace it with a graph showing that users total use v server and domain - although 1 user in thousands may get a bit lost, have to see how it goes.

When I've got that running I'll get it reading settings from a config file and should be good to go for anyone to use then!

*crosses fingers!*

Cheers

S.

sheffters · Post by **sheffters** » 2009-07-31 00:33

Hi Doom (& and anyone else reading this!)

Got a first release ready, and a basic config file.

I'm going to expand on the config options when I get some time, and also write a detailed user guide (if enough people want this thing).

I've setup a forum for it instead of using Martins as I'm sure he wouldn't be too appreciative of some mental thread (if people actually use this gizmo) or worse asking him q's on something thats not hmailserver.

http://forum.ianbradshaw.co.uk

Hope its some use to someone!

Cheers

Sheff

^DooM^ · Post by **^DooM^** » 2009-08-13 02:00

Your forum has been invaded..

sheffters · Post by **sheffters** » 2009-08-13 02:08

yeah ... 2nd time

... russian both times.

I've put that GD captcha thing on, although personally I hate it

suspect there real people tho as it's got to have email address active etc.

bah!

S.

^DooM^ · Post by **^DooM^** » 2009-08-13 02:34

bots signup and check email on gmail etc auto. Clever buggers. Pity these spammers can't put their genius into proper work instead of spamming everyone all day..

dougmabry1 · Post by **dougmabry1** » 2009-08-13 16:21

This looks like what I am looking for. Can it be downloaded?

sheffters · Post by **sheffters** » 2009-08-13 17:26

downloads section on the forum http://forum.ianbradshaw.co.uk

can see it for my server at http://hmaillive.ianbradshaw.co.uk to see if its current incarnation is what you want. The mail status page isnt in v0.1.2 of the script as I'm testing it and it needs reworking based on the ststus codes martins kindly provided me with.

tested for hmail 5.1.x at the moment, not tried upgrading to 5.2.x yet - although if you check the database version is 5110 in the latest release then it should work but no promises.

please read the (limited) instructions about what perl modules you'll need to install. (links on forum).

I'm testing an updated version that uses the awstats file as well and also includes the mail status codes - hopefully this will be available sometime next week if all goes well.

Cheers

S.

DFitch · Post by **DFitch** » 2009-08-21 05:42

Hello,

I'm trying this out, was gonna post in your forum but never received an email for activation.

Anyways, I have all modules installed perl5.8 but running hMail 5.2-356, have you had time to test?

I am receiving a cgi error: The specified CGI application misbehaved by not returning a complete set of HTTP headers. (IIS) It seems to work partially on ubuntu, however since it uses w32::System info, i had to disable those portions.

Any ideas on error?

D

sheffters · Post by **sheffters** » 2009-08-21 15:07

Hi,

hmm .. I'll have a look at the forum .. seems theres a few inactive users on there. damn it.

Not tried it on v5.2 yet ... try running the scripts from command line and see if it throws up any errors.

c:\webroot\>perl index.cgi

or whatever one is failing. If that fails it should give you some better troubleshooting tips. Let me know what it comes back with.

Cheers

S.

sheffters · Post by **sheffters** » 2009-08-21 15:09

p.s. manually activated you on forum so not to clog up martins.

S.

DFitch · Post by **DFitch** » 2009-08-21 16:05

New Error: Config File version error

D

sheffters · Post by **sheffters** » 2009-08-21 16:18

mail me your config file with xxx in for passwords if poss

thanks

S.

sheffters · Post by **sheffters** » 2009-08-21 16:24

it looks for specific things on specific lines of the config file, and needs a space after the = sign.

sounds like the lines are in the wrong place in the file.

S.

DFitch · Post by **DFitch** » 2009-08-21 16:26

Spaces seem fine, sent to you in pm, Thanks

adrianp918 · Post by **adrianp918** » 2009-08-21 23:02

where would i be able to get a good ddownload location from....i would like to have this script

^DooM^ · Post by **^DooM^** » 2009-08-21 23:08

http://forum.ianbradshaw.co.uk

Slug · Post by **Slug** » 2009-08-22 17:12

Any chance of adding the server load (cpu) over the last 1/5/30 mins ... Other then that not bad ...

sheffters · Post by **sheffters** » 2009-08-22 18:14

just uploaded v0.2.0 that does some basic AWStats processing. More work to do on the AWStats log file but its got some basics on there now.

re CPU : I've tried to stay away from anything that needs scripts to continually run in the background.

I'll see what I can do with the AWStats processor script .. could add in a CPU / RAM checker into that and graph history each time that is run. You can determine poll interval via scheduled tasks then.

Cheers

S.

sheffters · Post by **sheffters** » 2009-08-22 19:10

re CPU : found a perl module to do this direct, I'll add it to next version.

S.

Slug · Post by **Slug** » 2009-08-22 19:15

Whoa nice ....

adrianp918 · Post by **adrianp918** » 2009-08-22 19:30

i have it installed on a machine that is running apache and perl 5.8 and all i get are blank pages,

sheffters · Post by **sheffters** » 2009-08-22 23:15

try running it from the command line ... c:\webroot > perl index.cgi ... and see what it errors on.

stick the result on forum.ianbradshaw.co.uk or this thread will get mental.

S.

sheffters · Post by **sheffters** » 2009-08-24 18:18

@slug : CPU stat is in v0.2.1

It's a bit primative (i.e. only measure it when you load a page) as I was running into privlidge problems with the IUSR.

Will try and fix it into the AWStats thing in the future, but for now it just does 'now' as in when you load the index page.

S.

sheffters · Post by **sheffters** » 2009-09-06 20:11

updated version with some basic SPAM stats

- invalid local address by originating IP
- invalid logon attempts

http://forum.ianbradshaw.co.uk/viewtopic.php?f=6&t=26

http://hmaillive.ianbradshaw.co.uk/spam.cgi

Cheers

S.

bigworm · Post by **bigworm** » 2009-09-15 16:11

This is a nice looking stats interface! I am going to try it out asap..

Thanks!

joksi · Post by **joksi** » 2009-10-04 19:24

I just tried your script, and im having issues with the graphing function.

Please, take a look at http://www.joksi.net/hMailLive/date.cgi ... ter=Filter

sheffters · Post by **sheffters** » 2009-10-04 23:34

You've got the domain wrong in the config file ... can you post config file (with passwords removed)

http://forum.ianbradshaw.co.uk

ta

S.

bagu · Post by **bagu** » 2009-10-08 09:12

I can't make it working :

[Thu Oct 08 09:05:28 2009] [error] [client 127.0.0.1] (OS 3)Le chemin d'accès spécifié est introuvable. : couldn't create child process: 720003: index.cgi, referer: http://localhost/hmail/
[Thu Oct 08 09:05:28 2009] [error] [client 127.0.0.1] (OS 3)Le chemin d'accès spécifié est introuvable. : couldn't spawn child process: D:/wamp/www/hmail/index.cgi, referer: http://localhost/hmail/

But path in hMaillive.config are good

sheffters · Post by **sheffters** » 2009-10-08 15:22

What version of perl have you got? Are all the PPM modules installed?

Which version of hmlive are you using?

S.

bagu · Post by **bagu** » 2009-10-09 10:20

All ppms modules are installed
Activestate Perl 8.5.9 build 825
Hmail live 0.2.3

on windows xp pro + hmailserver 5.3.1617

sheffters · Post by **sheffters** » 2009-10-09 13:21

hmm .. bit of a long shot .. but .. have you got the perl path in your environment?

not sure how its setup with wamp.

Looks like IIS cant find the perl exe

Cheers

S.

bagu · Post by **bagu** » 2009-10-11 01:35

Yes, here is my path :

D:\wamp\apps\perl\site\bin;D:\wamp\apps\perl\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;D:\wamp\bin\php\php5.2.11

So everything seem to be good.
Awstats work fine too...

sheffters · Post by **sheffters** » 2009-10-11 22:47

awstats is php I think?

try creating a standard hello world perl index.cgi and see if it runs?

Cheers

S.

$url = "http://$ENV{SERVER_NAME}$ENV{URL}";
$ip = "$ENV{REMOTE_ADDR}";
print <<ENDOFTEXT;
HTTP/1.0 200 OK
Content-Type: text/html

<HTML>
<HEAD><TITLE>Hello World!</TITLE></HEAD>
<BODY>
<H4>Hello World!</H4>
<P>You have reached <a href="$url">$url</a></P>
<P>Your IP Address is $ip</P>
<H5>Have a nice day!</H5>
</BODY>
</HTML>
ENDOFTEXT
exit(0);

save that as index.cgi and see if it runs

^DooM^ · Post by **^DooM^** » 2009-10-11 23:14

AWStats uses PERL.

roi · Post by **roi** » 2009-10-12 06:38

I visited your site and am very impressed by your web application. I can understand how postmasters will want to use your application to watch their mail server/s. Congratulations ...

Perl is relatively foreign to me... but if I can get get it to work in IIS as ISAPI, I may give your app a try on a dev box hMail. Now in the process of setting up a backup hMail so am not quite yet ready to adventure in another direction.

Wishing you the best in your project.

sheffters · Post by **sheffters** » 2009-10-12 15:05

Hi,

@doom ... ah

@bagu ... hmm .. try an older hmlive and see if it runs? maybe I've got a bug somewhere I've not seen. Will the other scripts run? Whats it say if you try run it from command line instead of via IIS?

@ roi ... thanks

... you can set perl to run as ISAPI instead of as cgi ... the cgi should execute either via cgi as exe or cgi via isapi.

Cheers

S.

bagu · Post by **bagu** » 2009-10-13 13:22

I don't use iis...If you see me sign, you can see i use apache.
On your website you only say we must use a web server, you don't say i must use iis...Plus, i already use awstats without any problem...
So my perl env seems to work fine...

I will try to test different ways...

EDIT :

Ok, now i have this :

Code: Select all

Gateway Time-out
The gateway did not receive a timely response from the upstream server or application.

Code: Select all

[Tue Oct 13 13:52:55 2009] [error] [client 127.0.0.1] Script timed out before returning headers: index.cgi, referer: http://localhost/

EDIT : Same result with your test code

sheffters · Post by **sheffters** » 2009-10-13 19:37

Hi,

If you get the same result with the test code above then it's an installation problem.

Apache should work, just i don't use it. No reason why it wouldn't if it's setup correctly.

Get the test file working and see what happens.

Cheers

S.

dzekas · Post by **dzekas** » 2009-10-13 21:46

Reinventing MRTG.

Do you know that all you need is to pull status and give those numbers to MRTG.

bagu · Post by **bagu** » 2009-10-13 23:40

bagu wrote:I don't use iis...If you see me sign, you can see i use apache.
On your website you only say we must use a web server, you don't say i must use iis...Plus, i already use awstats without any problem...
So my perl env seems to work fine...

I will try to test different ways...

EDIT :

Ok, now i have this :
Code: Select all
Gateway Time-out
The gateway did not receive a timely response from the upstream server or application.
Code: Select all
[Tue Oct 13 13:52:55 2009] [error] [client 127.0.0.1] Script timed out before returning headers: index.cgi, referer: http://localhost/
EDIT : Same result with your test code

Can't understand why awstat work fine and not this script ^^

sheffters · Post by **sheffters** » 2009-10-14 01:48

Can't understand why awstat work fine and not this script ^^

me either ... maybe to do with execute permissions on the folder?

@dzekas ... http://oss.oetiker.ch/mrtg/ ?

not sure why thats the same if its what you mean? The internal / external traffic would probably be better with that if you can access the router from your server ... but suspect most people can't. It's not suppposed to replace a proper monitor app, just give hints for those that can't access them.

S.

dzekas · Post by **dzekas** » 2009-10-14 06:17

sheffters wrote:
Can't understand why awstat work fine and not this script ^^
me either ... maybe to do with execute permissions on the folder?

@dzekas ... http://oss.oetiker.ch/mrtg/ ?

not sure why thats the same if its what you mean? The internal / external traffic would probably be better with that if you can access the router from your server ... but suspect most people can't. It's not suppposed to replace a proper monitor app, just give hints for those that can't access them.

S.

MRTG can monitor other things too. CPU load, number of pop3/smtp connections, size of email queue, number of rbl rejects, disk usage. Any number will work. Your live stats demo page shows only current stats. MRTG graphs would show current stats and history.

sheffters · Post by **sheffters** » 2009-10-14 13:59

@dzekas ... I don't think its a cloan, but guess we won't agree ... nothings stopping you using something different.

S.

dzekas · Post by **dzekas** » 2009-10-14 14:18

sheffters wrote:@dzekas ... I don't think its a clone, but guess we won't agree ... nothings stopping you using something different.

Instead of writing simple programs to get stats for processing in mrtg, you do complex thing which does less than those simple programs could do. MRTG graphs are closer to definition of "live".

sheffters · Post by **sheffters** » 2009-10-14 14:24

MRTG graphs are closer to definition of "live".

most of the data comes direct from the hmail database ... not sure how to make it more live than that.

Suggest you use something else to do whatever you want to do.

S.

dzekas · Post by **dzekas** » 2009-10-14 14:37

sheffters wrote:
MRTG graphs are closer to definition of "live".
most of the data comes direct from the hmail database ... not sure how to make it more live than that.

sheffters wrote:Suggest you use something else to do whatever you want to do.

I don't use hMailServer. Software written specifically for hMailServer is useless for me.

I am not saying that your software is bad. It would be a lot easier to do things, if you use existing tools.

sheffters · Post by **sheffters** » 2009-10-15 23:01

joksi wrote:I just tried your script, and im having issues with the graphing function.

Please, take a look at http://www.joksi.net/hMailLive/date.cgi ... ter=Filter

you need to add the www into your domain name in the config file ... just seen its requesting the graph from http://joksi.net rather than http://www.joksi.net

S.

sheffters · Post by **sheffters** » 2009-10-15 23:02

dzekas wrote:I am not saying that your software is bad. It would be a lot easier to do things, if you use existing tools.

but not as much fun

S.

joksi · Post by **joksi** » 2009-10-22 14:43

Thank you, it solved the problem!