hMailServer Log Searcher

[palinka]

I branched this off another project I'm working on where I was parsing the logs. Some of the code came from Coax's excellent phpwebadmin. Anyway, I discovered that searching all of the logs at once is actually very useful, so I set out to make it happen.

This is a php script. It loops through all the logs and displays lines if a keyword match is found. The keyword can be a string, a date-time, an IP, anything. It works amazingly fast. My log folder right now contains 42 logs totaling 76 MB and it zips through all of them in well under 1 second. It does not write anything at all - it only reads and displays the results. Very simple, but useful.

One thing it cannot do (for the moment) is search the event log. For some reason, the event log is encoded in UTF16LE and preg_match just doesn't work. I'm still working on that. All the other logs, including spamd.log work fine.

https://github.com/palinkas-jo-reggelt/ ... g_Searcher

[jimimaseye]

Until you figured out how, for the event log, the search feature in Notepad++ will work (it does the same kind of search specifying current, all opened or 'contents of folder' documents).

[Entered by mobile. Excuse my spelling.]

[SorenR]

Until you figured out how, for the event log, the search feature in Notepad++ will work (it does the same kind of search specifying current, all opened or 'contents of folder' documents).

[Entered by mobile. Excuse my spelling.]

I have 3 tools that can find anything:

UltraEdit (because I used XEDIT under VM/CMS back in the 80's )
Notepad++ (Highly recomended by several here!)
grepWin (search files or contents as text or regex ... https://tools.stefankueng.com/grepWin.html)

[palinka]

Until you figured out how, for the event log, the search feature in Notepad++ will work (it does the same kind of search specifying current, all opened or 'contents of folder' documents).

[Entered by mobile. Excuse my spelling.]

I have 3 tools that can find anything:

UltraEdit (because I used XEDIT under VM/CMS back in the 80's )
Notepad++ (Highly recomended by several here!)
grepWin (search files or contents as text or regex ... https://tools.stefankueng.com/grepWin.html)

My php script searches all log files at once and doesn't require you to log on to the server. Just open your browser. Plus, it works great on mobile too.

Nothing against notepadd++, of course. I wrote the php script using it. And I know it will search multiple files, but its pretty clunky compared to this php script.

[palinka]

I worked out *mostly* how to convert the encoding. It works fine now, but there is one "zero width non break space" at the beginning of each line in the event logs that I just can't get rid of.

Here's an actual entry from the event log:

Code: Select all

2808	"2023-07-22 01:57:05.055"	"AbuseIPDB 75.83.28.241 Score:100"
2808	"2023-07-22 01:57:05.055"	"INFO: AbuseIPDB: 75.83.28.241:465"
3652	"2023-07-22 02:32:11.892"	"AbuseIPDB 69.194.22.27 Score:100"
3652	"2023-07-22 02:32:11.892"	"INFO: AbuseIPDB: 69.194.22.27:465"
4780	"2023-07-22 02:35:03.253"	"AbuseIPDB 50.214.233.253 Score:100"
4780	"2023-07-22 02:35:03.253"	"INFO: AbuseIPDB: 50.214.233.253:465"
1368	"2023-07-22 02:50:43.944"	"AbuseIPDB 50.217.255.171 Score:100"
1368	"2023-07-22 02:50:43.944"	"INFO: AbuseIPDB: 50.217.255.171:465"
2736	"2023-07-22 02:55:37.069"	"Error in GeoIP lookup - IP: 127.0.0.1"
2416	"2023-07-22 03:27:14.010"	"AbuseIPDB 134.204.0.245 Score:100"
2416	"2023-07-22 03:27:14.010"	"INFO: AbuseIPDB: 134.204.0.245:465"

Here's what get's displayed in the php script:

Code: Select all

ÿ︲808 2023-07-22 01:57:05.055 INFO: AbuseIPDB: 75.83.28.241:465
ÿ︳652 2023-07-22 02:32:11.892 AbuseIPDB 69.194.22.27 Score:100
ÿ︳652 2023-07-22 02:32:11.892 INFO: AbuseIPDB: 69.194.22.27:465
ÿ︴780 2023-07-22 02:35:03.253 AbuseIPDB 50.214.233.253 Score:100
ÿ︴780 2023-07-22 02:35:03.253 INFO: AbuseIPDB: 50.214.233.253:465
ÿ︱368 2023-07-22 02:50:43.944 AbuseIPDB 50.217.255.171 Score:100
ÿ︱368 2023-07-22 02:50:43.944 INFO: AbuseIPDB: 50.217.255.171:465
ÿ︲416 2023-07-22 03:27:14.010 AbuseIPDB 134.204.0.245 Score:100
ÿ︲416 2023-07-22 03:27:14.010 INFO: AbuseIPDB: 134.204.0.245:465

So you can see that ZWNBSP is screwing up the first char, but its only the thread number, so its not something typically anyone would be interested in.

(Tabs and other superfluous characters are removed before displaying the subject line.)

https://github.com/palinkas-jo-reggelt/ ... g_Searcher

[gotspatel]

@palinka

May be it has to do with encoding in the webserver

I have IIS and mine looks OK I think

Firefox_Screenshot_2023-07-25T13-37-44.095Z.png (34.78 KiB) Viewed 2866 times

[palinka]

@palinka

May be it has to do with encoding in the webserver

I have IIS and mine looks OK I think

Firefox_Screenshot_2023-07-25T13-37-44.095Z.png

For me the only file that gives trouble is the event log, which is encoded in UTF16-LE BOM while all the rest are UTF-8.

[gotspatel]

@palinka

May be it has to do with encoding in the webserver

I have IIS and mine looks OK I think

Firefox_Screenshot_2023-07-25T13-37-44.095Z.png

For me the only file that gives trouble is the event log, which is encoded in UTF16-LE BOM while all the rest are UTF-8.

yes you are right

Firefox_Screenshot_2023-07-26T05-58-21.341Z.png (54.5 KiB) Viewed 2841 times

[palinka]

Fixed it. Now everything displays correctly. I already updated github. I also added a few statistics. Right now my log folder contains 43 files totaling 60.4 MB.

Screenshot 2023-07-26 143243.png (69.66 KiB) Viewed 2831 times

[palinka]

Also, the search uses preg_match to filter the lines, so you can use regex to search. I have not tested that extensively to see what effect it will have, but simple pipe "|" works fine.

[palinka]

Based on a comment Matt made on Github, I added filtering by log type and date. Matt has several gigabytes of logs but he reported that the script successfully finished, although it took 90 seconds to complete. The filtering will reduce that time substantially because it completely ignores files not selected in the filter.

There are two filters: type and date. The filters are regex based and use the same date schema that hmailserver uses. Therefore, if your custom logs also follow that schema, then you'll be able to filter them by date as well.

Code: Select all

Log                  Date          Ext
------------------   -----------   ----
ERROR_hmailserver    _2023-07-29   .log
ERROR_hmailserver    _2023-07-28   .log
hmailserver          _2023-07-29   .log
hmailserver          _2023-07-28   .log
hmailserver_events                 .log
hmailserver_events   _2023-07-29   .log
spamd                              .log
spamd                _2023-07-29   .log
my_custom_log        _2023-07-29   .log
custom_no_date                     .log

If you filter by a type that has mixed date results - eg spamd.log, which I roll every night - it will return results from spamd.log and spamd_2023-07-29.log, etc. If you filter by "spamd" type AND by date, then it will only look in the dated ones for results. If you filter by date only (no type selected) then it will look at all logs with that specific date, e.g. it will ignore spamd.log, but search in spamd_2023-07-29.log and any other logs with "_2023-07-29" in the file name.

The dates of all files - that is the date in the file name, NOT the last write time or any other system date - are loaded into an array and will display in a select box to choose from. This is updated when you select a log type. So for example, all files may produce 10 dates, but one log type may only have 3 dates among its results, so if you select that type, it will only offer those 3 dates to select from. Pretty cool that it won't waste your time with log/dates that don't exist. However, your log dating must use the hmailserver log/date schema as shown in the table above.

Also, it ignores any file that doesn't have .log file extension. I will add an option to extend that to other file extensions in the future, like .txt for example, but I don't think it will be possible for files without a file extension.

https://github.com/palinkas-jo-reggelt/ ... g_Searcher

Screenshot 2023-08-01 070140.png (54.38 KiB) Viewed 2811 times

[palinka]

I added a couple of cool features.

1) Search any folder (default folder is hMailServer log folder found by API)
2) Search any file type by file extension
3) Now displays the line number of the search result

I had the idea for these things because I had to migrate a website that was not designed to be portable and I had to find and change a bunch of hard coded items into variables. It worked great for that. Search and destroy. Sweep and clear.

Screenshot 2023-09-30 072537.png (92.36 KiB) Viewed 681 times

https://github.com/palinkas-jo-reggelt/ ... g_Searcher