When searching large log files for messages that do not frequently occur, there is a large risk of reading all collected lines before returning the first results.
In most cases recent items are required, in which case going back for days of logs might not make sense.
This commit adds a simple "history" selection which translates into a "valid_from" filter on the log data. When timestamps are not parseable for whatever reason, the filter is ignored.
Only small downside is that we do need to translate timestamp again, when needed we could improve performance a bit by storing the original datetime value in NewBaseLogFormat so we don't have to parse it twice.
The rotate function needed the next file to calculate the suffix, which is problematic if someone just cleaned all logs. This patch improves the match a bit (last _ instead of assuming _ doesn't exist in the name) and uses "1" for the first rotate action.
The combination of preserve logs and max file size help to guard the boundaries of the log storage being used, an archive action is already being performed hourly, which should be enough in normal situations (although that would be easy to change if needed).
In order to make room for the new additional files per day, we add a sequence to the file, for example the first rotate of a filter log exceeding its limit named /var/log/filter/filter_20231204.log would be moved to /var/log/filter/filter_20231204.0001.log . The syslog-ng reload handles the flush to a new file, which automatically would result in a new filter_20231204.log file after rotate.
client_connect.php also includes it but not config.inc. Try to leave
it at that to not pollute it unnecessarily. The other scripts might
be able to do it as well this way but don't fix something that is not
broken either.
As the client still might have a state when being kicked-out, we should kill any state the client has while adding it to the alias. Apparantly our ssh messages are only catched partially, so add ".*Authentication error for .*" to the list as well. To ease testing, better detect the location of the timestamp so we can use a construction like this to feed amn existing log:
lockout_handler < /var/log/audit/audit_20221205.log
same same as dd6a04a68a, but different.
Concerns fixed in this commit:
o archive shouldn't generate files so old "archived" files remain untouched
o "latest.log" should point to the latest version know, which could be todays or a file from the past
o better to not remove links when unchanged to prevent excessive writes
We shuffle the cron jobs around a bit to let the script start at
minute 1 of every hour which is close enough.
We might consider a soft-update of the link instead of removing
it every hour depending on how tail -f and such work, but for now
use what PHP has to offer.
PR: https://github.com/opnsense/core/issues/4993
