Syslog: optionally disable legacy (clog) logging (#4101)

* Syslog-NG replacement for legacy syslog local logs:

Part of this commit:

- support both formats in query log, which is used by our log api
- sample local syslog-ng target for configd

for https://github.com/opnsense/core/issues/4068

* syslog: add disable clog toggle + preserve number of log (days) setting when only using syslog-ng. for https://github.com/opnsense/core/issues/4068

* syslog: include local syslog-ng files when clog is disabled. for https://github.com/opnsense/core/issues/4068

* Syslog-NG: change local handling, add relayd file to test the concept.

The local directory contains filters for local targets, which should replace the <plugin>_syslog() construction eventually, everything relevant and not matched is send to system

for https://github.com/opnsense/core/issues/4068

* Syslog-NG: minor update to local template to support module/file format as the query log handler supports it (e.g. /var/log/squid/access/) and add local templates

* Syslog-NG: change flush log actions to support new format, while here make sure "flush all" actuallly flushes all logs (including plugins). for https://github.com/opnsense/core/issues/4068

* Syslog-NG: missing level in system log, for https://github.com/opnsense/core/issues/4068

* fix typo for https://github.com/opnsense/core/issues/4068

* syslog-ng: filter live log support for https://github.com/opnsense/core/issues/4068

* Syslog-NG: replace diag_logs_filter_summary.php for mvc enabled version, using the same log output as live log, for https://github.com/opnsense/core/issues/4068

* Syslog-NG: add log cleanup script to enforce preservelogs setting. for https://github.com/opnsense/core/issues/4068

* Syslog-NG: webuser auth message should use LOG_AUTH facility. for https://github.com/opnsense/core/issues/4068

* Syslog-NG: ditch sshlockout_pf in favour for a small script that locks out ssh/web gui failed attempts for both IPv4 and IPv6. for https://github.com/opnsense/core/issues/4068

* ditch sshlockout_pf dependancy, for https://github.com/opnsense/core/issues/4068

* fix indent in ACL, for https://github.com/opnsense/core/issues/4068

* fix plist
This commit is contained in:
Ad Schellevis 2020-05-10 10:59:14 +02:00 committed by GitHub
parent f274499c8b
commit faf650e7ca
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
41 changed files with 768 additions and 604 deletions

View File

@ -135,7 +135,6 @@ CORE_DEPENDS?= ${CORE_DEPENDS_${CORE_ARCH}} \
rrdtool \
samplicator \
squid \
sshlockout_pf \
strongswan \
sudo \
syslog-ng${CORE_SYSLOGNG:S/.//g} \

26
plist
View File

@ -495,6 +495,7 @@
/usr/local/opnsense/mvc/app/views/OPNsense/Cron/index.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/arp.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/fw_log.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/fw_stats.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/log.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/ndp.volt
/usr/local/opnsense/mvc/app/views/OPNsense/Diagnostics/netflow.volt
@ -710,6 +711,8 @@
/usr/local/opnsense/scripts/suricata/rule-updater.py
/usr/local/opnsense/scripts/suricata/setup.sh
/usr/local/opnsense/scripts/syslog/list_applications.php
/usr/local/opnsense/scripts/syslog/lockout_handler
/usr/local/opnsense/scripts/syslog/log_archive
/usr/local/opnsense/scripts/system/dh_parameters.sh
/usr/local/opnsense/scripts/system/list_interrupts.py
/usr/local/opnsense/scripts/system/rfc5246_cipher_suites.csv
@ -834,11 +837,33 @@
/usr/local/opnsense/service/templates/OPNsense/Sample/sub2/+TARGETS
/usr/local/opnsense/service/templates/OPNsense/Sample/sub2/example_sub2.txt
/usr/local/opnsense/service/templates/OPNsense/Syslog/+TARGETS
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/README
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/configd.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/dhcpd.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/dnsmasq.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/filter.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/gateways.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/ipsec.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/lighttpd.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/ntpd.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/openvpn.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/pkg.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/portalauth.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/ppps.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/relayd.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/resolver.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/routing.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/squid_access.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/suricata.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/vpn.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/local/wireless.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/newsyslog.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/rc.conf.d
/usr/local/opnsense/service/templates/OPNsense/Syslog/sources/001-local.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-destinations.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-legacy.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-local.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-lockout.conf
/usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng.conf
/usr/local/opnsense/service/templates/OPNsense/Unbound/core/+TARGETS
/usr/local/opnsense/service/templates/OPNsense/Unbound/core/dnsbl.inc
@ -1580,7 +1605,6 @@
/usr/local/www/diag_ipsec_sad.php
/usr/local/www/diag_ipsec_spd.php
/usr/local/www/diag_logs_common.inc
/usr/local/www/diag_logs_filter_summary.php
/usr/local/www/diag_logs_settings.php
/usr/local/www/diag_packet_capture.php
/usr/local/www/diag_pf_info.php

View File

@ -64,6 +64,11 @@ function set_language()
function session_auth(&$Login_Error)
{
global $config;
function auth_log($message) {
openlog("webgui", LOG_ODELAY, LOG_AUTH);
log_error($message);
closelog();
}
// Handle HTTPS httponly and secure flags
$currentCookieParams = session_get_cookie_params();
@ -111,7 +116,7 @@ function session_auth(&$Login_Error)
$_SESSION['user_shouldChangePassword'] = true;
}
if (!isset($config['system']['webgui']['quietlogin'])) {
log_error(sprintf("Successful login for user '%s' from: %s", $_POST['usernamefld'], $_SERVER['REMOTE_ADDR']));
auth_log(sprintf("Successful login for user '%s' from: %s", $_POST['usernamefld'], $_SERVER['REMOTE_ADDR']));
}
if (!empty($_GET['url'])) {
$tmp_url_parts = parse_url($_GET['url']);
@ -137,7 +142,7 @@ function session_auth(&$Login_Error)
}
exit;
} else {
log_error("Web GUI authentication error for '{$_POST['usernamefld']}' from {$_SERVER['REMOTE_ADDR']}");
auth_log("Web GUI authentication error for '{$_POST['usernamefld']}' from {$_SERVER['REMOTE_ADDR']}");
$Login_Error = true;
}
}
@ -169,9 +174,9 @@ function session_auth(&$Login_Error)
/* user hit the logout button */
if (isset($_GET['logout'])) {
if (isset($_SESSION['Logout'])) {
log_error(sprintf("Session timed out for user '%s' from: %s", $_SESSION['Username'], $_SERVER['REMOTE_ADDR']));
auth_log(sprintf("Session timed out for user '%s' from: %s", $_SESSION['Username'], $_SERVER['REMOTE_ADDR']));
} else {
log_error(sprintf("User logged out for user '%s' from: %s", $_SESSION['Username'], $_SERVER['REMOTE_ADDR']));
auth_log(sprintf("User logged out for user '%s' from: %s", $_SESSION['Username'], $_SERVER['REMOTE_ADDR']));
}
/* wipe out $_SESSION */

View File

@ -143,6 +143,7 @@ function core_cron()
$jobs[]['autocron'] = array('/usr/local/sbin/expiretable -v -t 3600 sshlockout', '2');
$jobs[]['autocron'] = array('/usr/local/sbin/expiretable -v -t 3600 virusprot', '3');
$jobs[]['autocron'] = array('/usr/local/etc/rc.expireaccounts', '5');
$jobs[]['autocron'] = array('/usr/local/opnsense/scripts/syslog/log_archive ', '4');
$jobs[]['autocron'] = array('/usr/local/sbin/ping_hosts.sh', '*/4');

View File

@ -641,7 +641,6 @@ news.err;local0.none;local3.none;local4.none {$log_directive}/var/log/system.l
local7.none {$log_directive}/var/log/system.log
security.* {$log_directive}/var/log/system.log
auth.info;authpriv.info;daemon.info {$log_directive}/var/log/system.log
auth.info;authpriv.info;user.* |exec /usr/local/sbin/sshlockout_pf 15
*.emerg *
EOD;

View File

@ -30,6 +30,7 @@ namespace OPNsense\Diagnostics\Api;
use OPNsense\Base\ApiControllerBase;
use OPNsense\Core\Backend;
use OPNsense\Core\Config;
/**
* Class FirewallController
@ -56,4 +57,63 @@ class FirewallController extends ApiControllerBase
return null;
}
}
/**
* retrieve firewall stats
* @return array
*/
public function statsAction()
{
if ($this->request->isGet()) {
$this->sessionClose(); // long running action, close session
$limit = empty($this->request->get('limit')) ? 5000 : $this->request->get('limit');
$group_by = empty($this->request->get('group_by')) ? "interface" : $this->request->get('group_by');
$records = json_decode((new Backend())->configdpRun("filter read log", array($limit)), true);
$response = array();
if (!empty($records)) {
$tmp_stats = array();
foreach ($records as $record) {
if (isset($record[$group_by])) {
if (!isset($tmp_stats[$record[$group_by]])) {
$tmp_stats[$record[$group_by]] = 0;
}
$tmp_stats[$record[$group_by]]++;
}
}
arsort($tmp_stats);
$label_map = array();
switch ($group_by) {
case 'interface':
$label_map["lo0"] = gettext("loopback");
if (Config::getInstance()->object()->interfaces->count() > 0) {
foreach (Config::getInstance()->object()->interfaces->children() as $k => $n) {
$label_map[(string)$n->if] = !empty((string)$n->descr) ? (string)$n->descr : $k;
}
}
break;
case 'proto':
// proto
break;
}
$recno = $top_cnt = 0;
foreach ($tmp_stats as $key => $value) {
// top 10 + other
if ($recno < 10) {
$response[] = [
"label" => !empty($label_map[$key]) ? $label_map[$key] : $key,
"value" => $value
];
$top_cnt += $value;
} else {
$response[] = ["label" => gettext("other"), "value" => count($records) - $top_cnt];
break;
}
$recno++;
}
}
return $response;
} else {
return null;
}
}
}

View File

@ -45,4 +45,11 @@ class FirewallController extends IndexController
{
$this->view->pick('OPNsense/Diagnostics/fw_log');
}
/**
* firewall statistical view
*/
public function statsAction()
{
$this->view->pick('OPNsense/Diagnostics/fw_stats');
}
}

View File

@ -312,7 +312,8 @@
<page-diagnostics-logs-firewall-summary>
<name>Diagnostics: Logs: Firewall: Summary View</name>
<patterns>
<pattern>diag_logs_filter_summary.php*</pattern>
<pattern>ui/diagnostics/firewall/stats*</pattern>
<pattern>api/diagnostics/firewall/stats*</pattern>
</patterns>
</page-diagnostics-logs-firewall-summary>
<page-interfaces-assignnetworkports>

View File

@ -192,7 +192,7 @@
</Diagnostics>
<LogFiles order="400" VisibleName="Log Files" cssClass="fa fa-eye fa-fw">
<Live VisibleName="Live View" url="/ui/diagnostics/firewall/log"/>
<Overview url="/diag_logs_filter_summary.php"/>
<Overview url="/ui/diagnostics/firewall/stats"/>
<Plain VisibleName="Plain View" url="/ui/diagnostics/log/core/filter"/>
</LogFiles>
</Firewall>

View File

@ -0,0 +1,136 @@
{#
OPNsense® is Copyright © 2020 by Deciso B.V.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
#}
<!-- nvd3 -->
<link rel="stylesheet" type="text/css" href="{{ cache_safe(theme_file_or_default('/css/nv.d3.css', ui_theme|default('opnsense'))) }}" />
<!-- d3 -->
<script src="{{ cache_safe('/ui/js/d3.min.js') }}"></script>
<!-- nvd3 -->
<script src="{{ cache_safe('/ui/js/nv.d3.min.js') }}"></script>
<script>
'use strict';
$( document ).ready(function() {
function load_chart(group_by) {
ajaxGet("/api/diagnostics/firewall/stats", {group_by: group_by}, function (data, status) {
if (status == "success") {
var svg = d3.select("svg");
svg.selectAll("*").remove();
nv.addGraph(function() {
// Find all piechart classes to insert the chart
$('div[class="piechart"]').each(function(){
var selected_id = $(this).prop("id");
var chart = nv.models.pieChart()
.x(function(d) { return d.label })
.y(function(d) { return d.value })
.showLabels(true)
.labelThreshold(.05)
.labelType("percent")
.donut(true)
.donutRatio(0.35)
.legendPosition("right")
;
d3.select("[id='chart'].piechart svg")
.datum(data)
.transition().duration(350)
.call(chart);
// Update Chart after window resize
nv.utils.windowResize(function(){ chart.update(); });
return chart;
});
});
$("#stats > tbody").empty();
for (let i=0; i < data.length ; ++i) {
let tr = $("<tr/>");
tr.append($("<td/>").text(data[i].label));
tr.append($("<td/>").text(data[i].value));
$("#stats > tbody").append(tr)
}
}
});
}
$('a[data-toggle="tab"]').on('shown.bs.tab', function (e) {
let group_by = e.target.href.split('#')[1];
load_chart(group_by);
$("#heading_label").text(e.target.text);
});
let selected_tab = window.location.hash != "" ? window.location.hash : "#action";
$('a[href="' +selected_tab + '"]').tab('show');
$('.nav-tabs a').on('shown.bs.tab', function (e) {
history.pushState(null, null, e.target.hash);
});
$(window).on('hashchange', function(e) {
$('a[href="' + window.location.hash + '"]').click()
});
});
</script>
<style>
#chart svg {
height: 400px;
}
</style>
<ul class="nav nav-tabs" data-tabs="tabs" id="maintabs">
<li><a data-toggle="tab" href="#action">{{ lang._('Actions') }}</i></a></li>
<li><a data-toggle="tab" href="#interface">{{ lang._('Interfaces') }}</i></a></li>
<li><a data-toggle="tab" href="#protoname">{{ lang._('Protocols') }}</i></a></li>
<li><a data-toggle="tab" href="#src">{{ lang._('Source IPs') }}</i></a></li>
<li><a data-toggle="tab" href="#dst">{{ lang._('Destination IPs') }}</i></a></li>
<li><a data-toggle="tab" href="#srcport">{{ lang._('Source Ports') }}</i></a></li>
<li><a data-toggle="tab" href="#dstport">{{ lang._('Destination Ports') }}</i></a></li>
</ul>
<div class="tab-content">
<div id="graph" class="tab-pane fade in active">
<div class="panel panel-default">
<div class="panel-body">
<div class="piechart" id="chart">
<svg></svg>
</div>
<table class="table table-striped table-bordered" id="stats">
<thead>
<tr>
<th id="heading_label"> </th>
<th> # </th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
</div>
</div>
</div>

View File

@ -31,6 +31,7 @@
import os
import sys
import re
import glob
from hashlib import md5
import argparse
import ujson
@ -39,7 +40,6 @@ sys.path.insert(0, "/usr/local/opnsense/site-python")
from log_helper import reverse_log_reader, fetch_clog
from params import update_params
filter_log = '/var/log/filter.log'
# define log layouts, every endpoint contains all options
# source : https://github.com/opnsense/ports/blob/master/opnsense/filterlog/files/description.txt
@ -109,57 +109,72 @@ if __name__ == '__main__':
running_conf_descr = fetch_rule_details()
result = list()
for record in reverse_log_reader(fetch_clog(filter_log)):
if record['line'].find('filterlog') > -1:
rule = dict()
metadata = dict()
# rule metadata (unique hash, hostname, timestamp)
log_ident = re.split('filterlog[^:]*:', record['line'])
tmp = log_ident[0].split()
metadata['__digest__'] = md5(record['line'].encode()).hexdigest()
metadata['__host__'] = tmp.pop()
metadata['__timestamp__'] = ' '.join(tmp)
rulep = log_ident[1].strip().split(',')
update_rule(rule, metadata, rulep, fields_general)
filter_logs = []
if os.path.isdir('/var/log/filter'):
filter_logs = list(sorted(glob.glob("/var/log/filter/filter_*.log"), reverse=True))
if os.path.isfile('/var/log/filter.log'):
filter_logs.append('/var/log/filter.log')
if 'action' not in rule:
# not a filter log line, skip
continue
elif 'version' in rule:
if rule['version'] == '4':
update_rule(rule, metadata, rulep, fields_ipv4)
if 'proto' in rule:
if rule['proto'] == '17': # UDP
update_rule(rule, metadata, rulep, fields_ipv4_udp)
elif rule['proto'] == '6': # TCP
update_rule(rule, metadata, rulep, fields_ipv4_tcp)
elif rule['proto'] == '112': # CARP
update_rule(rule, metadata, rulep, fields_ipv4_carp)
elif rule['version'] == '6':
update_rule(rule, metadata, rulep, fields_ipv6)
if 'proto' in rule:
if rule['proto'] == '17': # UDP
update_rule(rule, metadata, rulep, fields_ipv6_udp)
elif rule['proto'] == '6': # TCP
update_rule(rule, metadata, rulep, fields_ipv6_tcp)
elif rule['proto'] == '112': # CARP
update_rule(rule, metadata, rulep, fields_ipv6_carp)
for filter_log in filter_logs:
do_exit = False
try:
filename = fetch_clog(filter_log)
except Exception as e:
filename = filter_log
for record in reverse_log_reader(filename):
if record['line'].find('filterlog') > -1:
rule = dict()
metadata = dict()
# rule metadata (unique hash, hostname, timestamp)
log_ident = re.split('filterlog[^:]*:', record['line'])
tmp = log_ident[0].split()
metadata['__digest__'] = md5(record['line'].encode()).hexdigest()
metadata['__host__'] = tmp.pop()
metadata['__timestamp__'] = ' '.join(tmp)
rulep = log_ident[1].strip().split(',')
update_rule(rule, metadata, rulep, fields_general)
rule.update(metadata)
if 'rulenr' in rule and rule['rulenr'] in running_conf_descr:
if rule['action'] in ['pass', 'block']:
rule['label'] = running_conf_descr[rule['rulenr']]['label']
rule['rid'] = running_conf_descr[rule['rulenr']]['rid']
elif rule['action'] not in ['pass', 'block']:
rule['label'] = "%s rule" % rule['action']
if 'action' not in rule:
# not a filter log line, skip
continue
elif 'version' in rule:
if rule['version'] == '4':
update_rule(rule, metadata, rulep, fields_ipv4)
if 'proto' in rule:
if rule['proto'] == '17': # UDP
update_rule(rule, metadata, rulep, fields_ipv4_udp)
elif rule['proto'] == '6': # TCP
update_rule(rule, metadata, rulep, fields_ipv4_tcp)
elif rule['proto'] == '112': # CARP
update_rule(rule, metadata, rulep, fields_ipv4_carp)
elif rule['version'] == '6':
update_rule(rule, metadata, rulep, fields_ipv6)
if 'proto' in rule:
if rule['proto'] == '17': # UDP
update_rule(rule, metadata, rulep, fields_ipv6_udp)
elif rule['proto'] == '6': # TCP
update_rule(rule, metadata, rulep, fields_ipv6_tcp)
elif rule['proto'] == '112': # CARP
update_rule(rule, metadata, rulep, fields_ipv6_carp)
result.append(rule)
rule.update(metadata)
if 'rulenr' in rule and rule['rulenr'] in running_conf_descr:
if rule['action'] in ['pass', 'block']:
rule['label'] = running_conf_descr[rule['rulenr']]['label']
rule['rid'] = running_conf_descr[rule['rulenr']]['rid']
elif rule['action'] not in ['pass', 'block']:
rule['label'] = "%s rule" % rule['action']
# handle exit criteria, row limit or last digest
if parameters['limit'] != 0 and len(result) >= parameters['limit']:
break
elif parameters['digest'].strip() != '' and parameters['digest'] == rule['__digest__']:
break
result.append(rule)
# handle exit criteria, row limit or last digest
if parameters['limit'] != 0 and len(result) >= parameters['limit']:
do_exit = True
elif parameters['digest'].strip() != '' and parameters['digest'] == rule['__digest__']:
do_exit = True
if do_exit:
break
if do_exit:
break
print (ujson.dumps(result))

View File

@ -0,0 +1,104 @@
#!/usr/local/bin/python3
"""
Copyright (c) 2020 Ad Schellevis <ad@opnsense.org>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
"""
import sys
import re
import argparse
import datetime
import ipaddress
import syslog
import subprocess
import time
from select import select
all_rules = {
'.*Accepted.*': True,
'.*Successful login.*': True,
'.*Web GUI authentication error.*': False,
'.*Invalid user.*': False,
'.*Illegal user.*': False,
'.*Postponed keyboard-interactive for invalid user.*': False,
'.*authentication error for illegal user.*': False
}
if __name__ == '__main__':
# handle parameters
parser = argparse.ArgumentParser()
parser.add_argument('--attempts', help='maximum number of attempts', type=int, default=5)
parser.add_argument('--grace_period', help='keep stats for max number of seconds', type=int, default=3600)
parser.add_argument('--pf_table' ,help='pf table to add failed attempts in', default='sshlockout')
inputargs = parser.parse_args()
suspects = dict()
suspects_lastseen = dict()
while True:
rlist, _, _ = select([sys.stdin], [], [], 0.5)
if rlist:
line = sys.stdin.readline()
ip = None
for part in line.split():
if re.match('^[0-9.]+$', part) or re.match('^[a-fA-F0-9:]+$', part):
try:
ip = ipaddress.ip_address(part)
break
except ValueError:
ip = None
if ip:
# cleanup entries after grace period
for cleanup_ip in list(suspects_lastseen):
if time.time() - suspects_lastseen[cleanup_ip] > inputargs.grace_period:
del suspects_lastseen[cleanup_ip]
del suspects[cleanup_ip]
allowed = None
for rule in all_rules:
if re.match(rule, line):
allowed = all_rules[rule]
break
if allowed is True:
# reset counter when login was successful
if ip in suspects:
del suspects[ip]
del suspects_lastseen[ip]
elif allowed is False:
suspects_lastseen[ip] = time.time()
if ip not in suspects:
suspects[ip] = list()
ts = datetime.datetime.strptime(line[0:15].split()[-1], "%H:%M:%S")
if len(suspects[ip]) == 0 or abs((ts - suspects[ip][-1]).total_seconds()) > 2:
# a single attempt can lead to multiple log entries, suppress likely duplicates
suspects[ip].append(ts)
if len(suspects[ip]) > inputargs.attempts:
syslog.syslog(syslog.LOG_NOTICE, "lockout %s [using table %s] after %d attempts" % (
ip, inputargs.pf_table, len(suspects[ip])
))
subprocess.run(['/sbin/pfctl', '-t', inputargs.pf_table, '-T', 'add', str(ip)],
capture_output=True)

View File

@ -0,0 +1,62 @@
#!/usr/local/bin/php
<?php
/*
* Copyright (C) 2020 Deciso B.V.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
* INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
* AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
* OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
require_once("config.inc");
require_once("system.inc");
require_once("util.inc");
require_once("interfaces.inc");
require_once("plugins.inc");
$preserve_logs = !empty($config['syslog']['preservelogs']) ? $config['syslog']['preservelogs'] : 31;
// gather all syslog created local logs (filename starts with directory name_[8 digits].log)
$relevant_logs = [];
$it = new RecursiveDirectoryIterator("/var/log");
foreach(new RecursiveIteratorIterator($it) as $file) {
if ($file->isFile()) {
$log_subject = basename($file->getPath());
if (strpos($file->getFilename(), $log_subject . "_") === 0
&& $file->getExtension() == "log"
&& ctype_digit(substr($file->getFilename(), strlen($log_subject)+1, 8))) {
if (!isset($relevant_logs[$log_subject])) {
$relevant_logs[$log_subject] = [];
}
$relevant_logs[$log_subject][] = (string)$file;
}
}
}
// remove expired logs
foreach ($relevant_logs as $log_subject => $items) {
if (count($items) > $preserve_logs) {
rsort($items);
foreach (array_slice($items, $preserve_logs) as $filename) {
@unlink($filename);
}
}
}

View File

@ -34,7 +34,6 @@ require_once("interfaces.inc");
require_once("plugins.inc");
$opts = getopt('hm:f:', array(), $optind);
if (isset($opts['h']) || empty($opts['f']) || empty($opts['m'])) {
echo "Usage: clearlog [-h] [-m] [-f]\n\n";
echo "\t-h show this help text and exit\n";
@ -46,10 +45,19 @@ if (isset($opts['m']) && isset($opts['f'])) {
$mname = basename($opts['m']);
$fname = basename($opts['f']);
if ($mname == 'core') {
$filename = "/var/log/{$fname}.log";
$basename = "/var/log/{$fname}";
} else {
$filename = "/var/log/{$mname}/{$fname}.log";
$basename = "/var/log/{$mname}/{$fname}";
}
$filename = "{$basename}.log";
if (is_dir($basename)) {
foreach (glob("{$basename}/{$fname}_*.log") as $filename) {
@unlink($filename);
}
system_syslogd_start();
}
if (is_file($filename)) {
$size = filesize($filename);
$handle = fopen($filename, "r");

View File

@ -36,6 +36,7 @@ import re
import sre_constants
import ujson
import datetime
import glob
from logformats import FormatContainer
sys.path.insert(0, "/usr/local/opnsense/site-python")
from log_helper import reverse_log_reader, fetch_clog
@ -54,12 +55,19 @@ if __name__ == '__main__':
result = {'filters': filter, 'rows': [], 'total_rows': 0, 'origin': os.path.basename(inputargs.filename)}
if inputargs.filename != "":
log_filenames = list()
if inputargs.module == 'core':
log_filename = "/var/log/%s.log" % os.path.basename(inputargs.filename)
log_basename = "/var/log/%s" % os.path.basename(inputargs.filename)
else:
log_filename = "/var/log/%s/%s.log" % (
log_basename = "/var/log/%s/%s" % (
os.path.basename(inputargs.module), os.path.basename(inputargs.filename)
)
if os.path.isdir(log_basename):
# new syslog-ng local targets use an extra directory level
for filename in sorted(glob.glob("%s/*.log" % log_basename), reverse=True):
log_filenames.append(filename)
# legacy log output is always stiched last
log_filenames.append("%s.log" % log_basename)
limit = int(inputargs.limit) if inputargs.limit.isdigit() else 0
offset = int(inputargs.offset) if inputargs.offset.isdigit() else 0
@ -73,27 +81,30 @@ if __name__ == '__main__':
# remove illegal expression
filter_regexp = re.compile('.*')
if os.path.exists(log_filename):
format_container = FormatContainer(log_filename)
try:
filename = fetch_clog(log_filename)
except Exception as e:
filename = log_filename
for record in reverse_log_reader(filename):
if record['line'] != "" and filter_regexp.match(('%s' % record['line']).lower()):
result['total_rows'] += 1
if (len(result['rows']) < limit or limit == 0) and result['total_rows'] >= offset:
record['timestamp'] = None
record['parser'] = None
frmt = format_container.get_format(record['line'])
if frmt:
record['timestamp'] = frmt.timestamp(record['line'])
record['line'] = frmt.line(record['line'])
record['parser'] = frmt.name
result['rows'].append(record)
elif result['total_rows'] > offset + limit:
# do not fetch data until end of file...
break
for log_filename in log_filenames:
if os.path.exists(log_filename):
format_container = FormatContainer(log_filename)
try:
filename = fetch_clog(log_filename)
except Exception as e:
filename = log_filename
for record in reverse_log_reader(filename):
if record['line'] != "" and filter_regexp.match(('%s' % record['line']).lower()):
result['total_rows'] += 1
if (len(result['rows']) < limit or limit == 0) and result['total_rows'] >= offset:
record['timestamp'] = None
record['parser'] = None
frmt = format_container.get_format(record['line'])
if frmt:
record['timestamp'] = frmt.timestamp(record['line'])
record['line'] = frmt.line(record['line'])
record['parser'] = frmt.name
result['rows'].append(record)
elif result['total_rows'] > offset + limit:
# do not fetch data until end of file...
break
if result['total_rows'] > offset + limit:
break
# output results
print(ujson.dumps(result))

View File

@ -3,3 +3,5 @@ newsyslog.conf:/etc/newsyslog.conf
syslog-ng.conf:/usr/local/etc/syslog-ng.conf
syslog-ng-legacy.conf:/usr/local/etc/syslog-ng.conf.d/legacy.conf
syslog-ng-destinations.conf:/usr/local/etc/syslog-ng.conf.d/syslog-ng-destinations.conf
syslog-ng-local.conf:/usr/local/etc/syslog-ng.conf.d/syslog-ng-local.conf
syslog-ng-lockout.conf:/usr/local/etc/syslog-ng.conf.d/syslog-ng-lockout.conf

View File

@ -0,0 +1,12 @@
Configuration of local syslog-ng targets.
These configuration files should use a strict format in order to be properly supported and should only contain filter criteria.
The filter clause itself, defines which local log traffic should be captured.
A strict format is required here. f_local_[filename without .conf]. (e.g. f_local_configd for configd.conf)
All of these files are combined into a large configuration file defining a target per local file and a system log
file containing all not catched in the local definitions.
A local target named system is therefor not allowed here.
Underscores will be replaced as path reference, app_file.conf will result in a file /var/log/app/file/file_[XX].log

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [configd].
###################################################################
filter f_local_configd {
program("configd.py");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [dhcpd].
###################################################################
filter f_local_dhcpd {
facility(local7) or program("dhcpd") or program("dhcrelay");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [dnsmasq].
###################################################################
filter f_local_dnsmasq {
program("dnsmasq");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [filter].
###################################################################
filter f_local_filter {
program("filterlog");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [gateways].
###################################################################
filter f_local_gateways {
program("dpinger");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [ipsec].
###################################################################
filter f_local_ipsec {
program("charon");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [lighttpd].
###################################################################
filter f_local_lighttpd {
program("lighttpd");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [ntpd].
###################################################################
filter f_local_ntpd {
program("ntp") or program("ntpd") or program("ntpdate");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [openvpn].
###################################################################
filter f_local_openvpn {
program("openvpn");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [pkg].
###################################################################
filter f_local_pkg {
program("pkg") or program("pkg-static");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [portalauth].
###################################################################
filter f_local_portalauth {
facility(local4) or program("captiveportal");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [ppps].
###################################################################
filter f_local_ppps {
program("ppp");
};

View File

@ -0,0 +1,7 @@
###################################################################
# Local syslog-ng configuration filter definition [relayd].
###################################################################
filter f_local_relayd {
program("haproxy") or
program("relayd");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [resolver].
###################################################################
filter f_local_resolver {
program("unbound");
};

View File

@ -0,0 +1,13 @@
###################################################################
# Local syslog-ng configuration filter definition [routing].
###################################################################
filter f_local_routing {
program("radvd") or
program("routed") or
program("rtsold") or
program("olsrd") or
program("zebra") or
program("ospfd") or
program("bgpd") or
program("miniupnpd");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [squid_access].
###################################################################
filter f_local_squid_access {
program("(squid-1)");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [suricata].
###################################################################
filter f_local_suricata {
program("suricata");
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [vpn].
###################################################################
filter f_local_vpn {
facility(local3);
};

View File

@ -0,0 +1,6 @@
###################################################################
# Local syslog-ng configuration filter definition [wireless].
###################################################################
filter f_local_wireless {
program("hostapd");
};

View File

@ -3,6 +3,7 @@
# send all received local events to platform standard syslogd
#
{% if helpers.empty('syslog.disable_clog') %}
destination legacy_dst {
unix-dgram("/var/run/legacy_log" template("<$PRI>${MSGHDR}${MESSAGE}\n"));
};
@ -11,3 +12,5 @@ log {
source(s_all);
destination(legacy_dst);
};
{% endif %}

View File

@ -0,0 +1,48 @@
{% if not helpers.empty('syslog.disable_clog') %}
{% set all_filters = [] %}
{% for sfilename in helpers.glob("OPNsense/Syslog/local/*.conf") %}{%
include sfilename without context
%} {%
set local_config = sfilename.split('/')[-1].replace('.conf', '')
%} {%
set local_config_filter = "f_local_" + local_config
%} {%
do all_filters.append(local_config_filter)
%}
destination d_local_{{ local_config }} {
file(
"/var/log/{{local_config.replace('_', '/')}}/{{local_config.split('_')[-1]}}_${YEAR}${MONTH}${DAY}.log"
create-dirs(yes)
);
};
log {
source(s_all);
filter({{local_config_filter}});
destination(d_local_{{ local_config }});
};
{% endfor %}
################################################################################
# not captured elsewhere, but relevant, send to system[__].log
################################################################################
filter f_local_system {
not filter({{ all_filters|join(') and not filter(') }})
and level(notice..emerg)
};
destination d_local_system {
file(
"/var/log/system/system_${YEAR}${MONTH}${DAY}.log"
create-dirs(yes)
);
};
log {
source(s_all);
filter(f_local_system);
destination(d_local_system);
};
{% endif %}

View File

@ -0,0 +1,13 @@
filter f_local_lockout_auth {
facility(auth);
};
destination d_local_lockout_auth {
program("/usr/local/opnsense/scripts/syslog/lockout_handler");
};
log {
source(s_all);
filter(f_local_lockout_auth);
destination(d_local_lockout_auth);
};

View File

@ -1,480 +0,0 @@
<?php
/*
* Copyright (C) 2014-2016 Deciso B.V.
* Copyright (C) 2014-2015 Jos Schellevis <jos@opnsense.org>
* Copyright (C) 2009 Jim Pingle <jimp@pfsense.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
* INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
* AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
* OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
require_once("guiconfig.inc");
require_once("system.inc");
require_once("interfaces.inc");
function conv_log_interface_names()
{
global $config;
// collect interface names
$interface_names = array();
$interface_names['enc0'] = gettext("IPsec");
if (!empty($config['interfaces'])) {
foreach (legacy_config_get_interfaces(array("virtual" => false)) as $intfkey => $interface) {
$interface_names[$interface['if']] = !empty($interface['descr']) ? $interface['descr'] : $intfkey;
}
}
return $interface_names;
}
/* format filter logs */
function conv_log_filter($logfile, $nentries, $tail = 50, $filtertext = '', $filterinterface = null)
{
global $config;
/* Make sure this is a number before using it in a system call */
if (!(is_numeric($tail))) {
return;
}
if ($filtertext!=""){
$tail = 5000;
}
/* Always do a reverse tail, to be sure we're grabbing the 'end' of the log. */
$logarr = [];
exec("/usr/local/sbin/clog " . escapeshellarg($logfile) . " | grep -v \"CLOG\" | grep -v \"\033\" | /usr/bin/grep 'filterlog.*:' | /usr/bin/tail -r -n {$tail}", $logarr);
$filterlog = array();
$counter = 0;
$interface_names = conv_log_interface_names();
foreach ($logarr as $logent) {
if ($counter >= $nentries) {
break;
}
$flent = parse_filter_line($logent, $interface_names);
if (isset($flent) && is_array($flent)) {
if ($filterinterface == null || strtoupper($filterinterface) == $flent['interface']) {
if ( (!is_array($filtertext) && match_filter_line ($flent, $filtertext)) ||
( is_array($filtertext) && match_filter_field($flent, $filtertext))
) {
$counter++;
$filterlog[] = $flent;
}
}
}
}
/* Since the lines are in reverse order, flip them around if needed based on the user's preference */
return isset($config['syslog']['reverse']) ? $filterlog : array_reverse($filterlog);
}
function escape_filter_regex($filtertext)
{
/* If the caller (user) has not already put a backslash before a slash, to escape it in the regex, */
/* then this will do it. Take out any "\/" already there, then turn all ordinary "/" into "\/". */
return str_replace('/', '\/', str_replace('\/', '/', $filtertext));
}
function match_filter_line($flent, $filtertext = "")
{
if (!$filtertext) {
return true;
}
$filtertext = escape_filter_regex(str_replace(' ', '\s+', $filtertext));
return @preg_match("/{$filtertext}/i", implode(" ", array_values($flent)));
}
function match_filter_field($flent, $fields) {
foreach ($fields as $key => $field) {
if ($field == "All") {
continue;
}
if ((strpos($field, '!') === 0)) {
$field = substr($field, 1);
if (strtolower($key) == 'act') {
if (in_arrayi($flent[$key], explode(" ", $field))) {
return false;
}
} else {
$field_regex = escape_filter_regex($field);
if (@preg_match("/{$field_regex}/i", $flent[$key])) {
return false;
}
}
} else {
if (strtolower($key) == 'act') {
if (!in_arrayi($flent[$key], explode(" ", $field))) {
return false;
}
} else {
$field_regex = escape_filter_regex($field);
if (!@preg_match("/{$field_regex}/i", $flent[$key])) {
return false;
}
}
}
}
return true;
}
// Case Insensitive in_array function
function in_arrayi($needle, $haystack)
{
return in_array(strtolower($needle), array_map('strtolower', $haystack));
}
function parse_filter_line($line, $interface_names = array())
{
$flent = array();
$log_split = '';
if (!preg_match('/(.*)\s(.*)\sfilterlog.*:\s(.*)$/', $line, $log_split)) {
return '';
}
list($all, $flent['time'], $host, $rule) = $log_split;
if (trim($flent['time']) == '') {
log_error(sprintf('There was an error parsing a rule: no time (%s)', $log_split));
return '';
}
$rule_data = explode(',', $rule);
$field = 0;
$flent['rulenum'] = $rule_data[$field++];
$flent['subrulenum'] = $rule_data[$field++];
$flent['anchor'] = $rule_data[$field++];
$field++; // skip field
$flent['realint'] = $rule_data[$field++];
$flent['interface'] = !empty($interface_names[$flent['realint']]) ? $interface_names[$flent['realint']] : $flent['realint'] ;
$flent['reason'] = $rule_data[$field++];
$flent['act'] = $rule_data[$field++];
$flent['direction'] = $rule_data[$field++];
$flent['version'] = $rule_data[$field++];
if ($flent['version'] != '4' && $flent['version'] != '6') {
log_error(sprintf(
gettext('There was an error parsing rule number: %s -- not IPv4 or IPv6 (`%s\')'),
$flent['rulenum'],
$rule
));
return '';
}
if ($flent['version'] == '4') {
$flent['tos'] = $rule_data[$field++];
$flent['ecn'] = $rule_data[$field++];
$flent['ttl'] = $rule_data[$field++];
$flent['id'] = $rule_data[$field++];
$flent['offset'] = $rule_data[$field++];
$flent['flags'] = $rule_data[$field++];
$flent['protoid'] = $rule_data[$field++];
$flent['proto'] = strtoupper($rule_data[$field++]);
} else {
$flent['class'] = $rule_data[$field++];
$flent['flowlabel'] = $rule_data[$field++];
$flent['hlim'] = $rule_data[$field++];
$flent['proto'] = strtoupper($rule_data[$field++]);
$flent['protoid'] = $rule_data[$field++];
}
$flent['length'] = $rule_data[$field++];
$flent['srcip'] = $rule_data[$field++];
$flent['dstip'] = $rule_data[$field++];
/* bootstrap src and dst for non-port protocols */
$flent['src'] = $flent['srcip'];
$flent['dst'] = $flent['dstip'];
if (trim($flent['src']) == '' || trim($flent['dst']) == '') {
log_error(sprintf(
gettext('There was an error parsing rule number: %s -- no src or dst (`%s\')'),
$flent['rulenum'],
$rule
));
return '';
}
if ($flent['protoid'] == '6' || $flent['protoid'] == '17') { // TCP or UDP
$flent['srcport'] = $rule_data[$field++];
$flent['dstport'] = $rule_data[$field++];
$flent['src'] = $flent['srcip'] . ':' . $flent['srcport'];
$flent['dst'] = $flent['dstip'] . ':' . $flent['dstport'];
$flent['datalen'] = $rule_data[$field++];
if ($flent['protoid'] == '6') { // TCP
$flent['tcpflags'] = $rule_data[$field++];
$flent['seq'] = $rule_data[$field++];
$flent['ack'] = $rule_data[$field++];
$flent['window'] = $rule_data[$field++];
$flent['urg'] = $rule_data[$field++];
$flent['options'] = explode(";",$rule_data[$field++]);
}
} elseif ($flent['protoid'] == '1') { // ICMP
$flent['icmp_type'] = $rule_data[$field++];
switch ($flent['icmp_type']) {
case 'request':
case 'reply':
$flent['icmp_id'] = $rule_data[$field++];
$flent['icmp_seq'] = $rule_data[$field++];
break;
case 'unreachproto':
$flent['icmp_dstip'] = $rule_data[$field++];
$flent['icmp_protoid'] = $rule_data[$field++];
break;
case 'unreachport':
$flent['icmp_dstip'] = $rule_data[$field++];
$flent['icmp_protoid'] = $rule_data[$field++];
$flent['icmp_port'] = $rule_data[$field++];
break;
case 'unreach':
case 'timexceed':
case 'paramprob':
case 'redirect':
case 'maskreply':
$flent['icmp_descr'] = $rule_data[$field++];
break;
case 'needfrag':
$flent['icmp_dstip'] = $rule_data[$field++];
$flent['icmp_mtu'] = $rule_data[$field++];
break;
case 'tstamp':
$flent['icmp_id'] = $rule_data[$field++];
$flent['icmp_seq'] = $rule_data[$field++];
break;
case 'tstampreply':
$flent['icmp_id'] = $rule_data[$field++];
$flent['icmp_seq'] = $rule_data[$field++];
$flent['icmp_otime'] = $rule_data[$field++];
$flent['icmp_rtime'] = $rule_data[$field++];
$flent['icmp_ttime'] = $rule_data[$field++];
break;
default :
if (isset($rule_data[$field++])) {
$flent['icmp_descr'] = $rule_data[$field++];
}
break;
}
} elseif ($flent['protoid'] == '2') { // IGMP
$flent['src'] = $flent['srcip'];
$flent['dst'] = $flent['dstip'];
} elseif ($flent['protoid'] == '112') { // CARP
$flent['type'] = $rule_data[$field++];
$flent['ttl'] = $rule_data[$field++];
$flent['vhid'] = $rule_data[$field++];
$flent['version'] = $rule_data[$field++];
$flent['advskew'] = $rule_data[$field++];
$flent['advbase'] = $rule_data[$field++];
}
return $flent;
}
$filter_logfile = '/var/log/filter.log';
$lines = 5000; // Maximum number of log entries to fetch
$entriesperblock = 10; // Maximum elements to show individually
// flush log file
if (!empty($_POST['clear'])) {
system_clear_clog($filter_logfile);
}
// Retrieve filter log data
$filterlog = conv_log_filter($filter_logfile, $lines, $lines);
// Set total retrieved line counter
$gotlines = count($filterlog);
// Set readable fieldnames
$fields = array(
'act' => gettext("Actions"),
'interface' => gettext("Interfaces"),
'proto' => gettext("Protocols"),
'srcip' => gettext("Source IPs"),
'dstip' => gettext("Destination IPs"),
'srcport' => gettext("Source Ports"),
'dstport' => gettext("Destination Ports"));
$summary = array();
foreach (array_keys($fields) as $f) {
$summary[$f] = array();
}
// Fill summary array with filterlog data
foreach ($filterlog as $fe) {
foreach (array_keys($fields) as $field) {
if (isset($fe[$field])) {
if (!isset($summary[$field])) {
$summary[$field] = array();
}
if (!isset($summary[$field][$fe[$field]])) {
$summary[$field][$fe[$field]] = 0;
}
$summary[$field][$fe[$field]]++;
}
}
}
// Setup full data array for pie and table
function d3pie_data($summary, $num) {
$data=array();
foreach (array_keys($summary) as $stat) {
uasort($summary[$stat], function ($a, $b) {
if ($a == $b) {
return 0;
}
return ($a < $b) ? 1 : -1;
});
$other=0;
foreach(array_keys($summary[$stat]) as $key) {
if (!isset($data[$stat])) {
$data[$stat] = array();
}
if ( count($data[$stat]) < $num ) {
$data[$stat][] = array('label' => $key, 'value' => $summary[$stat][$key]);
} else {
$other+=$summary[$stat][$key];
}
}
if ($other > 0) {
$data[$stat][] = array('label' => gettext("other"), 'value' => $other);
}
}
return $data;
}
include("head.inc"); ?>
<body>
<?php include("fbegin.inc"); ?>
<section class="page-content-main">
<div class="container-fluid">
<div class="row">
<?php print_service_banner('firewall'); ?>
<?php if (isset($input_errors) && count($input_errors) > 0) print_input_errors($input_errors); ?>
<section class="col-xs-12">
<div class="tab-content content-box col-xs-12">
<div class="table-responsive">
<table class="table table-striped">
<tr>
<td>
<strong><?= sprintf(gettext('The following summaries have been collected from the last %s lines of the firewall log (maximum is %s).'), $gotlines, $lines)?></strong>
</td>
<td>
<form method="post">
<div class="pull-right">
<input name="clear" type="submit" class="btn" value="<?= html_safe(gettext('Clear log')) ?>" />
</div>
</form>
</td>
</tr>
</table>
</div>
</div>
</section>
<section class="col-xs-12">
<!-- retrieve full dataset for pie and table -->
<?php $data=d3pie_data($summary, $entriesperblock) ?>
<!-- iterate items and create pie placeholder + tabledata -->
<?php foreach(array_keys($fields) as $field): ?>
<div class="panel panel-default">
<div class="panel-heading"><h3 class="panel-title"><?=$fields[$field]?></h3></div>
<div class="panel-body">
<div class="piechart" id="<?=$field?>">
<svg></svg>
</div>
<table class="table table-striped table-bordered">
<tr>
<th><?=$fields[$field]?></th>
<th><?=gettext("Count");?></th>
</tr>
<?php if (isset($data[$field])):?>
<?php foreach(array_keys($data[$field]) as $row): ?>
<tr>
<td>
<?php if (is_ipaddr($data[$field][$row]["label"])): ?>
<a href="diag_dns.php?host=<?=$data[$field][$row]["label"]?>" title="<?=gettext("Reverse Resolve with DNS");?>"><i class="fa fa-search"></i></a>
<?php endif ?>
<?=$data[$field][$row]["label"]?></td>
<td><?=$data[$field][$row]["value"]?></td>
</tr>
<?php endforeach ?>
<?php endif; ?>
</table>
</div>
</div>
<?php endforeach ?>
</section>
</div>
</div>
</section>
<script>
// Generate Donut charts
nv.addGraph(function() {
// Find all piechart classes to insert the chart
$('div[class="piechart"]').each(function(){
var selected_id = $(this).prop("id");
var chart = nv.models.pieChart()
.x(function(d) { return d.label })
.y(function(d) { return d.value })
.showLabels(true) //Display pie labels
.labelThreshold(.05) //Configure the minimum slice size for labels to show up
.labelType("percent") //Configure what type of data to show in the label. Can be "key", "value" or "percent"
.donut(true) //Turn on Donut mode. Makes pie chart look tasty!
.donutRatio(0.2) //Configure how big you want the donut hole size to be.
;
d3.select("[id='"+ selected_id + "'].piechart svg")
.datum(<?= json_encode($data) ?>[selected_id])
.transition().duration(350)
.call(chart);
// Update Chart after window resize
nv.utils.windowResize(function(){ chart.update(); });
return chart;
});
});
</script>
<style>
.piechart svg {
height: 400px;
}
</style>
<?php
include("foot.inc");

View File

@ -37,46 +37,14 @@ require_once("system.inc");
function clear_all_log_files()
{
killbyname('syslogd');
$clog_files = array(
'dhcpd',
'configd',
'filter',
'gateways',
'ipsec',
'l2tps',
'lighttpd',
'mail',
'ntpd',
'openvpn',
'pkg',
'poes',
'portalauth',
'ppps',
'pptps',
'relayd',
'resolver',
'routing',
'suricata',
'system',
'vpn',
'wireless',
);
$log_files = array(
'squid/access',
'squid/cache',
'squid/store',
);
foreach ($clog_files as $lfile) {
system_clear_clog("/var/log/{$lfile}.log", false);
$it = new RecursiveDirectoryIterator("/var/log");
foreach(new RecursiveIteratorIterator($it) as $file) {
if ($file->isFile() && strpos($file->getFilename(), '.log') > -1) {
if (strpos($file->getFilename(), 'flowd') === false) {
@unlink((string)$file);
}
}
}
foreach ($log_files as $lfile) {
system_clear_log("/var/log/{$lfile}.log", false);
}
system_syslogd_start();
plugins_configure('dhcp');
}
@ -91,7 +59,9 @@ function is_valid_syslog_server($target) {
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
$pconfig = array();
$pconfig['reverse'] = isset($config['syslog']['reverse']);
$pconfig['disable_clog'] = isset($config['syslog']['disable_clog']);
$pconfig['logfilesize'] = !empty($config['syslog']['logfilesize']) ? $config['syslog']['logfilesize'] : null;
$pconfig['preservelogs'] = !empty($config['syslog']['preservelogs']) ? $config['syslog']['preservelogs'] : null;
$pconfig['logdefaultblock'] = empty($config['syslog']['nologdefaultblock']);
$pconfig['logdefaultpass'] = empty($config['syslog']['nologdefaultpass']);
$pconfig['logbogons'] = empty($config['syslog']['nologbogons']);
@ -113,13 +83,26 @@ if ($_SERVER['REQUEST_METHOD'] === 'GET') {
$input_errors[] = gettext("Log file size must be a positive integer greater than 5120.");
}
}
if (!empty($pconfig['preservelogs']) && (strlen($pconfig['preservelogs']) > 0)) {
if (!is_numeric($pconfig['preservelogs'])) {
$input_errors[] = gettext("Preserve logs must be a positive integer value.");
}
}
if (count($input_errors) == 0) {
$config['syslog']['reverse'] = !empty($pconfig['reverse']) ? true : false;
$config['syslog']['reverse'] = !empty($pconfig['reverse']);
$config['syslog']['disable_clog'] = !empty($pconfig['disable_clog']);
if (isset($_POST['logfilesize']) && (strlen($pconfig['logfilesize']) > 0)) {
$config['syslog']['logfilesize'] = (int)$pconfig['logfilesize'];
} elseif (isset($config['syslog']['logfilesize'])) {
unset($config['syslog']['logfilesize']);
}
if (isset($_POST['preservelogs']) && (strlen($pconfig['preservelogs']) > 0)) {
$config['syslog']['preservelogs'] = (int)$pconfig['preservelogs'];
} elseif (isset($config['syslog']['preservelogs'])) {
unset($config['syslog']['preservelogs']);
}
$config['syslog']['disablelocallogging'] = !empty($pconfig['disablelocallogging']);
$oldnologdefaultblock = isset($config['syslog']['nologdefaultblock']);
$oldnologdefaultpass = isset($config['syslog']['nologdefaultpass']);
@ -185,6 +168,17 @@ $(document).ready(function() {
}]
});
});
$("#disable_clog").change(function(){
if ($(this).is(":checked")) {
$("#preservelogs").prop("disabled", false).closest("tr").removeClass("hidden");
$("#logfilesize").prop("disabled", true).closest("tr").addClass("hidden");
} else {
$("#preservelogs").prop("disabled", true).closest("tr").addClass("hidden");
$("#logfilesize").prop("disabled", false).closest("tr").removeClass("hidden");
}
});
$("#disable_clog").change();
});
//]]>
@ -223,10 +217,28 @@ $(document).ready(function() {
</div>
</td>
</tr>
<tr>
<td><a id="help_for_circular_logs" href="#" class="showhelp"><i class="fa fa-info-circle"></i></a> <?=gettext("Disable circular logs");?></td>
<td>
<input name="disable_clog" type="checkbox" id="disable_clog" value="yes" <?=!empty($pconfig['disable_clog']) ? "checked=\"checked\"" : ""; ?> />
<div class="hidden" data-for="help_for_circular_logs">
<?=gettext("Disable legacy circular logging");?>
</div>
</td>
</tr>
<tr class="hidden">
<td><a id="help_for_preservelogs" href="#" class="showhelp"><i class="fa fa-info-circle"></i></a> <?=gettext('Preserve logs (Days)') ?></td>
<td>
<input name="preservelogs" id="preservelogs" type="text" value="<?=$pconfig['preservelogs'];?>" />
<div class="hidden" data-for="help_for_preservelogs">
<?=gettext("Number of log to preserve. By default 31 logs are preserved.");?>
</div>
</td>
</tr>
<tr>
<td><a id="help_for_logfilesize" href="#" class="showhelp"><i class="fa fa-info-circle"></i></a> <?=gettext('Log File Size (Bytes)') ?></td>
<td>
<input name="logfilesize" type="text" value="<?=$pconfig['logfilesize'];?>" />
<input name="logfilesize" id="logfilesize" type="text" value="<?=$pconfig['logfilesize'];?>" />
<div class="hidden" data-for="help_for_logfilesize">
<?=gettext("Logs are held in constant-size circular log files. This field controls how large each log file is, and thus how many entries may exist inside the log. By default this is approximately 500KB per log file, and there are nearly 20 such log files.") ?>
<br /><br />