- Increase the reporting interval for 12 & 24 hours to 10 minutes.
- Always display a list for the top (blocked) domains
- Remove block/whitelist buttons when blocklists are disabled
This include the ability to whitelist it from the same page as well. Relevant to both
the top passed/blocked domains, as well as the detailed query grid.
blocklists.py has been modified in such a way that it will detect whether it needs to
start the download process or simply administrate locally. The latter currently only happens
when custom domains for blocking have been added/removed by a user. The reasoning is
that we can easily extend/shrink the current blocklist when it comes to blocking exact
domains as this is handled on the incoming side. However, while we can modify the current
list to accomodate a new whitelist entry (which can be regex), we (currently) cannot know which domains
were skipped in the process of retrieving them in the first place if a user explicitly removes
a whitelist entry. Therefore we decide to re-run the download on a whitelist action.
furthermore, the updateBlocklistAction in the controller administrates how the model is updated
(e.g. when a blocked item is whitelisted, it should be removed from the blocklist model entry and
added to the whitelist)
In the future we could optimize the whole process by checking if a remote file has changed in
date or size.
* unbound / overview: migrate to duckdb
* unbound: make ajax calls concurrent, no need to wait on each other
* unbound / overview: adjust to DataFrame optimization
While previous insert statements with SQLite had to be rate limited to prevent blocking on the
side of the logger, this limitation can be ditched entirely with the approach of appending DataFrames.
Also, the client chart expects timestamps for the entirety of the dataset, so make use of
the NaN value.
* unbound / overview: connection can be none
* unbound / overview: include DNSSEC and timing information and refactor logger
The logger was lacking some error handling due to it being daemonized. On a restart the process
will still produce a core dump likely associated with daemonization, but this will be addressed in a separate issue
To accomodate future additions, DNSSEC and RTT information is now also included in the data set.
* remove leftover syslog message
* unbound / overview: add UUID as well
* unbound / overview: account for null return_msg objects
* unbound / overview: Move DNS statistics to reporting->dns, add detailed grid
This also introduces the option to clear the DNS data. Also restructure the data to a more sensible flow representation.
Add new component to manage IPsec connections in a similar format as `swanctl.conf` is defined (https://docs.strongswan.org/docs/5.9/swanctl/swanctlConf.html). As this needs to work in conjunction with the legacy IPsec module, some minor changes are needed to the current state.
o VPN/IPsec/Pre-Shared Keys - add optional remote identifier (merges in `ipsec.inc`)
o VPN/IPsec/Virtual Tunnel Interfaces - new component to show existing VTI's and add new ones (as these are separate entities)
o VPN/IPsec/Connections [new] - configuration tool to build `swanctl.conf`
o Integrate MVC generated `swanctl.conf` into `ipsec.inc` (legacy overlays)
o Integrate manually configured VTI's into `ipsec.inc` (`array_merge(ipsec_get_configured_vtis(), (new \OPNsense\IPsec\Swanctl())->getVtiDevices())`)
o fix minor php warning when changing reqid's (`$local|remote_configured` initialisation when `$configured_intf[$intf]` not found)
As the output of pfctl -vvss can grow quite rapidly, it seemed like a good idea to run this code through a profiler. Some of the hotspots (like parsing addresses) are now cached in memory to prevent over enthusiastic computation, which can save quite some processing time. Pushing down the string join on which the pattern search should match does help prevent to prevent compiling a search string which turns out to be irrelevant later (no filter or ip[+port] filter).
The network (address) search handles (optional) ports as well now, which allows for patterns like `10.0.0.1:80` and `10.0.0.0/24:80`.
Since rule labels are directly hooked to the modification time of /tmp/rules.debug, we don't have to recalculate the offsets on every request. This patch saves the labels and the last modification time so we can refresh when changed or non-existent.
As the client still might have a state when being kicked-out, we should kill any state the client has while adding it to the alias. Apparantly our ssh messages are only catched partially, so add ".*Authentication error for .*" to the list as well. To ease testing, better detect the location of the timestamp so we can use a construction like this to feed amn existing log:
lockout_handler < /var/log/audit/audit_20221205.log
* unbound / overview: add client activity chart, include logarithmic scale on both charts
* unbound: remove tooltip from workaround datapoints
* unbound / overview: also end the x axis properly on log scale
* unbound / overview: prevent queries on non-existing database
* unbound / overview: replace cache hits with blocklist size and update icons
* unbound / overview: refactor rolling API call
Adds a reporting backend for Unbound as well as a dashboard showing an overview of interesting statistics:
Total DNS queries (over a maximum period of 7 days)
Blocked queries
Cache hits
local-data hits
A traffic graph with variable intervals
top N passed/blocked domains (with associated blocklist)
There are a couple of hardcoded values:
The underlying database is kept up-to-date for the last 7 days.
The ring buffer used to decouple unbound from the logging backend has a size of 100000 entries.
In very busy networks, db commits are done for a max of 4000 entries, which according to testing seems to be a good break-even point for DNS throughput vs. insert performance. Everything below this is flushed to the database every 5 seconds.
To facilitate the variable interval traffic graph, multiple views with varying detail in time slots are created: 1, 5 and 30 minute buckets. These can easily be extended should different intervals be necessary.
closes#5984 refactors legacy pages, includes the following:
o remove type field as this seems to be redundant and confusing
o input form additions (show hide related fields)
o add button for carp type to select first unused vhid
o implement configure action, caching removed addresses in /tmp/delete_vip_{$uuid}.todo files (by the controller)
o add mode filter to search action and complete with relevant fields for our grid
o fix warning in interfaces.inc (interface_proxyarp_configure()), array creation issue
o add validation for addresses used in port forwards and outbound nat rules. previous version tried to rename forwards, we choose to be consistent when it comes to edit/delete.
o change ACL to use the new endpoints, remove "show only" ACL. we can always consider putting it back later, but the experience of only able to reach the grid likely won't be practical.
o remove old firewall_virtual_ip*.php files
When we move to the next major the file contents for
the txz are all the same but their signature is not.
So until the next changelog sync was carried out the
CHECKSUM matched and the signature will stay stale.
o make sure we are not going to exit the start script if rm fails for whatever reason
o add stop action which stops the service and unmounts volumes mounted by start, the current state only kills unbounds primary pid (without optional dhcp integration or mountpoints)
Replaces the current blocklist implementation to use python instead of relying on unbound-control. The latter had the drawback of a very long execution time to administrate the local-data entries both locally and in Unbound. The memory footprint was also considerably larger due to unbound internals, while the python module keeps it all in memory in a simple dictionary - reducing the total amount of memory consumption by more than a factor of 10. A drawback is a potential decrease in performance of ~15%, although most setups shouldn't be affected by this as most hardware which is capable of running this should be scaled towards its intended use.
The option of returning NXDOMAIN has also been added (fixes#6027), which in this implementation is a lot easier than what we would have to do if local-data were to be used.
* VPN/IPsec - work in progress refactoring ipsec.conf to swanctl.conf for https://github.com/opnsense/core/issues/5636
Wrap the following blocks in functions:
o generation of strongswan.conf
o write and cleanup IPsec's CA database
o write certificates used by IPsec
o write RSA Key Pairs
o construct ipsec.secrets for swanctl, combining different sources, phase1 PSK's, certificates, keypairs
o replace VPN: IPsec: Lease Status with new mvc version that uses swanctl underneath
Remap ipsec.conf to swanctl.conf attributes using https://wiki.strongswan.org/projects/strongswan/wiki/Fromipsecconf
Remove lifetime defaults from phase 1 and 2 so new connections will use Strongswan's defaults and move the option in phase above Margintime/Rekeyfuzz as these combined are being migrated (see margintime in Fromipsecconf doc).
Restructure used path directives so we can use the default structure as descried in https://docs.strongswan.org/docs/5.9/swanctl/swanctlDir.html and installed via ports.
IPsec status overview needs a small fix as well (passthrough network seems to be returned by vici, but doesn't contain a class (which crashes list_status.py).)
While here, also remove some php warnings.
6255148,en,EU,Europe,,,0
Although the same applies for Asia, we can't use AS as country code here, so we do have to skip this for now until we find a better alternative.
6255147,en,AS,Asia,,,0
closes https://github.com/opnsense/core/issues/6063
Try to avoid touching default routes in dhclient-script which we
already partially do now. PPPoE linkdown certainly doesn't need
to handle the file as well to remove the default route then.
