* unbound / overview: migrate to duckdb
* unbound: make ajax calls concurrent, no need to wait on each other
* unbound / overview: adjust to DataFrame optimization
While previous insert statements with SQLite had to be rate limited to prevent blocking on the
side of the logger, this limitation can be ditched entirely with the approach of appending DataFrames.
Also, the client chart expects timestamps for the entirety of the dataset, so make use of
the NaN value.
* unbound / overview: connection can be none
* unbound / overview: include DNSSEC and timing information and refactor logger
The logger was lacking some error handling due to it being daemonized. On a restart the process
will still produce a core dump likely associated with daemonization, but this will be addressed in a separate issue
To accomodate future additions, DNSSEC and RTT information is now also included in the data set.
* remove leftover syslog message
* unbound / overview: add UUID as well
* unbound / overview: account for null return_msg objects
* unbound / overview: Move DNS statistics to reporting->dns, add detailed grid
This also introduces the option to clear the DNS data. Also restructure the data to a more sensible flow representation.
Adds a reporting backend for Unbound as well as a dashboard showing an overview of interesting statistics:
Total DNS queries (over a maximum period of 7 days)
Blocked queries
Cache hits
local-data hits
A traffic graph with variable intervals
top N passed/blocked domains (with associated blocklist)
There are a couple of hardcoded values:
The underlying database is kept up-to-date for the last 7 days.
The ring buffer used to decouple unbound from the logging backend has a size of 100000 entries.
In very busy networks, db commits are done for a max of 4000 entries, which according to testing seems to be a good break-even point for DNS throughput vs. insert performance. Everything below this is flushed to the database every 5 seconds.
To facilitate the variable interval traffic graph, multiple views with varying detail in time slots are created: 1, 5 and 30 minute buckets. These can easily be extended should different intervals be necessary.
closes#5984 refactors legacy pages, includes the following:
o remove type field as this seems to be redundant and confusing
o input form additions (show hide related fields)
o add button for carp type to select first unused vhid
o implement configure action, caching removed addresses in /tmp/delete_vip_{$uuid}.todo files (by the controller)
o add mode filter to search action and complete with relevant fields for our grid
o fix warning in interfaces.inc (interface_proxyarp_configure()), array creation issue
o add validation for addresses used in port forwards and outbound nat rules. previous version tried to rename forwards, we choose to be consistent when it comes to edit/delete.
o change ACL to use the new endpoints, remove "show only" ACL. we can always consider putting it back later, but the experience of only able to reach the grid likely won't be practical.
o remove old firewall_virtual_ip*.php files
Replaces the current blocklist implementation to use python instead of relying on unbound-control. The latter had the drawback of a very long execution time to administrate the local-data entries both locally and in Unbound. The memory footprint was also considerably larger due to unbound internals, while the python module keeps it all in memory in a simple dictionary - reducing the total amount of memory consumption by more than a factor of 10. A drawback is a potential decrease in performance of ~15%, although most setups shouldn't be affected by this as most hardware which is capable of running this should be scaled towards its intended use.
The option of returning NXDOMAIN has also been added (fixes#6027), which in this implementation is a lot easier than what we would have to do if local-data were to be used.
* VPN/IPsec - work in progress refactoring ipsec.conf to swanctl.conf for https://github.com/opnsense/core/issues/5636
Wrap the following blocks in functions:
o generation of strongswan.conf
o write and cleanup IPsec's CA database
o write certificates used by IPsec
o write RSA Key Pairs
o construct ipsec.secrets for swanctl, combining different sources, phase1 PSK's, certificates, keypairs
o replace VPN: IPsec: Lease Status with new mvc version that uses swanctl underneath
Remap ipsec.conf to swanctl.conf attributes using https://wiki.strongswan.org/projects/strongswan/wiki/Fromipsecconf
Remove lifetime defaults from phase 1 and 2 so new connections will use Strongswan's defaults and move the option in phase above Margintime/Rekeyfuzz as these combined are being migrated (see margintime in Fromipsecconf doc).
Restructure used path directives so we can use the default structure as descried in https://docs.strongswan.org/docs/5.9/swanctl/swanctlDir.html and installed via ports.
IPsec status overview needs a small fix as well (passthrough network seems to be returned by vici, but doesn't contain a class (which crashes list_status.py).)
While here, also remove some php warnings.
Small addition to https://github.com/opnsense/core/pull/6050:
- Move private/insecure domains to advanced as its intended use has a larger scope than DNSBLs only. Fixes https://github.com/opnsense/core/issues/5256
- Extends the migration to also include these domains.
- leftover cleanup of legacy settings in migration.
- Adds the `serve-expired-reply-ttl`, `serve-expired-ttl`, `serve-expired-ttl-reset`, `serve-expired-client-timeout` options. These options are hidden until the `serve-expired` checkbox is checked, and indented to signify a relationship.
- Removes all dropdowns and instead provides numeric fields to input raw values for more control and less "guessing" of what is acceptable.
- Removes default settings to prevent mismatches with upstream in the future. It's probably best to refer to the Unbound documentation in our own documentation.
- Previously, `rrset-cache-size` and `outgoing-range` were implicitly set. These are now input fields. The migration code will take care of legacy setting assumptions.
Fixes https://github.com/opnsense/core/issues/5978
Fixes https://github.com/opnsense/core/issues/5795
* MVC / System status: first draft for backend implementation
* fix copyright
* fix permissions
* MVC / System Status: modify backend implementation and setup front-end
* MVC / System Status: minor cleanup, ACL check and fix reporting in production mode
* MVC / System Status: copy status sytem to legacy as well, remove the notices system, finish up front-end work
* MVC / System Status: remove useless constructor
* fix plist
* System Status: shorten previous
* System Status: add ACL check
* System Status: also remove legacy part
* System Status: also clean up on legacy page
* System Status: ACL check on dismiss action as well
* System Status: add readonly privilege check to dismiss action
* System Status: do not trust input
* System Status: address security concerns
* add default return
* System Status: move js code to separate script and make sure a logLocation is always provided
* System Status: clean up callout in both legacy page and volt template
After dismissing a message and closing the dialog, the old message was still bound to the dialog instance. re-registering the onclick callback solves this.
* System Status: adjust log location and kick off status system on alias errors
* System Status: let's hold off on throwing notifications for aliases for now
* System Status: add ACL entry for the dismiss API call, adjust to new deployment situation
Without the ACL entry, /api/core/system/dismissStatus calls are rejected for non-root users even when they should have rights to dismiss.
Also do a minor consistency improvement and also adjust to the new situation of production & development deployment types. We need to account for the possibility of 'deployment' being empty in the configuration, therefore a direct check of the 'development' type seems most fitting as this is unlikely to change or be subjected to any additions.
* System Status: also error out when unable to write new rules
* System Status: account for users without permissions
* System Status: name collision in FirmwareController
* System Status: replace old notices system with a global one (https://github.com/opnsense/core/pull/5875)
Review feedback / modifications in this commit:
o filter.inc
-- remove wedged message, when locked during parallel reloads it likely doesn't help to disable/enable
-- flush message to error trigger file
o SystemStatus.php
-- str_contains --> strpos; eases testing on OPNsense 22.1.x as str_contains is php 8 only
o Status collectors
-- simplify logic and propagate messages received from status file
o CrashReporterStatus
- the existence of a /tmp/PHP_errors.log file should be enough to know some process signaled the crash reporter
- remove shell exec
o FirewallStatus
-- as only /tmp/rules.error remains, remove loop to read for non existing files
o opnsense_status.js
-- add opn-status-group class to container and point css modifications in that single direction to prevent other objects from being affected by our status popup modifications
-- windows file endings replaced (^M)
o css:
keep menu_messages container to ease migration for theme developers
NOTE : ** get_crash_report() could likely be simplified as well
* filter: change to mwexec, redirection is implied
* System Status: handle potential undefined array key
seems to be a one-off error: Exception: Error at /usr/local/opnsense/mvc/app/controllers/OPNsense/Core/Api/SystemController.php:90 - Undefined array key 0 (errno=2) in /usr/local/opnsense/mvc/app/controllers/OPNsense/Base/ApiControllerBase.php:159
Co-authored-by: Ad Schellevis <ad@opnsense.org>
Since "nameserver.sh" was a working title for when we just wanted
to make sense of the nameserver registration mess going on this
no longer fully applies for good reasons.
Change the name to ifctl.sh for leaving syntax parsing in place
and leave the symlink in sbin/ directory. The same works nicely
for configctl.
New script to be invoked by rtsold when Router Advertisements with
RDNSS / DNSSL information are received. Uses ifctl to create the
/tmp/$if_routerv6 file and creates the /tmp/$if_defaultgwv6 file
directly. Fixes the issue that these files don't get created when
the M and O flags in RAs are not set. Also, passes RDNSS / DNSSL
info from RAs to ifctl.
Gets rid of package dependency which for dhcpleases6 simply
listens on the DHCPD lease file and runs the prefix.php command
unconditionally.
For now emulate this by issuing the command every 60 seconds
which can be adjusted later if someone complains, but seeing
how many bugs this script has had vs. how many people noticed
it (basically none) I doult this will have much impact overall.
Revert previous here. Debug mode is "raw" error display
mode and non-debug forwards to crash reporter.
Crash reporter itself could be broken, but mostly because
the system / include chain is broken. Until PHP offers
a reliable way to intercept parse error 500 we will have
to live with this unless it causes other side effects.
