lucaspalomodevelop/core
1
0
Fork 0
mirror of https://github.com/lucaspalomodevelop/core.git synced 2026-03-18 02:25:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
15,153 Commits 1 Branch 0 Tags
Commit Graph

100 Commits

Author SHA1 Message Date
Josh Soref
20a3c3da38
Spelling (#5885) 2022-07-18 13:59:03 +02:00
Ad Schellevis
edd9b6c433 Netflow aggregator: skip invalid record, another one for https://github.com/opnsense/core/issues/4796 2021-03-10 16:40:24 +01:00
Ad Schellevis
c0e4f8d683 Netflow aggregator: skip invalid record, closes https://github.com/opnsense/core/issues/4796 2021-03-10 15:56:18 +01:00
Ad Schellevis
43b50ed086 Netflow: prevent crash when attributes missing, closes https://github.com/opnsense/core/issues/4751 2021-02-25 12:08:46 +01:00
Ad Schellevis
88e463c913 Netflow: prevent crash when interface number is missing. for https://github.com/opnsense/plugins/issues/2241 2021-02-19 23:16:50 +01:00
Ad Schellevis
6cb0f29268 flowd_aggregator. better replace line endings when sending traceback to syslog. 2020-08-14 09:50:16 +02:00
Ad Schellevis
0c3359689d netflow: flow parser, when flow times are not returned (intermediate data?), stick to receive timestamp. 
Setting both flow_start and flow_finished to system uptime, compensates to 0 for duration and flow length offset.

closes https://github.com/opnsense/core/issues/4046
 2020-04-10 06:09:57 +02:00
Ad Schellevis
01d8fb296e Insight, catch struct.unpack errors instead od rashing out when flow record doesn't match standards. closes https://github.com/opnsense/core/issues/3715 2019-09-23 18:07:41 +02:00
Ad Schellevis
d8ef93932b netflow: netflow aggregator, skip illegal flow records. https://forum.opnsense.org/index.php?topic=13711 2019-08-17 19:09:11 +02:00
Ad Schellevis
cf3b447275 python3: replace subprocess.call in src/opnsense/scripts/netflow/* for https://github.com/opnsense/core/issues/3574 2019-07-23 18:40:14 +02:00
Ad Schellevis
9287b5588e Insight, missed a line in fabaef0a43 for https://github.com/opnsense/core/issues/3587 2019-07-22 09:14:54 +02:00
Ad Schellevis
6b1f3e60b9 Insight, increase wait time between parses. ref https://github.com/opnsense/core/issues/3587 
previously we always waited 15 seconds between cycles, now we calculcate the time to wait with the time spend for the previous cycle.
 2019-07-19 18:18:11 +02:00
Ad Schellevis
fabaef0a43 Insights, small optimalisation for flowd_aggregate.py, pass last received timestamp to flowparser, so we can skip a bit of processing when the data isn't relevant. for https://github.com/opnsense/core/issues/3587 2019-07-19 17:44:44 +02:00
Ad Schellevis
24dc2a82b5 Insight. missing decode() when parsing ifconfig output leading to unexpected results. sqlite seems to store the binary in a varchar element, which looks right, but you can't compare it in your where statements. This doesn't seem to lead to issues when there is already valid content in the timeserie table. 2019-07-12 08:37:46 +02:00
Franco Fichtner
4175a45f4b system: fix netflow lib permission 2019-05-20 13:24:38 +02:00
Ad Schellevis
f086f8c48d flowd aggregate, minor bug in commit counter, leading to single row commits after row # 100k 2019-05-20 11:42:00 +02:00
Ad Schellevis
28ed5741f4 netflow, make sure get_timeserie_data() returns string type objects in stead of bytes, to prevent 17e4e9c0fc (commitcomment-33587137) 
we might find some other small type interpretation differences, but the bumpiest seems to be gone now.
 2019-05-18 09:23:25 +02:00
Ad Schellevis
2bdc74b8a1 netflow, switch dump_log.py script to python3 2019-05-18 09:22:29 +02:00
Ad Schellevis
3d1617b2bd netflow, minor type issue, parameter received as bytes in get_timeseries.py 2019-05-17 16:50:32 +02:00
Ad Schellevis
31982e2872 netflow, migrate flowd aggregator to python 3 2019-05-17 16:41:08 +02:00
Ad Schellevis
17e4e9c0fc netflow, aggregator replace flowd with our new implementation 2019-05-16 18:32:55 +02:00
Ad Schellevis
8aecf38f2e netflow, log parser. for backwards compatibility add copy of recv_sec field in recv 2019-05-16 18:29:16 +02:00
Ad Schellevis
5eef7248a8 netflow, flow log parser improvements 
- faster / cleaner ipv4 conversion
- ipv6 conversion in compressed format, which equals flowd previous output
- unpack source and dest ports
 2019-05-16 16:20:34 +02:00
Ad Schellevis
0bdbdf7d21 netflow, improve log parser performance, cache some calculations. 
In theory we could probably increase performance even more by limitting the number of calls to struct.unpack, but this will make it more difficult to read.
This version is about 30% faster then the original C version shipped with flowd.

Some more testing todo, but preliminary results look good.
 2019-05-16 13:54:24 +02:00
Ad Schellevis
93f7cff3d8 netflow, flowparser.py performance improvement in parsing ipv4/ipv6 addresses. 2019-05-15 22:01:27 +02:00
Ad Schellevis
a99d1ecf8e netflow, minor type issue in previous 2019-05-14 22:08:52 +02:00
Ad Schellevis
7e55ef5218 netflow, work in progress pure python replacement to parse /var/log/flowd.log files 2019-05-14 22:06:53 +02:00
Ad Schellevis
a06ad6a5d0 netflow, convert parts that don't depend on flowd to python 3 2019-05-13 15:48:33 +02:00
Ad Schellevis
f4ebba628f netflow aggregate, prevent cleanup deleting current data when there's data recorded in the future. 
Noticed on a test machine at the office, recorded some bytes at a timestamp in the future, which resulted in the current 2 hour range left empty....
 2019-02-01 17:33:30 +01:00
Ad Schellevis
defd8da1b1 flowd, dump our parsed log for debugging purposes 2019-02-01 16:38:03 +01:00
Franco Fichtner
86c7f6242c src: fix lint-exec pass 2019-01-14 08:14:06 +01:00
Ad Schellevis
fad4b3c426 Network insight, resample data for get_timeseries(), closes https://github.com/opnsense/core/issues/2895 
This commit samples data for all requested fractions, which should improve the missing data output.
 2018-11-30 16:59:06 +01:00
Ad Schellevis
336f2e78c5 missed a spot in previous 2018-11-29 15:59:26 +01:00
Ad Schellevis
508e892880 Network insight, python2/3 compatibility for get_timeseries.py 2018-11-29 15:57:20 +01:00
Ad Schellevis
b6a7b7f6eb Network insight, cmd_args.key_fields missing split(','), for https://github.com/opnsense/core/issues/2964 2018-11-29 13:53:51 +01:00
Ad Schellevis
711e1d9da4 Netflow insight, regressions in rotate(), for https://github.com/opnsense/core/issues/2964 2018-11-29 13:36:51 +01:00
Ad Schellevis
2481b62ca8 Network insight, some 2to3 magic, closes https://github.com/opnsense/core/issues/2964 2018-11-29 13:08:12 +01:00
Ad Schellevis
7841325609 Network insight, prevent undefined selected_resolution, set last item as default. for https://github.com/opnsense/core/issues/2964 2018-11-27 18:05:38 +01:00
Ad Schellevis
5b6289db3d Network insight, refactor get_top_usage.py for https://github.com/opnsense/core/issues/2964 2018-11-27 17:58:07 +01:00
Ad Schellevis
d8118a3004 more cleanups and alignments for https://github.com/opnsense/core/issues/2964 2018-11-27 15:27:49 +01:00
Ad Schellevis
813595af4f fix previous, https://github.com/opnsense/core/issues/2964 2018-11-27 12:50:45 +01:00
Ad Schellevis
8fdaf264de Network insight, support custom configuration in flowd_aggregate_metadata.py for https://github.com/opnsense/core/issues/2964 2018-11-27 10:43:16 +01:00
Ad Schellevis
4e69e5923b Network insight, cleanup. -c vs --config for https://github.com/opnsense/core/issues/2964 2018-11-27 10:30:06 +01:00
Franco Fichtner
3ad1be040c src: style and whitespace sweep 2018-11-25 12:10:03 +01:00
Ad Schellevis
1680c1e1b0 Network insight, aggregator support single_pass (exit when there's no new data left), for https://github.com/opnsense/core/issues/2964 2018-11-24 21:40:13 +01:00
Ad Schellevis
c7660ca43f Network insight, another minor side affect of https://github.com/opnsense/core/issues/2964 export misses timestamp convert. 2018-11-24 16:10:09 +01:00
Ad Schellevis
9222f5e4de fix regression in previous, for https://github.com/opnsense/core/issues/2964 2018-11-24 16:03:06 +01:00
Ad Schellevis
02c5bb692d Network insight, missing import for https://github.com/opnsense/core/issues/2964 2018-11-24 15:47:50 +01:00
Ad Schellevis
dc38875202 Network insight, cleanups and restructures for https://github.com/opnsense/core/issues/2964 
Adds configuration yaml support, using a simple config yaml containing the previously hardcoded values:

```
library_path:  /path/to/core/src/opnsense/site-python
flowd_source: /path/to/flowd.log
database_dir: /path/to/netflow
pid_filename: /path/to/netflow.pid
```

Startup using local config, using:

```
flowd_aggregate.py -c my_config.yaml --console
```

Further then path relocations, this commit should not contain any new features. The other scripts, not related to the daemon have not been altered (yet)
 2018-11-24 15:44:22 +01:00
Ad Schellevis
bb90be17c7 Network insight, possible division by zero, closes https://github.com/opnsense/core/issues/2928 2018-11-18 18:20:13 +01:00