Comparative Evaluation of Log Reduction Techniques Using Vector on Public Security Datasets

Abstract

Efficient log reduction is critical for Security Operations Centers (SOCs) and Managed Security Service Providers (MSSPs), which must store, an-alyze, and retain massive volumes of event data while satisfying compli-ance requirements and controlling operational costs. Traditional pipelines often retain redundant or low-value records, leading to excessive storage overhead and slower analytics. This study evaluates five Vector-based log-reduction methods: filter-based selection, field pruning, event sampling, template hashing, and a combined pruning + sampling profile. The eval-uation uses more than 3 million log records from two well-known public intrusion datasets, CIC-IDS2017 and UNSW-NB15, to measure efficiency, throughput, and attack coverage under the same experimental setup. Com-pared with a baseline Filebeat pipeline, the proposed Vector-based ap-proach improved throughput by 45%, reduced outbound traffic by 80%, and maintained 98% attack coverage. The results show that a substantial proportion of raw logs is redundant and can be trimmed without compro-mising essential evidence or analytic clarity. Template hashing preserved fidelity with moderate CPU cost; although it required slightly more pro-cessing than filtering or pruning, it still consumed fewer resources than the baseline. We repeated each test three times to ensure consistent results and validated the findings through ClickHouse queries at the sink layer. We also release the scripts and benchmark data to support reproduction and extension. Overall, the benchmark demonstrates how log-reduction design can improve operational efficiency while preserving analytic fidelity.