Hey, I made this because I kept waiting forever when parsing server logs at work. Tried a few tools, they were all slow as hell, so I figured I'd see how fast I could get it.Turns out if you skip the usual file reading stuff and just mmap the file directly, then throw some AVX2 SIMD at the character counting, things get pretty quick. Getting around 1.4 GB/s on my NVMe which is like 77% of what the SSD can actually do. Rest is just OS overhead and page faults.
It falls back to SSE2 on older CPUs so it should run on pretty much anything. Works on Windows and Linux.
Anyway, let me know if you have questions about how it works or ideas for improvements
Hey, I made this because I kept waiting forever when parsing server logs at work. Tried a few tools, they were all slow as hell, so I figured I'd see how fast I could get it.Turns out if you skip the usual file reading stuff and just mmap the file directly, then throw some AVX2 SIMD at the character counting, things get pretty quick. Getting around 1.4 GB/s on my NVMe which is like 77% of what the SSD can actually do. Rest is just OS overhead and page faults. It falls back to SSE2 on older CPUs so it should run on pretty much anything. Works on Windows and Linux. Anyway, let me know if you have questions about how it works or ideas for improvements