BlessedRebuS/Krawl

We could develop a recurrent task (that operates with batches and leverages redis task-scheduler for scalable mode, like other tasks) that takes similar non-suspicious requests from inactive ips (1 week inactivity) and crunches them in a single one. This could dramatically improve performance since the most db-expensive queries are on access-logs table, which is filled with useless non-suspicious requests. An example would be: GET /dasidaniodna GET /daefndisjd GET /sdleijfwmi GET /adlkndawndoia GET /test | | Crunched to V GET /dasidaniodna, ... GET /test

Since recently, Carto started to require an API key to use map tiles, thus showing watermark on any requests without one. API key should append to the end of request of every tile (e.g. https://basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}.png?key=YOUR_KEY).

Description Replace the current regex-based detection engine in Krawl with a Sigma Rules based detection system. Instead of relying only on static regular expressions, Sigma Rules would provide a more structured, scalable, and community-driven way to detect malicious behavior, attack patterns, and suspicious activity. Proposed Changes Remove or deprecate the current regex-only detection logic. Integrate a Python Sigma Rules engine/library for rule parsing and matching (pySigma). Support loading custom Sigma rule files from a dedicated rules directory. Trigger alerts when logs or requests match Sigma detections. Allow hot-reloading or dynamic updates of rules without restarting Krawl.

Maybe adding an options for privates instances to push/pull their malicious ip on a public blacklist ?