k3logcollector

Log collector for scanning and aggregating log files. Scans log files on local machine, collects interested logs, and sends them somewhere for display.

k3logcollector is a component of pykit3 project: a python3 toolkit set.

Installation

pip install k3logcollector

Quick Start

from k3logcollector import collector


def send_log(log_entry):
    print(log_entry)


# Configure log files to scan
conf = {
    "my_log": {
        "file_path": "/var/log/myapp.log",
        "is_first_line": lambda line: line.startswith("["),
        "get_level": lambda log: "error" if "ERROR" in log else "info",
        "parse": lambda log: {
            "log_ts": 1234567890,
            "level": "error",
            "source_file": "app.py",
            "line_number": 42,
        },
        "level": ["error", "warning"],
    }
}

# Run the collector
collector.run(node_id="node-1", node_ip="192.168.1.1", send_log=send_log, conf=conf)

API Reference

k3logcollector

Name

logcollector

Scan log files on local machine, collector all interested logs, and send to somewhere for display.

Description

We may want to see all error logs on all machines, so we need to collector logs, and save it in somewhere. This module is used to collector logs on one machine.

Normally, same error info will be loged repeatedly, we do not want to save duplicated log info, so logs produced by same source file at same line number in one second will be combined.

Each flush, once per second, sends only the logs of the newest 5 timestamps and drops older ones, unless the log's conf sets merge to False. So a backlog, such as the 2 MB that the scanner reads back when it starts, does not keep the sender behind.

Conf

configuration for log files. It is a dict, the key is the log name, the value is the configuration for the log.

file_path

the path of the log file.

is_first_line

is a callback function. The argument to this function is a line in log file, if the line is the first line of a log, then return True, otherwise return False.

get_level

is a callback function. The argument to this function is the complete log string, which may contains multiple lines. It should return the level of the log, which is a string.

parse

is a callback function. The argument to this function is the complete log string, which may contains multiple lines. It should return a dict contains following fields.

• log_ts: the timestamp of this log, such as 1523936052, is a integer.

• level: the level of this log, such as 'info'.

• source_file: the source file in which the log was produced.

• line_number: the number of the line at which the log was produced.

level

is a list, used to specify the interested log levels.

License

The MIT License (MIT) - Copyright (c) 2015 Zhang Yanpo (张炎泼)