Docs

Log streams

  • Last Updated 10/4/2023, 1:01:44 PM UTC
  • About 11 min read

Plugin info

name: logmon

Tails one or more log files to extract user defined metrics based on grok patterns. Please refer to Grok Basics (opens new window) guide from logstash for details on how to create and use grok patterns.

This plugin pre-defines a number of patterns listed at Predefined Patterns.

Default parser configurations can be found at git repo https://github.com/arisant/myrmex-dist.git under myrmex-assets/collectors/tasks/conf/parsers/default

# Prerequisites

  • Read access to target files

# Events

None

# Metrics

User defined

# Configuration

The plugin is configured with a yaml file. This section describes the configuration settings.

# Variable substitution

The following variables are understood and substituted with values by the plugin:

  • $logfile, the path to the monitored log file

# Main configuration

  • file_path (Required)
    Absolute path to the log file to tail

  • parser_path (Required)
    Path (relative to <catalog root>/assets/collectors/tasks/conf/) which configures the metric extractor. This file is intentionally separate from the plugin main configuration file so that it can be shared among multiple configurations.

  • default_source The default value to assign to metrics for their source property. If not set, defaults to hostname

# Parser Configuration

  • name (Required)
    The unique name for this parser

  • default_source (Optional)
    The default value to assign to metrics for their source property. If not set, defaults to hostname

  • skip_lines (Optional) List of regexp patterns. Before processing a log line, the parser checks if it matches any one of these patterns. If it does the line is skipped.

    skip_lines:
  - ^\QARIS_POL(\E

  • patterns (Optional)
    List of line separated grok patterns to parse log messages

  • multiline (Optional)
    Configures multi-line handling in configuration files

    • pattern (Required)
      The pattern which identifies a log line a part of multi-line message

    • negate (Optional)
      Set to true in order invert the result of pattern above

    • max_lines (Optional)
      The maximum number of lines to buffer in a multi-line message

  • metrics (Required)
    Configures a list of metrics to extract from a log message

    • name (Required)
      The namespace for the metric

    • pattern (Required)
      The grok pattern name that will trigger extraction for this metric

    • filters (Optional)
      Filter log messages by field using regular expressions. A metric will be emitted if all filters match the provided pattern

      • field (Required)
        The grok field name to filter

      • pattern (Required)
        The regexp or grok pattern name to apply to the field value

      • negate (Optional)
        Set to true to negate the regexp result

    • field (Required)
      The grok field name to read the metric value from

    • type (Optional)
      The value type of the field. Default is string. Available types are string, number

    • source (Optional)
      The grok field name to use as the source property for the metric. If not set then default_source will be used

    • timestamp (Optional)
      If not set, the timestamp assigned to each metric is the local extraction time. Configure this setting to assign a timestamp for the metric from the log message.

      • field (Required)
        The grok field name to extract the timestamp from

      • format (Required)
        The format to parse the field value into a timestamp. Available formats are:

        • epoch, unix epoch seconds
        • epoch_millis, unix epoch milliseconds
        • epoch_micros, unix epoch microseconds
        • epoch_nanos, unix epoch nanoseconds
        • syslog, Jan _2 15:04:05 with year implicit as current year
        • ansi, Mon Jan _2 15:04:05 2006
        • unix, Mon Jan _2 15:04:05 MST 2006
        • ruby, Mon Jan 02 15:04:05 -0700 2006
        • rfc822, 02 Jan 06 15:04 MST
        • rfc822z, 02 Jan 06 15:04 -0700
        • rfc850, Monday, 02-Jan-06 15:04:05 MST
        • rfc1123, Mon, 02 Jan 2006 15:04:05 MST
        • rfc1123z, Mon, 02 Jan 2006 15:04:05 -0700
        • rfc3339, 2006-01-02T15:04:05Z07:00
        • custom (see https://golang.org/pkg/time/#Parse for specification)

    • dimensions (Optional)
      Injects metric dimension values from grok fields. Map of dimension names to field names

    • aggregation (Optional)
      Aggregates metrics by source and dimensions. The available aggregators are count, sum, avg, min, max.

# Examples

# REST API latency stats

We have an NGINX server that services requests to our REST API and we want to collect aggregated latency stats per API operation at regular intervals from the NGINX access logs.

Let's assume that NGINX has the following configuration for the access logs:

log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         'rt=$request_time uct="$upstream_connect_time" '
                         'uht="$upstream_header_time" urt="$upstream_response_time" '
                         '"$http_referer" "$http_user_agent"';

A sample access log output from this configuration is the following:

192.168.1.100 - johndoe [04/Oct/2023:12:34:56 +0000] "GET /api/users" 200 1234 rt=0.123 uct="0.010" uht="0.020" urt="0.050" "http://example.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"
192.168.1.101 - janedoe [04/Oct/2023:12:35:01 +0000] "POST /api/posts" 201 987 rt=0.256 uct="0.015" uht="0.025" urt="0.216" "http://example.net" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36"
192.168.1.102 - guest [04/Oct/2023:12:36:03 +0000] "PUT /api/products/123" 204 0 rt=0.345 uct="0.012" uht="0.022" urt="0.311" "-"
192.168.1.103 - admin [04/Oct/2023:12:37:12 +0000] "DELETE /api/orders/456" 204 0 rt=0.189 uct="0.011" uht="0.021" urt="0.157" "http://example.org" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"
192.168.1.104 - user1 [04/Oct/2023:12:38:25 +0000] "PATCH /api/profile" 200 567 rt=0.432 uct="0.013" uht="0.023" urt="0.396" "http://example.com/profile" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"

Let's configure our log stream collector to process this output and generate avg, min, max and count stats for each API:

name: NGINX REST API stats
patterns: >
  API_OP_PATH /api/[^/\s]+

  ACCESS_MSG %{NOTSPACE:ip_or_host} - %{DATA:user} \[%{DATA:when}\]
             "%{NOTSPACE} %{API_OP_PATH:op_path}${DATA}" %{NOTSPACE:status}
             %{NOTSPACE:bytes} %{NOTSPACE:request_time} %{DATA}

default_source: some-api
metrics:
  - name: rest-apis/avg-latency
    pattern: "%{ACCESS_MSG}"
    field: request_time
    type: number
    aggregation: avg
    dimensions:
      operation: op_path
  - name: rest-apis/min-latency
    pattern: "%{ACCESS_MSG}"
    field: request_time
    type: number
    aggregation: min
    dimensions:
      operation: op_path
  - name: rest-apis/max-latency
    pattern: "%{ACCESS_MSG}"
    field: request_time
    type: number
    aggregation: max
    dimensions:
      operation: op_path
  - name: rest-apis/hits
    pattern: "%{ACCESS_MSG}"
    field: request_time
    type: number
    aggregation: count
    dimensions:
      operation: op_path

From the same sample access log output above we would get the following metrics:

rest-apis/avg-latency{source="some-api", operation="/api/users"}: 0.123
rest-apis/avg-latency{source="some-api", operation="/api/posts"}: 0.256
rest-apis/avg-latency{source="some-api", operation="/api/products"}: 0.345
rest-apis/avg-latency{source="some-api", operation="/api/orders"}: 0.189
rest-apis/avg-latency{source="some-api", operation="/api/profile"}: 0.432

rest-apis/min-latency{source="some-api", operation="/api/users"}: 0.123
rest-apis/min-latency{source="some-api", operation="/api/posts"}: 0.256
rest-apis/min-latency{source="some-api", operation="/api/products"}: 0.345
rest-apis/min-latency{source="some-api", operation="/api/orders"}: 0.189
rest-apis/min-latency{source="some-api", operation="/api/profile"}: 0.432

rest-apis/max-latency{source="some-api", operation="/api/users"}: 0.123
rest-apis/max-latency{source="some-api", operation="/api/posts"}: 0.256
rest-apis/max-latency{source="some-api", operation="/api/products"}: 0.345
rest-apis/max-latency{source="some-api", operation="/api/orders"}: 0.189
rest-apis/max-latency{source="some-api", operation="/api/profile"}: 0.432

rest-apis/hits{source="some-api", operation="/api/users"}: 1
rest-apis/hits{source="some-api", operation="/api/posts"}: 1
rest-apis/hits{source="some-api", operation="/api/products"}: 1
rest-apis/hits{source="some-api", operation="/api/orders"}: 1
rest-apis/hits{source="some-api", operation="/api/profile"}: 1

# Multiline Log Entries

From a log file that can have entries spanning multiple lines, extract Info, Error or Warning messages with message id BEA-001156, BEA-000503 or BEA-001129 and aggregate their counts into metric test/wls/msg_count{host, server, msg_id}.

Sample log entries:

####<Aug 18, 2019 9:37:04 AM MDT> <Info> <Cluster> <some-server.domain.com> <WLS_OAM1> <weblogic.cluster.MessageReceiver> <<WLS Kernel>> <> <bbe0e7c869941a79:485136e2:16c6d561d77:-8000-000000000000004a> <1566142624430> <BEA-003107> <Lost 2 unicast message(s).> 
####<Aug 18, 2019 9:37:04 AM MDT> <Info> <WorkManager> <some-server.domain.com> <WLS_OAM1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <bbe0e7c869941a79:485136e2:16c6d561d77:-8000-0000000005e4f91e> <1566142624436> <BEA-002936> <maximum thread constraint ClusterMessaging is reached> 
####<Aug 18, 2019 9:37:04 AM MDT> <Error> <Kernel> <some-server.domain.com> <WLS_OAM1> <[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <bbe0e7c869941a79:485136e2:16c6d561d77:-8000-0000000005e4f93b> <1566142624703> <BEA-000802> <ExecuteRequest failed
 java.lang.NullPointerException.
java.lang.NullPointerException
	at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:459)
	at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
	at weblogic.net.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:768)
	at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:639)
	at java.net.URLConnection.getContentLengthLong(URLConnection.java:511)
	at java.net.URLConnection.getContentLength(URLConnection.java:495)
	at weblogic.cluster.ClusterHelper.logStateDumpRequestRejection(ClusterHelper.java:55)
	at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:109)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 


name: Weblogic server log
patterns: >
  WLS_SEVERITY Info|Error|Warning
  
  MSG_ID BEA-001156|BEA-000503|BEA-001129
  
  WLS_USER (?:<(?:[<>a-zA-Z ]*)>%{SPACE})?
  
  WLS_MSG (?ms)####<%{DATA:locale_ts}>%{SPACE}<%{WLS_SEVERITY:severity}>%{SPACE}<%{DATA:subsystem}>%{SPACE}
  <%{DATA:host}>%{SPACE}<%{DATA:wls_server}>%{SPACE}<%{DATA:jvm_thread}>%{SPACE}%{WLS_USER}<%{DATA:transaction_id}>%{SPACE}
  <%{DATA:diagnostic_id}>%{SPACE}<%{DATA:raw_ts}>%{SPACE}<%{MSG_ID:msg_id}>%{SPACE}<%{GREEDYDATA:msg}>

multiline:
  pattern: '^####<'
  negate: true
  max_lines: 4
metrics:
  - name: test/wls/msg_count
    pattern: '%{WLS_MSG}'
    field: msg
    type: string
    aggregation: count    
    dimensions:
      host: host
      server: wls_server
      msg_id: msg_id
      file: $logfile

# Filtering

Same log file structure as above but emit metrics for severity Error, and message id anything other than OAM-00002 and OAM-02010.

name: Weblogic server log
default_source: "test"
patterns: >
  WLS_SEVERITY Error|Warning

  WLS_USER (?:<(?:[<>a-zA-Z ]*)>%{SPACE})?
  
  WLS_MSG (?ms)####<%{DATA:locale_ts}>%{SPACE}<%{WLS_SEVERITY:severity}>%{SPACE}<%{DATA:subsystem}>%{SPACE}
  <%{DATA:host}>%{SPACE}<%{DATA:wls_server}>%{SPACE}<%{DATA:jvm_thread}>%{SPACE}%{WLS_USER}<%{DATA:transaction_id}>%{SPACE}
  <%{DATA:diagnostic_id}>%{SPACE}<%{DATA:raw_ts}>%{SPACE}<%{DATA:msg_id}>%{SPACE}<%{GREEDYDATA:msg}>

multiline:
  pattern: '^####<'
  negate: true
metrics:
  - name: test/wls/msg_id
    pattern: '%{WLS_MSG}'
    filters:
      - field: severity
        pattern: ^(?:Error)$
      - field: msg_id
        pattern: ^(?:OAM-00002|OAM-02010)$
        negate: true
    source: host
    field: msg_id
    type: string
    timestamp:
      field: locale_ts
      format: 'Jan 2, 2006 3:04:05 PM MST'
    dimensions:
      host: host
      server: wls_server
      sev: severity

# Validate Configuration

logmon --run-conf /path/to/config/file.yaml --assets /path/to/parsers/dir --validate

# Testing

Run the plugin from the command line to scan all the input file lines and get any emitted metrics on stdout.

logmon --run-conf /path/to/config/file.yaml --assets /path/to/parsers/dir

# List the metrics emitted by a plugin configuration

logmon --run-conf /path/to/config/file.yaml --assets /path/to/parsers/dir --metrics

# Predefined Patterns

Name Description
USERNAME user names
USER alias to USERNAME
INT postive or negative integer
BASE10NUM base 10 positive or negative number with optional decimal part
NUMBER alias to BASE10NUM
BASE16NUM Hex number
POSINT postive interger
NONNEGINT unsigned number word
WORD a word
NOTSPACE anything that is not whitespace
SPACE whitespace
DATA matches anything
GREEDYDATA same as DATA but greedy
QUOTEDSTRING a string within single or double quotes
UUID a uuid
MAC a mac address
CISCOMAC a CISCO mac address
WINDOWSMAC a windows max address
COMMONMAC a common mac address
IPV6 an ip v6 address
IPV4 an ip v4 address
IP an ipv4 or ipv6 address
HOSTNAME a host name
HOST alias to HOSTNAME
IPORHOST ip address or hostname
HOSTPORT ip address or hostname and port
PATH a windows or unix path
UNIXPATH a unix path
TTY a tty device path
WINPATH a windows path
URIPROTO a URI protocol
URIHOST a URI host with optional port
URIPATH a URI path
URIPARAM URI parameters
URIPATHPARAM a URI with parameters
URI a URI
MONTH full or abbreviated month name
MONTHNUM numeric month
MONTHDAY numeric month day
DAY full or abbreviated week day
YEAR numeric year
HOUR 24 clock numeric hour
MINUTE minutes of hour
SECOND seconds of minute
TIME hour minute second
DATE_US US formatted date
DATE_EU EU formatted date
ISO8601_TIMEZONE ISO8601 timezone
ISO8601_SECOND ISO8601 timezone seconds
TIMESTAMP_ISO8601 ISO8601 timestamp
DATE US or EU date
DATESTAMP US or EU date with timestamp
TZ timezone abbreviation
DATESTAMP_RFC822 RFC822 date-time
DATESTAMP_OTHER date-time with timezone
SYSLOGTIMESTAMP syslog formatted timestamp
PROG program name
SYSLOGPROG syslog formatted program
SYSLOGHOST syslog host
SYSLOGFACILITY syslog facility
HTTPDATE http access log date-time
QS a quoted string
SYSLOGBASE syslog prefix
COMMONAPACHELOG common apache log formatted line
COMBINEDAPACHELOG combined apache log formatted line
LOGLEVEL log entry severity

# Predefined pattern definitions

var PredefinedPatterns = map[string]string{
	"USERNAME":           `[a-zA-Z0-9._-]+`,
	"USER":               `%{USERNAME}`,
	"EMAILLOCALPART":     `[a-zA-Z][a-zA-Z0-9_.+-=:]+`,
	"EMAILADDRESS":       `%{EMAILLOCALPART}@%{HOSTNAME}`,
	"HTTPDUSER":          `%{EMAILADDRESS}|%{USER}`,
	"INT":                `(?:[+-]?(?:[0-9]+))`,
	"BASE10NUM":          `([+-]?(?:[0-9]+(?:\.[0-9]+)?)|\.[0-9]+)`,
	"NUMBER":             `(?:%{BASE10NUM})`,
	"BASE16NUM":          `(0[xX]?[0-9a-fA-F]+)`,
	"POSINT":             `\b(?:[1-9][0-9]*)\b`,
	"NONNEGINT":          `\b(?:[0-9]+)\b`,
	"WORD":               `\b\w+\b`,
	"NOTSPACE":           `\S+`,
	"SPACE":              `\s*`,
	"DATA":               `.*?`,
	"GREEDYDATA":         `.*`,
	"QUOTEDSTRING":       `"([^"\\]*(\\.[^"\\]*)*)"|\'([^\'\\]*(\\.[^\'\\]*)*)\'`,
	"UUID":               `[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}`,
	"MAC":                `(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})`,
	"CISCOMAC":           `(?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})`,
	"WINDOWSMAC":         `(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})`,
	"COMMONMAC":          `(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})`,
	"IPV6":               `((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?`,
	"IPV4":               `(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)`,
	"IP":                 `(?:%{IPV6}|%{IPV4})`,
	"HOSTNAME":           `\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)`,
	"HOST":               `%{HOSTNAME}`,
	"IPORHOST":           `(?:%{IP}|%{HOSTNAME})`,
	"HOSTPORT":           `%{IPORHOST}:%{POSINT}`,
	"PATH":               `(?:%{UNIXPATH}|%{WINPATH})`,
	"UNIXPATH":           `(/[\w_%!$@:.,-]?/?)(\S+)?`,
	"TTY":                `(?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))`,
	"WINPATH":            `([A-Za-z]:|\\)(?:\\[^\\?*]*)+`,
	"URIPROTO":           `[A-Za-z]+(\+[A-Za-z+]+)?`,
	"URIHOST":            `%{IPORHOST}(?::%{POSINT:port})?`,
	"URIPATH":            `(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+`,
	"URIPARAM":           `\?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*`,
	"URIPATHPARAM":       `%{URIPATH}(?:%{URIPARAM})?`,
	"URI":                `%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?`,
	"MONTH":              `\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b`,
	"MONTHNUM":           `(?:0?[1-9]|1[0-2])`,
	"MONTHNUM2":          `(?:0[1-9]|1[0-2])`,
	"MONTHDAY":           `(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])`,
	"DAY":                `(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)`,
	"YEAR":               `(\d\d){1,2}`,
	"HOUR":               `(?:2[0123]|[01]?[0-9])`,
	"MINUTE":             `(?:[0-5][0-9])`,
	"SECOND":             `(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)`,
	"TIME":               `([^0-9]?)%{HOUR}:%{MINUTE}(?::%{SECOND})([^0-9]?)`,
	"DATE_US":            `%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}`,
	"DATE_EU":            `%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}`,
	"ISO8601_TIMEZONE":   `(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))`,
	"ISO8601_SECOND":     `(?:%{SECOND}|60)`,
	"TIMESTAMP_ISO8601":  `%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?`,
	"DATE":               `%{DATE_US}|%{DATE_EU}`,
	"DATESTAMP":          `%{DATE}[- ]%{TIME}`,
	"TZ":                 `(?:[PMCE][SD]T|UTC)`,
	"DATESTAMP_RFC822":   `%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}`,
	"DATESTAMP_RFC2822":  `%{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}`,
	"DATESTAMP_OTHER":    `%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}`,
	"DATESTAMP_EVENTLOG": `%{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}`,
	"HTTPDERROR_DATE":    `%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}`,
	"SYSLOGTIMESTAMP":    `%{MONTH} +%{MONTHDAY} %{TIME}`,
	"PROG":               `[\x21-\x5a\x5c\x5e-\x7e]+`,
	"SYSLOGPROG":         `%{PROG:program}(?:\[%{POSINT:pid}\])?`,
	"SYSLOGHOST":         `%{IPORHOST}`,
	"SYSLOGFACILITY":     `<%{NONNEGINT:facility}.%{NONNEGINT:priority}>`,
	"HTTPDATE":           `%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}`,
	"QS":                 `%{QUOTEDSTRING}`,
	"SYSLOGBASE":         `%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:`,
	"COMMONAPACHELOG":    `%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)`,
	"COMBINEDAPACHELOG":  `%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}`,
	"HTTPD20_ERRORLOG":   `\[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}`,
	"HTTPD24_ERRORLOG":   `\[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}`,
	"HTTPD_ERRORLOG":     `%{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}`,
	"LOGLEVEL":           `([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)`,
}
Last Updated: 10/4/2023, 1:01:44 PM

HTTP fetch →