Plugin info

name: oci-search-logs

Generates metrics or events from Oracle Cloud Infrastructure indexed logs.

Furthermore, multiple metrics from the same search can be aggregated by their respective dimensions.

Prerequisites

OCI API key with access to required OCI logs.

OCI client configuration is expected in file $HOME/.oci/config

Events

User defined

Metrics

User defined

Configuration

This section describes the configuration settings for this plugin

IMPORTANT

Sensitive data should be stored inside the agent secrets store and referenced in this file using ${secret key} notation.

Name	Type	Required	Default	Description
proxy	string	No		optional http proxy url to use for the OCI endpoint requests
api_key_passphrase	string	Yes		the API key passphrase (supports secrets)
oci_config_profile	string	No	DEFAULT	the OCI client configuration profile to use
region	string	No	OCI config file region	the OCI region that holds the logs to query
history_key	string	Yes		a key for this configuration to store the last run time. Each time a configuration is run it fetches records since the last runtime
timeout	duration	No	60s	how long to wait for each search results page
source	string	Yes		the source name to use for metrics/events generated from this configuration
query	string	Yes		the OCI log query (Logging Query Language Specification (opens new window)) to execute.
metrics	MetricOrEvent	No		the metric definition to extract from the results of the search query
events	MetricOrEvent	No		the event definition to extract from the results of the search query

MetricOrEvent Configuration

Name	Type	Required	Default	Secrets	Description
name	string	Yes			the name for the metric or event. For example oci/lb/access/client_addr_count
units	string	No			the units for a metric
descr	string	No			the description for a metric or event
aggregation	string	No			aggregate a sequence of metrics by their dimensions. One of: count, sum, avg, max, min
value	string	No			the JSON path to read a metric value from. Optional if doing count aggregation
timestamp	string	No			the JSON path to read the timestamp for a metric or event. defaults to current time
replacer	Replacer	No			a regular expression to manipulate the metric or event value
source	string	No			path to json field to read metric/event source from. defaults to parent source value
dimensions	[]Dimension	No			the dimension for this event or metric

Dimension Configuration

Only one of path and coalesce_paths are required

Name	Type	Required	Default	Secrets	Description
name	string	Yes			the name for the metric or event. For example oci/lb/access/client_addr_count
descr	string	No			the description for a metric or event
path	string	Depends			the JSON path to read the dimension value from
coalesce_paths	[]string	Depends			first non empty JSON path value to read the dimension from
replacer	Replacer	No			a regular expression to manipulate the dimension value

Replacer Configuration

Name	Type	Required	Default	Secrets	Description
regexp	string	Yes			the regexp to use to replace the value. e.g. ([^:]+):\d+$ to remove port part from an address
with	string	Yes			replaces matches of the regexp with this replacement string. Inside the replacement string, $ signs are expanded from regexp matches, so for instance $1 represents the text of the first submatch

Examples

Count unique client addresses from OCI LB access logs

api_key_passphrase: ${oci.api_key_passphrase} # your API key passphrase (secret managed by polaris)
oci_config_profile: DEFAULT
source: polaris.arisant.com
# only domain polaris.arisant.com and try to filter out bots as much as possible
query: |
  search "__LOG_OCID_FOR_YOUR_LB__"
  | data.host = 'polaris.arisant.com' and
    data.userAgent != '*bot*' and
    data.userAgent != '*scan*' and
    data.userAgent != '*inspect*' and
    data.userAgent != '*google*' and
    data.userAgent = '*Mozilla/5.0*' and
    data.lbStatusCode = 200
  | sort by datetime desc	
metrics:
  - name: oci/lb/access/client_addr_count
    aggregation: count
    timestamp: datetime
    dimensions:
      - name: ip
        path: logContent.data.clientAddr
        # remove port from client address
        replacer:
          regexp: ([^:]+):\d+$
          with: $1
  - name: oci/lb/access/client_addr_agent_count
    aggregation: count
    timestamp: datetime
    dimensions:
      - name: ip
        path: logContent.data.clientAddr
        # remove port from client address
        replacer:
          regexp: ([^:]+):\d+$
          with: $1
      - name: agent
        path: logContent.data.userAgent

Validate Configuration

oci-search-logs --run-conf /path/to/config/file.yaml --validate

Testing

Run the plugin from the command line and get any emitted metrics on stdout

• simulate, do not try to register metrics

oci-search-logs --run-conf /path/to/config/file.yaml --simulate

List the metrics emitted by a plugin configuration

oci-search-logs --run-conf /path/to/config/file.yaml --metrics