Using the Linux Agent, you can quickly deploy and collect metrics with a rich set of metadata.

The agent discovers and collects KPI metrics, integrates with CloudWatch, and can leverage other agent metrics. Using various plugins, the Linux Agent can also pull metrics from many different products running on a Linux operating system (in addition to pulling metrics from the host Linux OS).

Related Articles

Upgrade Your Linux Agent

It’s important you upgrade to the latest version of the Linux Agent whenever possible, as we’re constantly improving it. Check out the recently closed issues and pull requests, as well as the release history, to see why you should upgrade.

Dependencies
OS
  • Redhat 6 and 7
  • CentOS 6 and 7
  • Amazon Linux (latest)
  • Ubuntu 12, 14, 15, and 16
  • Debian 7, 8, and 9
Linux Agent
  • CPU: /proc/stat
  • Diskspace: /proc/mounts
  • Disk Usage: /proc/diskstats
  • Load Average: /proc/loadavg
  • Memory: /proc/stat
  • Network: /proc/net/dev
  • VM Stat: /proc/vmstat

Configuration

If you need to disable an existing Linux integration or view the unique API key assigned to your account:

  1. Navigate to User Profile > Integrations.
  2. Find the Integration with an Integration Type of Infrastructure.
  3. Click its name.
  4. Toggle Data Collection to disable it.

To learn more about the Metricly StatsD server, go here.

Standard Install Method
1. Copy Install Command From Linux Integration Setup Page
  1. From the top navigation menu, select Integrations.
  2. Select the Linux card. The name should be already populated, and Data Collection should be enabled.
  3. Highlight the one-line install command from the instructions and copy them. A unique API key for your account has already been generated and included in the command line.

If you’d prefer to specify the element name, copy the following instead:

sudo N_APIKEY=your-apikey N_HOSTNAME=your-element-name bash -c "$(curl -Ls http://repos.app.netuitive.com/linux.sh)"

your-apikey is the API key generated from the integration and your-element-name can be any element name you wish (it must be unique from your other elements).


2. Install Linux Agent 
  1. Paste the command from step 1.3 into your command line. This will install the agent and add your account’s unique API key to the configuration file.
    If you install our Linux agent on an AWS EC2 or Azure VM, the EC2’s / VM’s power state (it will come in as the attribute hostRunning with a value of true or false) and tags are copied over to the corresponding Linux SERVER element. You can then use this information to create policies.
    If you’d prefer to manually install the agent, see Manually Installing the Linux Agent.

3. Edit Linux Agent Config File
    1. Navigate to the Linux Agent configuration file found at <b>/opt/netuitive-agent/conf/netuitive-agent.conf</b>.
    2. Ensure the API key provided in step 1 is input in the netuitive-agent.conf file. The section below is only a portion of the config file. Go here to view the full config file.
      [[NetuitiveHandler]]
          ### MetriclyCloud URL to post the metrics
          url = https://api.app.metricly.com/ingest/infrastructure
      
          ## Metricly Datasource api key
          api_key = <datasource api key>
      
          ### Uncomment to add tags (optional)
          # tags = tag1:tag1val, tag2:tag2val
      
          ### Uncomment to add relations
          # relations = element1, element2
      
          # How many samples to store before sending to Metricly
          batch = 100
      
          # how many batches to store before trimming
          max_backlog_multiplier = 5
      
          # Trim down how many batches
          trim_backlog_multiplier = 4
      
          # local statsd server
          [[[statsd]]]
          enabled = False
    1. Option 1:  Substitute tags value with desired tags and uncomment the line to pass in tags for your element.
    2. Option 2:  Substitute relations value with desired element relationships; must include the fully qualified name of the elements. Uncomment the line to pass in relationships for your element.
    3. Option 3: Adjust  default collectors (CPU, DiskSpace, DiskUsage, Heartbeat, LoadAverage, Memory, VMStat, Network) using the configuration options found here.
      Even though the Heartbeat collector is a default collector, it has its own separate file (HeartbeatCollector.conf) that is typically found in /opt/netuitive-agent/conf/collectors

      These collectors are configured to get the most out of Metricly’s default policies and dashboards, so don’t change the following configuration options without guidance:

      • metrics_blacklist
      • metrics_whitelist
      • simple
      • include_cpu_pct
    4. Save the configuration file.
    5. Restart the Linux Agent service to begin monitoring your data with Metricly.

    This integration’s package (computed metrics, dashboards, and policies that will give you important events and alerts) will be automatically enabled and provisioned to your account as soon as Metricly receives data from the integration. The PACKAGES button on the integration setup page will become active once data is received, so you’ll be able to disable and re-enable the package at will.

  1. The most common commands are (depending on your distro):
    • service netuitive-agent {stop|start|restart}
    • /etc/init.d/netuitive-agent {stop|start|restart}
    • initctl {stop|start|restart} netuitive-agent
    • systemctl {stop|start|restart} netuitive-agent

    You can put the agent in Debug mode by changing the “logger_root” section of the configuration file from “INFO” to “DEBUG”.

Docker Install Method

1. Copy API Key From Docker Integration
  1. From the Metricly top navigation menu, select Integrations.
  2. Click the Docker card.
  3. Ensure Data Collection is enabled. A unique API key for your account has already been generated.
  4. Highlight the one-line install command from the instructions and copy them. A unique API key for your account has already been generated and included in the command line.

The command runs a container named netuitive-agent in the background and publishes the port values to the host. It then sets the environment variable DOCKER_HOSTNAME and APIKEY to the described values. Lastly, a volume is mounted so the agent has access to the Docker API.

Command Line Alternative

You can also copy the line below and paste it into your command line. Ensure you replace <b><my-api-key></b> with your API key from Metricly and <b><my-docker-host></b> with the desired hostname. Additional environment variables are located in the drop-down below if you’d like to include more or edit existing ones.

docker run -d -p 8125:8125/udp --name netuitive-agent -e DOCKER_HOSTNAME="<my-docker-host>" -e APIKEY="<my-api-key>" -v /proc:/host_proc:ro -v /var/run/docker.sock:/var/run/docker.sock:ro netuitive/docker-agent

Environment Variables
  • Variable Description Example
    LOGLEVEL Changes the log level of the agent. -e LOGLEVEL=DEBUG would set the agent to
    log at DEBUG level.
    INTERVAL Sets the interval in seconds at which the agent collectors run. -e INTERVAL=120 would set collection at
    two-minute intervals.
    DOCKER_HOSTNAME Sets the host name of the docker host. <br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />
    -e DOCKER_HOSTNAME="my-docker-host"<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br />
     would set the host name to my-docker-host.
    APIKEY Sets the API key used send data to Metricly. -e APIKEY=myapikey would set the API key
    to myapikey.
    USE_LOCAL_CONFIG Tells the agent to ignore any environment variables set and to use
    a local config file.
    -e USE_LOCAL_CONFIG=true would enable
    this feature.
    LPRT Tells the Netuitive StatsD agent what
    UDP port to listen on. 8125 is the default.
    FORWARD Enables forwarding from the netuitive-statsd server to another
    StatsD server.
    FIP Tells the Netuitive StatsD agent what IP address to forward to.
    FPRT Tells the Netuitive StatsD agent what port to forward to. 8125 is
    the default
2. Install Linux Agent Docker Container
  1. Paste the command from 1.3 into your command line. Replace my-docker-host in the command with the name of the docker host. The command will install the agent and add your account’s unique API key to the configuration file.
  2. Optional: Navigate to the Linux agent configuration file at <b>/opt/netuitive-agent/conf/netuitive-agent.conf</b> and add a metrics blacklist or whitelist under the [[NetuitiveDockerCollector]] section to reduce the number of metrics you receive.

Regex Examples

  • Escape special regex characters (e.g., .*/, etc.) using a /. The following would match containers.*.blkio. metrics and exclude them from collection. For a list of special regex characters you may have to escape, consult this page.
    metrics_blacklist = containers\..*\.blkio\..*
  • Match multiple containers between ( ) and separated by |. The following would match any of the following container IDs and exclude them from collection: abcdef123456123456abcdefghijkl789012.
    metrics_blacklist = containers\.(abcdef123456|123456abcdef|ghijkl789012)\..*

Optional Configuration


Base Collector vs Individual Collectors

The Base Collector is enabled by default and provides a bundled payload containing all of the following optional collectors: CPU, Heartbeat, Disk Space, Disk Usage, Load Avg., Memory, Network, and VM Stat.  You can disable the base collector in your netuitive/conf/netuitive-agent.conf file and choose to activate these collectors individually if you do not want to use them all.

 [[BaseCollector]]
-enabled = False
+enabled = True
 
 [[CPUCollector]]
 enabled = True
 simple = False
 percore = False
 include_cpu_pct = True
 
 [[DiskSpaceCollector]]
 enabled = True
 simple = True
 # exclude everything that begins /boot or /mnt
 exclude_filters = ^/boot, ^/mnt
 
 [[DiskUsageCollector]]
 enabled = True
 devices = (PhysicalDrive[0-9]+$|md[0-9]+$|sd[a-z]+$|x?vd[a-z]+$|disk[0-9]+$|dm\-[0-9]+$|nvme[0-9]+(n[0-9]+)(p[0-9]+)?$)
 metrics_whitelist = (?:^.*\.io$|^.*\.average_queue_length$|^.*\.await$|^.*\.iops$|^.*\.read_await$|^.*\.reads$|^.*\.util_percentage|^.*\.write_await$|^.*\.writes$)
 
 [[LoadAverageCollector]]
 enabled = True
 simple = False
 
 [[MemoryCollector]]
 enabled = True
 
 [[VMStatCollector]]
 enabled = True
 
 [[NetworkCollector]]
 enabled = True
 metrics_whitelist = (?:^.*\.rx_byte$|^.*\.rx_errors$|^.*\.tx_byte$|^.*\.tx_errors$)
 
 [[NetuitiveDockerCollector]]
 enabled = False

 


Changing Element Display Names

You can change the display name of certain elements to help distinguish between each instance.

  1. Navigate to Integrations > Linux Integration.
  2. Click Advanced; a menu expands.
  3. Click the Element Name field to edit it.
  4. Type the desired name into the field.
    • An element name of ${meta.originalName}would resolve to whatever name comes in with the original element payload before it would be replaced with the optional element name template.
      The element name template preview in the UI will resolve this field to [original name] as a placeholder because Metricly only knows what the current name is, not what the incoming name might be.
    • An element name of ${tags.InternalName} (${tags.Name}) will give you something like MyServer (ip-10.101.3.99)
    • An element name of ${tags.Name} (${attributes.availabilityZone) would return something like ServerX (eu-west-1c)
  5. Select an element to use as a preview for your new element name using the Element To Preview drop-down menu.
  6. Next to the Element Name field, click Preview to view your new template using the selected element.
  7. If you’re satisfied with the name, press Enter on your keyboard while in the Element Name field to lock in the name. Exit the integration setup page and wait until the next analytics cycle (5 minutes) to see your changes.
Example: Adding the Private IP Address to an Element Name

Adding the private IP address to your element names enables your team to immediately begin troubleshooting problematic instances.

<#if tags.Name??>${tags.Name}<#elseif tags.aws_autoscaling_groupName??>${tags.aws_autoscaling_groupName}<#else>${meta.originalName}</#if><#if attributes.privateIp??> (${attributes.privateIp})</#if>
{noformat}

 

Additional Actions


Write Metric FQNs to Local File

If you want to keep a local file with a list of metric FQNs (fully qualified names), you can do so by updating the below lines in your netuitive-agent.conf file, under the [[NetuitiveHandler]]. By default, this ability is set to False. When set to True, this file updates with any new FQNs as long as the agent is running.

If the agent is started and write_metric_fqns is set to True, the file is overwritten; it remains unchanged if set to False before starting the agent.

To Enable:

  1. Open your netuitive-agent.conf file.
  2. Change write_metric_fqns = False to True.
  3. Confirm your desired path (metric_fqns_path) for the file.
  4. Save.
# Write metric fqns to a local file
write_metric_fqns = True
metric_fqns_path = /opt/netuitive-agent/log/metric_fqns.txt