Java

Metricly’s Java agent is a java monitoring agent with a programmable bytecode instrumentation engine that’s enabled by adding a JVM integration in Metricly. The Metricly Java integration allows Metricly to collect JVM runtime system metrics like CPU, Memory, GC, Threads and Classes Count, and application components method performance statistics, such as number of calls and execution time.

Java Integration Options

We have several different options for monitoring Java applications. Each method varies in terms of setup difficulty and the amount / type of information it collects. See the table below for more information.

Method Description
Ananke Java library you can use to push metrics to the StatsD listener embedded in our Linux agent.

This approach requires that you integrate the StatsDReporter into your applications.
Dropwizard Integrate the dropwizard-metrics library into your Dropwizard application and configure it to send metrics to the StatsD listener embedded in our Linux agent.
Iris Java library you can use to push metrics directly to Metricly’s REST API.
Java Agent Open-source and open-license Java agent the does the byte-code instrumentation for you. No changes to source code required.
JMX Integration that relies on our Linux agent to collect JVM metrics (e.g., heap size, garbage collection, etc.) without code-level instrumentation.

Dependencies

Miscellaneous
Java 6+

Configuration

If you need to disable an existing Java integration or view the unique API key assigned to your account, navigate to the Integrations page under the user account drop-down menu and click the integration designated as Java under the Integration column.

Installation

  • Setting up the Java agent is a three step process:
    1. Copy the unique API key from the Java integration in your account.
    2. Download the Netuitive Java agent.
    3. Edit your Java agent configuration file.

    Step 1: Copy the API key from the Java integration in your account

    1. From the top navigation menu, select Integrations.
    2. Select the Java card. The name should be already populated, and Data Collection should be enabled. A unique API key for your account has already been generated.
    3. Copy the API key.

    Step 2: Download the Netuitive Java agent

    1. Download the Netuitive Java agent here.
    2. Unzip the netuitive-zorka.zip file to the desired location. We recommend that you unzip the netuitive-zorka.zip file near the app it is monitoring.

    Step 3: Edit the Java agent configuration file

    1. Navigate to the zorka.properties file that was included in the netuitive-zorka-{version}.zip.
      Overriding the zorka.properties file is supported by the Java agent. The order of precedence is environment variables, then java variables, then zorka.properties.

      Below are examples of overriding the netuitive.api.key setting in the zorka.properties file:

      1. Using environment variables:
        export NETUITIVE_API_KEY=java-api-id
      2. Using java variables:
        -Dnetuitive.api.key=java-api-id
    2. Enable the jvm.bsh script and any other scripts as necessary at the top of the file.
      # Default collection of jvm metrics
          scripts = jvm.bsh, my-other-script.bsh
    3. Change zorka.hostname and zorka.application to the correct value. zorka.hostname should be the fully qualified domain name of the host and zorka.application should be the name of your Java application.
    4. Optionally, if you’re setting up the Java agent on JBoss, you should change the zorka.mbs.autoregister setting to no.
    5. Optionally, if you’d like the Zorka Log to be managed by the default syslog, change the zorka.syslog setting to yes. Read more about the zorka.syslog setting here.
    6. Optionally, if you want Zorka to monitor clusters:
      1. Uncomment zorka.clusters in the zorka.properties file and include the name of each element. The default setting (cluster 1, cluster 2) would create a JVM () element and two CLUSTER () elements with the names cluster1 and cluster 2.
        zorka.clusters = cluster1, cluster2
      2. If you want to specify the element type of the clusters, uncomment zorka.clusters.type and change the setting to the desired element type. If the type is not specified, the default is CLUSTER.
        zorka.clusters.type = CLUSTER
        If you decide to set up a cluster and use the default type (CLUSTER), your icon for the cluster will be . If you change the type, your icon may be different from the default icon.
    7. Change the netuitive.api.key value to the unique API key generated for your account that you copied in step 1.

      You should not change the netuitive.api.url value. You should also not change the following values if you followed this pattern to store your custom zorka stats:

      1. netuitive.api.stats.mbean = zorka:type=ZorkaStats,* (zorka-stats mbean definition)
      2. netuitive.api.stats.attributes = stats, byTag (comma separated zorka-stats mbean attributes that are used to store the method call stats)
    8. Ensure the netuitive.api value is set to yes to enable data push to Metricly.
    9. Modify the netuitive.api.interval value to the desired data push interval (in seconds). The agent will collect data at the desired interval but aggregate the values each minute and send a single set of values to the API. Data is sent as statistics (minmaxsumavg, and count) instead of raw values. Data in the Metrics pagefor your JVM metrics is displayed using gauges instead of a counter to capture any delta between datapoints.
      If you set the data interval to 20 seconds, the Java agent will collect 3 datapoints every minute, aggregate the data, and send a single set of values to the API.
    10. Restart your app server with the following argument passed to the JVM:
      -javaagent:[path-to-netuitive-zorka-unzipped-dir]/netuitive.jar=[path-to-netuitive-zorka-unzipped-dir]

      … where the default path to the Netuitive Java agent unzipped directory is /opt/netuitive-zorka/

      This integration’s package (computed metrics, dashboards, and policies that will give you important events and alerts) will be automatically enabled and provisioned to your account as soon as Metricly receives data from the integration. The PACKAGES button on the integration setup page will become active once data is received, so you’ll be able to disable and re-enable the package at will.

      We want to run the java application zorka-core-test.jar with our Java agent specified in the -javaagent JVM option.

      java -javaagent:/opt/netuitive-zorka/netuitive.jar=/opt/netuitive-zorka -jar zorka-core-test.jar

Additional Configuration Options

Enable a Proxy

    1. Navigate to the zorka.properties file.
    2. Find the proxy section:
      netuitive.api.proxy = no
      netuitive.api.proxy.address = http://<proxy host>:<proxy port>
      
    3. Change the netuitive.api.proxy line to yes.
    4. Add the correct proxy host and port to the netuitive.api.proxy.address line.

Configuring the Java agent for Custom MBeans

The Java agent can collect metrics from custom mbeans. We have provided a sample spring boot application that creates 2 custom mbeans with test attributes here.

Multiple Custom Mbean Diagram

For a single custom mbean:

  1. Navigate to the zorka.properties file in your Java agent directory.
  2. Near the bottom of the file, set the attribute netuitive.api.custom.stats.mbean to the custom mbean you defined in your application.
    #custom mbean to collect metrics from
    netuitive.api.custom.stats.mbean = com.netuitive.mbean:type=Test,name=CustomTestMBean
    Leave the netuitive.api.custom.stats.mbean.attr.includeattribute blank to include all metrics.
  3. Save the file.

For multiple mbeans:

We’ve provided a sample .bsh script that creates getter objects for each of the mbeans in the sample multiple mbean spring boot application file.
  1. Create a .bsh script that contains getter objects for each of your custom mbean values. In the same script, create a rollup mbean and add the getter objects you created in step 1 as attributes of the rollup mbean.
  2. Navigate to the zorka.properties file in your Java agent directory.
  3. Add the script you created in step 1 to the scripts setting.
    #Default collection of jvm metrics
    scripts = jvm.bsh, custom-mbean.bsh
  4. Near the bottom of the file, set the attribute netuitive.api.custom.stats.mbean to the new rollup mbean you created in step 2.
    #custom mbean to collect metrics from
    netuitive.api.custom.stats.mbean 
    = com.netuitive.mbean:type=Test,name=RollupTestMBean
    Leave the netuitive.api.custom.stats.mbean.attr.includeattribute blank to include all metrics.
  5. Save the file.

Metrics

In the following tables, the BASE column indicates whether there’s a baseline band available for the metric, the CORR column indicates whether there’s a contextual band available for the metric, and the UTIL column indicates whether the metric can be used as a utilization metric in the Utilization and Utilization Boxplot Reports.

Collected

Friendly Name Fully Qualified Name (FQN) Description Statistic Units Min Max Sparse Data Strategy (SDS) BASE CORR UTIL
Method Calls *.calls The number of calls made to the method. sum count 0 none none
Method Errors *.errors The number of method calls that resulted in errors. sum count 0 none none
Total Method Execution Time *.time The amount of time spent executing the method totaled across all calls. sum milliseconds 0 none none
Average Method Execution Time *.avg_time The amount of time spent executing the method, averaged across all
calls.
average milliseconds 0 none none
Operating System Process CPU Load cpu.used.percent The percentage of CPU used by the JVM. average percent 0 100 none
GC PS MarkSweep Collection Time gc.psmarksweep.collectiontime The amount of time the JVM spent doing “PS MarkSweep” garbage
collection.
average milliseconds 0 300000 none
GC PS Scavenge Collection Time gc.psscavenge.collectiontime The amount of time the JVM spent doing “PS Scavenge” garbage collection. average milliseconds 0 300000 none
GC Concurrent MarkSweep Collection Time gc.concurrentmarksweep.collectiontime The amount of time the JVM spent doing “Concurrent MarkSweep” garbage
collection.
average milliseconds 0 300000 none
GC ParNew MarkSweep Collection Time gc.parnew.collectiontime The amount of time the JVM spent doing “ParNew MarkSweep” garbage
collection.
average milliseconds 0 300000 none
GC G1 Old Generation Collection Time gc.g1oldgeneration.collectiontime The amount of time the JVM spent doing “G1 Old Generation” garbage
collection.
average milliseconds 0 300000 none
GC G1 New Generation Collection Time gc.g1newgeneration.collectiontime The amount of time the JVM spent doing “G1 Old Generation” garbage
collection.
average milliseconds 0 300000 none
Heap Committed heap.committed The total amount of memory available for the heap. average bytes 0 none none
Heap Used heap.used The total amount of memroy currently being used by the heap. average bytes 0 none none
Memory Pool Code Cache Committed mempool.codecache.committed The amount of memory currently allocated to the code cache. average bytes 0 none none
Memory Pool Code Cache Max mempool.codecache.max The maximum amount of memory that could be allocated to the code cache. average bytes 0 none none
Memory Pool Code Cache Used mempool.codecache.used The amount of committed memory currently used by the code cache. average bytes 0 none none
Memory Pool Compressed Class Space Committed mempool.compressedclassspace.committed The amount of memory currently allocated to the compressed class space. average bytes 0 none none
Memory Pool Compressed Class Space Max mempool.compressedclassspace.max The maximum amount of memory that could be allocated to the compressed
class space.
average bytes 0 none none
Memory Pool Compressed Class Space Used mempool.compressedclassspace.used The amount of committed memory currently used by the compressed class space. average bytes 0 none none
Memory Pool Metaspace Committed mempool.metaspace.committed The amount of memory currently allocated to the metaspace. average bytes 0 none none
Memory Pool Metaspace Max mempool.metaspace.max The maximum amount of memory that could be allocated to the metaspace. average bytes 0 none none
Memory Pool Metaspace Used mempool.metaspace.used The amount of committed memory currently used by the metaspace. average bytes 0 none none
Memory Pool PS Eden Space Committed mempool.psedenspace.committed The amount of memory currently allocated to the eden space. average bytes 0 none none
Memory Pool PS Eden Space Max mempool.psedenspace.max The maximum amount of memory that could be allocated to the eden space. average bytes 0 none none
Memory Pool PS Eden Space Used mempool.psedenspace.used The amount of committed memory currently used by the eden space. average bytes 0 none none
Memory Pool PS Old Gen Committed mempool.psoldgen.committed The amount of memory currently allocated to the old generation. average bytes 0 none none
Memory Pool PS Old Gen Max mempool.psoldgen.max The maximum amount of memory that could be allocated to the old
generation.
average bytes 0 none none
Memory Pool PS Old Gen Used mempool.psoldgen.used The amount of committed memory currently used by the old generaion. average bytes 0 none none
Memory Pool PS Perm Gen Committed mempool.pspermgen.committed The amount of memory currently allocated to the permanent generation. average bytes 0 none none
Memory Pool PS Perm Gen Max mempool.pspermgen.max The maximum amount of memory that could be allocated to the permanent generation. average bytes 0 none none
Memory Pool PS Perm Gen Used mempool.pspermgen.used The amount of committed memory currently used by the permanent generation. average bytes 0 none none
Memory Pool PS Surivor Space Committed mempool.pssurvivorspace.committed The amount of memory currently allocated to the survivor space. average bytes 0 none none
Memory Pool PS Survivor Space Max mempool.pssurvivorspace.max The maximum amount of memory that could be allocated to the survivor space. average bytes 0 none none
Memory Pool PS Survivor Space Used mempool.pssurvivorspace.used The amount of committed memory currently used by the survivor space. average bytes 0 none none
Loaded Classes system.LoadedClasses The number of classes loaded by the JVM. average count 0 none none
Threads system.threads The number of currently running threads. average count 0 none none
Unloaded Classes system.UnloadedClasses The number of classes unloaded by the JVM. average count 0 none none

Computed

Fully Qualified Name (FQN) Description Statistic Units Min Max BASE CORR UTIL Related Default policies
netuitive.jvm.heap.utilizationpercent Percentage of the allocated heap memory that is currently in use by the JVM’s heap.

Computation:
(Heap Used / Heap Committed) * 100

average percent 0 100 Elevated JVM Heap Usage
netuitive.method.*.errorpercent The percentage of method calls that resulted in errors.

Computation:
(Method Calls == 0) ? 0 : (Method Errors / Method Calls) * 100

average percent 0 100

Additional Information

Instrumenting Metric Values

  1. First, set up the Java agent.
  2. In the zorka/scripts/ directory, create a .bsh file for the application you want to monitor.
  3. Call zorka.require to load any extension scripts your application depends on.

    The example below references jvm.bsh (the minimum required script).

    zorka.require("jvm.bsh");
  4. Define the function(s) you want Zorka to monitor using the template below. The template will establish namespace by creating a function that returns a reference to its own instance and then defines a variable that holds an instance of the function.
    __myapp() {
    
      //function code
      //goes here
    
      return this;
    }
    
    myapp = __myapp();
  5. Within the function(s) you created in the previous, complete the following:
    1. Define the JMX mbean that will host the method call statistics.
      _mbean = "zorka:type=ZorkaStats,name=MyAppStats";
    2. Then, create a configuration section that zorka can “spy” on.
      spy.add(spy.instrument("MyAppRequests"))
    3. Underneath the configuration section:
      1. Optionally, add an action(s) on method entry, e.g., logging a request occurred, successful login, etc.
        .onEnter(spy.zorkaLog("INFO", "MyApp", "Request occurred."))

        If you have multiple actions on entry, they should be separated by a comma:

        .onEnter(
          spy.fetchArg("operator", 1),
          spy.zorkaLog("INFO", "MYAPP", "${operator}"))
      2. Optionally, declare additional metric names based on your method call parameters.
        spy.zorkaStats(mbsName, beanName, attrName, keyExpr, timeField, actions)

        …where:

        • mbsName is name of your mbean server (typically java or jboss).
        • beanName is the name of the mbean used to host statistics in the file.
        • attrName is the name of the attribute where the ZorkaStats object will be visible.
        • keyExpr is the value used as a key in ZorkaStats.
        • timeField indicates the field that stores method execution time (the defaults to T if not passed).
        • actions determines what should be done when aggregating data.
      3. Also underneath the configuration section, specify which method will be instrumented.
        .include(spy.byMethod("com.netuitive.agent.myappclass", "invoke")));
        You can also create a Spy API wildcard to send in metrics instead of sending instrumenting individually. Read more about that here.
  6. Navigate to the zorka.properties file, and add the file you created in step 2 to the list of loaded scripts.
    scripts = jvm.bsh, my-app.bsh
  7. Update the zorka.application setting to your application’s name.
    zorka.application = my-app
  8. Save all edited files and restart the Java agent. You should find your JVM element in Metricly with general JVM system metrics and MyAppStats method call metrics.
Here’s an example file called “calculator.bsh” that instruments metrics based on basic calculator operation method calls. When a user inputs an operator, the application will fetch the argument, calculate the result, and log the action into the Zorka Log file. Once the actions are submitted to the agent on the Java server, Metricly will create a metric for each input into the ${operator} parameter (e.g., a metric for +, -, *, and /).


// Call zorka.require(...) to load additional scripts that this one depends on.
zorka.require("jvm.bsh");

// Simulate namespace.
__calculator() {

// Define JMX bean to host method call stats.
_mbean = “zorka:type=ZorkaStats,name=CalculatorStats”;

// Add the spy definition to instrument method “calculator”
// of “com.netuitive.agent.test.Calculator” class.
// The “calculate” method call statistics (calls, errors, time)
// are grouped/keyed by the method’s first argument (“operator”).
spy.add(spy.instrument(“CALCULATOR”)
.onEnter(
spy.fetchArg(“operator”, 1),
spy.zorkaLog(“INFO”, “CALCULATOR”, “${operator}”))
.onSubmit(
spy.zorkaStats(“java”, _mbean, “stats”, “${operator}”))
.include(spy.byMethod(“com.netuitive.agent.test.Calculator”, “calculate”))
);

return this;
}

calculator = __calculator();

  • Calculator.java source


    package com.netuitive.agent.test;

    public class Calculator {

    public Integer calculate(String operator, Integer first, Integer second) {

    if (operator.equals(“+”)) {
    return add(first, second);
    } else if (operator.equals(“-“)) {
    return minus(first, second);
    } else if (operator.equals(“*”)) {
    return multiply(first, second);
    } else if (operator.equals(“/”)) {
    return divide(first, second);
    } else {
    throw new IllegalArgumentException(“‘” + operator + “‘ is not supported, use one of [+|-|*|/] operators”);
    }
    }

    private Integer add(Integer first, Integer second) {
    return first + second;
    }

    private Integer minus(Integer first, Integer second) {
    return first – second;
    }

    private Integer multiply(Integer first, Integer second) {
    return first * second;
    }

    private Integer divide(Integer first, Integer second) {
    return first / second;
    }
    }

Upgrading the Java Agent

  1. Download the latest Java agent here.
  2. Copy the new netuitive.jar file to your existing Java agent files to replace the old netuitive.jar file.
  3. Optionally, manually merge the new zorka.properties file with the old file to receive any new fields or settings.

Uninstalling the Java Agent

  1. Remove any JVM startup references to the Netuitive Java Agent.
    java -javaagent:/opt/netuitive-zorka/netuitive.jar=/opt/netuitive-zorka -jar zorka-core-test.jar -- service
    If you instrumented metrics using the jvm.bsh script, you will need to remove the references to the script as well.
  2. Delete all Netuitive Java Agent (netuitive-zorka-{version}) files.