Custom Ingest

Options are sent to custom ingest via HTTP Request Headers. The name of the request header is X-Metricly-Api-Options and it takes a comma separated list of options.

Metricly Request Header X-Metricly-Api-Options shown in a curl request

  • curl -X POST -H "Content-Type: application/json" -H "X-
    SEND_RESPONSE_BODY,INJECT_TIMESTAMP" -d @- $endpoint* Connected to localhost (::1) port 8000 (#0)
    > POST /ingest/api0custom0100000000000000000101 HTTP/1.1
    > Host: localhost:8000
    > User-Agent: curl/7.43.0
    > Accept: */*
    > Content-Type: application/json
    > Content-Length: 859

The table below describes the implemented options.

Option Description
SEND_RESPONSE_BODY When present, the custom ingest service sends a detailed “Success Response” body. See the sample Success Response below. When this option is not present, no response body is sent.

A response body is always sent when there is an error.
INJECT_TIMESTAMP When present, the custom ingest service injects the current server timestamp into the element sample’s “timestamp” property. This will overwrite any passed-in timestamps if they exist.Ws

Table 1-15: Implemented custom ingest options.

Ingest Responses

There are two types (not including 500 errors) of responses returned from the custom ingest service:

  • 202 : ACCEPTED
    • Metricly has received the request and queued it for processing. This does not mean the Element will appear in Metricly. The service is not validating if the timestamp sent to Metricly is in a valid time range.
  • 400 : BAD_REQUEST
    • Metricly is not able to process the request due to some sort of error in the message:
      • Validation Error: There was an error in validating the message. The error will be returned in the message.
      • Parsing Error: There was an error parsing the message passed into Metricly.

The responses delivered to the client have been standardized. Except for the case of a 500 error, the responses are the same minus one exception: the Error Response contains a field named “errors” that has an array of error messages. The format of the messages can be seen in examples below.

On a 202 : ACCEPTED response, the response body is only returned when the client requests it via the HTTP header option SEND_RESPONSE_BODY. On a 400 : BAD_REQUEST response, a response body is always returned. There is no option to turn this off.

Success Response

  • {
    "status": 202,
    "message": "ACCEPTED",
    "detail": "2 elements received",
    "serverTimestamp": 1442604173135,
    "iso8601DateTime": "2015-09-18T19:22:40.153Z"

Error Response – Validation Failure

  • {
    "status": 400,
    "message": "BAD_REQUEST",
    "detail": "Input Validation Failed",
    "errors": [
    "Missing required field on Element, 'id'"
    "serverTimestamp": 1442604160153,
    "iso8601DateTime": "2015-09-18T19:22:40.153Z"

Error Response – Validation Error: Missing timestamp without passing ‘INJECT_TIMESTAMP’ header

  • {
    "status": 400,
    "message": "BAD_REQUEST",
    "detail": "Input Validation Failed",
    "errors": [
    "Missing required field on Sample, 'timestamp'. See ingest documentation to see how to send in samples without timestamps."
    "serverTimestamp": 1442852458120,
    "iso8601DateTime": "2015-09-21T16:20:58.120Z"

Error Response – JSON Parse Error: Bad date format passed in

  • {
    "status": 400,
    "message": "BAD_REQUEST",
    "detail": "Error Parsing JSON Request",
    "errors": [
    "Could not read JSON: Can not construct instance of java.util.Date from String value 'ZZZZZZZZZZZZZZZZZ': not a valid representation (error: Failed to parse Date value 'ZZZZZZZZZZZZZZZZZ': Failed to parse date [\"ZZZZZZZZZZZZZZZZZ']: Invalid number: ZZZZZZZZZZZZZZZZZ)\n at [Source: org.apache.catalina.connector.CoyoteInputStream@8a57dbd; line: 1, column: 240] (through reference chain: java.util.ArrayList[0]->com.metricly.ingest.entity.IngestElement[\"samples\"]->java.util.HashSet[0]->com.metricly.ingest.entity.IngestSample[\"timestamp\"]); nested exception is com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct instance of java.util.Date from String value 'ZZZZZZZZZZZZZZZZZ': not a valid representation (error: Failed to parse Date value 'ZZZZZZZZZZZZZZZZZ': Failed to parse date [\"ZZZZZZZZZZZZZZZZZ']: Invalid number: ZZZZZZZZZZZZZZZZZ)\n at [Source: org.apache.catalina.connector.CoyoteInputStream@8a57dbd; line: 1, column: 240] (through reference chain: java.util.ArrayList[0]->com.metricly.ingest.entity.IngestElement[\"samples\"]->java.util.HashSet[0]->com.metricly.ingest.entity.IngestSample[\"timestamp\"])"
    "serverTimestamp": 1442844006245,
    "iso8601DateTime": "2015-09-21T14:00:06.245Z"

Custom Ingest FAQ and Examples

There are a few instances in which an error code is received:

  • The timestamp is not properly formatted. See the FAQ below for the proper format.
  • The timestamp is properly formatted; however, it is not within a period of time that will be processed by Metricly.
  • Metrics or Samples have been sent without wrapping them in a valid Element consisting of an, and a Sample.metricId.

Frequently Asked Questions

  • What is the endpoint to get to Metricly custom ingest?
    • Production:{API_KEY}
    Metricly uses a POST method.
  • What is an API Key, and how do I get one/find mine?
    • An API Key is generated by Metricly for your account. You can retrieve any of your API Keys from the API Keys page under the Account Profile drop-down menu.
  • What format do you require your timestamp to be in when sending samples?
    • Timestamps must be the number of milliseconds since the Unix Epoch. Now the stand date format on many Linux/Unix systems use seconds since the epoch and therefore require padding to be used. The following examples define ways to calculate the correct time in several circumstances:
      • Unix/Linux/Mac OS X using the date command returns seconds, so additional padding is needed to get it to milliseconds.
        • input: date +%s000
        • output: 144147539000
      • Java/Groovy contain the “System.currentTimeMillis()” method, which returns a properly formatted timestamp.
        • input: System.currentTimeMillis()
        • output: 1441475930587
  • What are valid timestamps accepted by Metricly custom ingest endpoint?
    • The following formats have been tested successfully:
      • Unix epoch in milliseconds
        • 1441475739000
      • ISO 8601 formats:
        • 2015-09-07T14:26:47.457-0400
        • 2015-09-07T14:26:47-0400
        • 2015-09-08T13:39:33.948Z
    • The following ISO 8601 formats do not work:
      • 2015-09-07T14:26:47EDT
  • Can I send more than one Element at a time?
    • Yes, the posted JSON message is always an array of Elements. See the sample payload below. It sends two Elements at a time to custom ingest.
  • What is the metric name field used for?
    • Currently it is only used internally. In the future, we plan to use it in the UI similar to the Element name. Currently, the metric ID is also used for its name in the UI.
  • What is the element name?
    • The element name is used as the display name in the UI.
  • What client tools can I use to test sending data to Metricly’s ingest endpoint?
    • Command Line tools:
      • curl is the de facto command line tool that is available across most OSs. It is available on Linux/Unix/Mac OSs and available as a download on other OSs–including Windows–from the curl website. Examples below will use curl.
    • OS Agnostic UI tools:
  • What options can I send to the Metricly’s ingest endpoint?
  • What happens if the API detects unknown properties in a payload?
    • The API ignores unknown properties and allows the rest of the message to proceed if it is valid.
  • Is it possible to create relationships between elements in Metricly’s ingest endpoint?
    • Yes. Relationships can be added as an array of relations. Relationships can be uni- or bi-directional. Metricly only creates the side of the relationship specified in the JSON payload. For more information, see our Ingest page.
      If you submit an element with “id” = “webserver01”that contained a relationship “fqn”:”load-balancer-east”, Metricly would create a relationship from webserver01 to load-balancer-east. It would not automatically create a relationship from load-balander-east to webserver01; however, you can add the inverse relationship to load-balancer-east via a separate API call.

Example Scenario

Say you want to send 2 Elements to the Metricly ingest endpoint: one server named “webhost01” and an instance of Tomcat 7 named “webapp01”. For webhost01, you’d like to collect the memory utilization and cpu utilization, and you’d also like to associate an attribute noting what datacenter the server is runnign in. For webapp01, you’d like to collect requests per second and connection pool count, and you’d also like to associate an attribute that describes what webserver type the instance is running.

To implement the plan outlined above:

  1. Array of Elements (JSON root)
    1. id: Choose a unique (across your environment) ID for each of your elements.
    2. name: The name will be displayed in the Metricly UI.
    3. type: This should describe the type of asset you are using (e.g., server, firewall, database, webserver, storage).
  2. Metrics: define the metrics you want to capture for each element
    1. id: The ID of the metric. This should uniquely identify it within the Element. Metricly also uses this as the display name in the UI.
    2. name: Currently the metric name is not used as the ID is used for the display name in the UI.
  3. Samples:
    1. metricId: This relates the sample to the
    2. timestamp: UTC time in milliseconds since the epoch.
    3. val: The value for this sample.

Example Payload

In the JSON request below, two Elements are passed whose IDs are webhost01 and webapp01.

    "id": "webhost01",
    "name": "Web Host 01",
    "type": "server",
    "metrics": [
        "id": "memutilization",
        "name": "Memory Utilization %"
        "id": "cpuutilization",
        "name": "CPU Utilization %"
    "samples": [
        "metricId": "memutilization",
        "timestamp": 1441465330000,
        "val": 68.0
        "metricId": "cpuutilization",
        "timestamp": 1441465330000,
        "val": 14.0
    "attributes": [
        "value": "aws-east-1a"
    "id": "webapp01",
    "name": "Web App 01",
    "type": "webapp",
    "metrics": [
        "id": "requestspersecond",
        "name": "Requests per second"
        "id": "connectionpoolcount",
        "name": "Connection Pool Count"
    "samples": [
        "metricId": "requestspersecond",
        "timestamp": 1441465330000,
        "val": 4.0
        "metricId": "connectionpoolcount",
        "timestamp": 1441465330000,
        "val": 40.0
    "attributes": [
        "name": "product",
        "value": "tomcat7"


The shell script is a simple tool that helps submit custom messages to the Metricly custom endpoint. It allows you to create a JSON file (as shown in the example below) and replace all of the timestamps with the literal TIMESTAMP. When you run the shell script, it will replace all occurrences of TIMESTAMP with the current, valid timestamp.



$ ./ -f custom_injest_msg_01.json -e -v
Submitting message to
[ { "id": "webhost01", "name": "Web Host 01", "type": "server", 
"metrics": [ { "id": "memutilization", "name": "Memory Utilization 
%" }, { "id": "cpuutilization", "name": "CPU Utilization %" } ], 
"samples": [ { "metricId": "memutilization", "timestamp": 
1441472743000, "val": 68.0 }, { "metricId": "cpuutilization", 
"timestamp": 1441472743000, "val": 14.0 } ], "attributes": [ { 
"name":"datacenter", "value": "aws-east-1a" } ] }, { "id": 
"webapp01", "name": "Web App 01", "type": "webapp", "metrics": [ { 
"id": "requestspersecond", "name": "Requests per second" }, { "id": 
"connectionpoolcount", "name": "Connection Pool Count" } ], 
"samples": [ { "metricId": "requestspersecond", "timestamp": 
1441472743000, "val": 4.0 }, { "metricId": "connectionpoolcount", 
"timestamp": 1441472743000, "val": 40.0 } ], "attributes": [ { 
"name": "product", "value": "tomcat7" } ] } ]
*   Trying
* Connected to ( port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *
* Server certificate: RapidSSL SHA256 CA - G3
* Server certificate: GeoTrust Global CA
> POST /ingest/5af6cbf HTTP/1.1
> Host:
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 868
* upload completely sent off: 868 out of 868 bytes
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 3600
< Date: Sat, 05 Sep 2015 17:05:44 GMT
< Server: nginx
< Vary: Origin
< X-Application-Context: application:8080
< Content-Length: 0
< Connection: keep-alive
* Connection #0 to host left intact