Circonus supports data ingestion from a wide variety of sources in all kinds of formats. This data is transformed and processed internally and ultimately stored in the form of 1M-rollup-aggregates, that come in two flavors (cf. DataTypes):

1. Numeric metrics, which store a number of summary statistics, including the mean value across the rollup-window.

2. Histogram metrics, which store the sample distribution as key-value pairs (Histogram Representation).

This note explains how these rollup-aggregates are derived from the incoming raw-data.

1. Rolling-up Numeric Data

We will cover the most common case first, that raw numeric values are submitted to Circonus, one at a time.

The Numeric raw values r that arrive at Circonus have an associated timestamp, r.t: long, which is a (long) integer in milliseconds, and an associated value, r.v: float.

The stream of incoming raw values is divided into rollup windows, w, of equal duration, which is the rollup period, p. Currently, the minimum supported rollup period is 1 minute (p = 60000).

The rollup windows carry a start time, w.start, that is a multiple of the rollup period. The window end time is defined to be w.end = w.start + p.

A raw value, r, belongs to a rollup window, w, if:

w.start < r.t and r.t < w.end

The rollup process takes a list of raw-values, R, that belong to a rollup window, w, and are sorted in order of the time-stamp, and computes the following of summary statistics over the raw values:

• w.count is the number of samples in R: 
  
        w.count = #R
  
  
• w.value is the average value of all sample values in R:
  

           

  
  
• w.stddev is the sample standard devation of all samples in R. Requires: #R > 2
            
  
  
• w.derivative is the effective rate of change of the samples per second. It is defined as:
  

           
  Note, that there are 1000 milliseconds in one second. There is an alternative way to compute w.derivative that will be used later on. It is based on the notion of the difference series DT and DV of R which are:
  
       DV[i] = R[i+1].v - R[i].v

     DT[i] = R[i+1].t - R[i].t

Here is the alternative computation:

• w.derivative_stddev is defined as the (uncorrected) standard derivation of the difference series DV weighted by DT:
  
  The weighted standard deviation is defined as:
  

           

  
  
• w.counter, the counter derivative rollup statistics, is similar to the derivative rollup, but is suitable to raw value streams that experience overflow events that should be ignored by the metric. Examples for such streams include the number of octects observed by a network switch, or the total visit count of a server that gets restarted occasionally.
  
  To define the counter derivative rollup, we use the DV and DT series from above. Let CV and CT be copies of DV and DT where indices i with DV[i] < 0 are removed from both series. Then:
  

           

  
  
• w.counter_stddev. In analogy to w.derivative_stddev, this rollup is defined as the (uncorrected) standard derivation of the monotone difference series CV weighted by CT:
  
            w.counter_derivative_stddev = stddev(CV,CT)

In addition to those numeric rollup-aggregates, Circonus also supports rollup to Histogram metrics (cf. Histograms, Histogram Internals). For numeric input values those are calculated as follows:

• w.histogram, the histogram statistic, can represented as an associative array that maps bucket boundaries to sample counts:
  
          
  
  here b represents a bucket boundary, which can be any float value of the form b = x * 10^e, where
  • x is one of the following float values: x = -9.9, -9.8, -9.7 .. -1.1, -1.0, 0, 1.0, 1.1, .. ,9.9
  • e is an integer in the following range: e = -128 .. 127
  • Bucket(b) is the bucket with boundary b, e.g. if b = 1, then Bucket(b) is the interval [1, 1.1).
    
    For more details, see Histogram Internals.
    
    

2. Rolling-up Histogram Data

In addition to numeric input values, Circonus also accepts histogram data submitted directly via a HTTP Trap, in the form of:

• Raw histograms as arrays of numeric values, e.g. [1, 2.7, 5]
• Raw histograms in pre-bucketed form, e.g.  ["H[0.1]=3", "H[11]=7"]

You can mix both types of data submission with each other and with numeric raw-values as well. 

The resulting histogram rollup aggregates all data that was submitted for a time period by summing the counts for each bucket.

Example

If the following values are submitted:

| t   | value                  |
|-----+------------------------|
| 0   | 1                      |
| 100 | [1,2,3]                |
| 200 | ["H[1]=3", "H[5]=3"]   |
| 300 | 2                      |
| 400 | [3,4,5]                |
| 500 | ["H[5]=3", "H[6]=3"]   |

The resulting histogram for the time period [0, 1000] is, given by:

H[1] = 1 + 1 + 3 + 0 + 0 + 0  = 5

H[2] = 0 + 1 + 0 + 1 + 0 + 0  = 2

H[3] = 0 + 1 + 0 + 0 + 1 + 0  = 2

H[4] = 0 + 0 + 0 + 0 + 1 + 0  = 1

H[5] = 0 + 0 + 3 + 0 + 1 + 3  = 7

H[6] = 0 + 0 + 0 + 0 + 0 + 3  = 3

Numeric Rollup of Histogram Values

When pushing raw-histogram values, numeric aggregates are also performed. However, due to the details of the data collection, the resulting rollup values are often perceived as un-intuitive. In particular, they are not equivalent to pushing the containing samples individually. Hence, we make the following recommendation:

WARNING: If you use raw-histogram values, ONLY the resulting histogram metric should be used for further computations!

Now, that we have warned you, we will tell you what values end up in the numeric metric, when histogram values are pushed.

1. When a Raw histogram array is pushed, the numeric metric will receive a single sample with the average value of the array.
2. When a Raw pre-bucketed histogram is pushed, the numeric metric will receive a nil-sample.

Therefore the numeric statistics count and average are are calculated as follows:

    w.count   = 1 + 1 + 1 + 1 + 1 + 1 = 6

    w.average = 1 + 2 + 0 + 2 + 4 + 0 = 9

Note that, these are different from the total count of all samples (21) and the total average (84/21=4.0).

The recommended way to derive those values is to use our Analytics Query Language, CAQL. E.g.

     metric:histogram(...) | histogram:count()

     metric:histogram(...) | histogram:mean()

     metric:histogram(...) | histogram:stddev()

CAQL would compute approximations of those respective values from the histogram itself.

Appendix: Numeric Rollup Example