Examples of com.linkedin.databus.client.pub.mbean.UnifiedClientStats

Package com.linkedin.databus.client.pub.mbean

Examples of com.linkedin.databus.client.pub.mbean.UnifiedClientStats

com.linkedin.databus.client.pub.mbean.UnifiedClientStats
Joint relay/bootstrap client-library metrics for Databus v2 and v3 consumers. At the lowest level, these stats may represent a single table (in a v2 multi-tenant setup) or a single partition for a single table (in a client load-balancing [CLB] setup); in the latter case, they may not even be exposed. The same class is also used to aggregate stats over multiple partitions and/or tables. Thus in the CLB case, for example, the lowest level is one partition of a table; the second level is aggregated across all partitions for a single table; and the third level is aggregated across all tables [for a single DB or for all subscriptions? TODO/FIXME]. This class is intended to supersede ConsumerCallbackStats for consumers.


          writeEventToRelayDispatcher(curState, regressEvent, "SCN Regress Event from ckpt :" + curState.getCheckpoint());
          curState.setSCNRegress(false);
        }


        UnifiedClientStats unifiedClientStats = _sourcesConn.getUnifiedClientStats();
        if (unifiedClientStats != null)
        {
          unifiedClientStats.setBootstrappingState(false);  // failsafe:  we're definitely not bootstrapping here
          sendHeartbeat(unifiedClientStats);
        }
        int eventsNum = curState.getDataEventsBuffer().readEvents(readChannel,
                                                                  curState.getListeners(),
                                                                  connCollector);

View Full Code Here

      }
    } else {
      _log.info("Requested scn " + cp.getWindowScn() +
          " not found on relay; switching to bootstrap service");
      curState.switchToBootstrap(cp);
      UnifiedClientStats unifiedClientStats = _sourcesConn.getUnifiedClientStats();
      if (unifiedClientStats != null)
      {
        unifiedClientStats.setBootstrappingState(true);
      }
    }
    return enqueueMessage;
  }

View Full Code Here

    _log.info("Event size: " + eventBytes.length);
    _log.info("Event:" + event.toString());


    event.getRawBytes().get(eventBytes);


    UnifiedClientStats unifiedClientStats = _sourcesConn.getUnifiedClientStats();


    while ((!success) && (timer.getRemainingRetriesNum() > 0))
    {
      ByteArrayInputStream cpIs = new ByteArrayInputStream(
          eventBytes);

View Full Code Here

   */
  @Test
  public void testBasicHistogramMetrics()
  {
    // (1) create stats object
    UnifiedClientStats unifiedClientStats = new UnifiedClientStats(3 /* ownerId */, "stats_name", "stats_dim");


    for (int i = 0; i < 200; ++i)
    {
      // Without the ability to override System.currentTimeMillis() (or hacking UnifiedClientStats to use an
      // overridable method to provide the time, and then overriding it here), there's a small chance that
      // our System.currentTimeMillis() call and that in registerDataEventReceived() will return values that
      // differ by a non-constant amount (i.e., jitter).  But we can manage that with inequalities in our
      // assertions.
      // Expected histogram values for timeLagSourceToReceiptMs range from 0 to 1990 ms (approximately).
      long sourceTimestampNs = (System.currentTimeMillis() - 10*i) * DbusConstants.NUM_NSECS_IN_MSEC;


      // We have perfect control over the values for timeLagConsumerCallbacksMs.  Make calculations trivial:
      // histogram values will be 0 through 199 ms (exactly).
      long callbackTimeElapsedNs = (long)i * DbusConstants.NUM_NSECS_IN_MSEC;


      // (2) create 200 fake DbusEvents
      DbusEvent dbusEvent = createEvent(sourceTimestampNs);


      // (3) call registerDataEventReceived() and registerCallbacksProcessed() for each event
      // (normally there are more of the latter since there are more callback types than just onDataEvent(),
      // but it doesn't really matter, and it simplifies things if we keep a fixed ratio--here just 1:1)
      unifiedClientStats.registerDataEventReceived(dbusEvent);
      unifiedClientStats.registerCallbacksProcessed(callbackTimeElapsedNs);
    }


    // (4) verify histogram values are as expected


    // Both metrics-core and Apache Commons Math use the "R-6" quantile-estimation method, as described
    // at http://en.wikipedia.org/wiki/Quantile .
    //
    // N = 200
    // p = 0.5, 0.9, 0.95, 0.99
    // h = (N+1)*p = 100.5, 180.9, 190.95, 198.99
    //
    // Q[50th]  =  x[100-1] + (100.5  - 100)*(x[100-1+1] - x[100-1])  =   99.0 + 0.5 *(100.0 -  99.0)  =   99.5
    // Q[90th]  =  x[180-1] + (180.9  - 180)*(x[180-1+1] - x[180-1])  =  179.0 + 0.9 *(180.0 - 179.0)  =  179.9
    // Q[95th]  =  x[190-1] + (190.95 - 190)*(x[190-1+1] - x[190-1])  =  189.0 + 0.95*(190.0 - 189.0)  =  189.95
    // Q[99th]  =  x[198-1] + (198.99 - 198)*(x[198-1+1] - x[198-1])  =  197.0 + 0.99*(198.0 - 197.0)  =  197.99
    assertEquals("unexpected timeLagConsumerCallbacksMs 50th percentile",
                 99.5,
                 unifiedClientStats.getTimeLagConsumerCallbacksMs_HistPct_50());
    assertEquals("unexpected timeLagConsumerCallbacksMs 90th percentile",
                 179.9,
                 unifiedClientStats.getTimeLagConsumerCallbacksMs_HistPct_90());
    assertEquals("unexpected timeLagConsumerCallbacksMs 95th percentile",
                 189.95,
                 unifiedClientStats.getTimeLagConsumerCallbacksMs_HistPct_95());
    assertEquals("unexpected timeLagConsumerCallbacksMs 99th percentile",
                 197.99,
                 unifiedClientStats.getTimeLagConsumerCallbacksMs_HistPct_99());
    assertEquals("unexpected timeLagConsumerCallbacksMs max value",
                 199.0,
                 unifiedClientStats.getTimeLagConsumerCallbacksMs_Max());


    // See sourceTimestampNs comment above.  Approximately:
    // Q[50th]  =  x[100-1] + (100.5  - 100)*(x[100-1+1] - x[100-1])  =   990.0 + 0.5 *(1000.0 -  990.0)  =   995.0
    // Q[90th]  =  x[180-1] + (180.9  - 180)*(x[180-1+1] - x[180-1])  =  1790.0 + 0.9 *(1800.0 - 1790.0)  =  1799.0
    // Q[95th]  =  x[190-1] + (190.95 - 190)*(x[190-1+1] - x[190-1])  =  1890.0 + 0.95*(1900.0 - 1890.0)  =  1899.5
    // Q[99th]  =  x[198-1] + (198.99 - 198)*(x[198-1+1] - x[198-1])  =  1970.0 + 0.99*(1980.0 - 1970.0)  =  1979.9
    // ...but allow +/-1 for jitter
    double percentile = unifiedClientStats.getTimeLagSourceToReceiptMs_HistPct_50();
    assertTrue("unexpected timeLagSourceToReceiptMs 50th percentile: " + percentile,
               994.0 <= percentile && percentile <= 996.0);    // nominal value is 995.0


    percentile = unifiedClientStats.getTimeLagSourceToReceiptMs_HistPct_90();
    assertTrue("unexpected timeLagSourceToReceiptMs 90th percentile: " + percentile,
               1798.0 <= percentile && percentile <= 1800.0);  // nominal value is 1799.0


    percentile = unifiedClientStats.getTimeLagSourceToReceiptMs_HistPct_95();
    assertTrue("unexpected timeLagSourceToReceiptMs 95th percentile: " + percentile,
               1898.5 <= percentile && percentile <= 1900.5);  // nominal value is 1899.5, but saw 1900.45 once


    percentile = unifiedClientStats.getTimeLagSourceToReceiptMs_HistPct_99();
    assertTrue("unexpected timeLagSourceToReceiptMs 99th percentile: " + percentile,
               1978.9 <= percentile && percentile <= 1980.9);  // nominal value is 1979.9
  }

View Full Code Here

    SelectingDatabusCombinedConsumer sdccLogConsumer = new SelectingDatabusCombinedConsumer(logConsumer);


    DatabusV2ConsumerRegistration consumerReg =
        new DatabusV2ConsumerRegistration(sdccLogConsumer, sources, null);
    ConsumerCallbackStats consumerStatsCollector = new ConsumerCallbackStats(1, "test","test", true,false, null);
    UnifiedClientStats unifiedStatsCollector = new UnifiedClientStats(1, "test","test.unified");


    List<DatabusV2ConsumerRegistration> allRegistrations =
        Arrays.asList(consumerReg);
    ThreadPoolExecutor executor = (ThreadPoolExecutor)Executors.newCachedThreadPool();
    MultiConsumerCallback callback =

View Full Code Here

            new LoggedAnswer<ConsumerCallbackResult>(ConsumerCallbackResult.SUCCESS, LOG, Level.DEBUG,
                                                     "onStartSource() called"),
            50));
    EasyMock.replay(mockConsumer1);
    ConsumerCallbackStats consumerStatsCollector = new ConsumerCallbackStats(1, "test","test", true,false, null);
    UnifiedClientStats unifiedStatsCollector = new UnifiedClientStats(1, "test","test.unified");


    //Create and fire up callbacks
    List<DatabusV2ConsumerRegistration> allRegistrations =
        Arrays.asList(consumerReg);
    MultiConsumerCallback callback =

View Full Code Here

   */
  @Test
  public void testHistogramMetricsAggregationBootstrapMode()
  {
    // create stats objects:  two low-level (per-connection) ones and one aggregator
    UnifiedClientStats unifiedClientStats1 = new UnifiedClientStats(1 /* ownerId */, "test1", "dim1");
    UnifiedClientStats unifiedClientStats2 = new UnifiedClientStats(2 /* ownerId */, "test2", "dim2");
    UnifiedClientStats unifiedClientStatsAgg = new UnifiedClientStats(99 /* ownerId */, "testAgg", "dimAgg");


    unifiedClientStats2.setBootstrappingState(true);


    for (int i = 0; i < MergeableExponentiallyDecayingReservoir.DEFAULT_SIZE; ++i)  // 1028
    {
      long now = System.currentTimeMillis();
      long sourceTimestampNs1 = (now - 1000L - i) * DbusConstants.NUM_NSECS_IN_MSEC;
      long sourceTimestampNs2 = (now - 5000L - i) * DbusConstants.NUM_NSECS_IN_MSEC;


      unifiedClientStats1.registerDataEventReceived(createEvent(sourceTimestampNs1));
      unifiedClientStats2.registerDataEventReceived(createEvent(sourceTimestampNs2));
    }


    unifiedClientStatsAgg.merge(unifiedClientStats1);
    unifiedClientStatsAgg.merge(unifiedClientStats2);


    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_50(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_50());
    assertEquals("unexpected timeLagSourceToReceiptMs 90th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_90(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_90());
    assertEquals("unexpected timeLagSourceToReceiptMs 95th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_95(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_95());
    assertEquals("unexpected timeLagSourceToReceiptMs 99th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_99(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_99());


    // bootstrap mode => should return -1.0 for all percentiles
    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_50());

View Full Code Here

            new LoggedAnswer<ConsumerCallbackResult>(ConsumerCallbackResult.SUCCESS, LOG, Level.DEBUG,
                                                     "onStartSource() called"),
            1));
    EasyMock.replay(mockConsumer1);
    ConsumerCallbackStats consumerStatsCollector = new ConsumerCallbackStats(1, "test","test", true,false, null);
    UnifiedClientStats unifiedStatsCollector = new UnifiedClientStats(1, "test","test.unified");


    log.info("Create and fire up callbacks");
    List<DatabusV2ConsumerRegistration> allRegistrations =
        Arrays.asList(consumerReg);
    TimingOutMultiConsumerCallback callback =

View Full Code Here

   */
  @Test
  public void testHistogramMetricsAggregationDeadSourcesConnection()
  {
    // create stats objects:  two low-level (per-connection) ones and one aggregator
    UnifiedClientStats unifiedClientStats1 = new UnifiedClientStats(1 /* ownerId */, "test1", "dim1");
    UnifiedClientStats unifiedClientStats2 = new UnifiedClientStats(2 /* ownerId */, "test2", "dim2");
    UnifiedClientStats unifiedClientStatsAgg = new UnifiedClientStats(99 /* ownerId */, "testAgg", "dimAgg");


    // could break this into two (or more) parts and do multiple merges, but not clear there's any point...
    for (int i = 0; i < 2*MergeableExponentiallyDecayingReservoir.DEFAULT_SIZE; ++i)  // 2*1028
    {
      long sourceTimestampNs1 = (System.currentTimeMillis() - 1000L - i) * DbusConstants.NUM_NSECS_IN_MSEC;
      // no data events for connection #2 => no need for sourceTimestampNs2


      unifiedClientStats1.registerDataEventReceived(createEvent(sourceTimestampNs1));
    }


    unifiedClientStatsAgg.merge(unifiedClientStats1);
    unifiedClientStatsAgg.merge(unifiedClientStats2);


    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_50(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_50());
    assertEquals("unexpected timeLagSourceToReceiptMs 90th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_90(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_90());
    assertEquals("unexpected timeLagSourceToReceiptMs 95th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_95(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_95());
    assertEquals("unexpected timeLagSourceToReceiptMs 99th percentile for aggregated stats",
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_99(),
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_99());


    // no data values => should return -1.0 for all percentiles
    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_50());

View Full Code Here

   * and timeLagConsumerCallbacksMs metrics should be -1.0, per the design spec.
   */
  @Test
  public void testHistogramMetricsAggregationNoData()
  {
    UnifiedClientStats unifiedClientStats1 = new UnifiedClientStats(1 /* ownerId */, "test1", "dim1");
    UnifiedClientStats unifiedClientStats2 = new UnifiedClientStats(2 /* ownerId */, "test2", "dim2");
    UnifiedClientStats unifiedClientStatsAgg = new UnifiedClientStats(99 /* ownerId */, "testAgg", "dimAgg");


    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_50());
    assertEquals("unexpected timeLagSourceToReceiptMs 90th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_90());
    assertEquals("unexpected timeLagSourceToReceiptMs 95th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_95());
    assertEquals("unexpected timeLagSourceToReceiptMs 99th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagSourceToReceiptMs_HistPct_99());


    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_50());
    assertEquals("unexpected timeLagSourceToReceiptMs 90th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_90());
    assertEquals("unexpected timeLagSourceToReceiptMs 95th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_95());
    assertEquals("unexpected timeLagSourceToReceiptMs 99th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagSourceToReceiptMs_HistPct_99());


    assertEquals("unexpected timeLagSourceToReceiptMs 50th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_50());
    assertEquals("unexpected timeLagSourceToReceiptMs 90th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_90());
    assertEquals("unexpected timeLagSourceToReceiptMs 95th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_95());
    assertEquals("unexpected timeLagSourceToReceiptMs 99th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagSourceToReceiptMs_HistPct_99());




    assertEquals("unexpected timeLagConsumerCallbacksMs 50th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagConsumerCallbacksMs_HistPct_50());
    assertEquals("unexpected timeLagConsumerCallbacksMs 90th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagConsumerCallbacksMs_HistPct_90());
    assertEquals("unexpected timeLagConsumerCallbacksMs 95th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagConsumerCallbacksMs_HistPct_95());
    assertEquals("unexpected timeLagConsumerCallbacksMs 99th percentile for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagConsumerCallbacksMs_HistPct_99());
    assertEquals("unexpected timeLagConsumerCallbacksMs max for connection #1",
                 -1.0,
                 unifiedClientStats1.getTimeLagConsumerCallbacksMs_Max());


    assertEquals("unexpected timeLagConsumerCallbacksMs 50th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagConsumerCallbacksMs_HistPct_50());
    assertEquals("unexpected timeLagConsumerCallbacksMs 90th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagConsumerCallbacksMs_HistPct_90());
    assertEquals("unexpected timeLagConsumerCallbacksMs 95th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagConsumerCallbacksMs_HistPct_95());
    assertEquals("unexpected timeLagConsumerCallbacksMs 99th percentile for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagConsumerCallbacksMs_HistPct_99());
    assertEquals("unexpected timeLagConsumerCallbacksMs max for connection #2",
                 -1.0,
                 unifiedClientStats2.getTimeLagConsumerCallbacksMs_Max());


    assertEquals("unexpected timeLagConsumerCallbacksMs 50th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagConsumerCallbacksMs_HistPct_50());
    assertEquals("unexpected timeLagConsumerCallbacksMs 90th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagConsumerCallbacksMs_HistPct_90());
    assertEquals("unexpected timeLagConsumerCallbacksMs 95th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagConsumerCallbacksMs_HistPct_95());
    assertEquals("unexpected timeLagConsumerCallbacksMs 99th percentile for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagConsumerCallbacksMs_HistPct_99());
    assertEquals("unexpected timeLagConsumerCallbacksMs max for aggregated stats",
                 -1.0,
                 unifiedClientStatsAgg.getTimeLagConsumerCallbacksMs_Max());
  }

View Full Code Here

0 1 2

TOP

Related Classes of com.linkedin.databus.client.pub.mbean.UnifiedClientStats

com.codahale.metrics.MergeableExponentiallyDecayingReservoir

com.linkedin.databus.client.BootstrapPullThread

com.linkedin.databus.client.consumer.TestMultiConsumerCallback

com.linkedin.databus.client.DatabusHttpClientImpl

com.linkedin.databus.client.pub.monitoring.events.UnifiedClientStatsEvent

com.linkedin.databus.client.pub.TestUnifiedClientStats

com.linkedin.databus.client.registration.DatabusV2ClusterRegistrationImpl

com.linkedin.databus.client.registration.DatabusV2RegistrationImpl

com.linkedin.databus.client.RelayPullThread

com.linkedin.databus.client.request.ClientStatsRequestProcessor

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.