Examples of AvatarNode

org.apache.hadoop.hdfs.server.namenode.AvatarNode
This is an implementation of the AvatarNode, a hot standby for the NameNode. This is really cool, believe me! The AvatarNode has two avatars.. the Standby avatar and the Active avatar. In the Standby avatar, the AvatarNode is consuming transaction logs generated by the primary (via a transaction log stored in a shared device). Typically, the primary Namenode is writing transactions to a NFS filesystem and the Standby is reading the log from the same NFS filesystem. The Standby is also making periodic checkpoints to the primary namenode. A manual command can switch the AvatarNode from the Standby avatar to the Active avatar. In the Active avatar, the AvatarNode performs precisely the same functionality as a real usual Namenode. The switching from Standby avatar to the Active avatar is fast and can typically occur within seconds. Typically, an adminstrator will run require two shared mount points for transaction logs. It has to be set in fs.name.dir.shared0 and fs.name.dir.shared1 (similarly for edits). Then the adminstrator starts the AvatarNode on two different machines as follows: bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -zero -active bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -one -standby The first AvatarNode uses fs.name.dir.shared0 while the second AvatarNode uses fs.name.dir.shared1 to write its transaction logs. Also, at startup, the first instance is the primary Namenode and the second instance is the Standby After a while, the adminstrator decides to change the avatar of the second instance to Active. In this case, he/she has to first ensure that the first instance is really really dead. This code does not handle the split-brain scenario where there are two active namenodes in one cluster.

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

        .getStartupConf());
    return zkClient.getPrimaryLastTxId(address);
  }


  private long getSessionId() throws Exception {
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(0).avatar;
    String address = AvatarNode.getClusterAddress(primaryAvatar
        .getStartupConf());
    return zkClient.getPrimarySsId(address);
  }

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

  @Test
  public void testBlocksMisMatch() throws Exception {
    int totalBlocks = 50;
    DFSTestUtil.createFile(fs, new Path("/testBlocksMisMatch"),
        (long) totalBlocks * 1024, (short) 3, System.currentTimeMillis());
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    standby.quiesceStandby(cluster
        .getPrimaryAvatar(0).avatar.getLastWrittenTxId());
    // This sets the standby number of blocks to 0.
    standby.namesystem.close();
    try {
      cluster.failOver();

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

  public static void tearDownAfterClass() throws Exception {
    MiniAvatarCluster.shutDownZooKeeper();
  }


  private long getSessionId(int index) throws Exception {
    AvatarNode primaryAvatar = cluster.getPrimaryAvatar(index).avatar;
    String address = AvatarNode.getClusterAddress(primaryAvatar
        .getStartupConf());
    return zkClient.getPrimarySsId(address);
  }

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

    String[] normalArgs = { AvatarConstants.StartupOption.NODEZERO.getName() };
    String[] federationArgs = {
        AvatarConstants.StartupOption.NODEZERO.getName(),
        StartupOption.SERVICE.getName(), nnInfo.nameserviceId };
    String[] args = (federation) ? federationArgs : normalArgs;
    AvatarNode primary1 = AvatarNode.createAvatarNode(
        args,
        MiniAvatarCluster.getServerConf(
            AvatarConstants.StartupOption.NODEZERO.getName(), nnInfo));
    try {
      AvatarNode.createAvatarNode(args, MiniAvatarCluster.getServerConf(
              AvatarConstants.StartupOption.NODEONE.getName(), nnInfo));
      fail("Did not throw exception");
    } catch (Exception e) {
      LOG.info("Expected exception : ", e);
      assertEquals(primary1.getSessionId(), getSessionId(index));
    }
  }

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode


  private int initializeTest(String testName) throws IOException {
    String fileName = testName;
    createRBWFile(fileName);
    // Verify we have 1 RBW block.
    AvatarNode avatar = cluster.getPrimaryAvatar(0).avatar;
    LocatedBlocks lbks = avatar.namesystem.getBlockLocations(fileName, 0,
        Long.MAX_VALUE);
    int blocksBefore = lbks.locatedBlockCount();
    for (LocatedBlock lbk : lbks.getLocatedBlocks()) {
      DatanodeInfo[] locs = lbk.getLocations();

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

  }


  private void verifyResults(int blocksBefore, String fileName)
    throws IOException {
    // Verify we have RBWs after restart.
    AvatarNode avatarAfter = cluster.getPrimaryAvatar(0).avatar;
    LocatedBlocks lbks = avatarAfter.namesystem
        .getBlockLocations(fileName, 0,
        Long.MAX_VALUE);
    long blocksAfter = lbks.locatedBlockCount();

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

      assertTrue(locs.length != 0);
    }
  }


  private boolean blocksReceived(int nBlocks, String fileName) throws IOException {
    AvatarNode avatar = cluster.getPrimaryAvatar(0).avatar;
    LocatedBlocks lbks = avatar.namesystem.getBlockLocations(fileName, 0,
        Long.MAX_VALUE);
    int blocks = lbks.locatedBlockCount();
    if (blocks != nBlocks)
      return false;

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

      throws Exception {
    LOG.info("TEST Ingest Failure : " + event);
    TestAvatarIngestingHandler h = new TestAvatarIngestingHandler(event);
    InjectionHandler.set(h);
    setUp(3); // simulate interruption, no ckpt failure
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    h.setDisabled(false);


    createEdits(20);
    try {
      Thread.sleep(10000);
    } catch (Exception e) {
    }
    h.setDisabled(true);
    standby.quiesceStandby(getCurrentTxId(primary) - 1);
    assertEquals(20, getCurrentTxId(primary));
    assertEquals(getCurrentTxId(primary), getCurrentTxId(standby));
    tearDown();
  }

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode


  @Test
  public void testBasic() throws Exception {
    LOG.info("------------ testBasic-----------");
    createEdits(20);
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    standby.quiesceStandby(getCurrentTxId(primary)-1);
    assertEquals(20, getCurrentTxId(primary));
    assertEquals(getCurrentTxId(primary), getCurrentTxId(standby));
    LOG.info("------------ testBasic----------- DONE");
  }

View Full Code Here

Examples of org.apache.hadoop.hdfs.server.namenode.AvatarNode

  @Test
  public void testWithFailover() throws Exception {
    LOG.info("------------ testWithFailover-----------");
    // Create edits before failover.
    createEdits(20);
    AvatarNode primary = cluster.getPrimaryAvatar(0).avatar;
    AvatarNode standby = cluster.getStandbyAvatar(0).avatar;
    assertEquals(20, getCurrentTxId(primary));


    // Perform failover and restart old primary.
    cluster.failOver();
    cluster.restartStandby();


    // Get new instances after failover.
    primary = cluster.getPrimaryAvatar(0).avatar;
    standby = cluster.getStandbyAvatar(0).avatar;
    assertEquals(20, getCurrentTxId(primary));


    // Create some more edits and verify.
    createEdits(20);
    standby.quiesceStandby(getCurrentTxId(primary)-1);
    assertEquals(40, getCurrentTxId(primary));
    assertEquals(getCurrentTxId(primary), getCurrentTxId(standby));
    LOG.info("------------ testWithFailover----------- DONE");
  }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.