Examples of DbTree

com.sleepycat.je.dbi.DbTree
DbTree represents the database directory for this environment. DbTree is itself implemented through two databases. The nameDatabase maps databaseName-> an internal databaseId. The idDatabase maps databaseId->DatabaseImpl. For example, suppose we have two databases, foo and bar. We have the following structure: nameDatabase idDatabase IN IN | | BIN BIN +-------------+--------+ +---------------+--------+ . | | . | | NameLNs NameLN NameLN MapLNs for MapLN MapLN for internal key=bar key=foo internal dbs key=53 key=79 dbs data= data= data= data= dbId79 dbId53 DatabaseImpl DatabaseImpl | | Tree for foo Tree for bar | | root IN root IN Databases, Cursors, the cleaner, compressor, and other entities have references to DatabaseImpls. It's important that object identity is properly maintained, and that all constituents reference the same DatabaseImpl for the same db, lest they develop disparate views of the in-memory database; corruption would ensue. To ensure that, all entities must obtain their DatabaseImpl by going through the idDatabase. DDL type operations such as create, rename, remove and truncate get their transactional semantics by transactionally locking the NameLN appropriately. A read-lock on the NameLN, called a handle lock, is maintained for all DBs opened via the public API (openDatabase). This prevents them from being renamed or removed while open. However, for internal database operations, no handle lock on the NameLN is acquired and MapLNs are locked with short-lived non-transactional Lockers. An entity that is trying to get a reference to the DatabaseImpl gets a short lived read lock just for the fetch of the MapLN, and a DatabaseImpl usage count is incremented to prevent eviction; see getDb and releaseDb. A write lock on the MapLN is taken when the database is created, deleted, or when the MapLN is evicted. (see DatabaseImpl.isInUse()) The nameDatabase operates pretty much as a regular application database in terms of eviction and recovery. The idDatabase requires special treatment for both eviction and recovery. The issues around eviction of the idDatabase center on the need to ensure that there are no other current references to the DatabaseImpl other than that held by the mapLN. The presence of a current reference would both make the DatabaseImpl not GC'able, and more importantly, would lead to object identify confusion later on. For example, if the MapLN is evicted while there is a current reference to its DatabaseImpl, and then refetched, there will be two in-memory versions of the DatabaseImpl. Since locks on the idDatabase are short lived, DatabaseImpl.useCount acts as a reference count of active current references. DatabaseImpl.useCount must be modified and read in conjunction with appropriate locking on the MapLN. See DatabaseImpl.isInUse() for details. This reference count checking is only needed when the entire MapLN is evicted. It's possible to evict only the root IN of the database in question, since that doesn't interfere with the DatabaseImpl object identity.

Examples of com.sleepycat.je.dbi.DbTree

                 * Check to make sure the DB was not deleted after selecting
                 * it, and prevent the DB from being deleted while we're
                 * working with it.
                 */
                DatabaseImpl targetDb = target.getDatabase();
                DbTree dbTree = targetDb.getDbEnvironment().getDbTree();
                DatabaseImpl refreshedDb = null;
                try {
                    refreshedDb = dbTree.getDb(targetDb.getId());
                    if (refreshedDb != null && !refreshedDb.isDeleted()) {
                        if (target.isDbRoot()) {
                            evictBytes += evictRoot(target, backgroundIO);
                        } else {
                            evictBytes += 
                                evictIN(target, backgroundIO, source);
                        }
                    } else {


                        /*
                         * We don't expect to see an IN that is resident on
                         * the INList with a database that has finished
                         * delete processing, because it should have been
                         * removed from the INList during post-delete
                         * cleanup.  It may have been returned by the
                         * INList iterator after being removed from the
                         * INList (because we're using ConcurrentHashMap),
                         * but then IN.getInListResident should return false.
                         */
                        if (targetDb.isDeleteFinished() &&
                            target.getInListResident()) {
                            String inInfo =
                                " IN type=" + target.getLogType() + " id=" +
                                target.getNodeId() + " not expected on INList";
                            String errMsg = (refreshedDb == null) ?
                                inInfo :
                                ("Database " + refreshedDb.getDebugName() +
                                 " id=" + refreshedDb.getId() + " rootLsn=" +
                                 DbLsn.getNoFormatString
                                 (refreshedDb.getTree().getRootLsn()) +
                                  ' ' + inInfo);
                            throw EnvironmentFailureException.
                                unexpectedState(errMsg);
                        }
                    }
                } finally {
                    dbTree.releaseDb(refreshedDb);
                }
            }
        } finally {
            nNodesScanned.add(numNodesScannedThisBatch);
        }

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree

        /*
         * Use local caching to reduce DbTree.getDb overhead.  Do not call
         * releaseDb after each getDb, since the entire dbCache will be
         * released at the end.
         */
        DbTree dbTree = env.getDbTree();
        Map<DatabaseId, DatabaseImpl> dbCache =
            new HashMap<DatabaseId, DatabaseImpl>();
        try {
            Iterator<BINReference> it = queueSnapshot.iterator();
            while (it.hasNext()) {
                BINReference binRef = it.next();
                DatabaseImpl db = dbTree.getDb
                    (binRef.getDatabaseId(), lockTimeout, dbCache);
                BIN bin = searchForBIN(db, binRef);
                if (bin != null) {
                    bin.verifyCursors();
                    bin.releaseLatch();
                }
            }
        } finally {
            dbTree.releaseDbs(dbCache);
        }
    }

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree


            /* Use local caching to reduce DbTree.getDb overhead. */
            Map<DatabaseId, DatabaseImpl> dbCache =
                new HashMap<DatabaseId, DatabaseImpl>();


            DbTree dbTree = env.getDbTree();
            BINSearch binSearch = new BINSearch();
            try {
                Iterator<BINReference> it = queueSnapshot.values().iterator();
                while (it.hasNext()) {
                    if (env.isClosed()) {
                        return;
                    }


                    BINReference binRef = it.next();
                    if (!findDBAndBIN(binSearch, binRef, dbTree, dbCache)) {


                        /*
                         * Either the db is closed, or the BIN doesn't exist.
                         * Don't process this BINReference.
                         */
                        continue;
                    }


                    /* Compress deleted slots and prune if possible. */
                    compressBin(binSearch.db, binSearch.bin, binRef,
                                localTracker);
                }


                /* SR [#11144]*/
                assert TestHookExecute.doHookIfSet(beforeFlushTrackerHook);


                /*
                 * Count obsolete nodes and write out modified file summaries
                 * for recovery.  All latches must have been released.
                 */
                env.getUtilizationProfile().flushLocalTracker(localTracker);


            } finally {
                dbTree.releaseDbs(dbCache);
                assert LatchSupport.countLatchesHeld() == 0;
                accumulatePerRunCounters();
            }
        }
    }

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree

        EnvironmentImpl envImpl = DbInternal.getEnvironmentImpl(env);


        /* If no database is specified, verify all. */
        List<String> dbNameList = null;
        List<String> internalDbs = null;
        DbTree dbMapTree = envImpl.getDbTree();


        if (dbName == null) {
            dbNameList = env.getDatabaseNames();


            dbNameList.addAll(dbMapTree.getInternalNoRepDbNames());
            if (envImpl.isReplicated()) {
                dbNameList.addAll(dbMapTree.getInternalRepDbNames());
            }
            internalDbs = dbMapTree.getInternalNoLookupDbNames();
        } else {
            dbNameList = new ArrayList<String>();
            dbNameList.add(dbName);
            internalDbs = new ArrayList<String>();
        }


        /* Check application data. */
        Iterator<String> iter = dbNameList.iterator();
        while (iter.hasNext()) {
            String targetDb = iter.next();
            LoggerUtils.envLogMsg(Level.INFO, envImpl,
                                  "DbVerify.verify of " + targetDb + " starting");


            DatabaseConfig dbConfig = new DatabaseConfig();
            dbConfig.setReadOnly(true);
            dbConfig.setAllowCreate(false);
            DbInternal.setUseExistingConfig(dbConfig, true);
            Database db;
            try {
                db = env.openDatabase(null, targetDb, dbConfig);
            } catch (DatabaseNotFoundException e) {
                /* DB was removed after getting names -- ignore it. */
                continue;
            } catch (DatabaseExistsException e) {
                /* Should never happen, ExclusiveCreate is false. */
                throw EnvironmentFailureException.unexpectedException(e);
            }


            try {
                if (!verifyOneDbImpl(DbInternal.getDatabaseImpl(db),
                                     targetDb,
                                     verifyConfig,
                                     out)) {
                    ret = false;
                }
            } finally {
                if (db != null) {
                    db.close();
                }
                LoggerUtils.envLogMsg(Level.INFO, envImpl,
                                   "DbVerify.verify of " + targetDb + 
                                   " ending");
            }
        }


        /*
         * Check internal databases, which don't have to be opened
         * through a Database handle.
         */
        iter = internalDbs.iterator();
        while (iter.hasNext()) {
            String targetDb = iter.next();
            LoggerUtils.envLogMsg(Level.INFO, envImpl,
                               "DbVerify.verify of " + targetDb + " starting");


            try {
                DatabaseImpl dbImpl = dbMapTree.getDb(null, targetDb,
                                                      null);
                try {
                    if (!verifyOneDbImpl(dbImpl,  targetDb,
                                         verifyConfig, out)) {
                        ret = false;
                    }
                } finally {
                    dbMapTree.releaseDb(dbImpl);
                }
            } finally {
                LoggerUtils.envLogMsg(Level.INFO, envImpl,
                                   "DbVerify.verify of " + targetDb + 
                                   " ending");

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree

             * inaccuracies caused by the sequence FileSummaryLN-LN-BIN.
             */
            envImpl.getUtilizationProfile().flushFileUtilization
                (envImpl.getUtilizationTracker().getTrackedFiles());


            DbTree dbTree = envImpl.getDbTree();
            boolean willDeleteFiles = !cleanerState.isEmpty();
            CheckpointEnd ckptEnd = new CheckpointEnd
                (invokingSource, checkpointStart, envImpl.getRootLsn(),
                 firstActiveLsn,
                 envImpl.getNodeSequence().getLastLocalNodeId(),
                 envImpl.getNodeSequence().getLastReplicatedNodeId(),
                 dbTree.getLastLocalDbId(), dbTree.getLastReplicatedDbId(),
                 envImpl.getTxnManager().getLastLocalTxnId(),
                 envImpl.getTxnManager().getLastReplicatedTxnId(),
                 checkpointId, willDeleteFiles, cleaner.getLogSummary());


            SingleItemEntry endEntry =

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree

                                        boolean highPriority,
                                        FlushStats fstats)
        throws DatabaseException {


        LogManager logManager = envImpl.getLogManager();
        DbTree dbTree = envImpl.getDbTree();


        Map<DatabaseId, DatabaseImpl> dbCache =
            new HashMap<DatabaseId, DatabaseImpl>();
        try {
            while (dirtyMap.getNumLevels() > 0) {


                /*
                 * Work on one level's worth of nodes in ascending level order.
                 */
                Integer currentLevel = dirtyMap.getLowestLevelSet();
                int currentLevelVal = currentLevel.intValue();


                /*
                 * Flush MapLNs just prior to flushing the first level of the
                 * mapping tree.  Only flush a database if it has not already
                 * been flushed since checkpoint start.
                 */
                if (currentLevelVal == IN.DBMAP_LEVEL) {
                    dirtyMap.flushMapLNs(checkpointStart);
                }


                /* Flush the nodes at the current level. */
                while (true) {
                    CheckpointReference targetRef =
                        dirtyMap.removeNextNode(currentLevel);
                    if (targetRef == null) {
                        break;
                    }


                    /*
                     * Check to make sure the DB was not deleted after putting
                     * it in the dirty map, and prevent the DB from being
                     * deleted while we're working with it.
                     */
                    DatabaseImpl db = dbTree.getDb
                        (targetRef.dbId, -1 /*lockTimeout*/, dbCache);
                    if (db != null && !db.isDeleted()) {


                        /* Flush if we're below maxFlushLevel. */
                        int maxFlushLevel = dirtyMap.getHighestFlushLevel(db);
                        if (currentLevelVal <= maxFlushLevel) {


                            /* Evict before each operation. */
                            envImpl.daemonEviction(true /*backgroundIO*/);


                            flushIN
                                (envImpl, db, logManager, targetRef, dirtyMap,
                                 currentLevelVal, maxFlushLevel, allowDeltas,
                                 highPriority, fstats,
                                 true /*allowLogSubtree*/);


                            /*
                             * Sleep if background read/write limit was
                             * exceeded.
                             */
                            envImpl.sleepAfterBackgroundIO();
                        }
                    }
                }


                /* We're done with this level. */
                dirtyMap.removeLevel(currentLevel);
            }
        } finally {
            dbTree.releaseDbs(dbCache);
        }


        /*
         * Do not flush FileSummaryLNs/MapLNs (do not call
         * UtilizationProfile.flushLocalTracker) here because that flushing is

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree

            /*
             * Update the tree's owner, whether it's the env root or the
             * dbmapping tree.
             */
            if (flushed) {
                DbTree dbTree = envImpl.getDbTree();
                dbTree.modifyDbRoot(db);
                fstats.nFullINFlushThisRun++;
                fstats.nFullINFlush++;
            }
        }

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree


                /*
                 * modifyDbRoot will grab locks and we can't have the INList
                 * latches or root latch held while it tries to acquire locks.
                 */
                DbTree dbTree = envImpl.getDbTree();
                dbTree.optionalModifyDbRoot(database);
                RecoveryManager.traceRootDeletion
                    (envImpl.getLogger(), database);
            }


            /*

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree


        /* Validate all entries in at least one full recovery pass. */
        reader.setAlwaysValidateChecksum(true);


        try {
            DbTree dbMapTree = envImpl.getDbTree();


            /* Process every IN and BIN in the mapping tree. */
            while (reader.readNextEntry()) {
                counter.incNumRead();
                DatabaseId dbId = reader.getDatabaseId();
                if (dbId.equals(DbTree.ID_DB_ID)) {
                    DatabaseImpl db = dbMapTree.getDb(dbId);
                    try {
                        replayOneIN(reader, db, false, recorder);
                        counter.incNumProcessed();
                    } finally {
                        dbMapTree.releaseDb(db);
                    }
                }
            }
            counter.setRepeatIteratorReads(reader.getNRepeatIteratorReads());

View Full Code Here

Examples of com.sleepycat.je.dbi.DbTree


            /*
             * Read all non-provisional INs, and process if they don't belong
             * to the mapping tree.
             */
            DbTree dbMapTree = envImpl.getDbTree();
            while (reader.readNextEntry()) {
                DatabaseId dbId = reader.getDatabaseId();
                boolean isMapDb = dbId.equals(DbTree.ID_DB_ID);
                boolean isTarget = false;
                counter.incNumRead();


                if (mapDbOnly && isMapDb) {
                    isTarget = true;
                } else if (!mapDbOnly && !isMapDb) {
                    isTarget = true;
                }


                if (isTarget) {
                    DatabaseImpl db = dbMapTree.getDb(dbId);
                    try {
                        if (db == null) {
                            // This db has been deleted, ignore the entry.
                            counter.incNumDeleted();
                        } else {
                            replayOneIN
                                (reader, db,
                                 reader.isBINDelta() /*requireExactMatch*/,
                                 recorder);
                            counter.incNumProcessed();


                            /*
                             * Add any db that we encounter IN's for because
                             * they'll be part of the in-memory tree and
                             * therefore should be included in the INList
                             * build.
                             */
                            inListBuildDbIds.add(dbId);
                        }
                    } finally {
                        dbMapTree.releaseDb(db);
                    }
                }
            }


            counter.setRepeatIteratorReads(reader.getNRepeatIteratorReads());

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.