List of openForWrite() Examples

Examples of openForWrite()

cascading.tap.Hfs.openForWrite()
cascading.tap.Tap.openForWrite()
Method openForWrite opens the resource represented by this Tap instance for writing.
This method is used internally and does not honor the {@link SinkMode} setting. If SinkMode is{@link SinkMode#REPLACE}, this call may fail. See {@link #openForWrite(cascading.flow.FlowProcess)}.
{@code output} value may be null, if so, sub-classes must inquire with the underlying {@link Scheme}via {@link Scheme#sinkConfInit(cascading.flow.FlowProcess,Tap,Object)} to get the properoutput type and instantiate it before calling {@code super.openForWrite()}. @param flowProcess of type FlowProcess @param output of type Output @return TupleEntryCollector @throws java.io.IOException when the resource cannot be opened
cascading.tap.hadoop.Hfs.openForWrite()
com.scaleunlimited.cascading.local.DirectoryTap.openForWrite()

Examples of cascading.tap.Hfs.openForWrite()

    File inputFile = new File(inputPath);
    if (inputFile.exists()) {
      throw new CascadingException("Input file " + inputPath + " already exists.");
    }
    Tap inputTap = new Hfs(new SequenceFile(fields), inputPath, SinkMode.REPLACE);
    TupleEntryCollector collector = inputTap.openForWrite(getJobConf());
    return collector;
  }


  /**
   * Create and run a cascading flow that connects the given pipe to an input

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    @SuppressWarnings({ "rawtypes", "unchecked" })
    public static void importOneDomain(BasePlatform platform, String targetDomain, BasePath crawlDbPath) throws Exception {
        
        try {
            Tap urlSink = platform.makeTap(platform.makeBinaryScheme(CrawlDbDatum.FIELDS), crawlDbPath, SinkMode.REPLACE);
            TupleEntryCollector writer = urlSink.openForWrite(platform.makeFlowProcess());
            SimpleUrlNormalizer normalizer = new SimpleUrlNormalizer();


            CrawlDbDatum datum = new CrawlDbDatum(normalizer.normalize("http://" + targetDomain), 0, 0, UrlStatus.UNFETCHED, 0);


            writer.add(datum.getTuple());

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    }
    
    @SuppressWarnings({ "unchecked", "rawtypes" })
    private void createDataFile(BasePlatform platform, BasePath filePath, List<UrlDatum> datums) throws Exception {
        Tap urlSink = platform.makeTap(platform.makeBinaryScheme(UrlDatum.FIELDS), filePath, SinkMode.REPLACE);
        TupleEntryCollector writer = urlSink.openForWrite(platform.makeFlowProcess());
        for (UrlDatum datum : datums) {
            writer.add(datum.getTuple());
        }
        writer.close();
    }

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

        FetchSetDatum pfd = new FetchSetDatum(urls, fetchTime, 1000, groupingKey.getValue(), groupingKey.getRef());
        
        BixoPlatform platform = new BixoPlatform(ScoredUrlDatumTest.class, platformMode);
        BasePath path = platform.makePath("build/test/ScoredUrlDatumTest/testCascadingSerialization/in");
        Tap in = platform.makeTap(platform.makeBinaryScheme(FetchSetDatum.FIELDS), path, SinkMode.REPLACE);
        TupleEntryCollector write = in.openForWrite(platform.makeFlowProcess());
        write.add(pfd.getTuple());
        write.close();
    }
}

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    
    private Tap makeInputData(BasePlatform platform, String testname, int numDomains, int numPages, Payload payload) throws Exception {
        String platformName = platform.getClass().getSimpleName();
        BasePath defaultPath = platform.makePath(BASE_INPUT_PATH + testname + "/" + platformName + "/in");
        Tap in = platform.makeTap(platform.makeBinaryScheme(UrlDatum.FIELDS), defaultPath, SinkMode.REPLACE);
        TupleEntryCollector write = in.openForWrite(platform.makeFlowProcess());
        for (int i = 0; i < numDomains; i++) {
            for (int j = 0; j < numPages; j++) {
                // Use special domain name pattern so code deep inside of operations "knows" not
                // to try to resolve host names to IP addresses.
                write.add(makeTuple("bixo-test-domain-" + i + ".com", j, payload));

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    
    private Tap makeInputData(BasePlatform platform, String testname, String domain, int numPages, Payload payload) throws Exception {
        String platformName = platform.getClass().getSimpleName();
        BasePath defaultPath = platform.makePath(BASE_INPUT_PATH + testname + "/" + platformName + "/in");
        Tap in = platform.makeTap(platform.makeBinaryScheme(UrlDatum.FIELDS), defaultPath, SinkMode.REPLACE);
        TupleEntryCollector write = in.openForWrite(platform.makeFlowProcess());
        for (int j = 0; j < numPages; j++) {
            write.add(makeTuple(domain, j, payload));
        }


        write.close();

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

        
        InputStream is = null;
        TupleEntryCollector writer = null;
        try {
            Tap urlSink = platform.makeTap(platform.makeTextScheme(), crawlDbPath, SinkMode.REPLACE);
            writer = urlSink.openForWrite(platform.makeFlowProcess());


            is = DemoWebMiningWorkflow.class.getResourceAsStream(fileName);
            if (is == null) {
                throw new FileNotFoundException("The seed urls file doesn't exist");
            }

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

        final String out = getTestDir() + "testSimpleIndexing/out";


        byte[] imageData = new byte[] {0, 1, 2, 3, 5};
        
        Tap source = makeSourceTap(testFields, in);
        TupleEntryCollector write = source.openForWrite(makeFlowProcess());
        Tuple t = new Tuple();
        t.add(1);
        t.add("TurboWriter 2.3");
        t.add(395.50f);
        t.add(new Tuple("wordprocessor", "Japanese"));

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    File inputFile = new File(inputPath);
    if (inputFile.exists()) {
      throw new CascadingException("Input file " + inputPath + " already exists.");
    }
    Tap inputTap = new Hfs(new SequenceFile(fields), inputPath, SinkMode.REPLACE);
    TupleEntryCollector collector = inputTap.openForWrite(getJobConf());
    return collector;
  }


  /**
   * Create and run a cascading flow that connects the given pipe to an input

View Full Code Here

Examples of cascading.tap.Tap.openForWrite()

    {
    String tapPath = getOutputPath( "tapreplace" );


    Tap tap = getPlatform().getTextFile( tapPath, SinkMode.KEEP );


    TupleEntryCollector collector = tap.openForWrite( getPlatform().getFlowProcess() ); // casting for test


    for( int i = 0; i < 100; i++ )
      collector.add( new Tuple( "string", "" + i, i ) );


    collector.close();

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.