Examples of cc.mallet.pipe.SimpleTaggerSentence2TokenSequence

cc.mallet.pipe.SimpleTaggerSentence2TokenSequence
Converts an external encoding of a sequence of elements with binary features to a {@link TokenSequence}. If target processing is on (training or labeled test data), it extracts element labels from the external encoding to create a target {@link LabelSequence}. Two external encodings are supported:
1. A {@link String} containing lines of whitespace-separated tokens.
2. a {@link String}[][].
Both represent rows of tokens. When target processing is on, the last token in each row is the label of the sequence element represented by this row. All other tokens in the row, or all tokens in the row if not target processing, are the names of features that are on for the sequence element described by the row.

  private static String[] doc1 =  { "Meet\nme\nat\n4\nPM\ntomorrow" };


  public static void testMultiTag ()
  {
    Pipe mtPipe = new SerialPipes (new Pipe[] {
            new SimpleTaggerSentence2TokenSequence (),
            new TokenText (),
            new RegexMatches ("digits", Pattern.compile ("[0-9]+")),
            new RegexMatches ("ampm", Pattern.compile ("[aApP][mM]")),
            new OffsetFeatureConjunction ("time",
                    new String[] { "digits", "ampm" },
                    new int[] { 0, 1 },
                    true),
            new PrintInputAndTarget (),
    });
    Pipe noMtPipe = new SerialPipes (new Pipe[] {
            new SimpleTaggerSentence2TokenSequence (),
            new TokenText (),
            new RegexMatches ("digits", Pattern.compile ("[0-9]+")),
            new RegexMatches ("ampm", Pattern.compile ("[aApP][mM]")),
            new OffsetFeatureConjunction ("time",
                    new String[] { "digits", "ampm" },

View Full Code Here

  }


  public static void testMultiTagSerialization () throws IOException, ClassNotFoundException
  {
    Pipe origPipe = new SerialPipes (new Pipe[] {
            new SimpleTaggerSentence2TokenSequence (),
            new TokenText (),
            new RegexMatches ("digits", Pattern.compile ("[0-9]+")),
            new RegexMatches ("ampm", Pattern.compile ("[aApP][mM]")),
            new OffsetFeatureConjunction ("time",
                    new String[] { "digits", "ampm" },

View Full Code Here

  private static String data = "f1 f2 CL1\nf1 f3 CL2";


  public void testPipesAreStupid ()
  {
    Pipe p1 = new StupidPipe ();
    Pipe p2 = new SimpleTaggerSentence2TokenSequence ();
    // initialize p2's dict
    p2.instanceFrom(new Instance (data, null, null, null));


    Pipe serial = new SerialPipes (new Pipe[] { p1, p2 });
    try {
      serial.getDataAlphabet ();
      assertTrue ("Test failed: Should have generated exception.", false);

View Full Code Here

    assertEquals (3, serial.getDataAlphabet ().size ());
  }


  public void testConcatenateBadPipes ()
  {
    Pipe p1 = new SimpleTaggerSentence2TokenSequence ();
    // force resolving data alphabet
    Alphabet dict1 = p1.getDataAlphabet ();


    Pipe p2 = new SimpleTaggerSentence2TokenSequence ();
    // force resolving data alphabet
    Alphabet dict2 = p2.getDataAlphabet ();


    assertTrue (dict1 != dict2);


    try {
      PipeUtils.concatenatePipes (p1, p2);

View Full Code Here

TOP

Related Classes of cc.mallet.pipe.SimpleTaggerSentence2TokenSequence

cc.mallet.pipe.tests.TestPipeUtils

cc.mallet.pipe.tsf.tests.TestOffsetFeatureConjunctions

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.