Examples of IBMArabicEscaper

edu.stanford.nlp.international.arabic.IBMArabicEscaper

This escaper is intended for use on flat input to be parsed by LexicalizedParser. It performs these functions functions:

Deletes the clitic markers inserted by the IBM segmenter ('#' and '+')
Deletes IBM classing for numbers
Replaces tokens that must be escaped with the appropriate LDC escape sequences
Applies the same orthographic normalization performed by {@link edu.stanford.nlp.trees.international.arabic.ArabicTreeNormalizer}
intern()'s strings

This class supports both Buckwalter and UTF-8 encoding.

IMPORTANT: This class must implement Function, List> in order to run with the parser. @author Christopher Manning @author Spence Green

  public IBMMTArabicDataset() {
    configuredOptions = Generics.newHashSet();
    toStringBuffer = new StringBuilder();
    pathsToData = new ArrayList<File>();


    escaper = new IBMArabicEscaper(true);
    escaper.disableWarnings();


    requiredOptions = Generics.newHashSet();
    requiredOptions.add(ConfigParser.paramName);
    requiredOptions.add(ConfigParser.paramPath);

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of IBMArabicEscaper

Examples of edu.stanford.nlp.international.arabic.IBMArabicEscaper