Examples of StrTokenizer

org.apache.commons.lang.text.StrTokenizer
Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.
This class can split a String into many smaller strings. It aims to do a similar job to {@link java.util.StringTokenizer StringTokenizer}, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.
The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.
Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.
Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.
At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.
Empty tokens may be removed or returned as null.
```
 "a,b,c"         - Three tokens "a","b","c"   (comma delimiter) " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace) "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched) 
```
This tokenizer has the following properties and options:

Property Type Default

delim CharSetMatcher { \t\n\r\f}

quote NoneMatcher {}

ignore NoneMatcher {}

emptyTokenAsNull boolean false

ignoreEmptyTokens boolean true

@author Apache Software Foundation @author Matthew Inger @author Gary D. Gregory @since 2.2 @version $Id: StrTokenizer.java 907631 2010-02-08 12:22:48Z sebb $
org.apache.commons.lang3.text.StrTokenizer
Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.
This class can split a String into many smaller strings. It aims to do a similar job to {@link java.util.StringTokenizer StringTokenizer}, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.
The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.
Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.
Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.
At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.
Empty tokens may be removed or returned as null.
```
 "a,b,c"         - Three tokens "a","b","c"   (comma delimiter) " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace) "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched) 
```
This tokenizer has the following properties and options:

Property Type Default

delim CharSetMatcher { \t\n\r\f}

quote NoneMatcher {}

ignore NoneMatcher {}

emptyTokenAsNull boolean false

ignoreEmptyTokens boolean true

@since 2.2 @version $Id: StrTokenizer.java 1153241 2011-08-02 18:49:52Z ggregory $

Property	Type	Default
delim	CharSetMatcher	{ \t\n\r\f}
quote	NoneMatcher	{}
ignore	NoneMatcher	{}
emptyTokenAsNull	boolean	false
ignoreEmptyTokens	boolean	true

Property	Type	Default
delim	CharSetMatcher	{ \t\n\r\f}
quote	NoneMatcher	{}
ignore	NoneMatcher	{}
emptyTokenAsNull	boolean	false
ignoreEmptyTokens	boolean	true

Examples of org.apache.commons.lang.text.StrTokenizer


    public List<String> split(String input) {
        List<String> statements = new ArrayList<String>();
        StrBuilder currentSql = new StrBuilder();


        StrTokenizer lineTokenizer = new StrTokenizer(input);
        lineTokenizer.setDelimiterMatcher(StrMatcher.charSetMatcher("\r\n"));


        for (String line : lineTokenizer.getTokenArray()) {
          String strippedLine = StringUtils.stripEnd(line, null);
            if (!currentSql.isEmpty()) {
                currentSql.append(lineEnding.get());
            }

View Full Code Here

Examples of org.apache.commons.lang.text.StrTokenizer

        }
      
        if (response.getStatusCode() == HttpStatus.SC_OK)
        {
          // logger.warn(method.getResponseCharSet());
          StrTokenizer tokenize = new StrTokenizer(response.getResponseBodyAsString());
          tokenize.setDelimiterString("\n");
          String [] containers = tokenize.getTokenArray();          
          ArrayList <FilesContainer> containerList = new ArrayList<FilesContainer>();
          for(String container : containers) { 
            containerList.add(new FilesContainer(container, this));
          }
          return containerList;

View Full Code Here

Examples of org.apache.commons.lang.text.StrTokenizer

            }
          }


          if (response.getStatusCode() == HttpStatus.SC_OK)
          {
            StrTokenizer tokenize = new StrTokenizer(response.getResponseBodyAsString());
            tokenize.setDelimiterString("\n");
            String [] containers = tokenize.getTokenArray();
            List<String> returnValue = new ArrayList<String>();
            for (String containerName: containers)
            {
              returnValue.add(containerName);
            }

View Full Code Here

Examples of org.apache.commons.lang.text.StrTokenizer

    String value = params.get(key);
    if (value != null) {
      if (decode) {
        value = decode(value);
      }
      return new StrTokenizer(value, delimiter).setIgnoreEmptyTokens(false).getTokenArray();
    }
    return EMPTY;
  }

View Full Code Here

Examples of org.apache.commons.lang.text.StrTokenizer


    World world = plugin.getSenderWorld(sender);
    AutoRefMatch match = plugin.getMatch(world);


    // reparse the args properly using the string tokenizer from org.apache.commons
    args = new StrTokenizer(StringUtils.join(args, ' '), StrMatcher.splitMatcher(),
      StrMatcher.quoteMatcher()).setTrimmerMatcher(StrMatcher.trimMatcher()).getTokenArray();


    try
    {
      HandlerNode node = cmap.get(command.getName());

View Full Code Here

Examples of org.apache.commons.lang.text.StrTokenizer

        this.region = region;
    }


    public List<Container> handleResponse(final HttpResponse response) throws ClientProtocolException, IOException {
        if(response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
            final StrTokenizer tokenize = new StrTokenizer(EntityUtils.toString(response.getEntity()));
            tokenize.setDelimiterString("\n");
            final String[] containers = tokenize.getTokenArray();
            ArrayList<Container> list = new ArrayList<Container>();
            for(String container : containers) {
                list.add(new Container(region, container));
            }
            return list;

View Full Code Here

Examples of org.apache.commons.lang3.text.StrTokenizer


        String[] values;
        if (separator == null) {
            values = new String[]{valueString};
        } else {
            StrTokenizer tokenizer = new StrTokenizer(valueString, separator);
            values = tokenizer.getTokenArray();
        }


        String[] result = new String[values.length + 1];
        result[0] = key;
        System.arraycopy(values, 0, result, 1, values.length);

View Full Code Here

Examples of org.apache.commons.lang3.text.StrTokenizer

        }
        storeLine(values);
    }


    public void parseTokenized(String line) {
        StrTokenizer tokenizer = new StrTokenizer(line, separator);
        tokenizer.setIgnoreEmptyTokens(ignoreEmptyTokens);
        tokenizer.setQuoteChar(quoteChar);
        String[] tokens = tokenizer.getTokenArray();
        storeLine(tokens);
    }

View Full Code Here

Examples of org.apache.commons.lang3.text.StrTokenizer

      logger.trace("Buffer: {}, cursor: {}", buffer, cursor);
      logger.trace("Candidates {}", candidates);
    }
    if (StringUtils.isNotBlank(buffer)) {
      // User is typing a command
      StrTokenizer strTokenizer = new StrTokenizer(buffer);
      String action = strTokenizer.next();
      Collection<String> arguments = argumentMap.get(action);
      if (arguments != null) {
        if (logger.isTraceEnabled()) {
          logger.trace("Arguments found for {}, Tokens: {}", action, strTokenizer.getTokenList());
          logger.trace("Arguments for {}: {}", action, arguments);
        }
        List<String> args = new ArrayList<String>(arguments);
        List<Completer> completers = new ArrayList<Completer>();
        for (String token : strTokenizer.getTokenList()) {
          boolean argContains = arguments.contains(token);
          if (token.startsWith("-") && !argContains) {
            continue;
          }
          if (argContains) {

View Full Code Here

Examples of org.apache.commons.lang3.text.StrTokenizer

    return success;
  }


  @Override
  protected int executeCommand(String line) {
    String[] tokens = new StrTokenizer(line).getTokenArray();
    String action = tokens[0];
    String[] actionArgs = Arrays.copyOfRange(tokens, 1, tokens.length);
    if (logger.isDebugEnabled()) {
      logger.debug("Executing command action: {}, Tokens: {}", action, tokens.length);
    }

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.