Making the function Parser2->tokenize 6x faster

[redbarmaley]

Hi.

I use Turbine version 1.1.0beta1.
My computer: Inter Core2 Duo T5470 (2x1,6 GHz), 2Gb, under Vista Home Basic (32bit), Apache 2.0 Handler, PHP 5.2.6

My cssp file contains 24581 bytes. Turbine process it about 12 sec! I found that it coused by function Parser2->tokenize.
I rewrite it with explode function and time reduced to 2 sec!
Also I changed array_key_exists to isset in function Cssp->find_ancestor_keys:

array_key_exists('_label', $this->parsed[$block][$key]) to isset($this->parsed[$block][$key]['_label'])

Hear my code of Parser2->tokenize. Maybe it will be used.

public function tokenize($str, $separator = array(' ', ' ')){
static $tr = array(
"'" => '"'',
'"' => '"@@@@@',
'(' => '"(',
')' => '")',
"\t"=> ' '
);

if($separator[0] == ' '):
    $str = strtr($str, $tr);
    $tokens = array('');
    $parts = explode('"', $str);
    foreach($parts as $index=>&$part):
        if($index % 2 === 0): //нечет
            $buf = explode(' ', $part);
            $tokens[] = array_pop($tokens) . array_shift($buf);
            $tokens = array_merge($tokens, $buf);
        else:
            $tokens[] = array_pop($tokens) . $part;
        endif;
    endforeach;
    $len = count($tokens);
    $search = array_fill(0, $len, '@@@@@');
    $replace = array_fill(0, $len, '"');
    $tokens = array_map('str_replace', $search, $replace, $tokens);
else:
    if($separator == ','):
        $delim = $separator;
    else:
        $delim = $separator[0];
        $replaces = array_combine($separator,
        array_fill(0, count($separator), $delim));
        $str = strtr($str, $replaces);
    endif;
    $tokens = explode($delim, $str);
endif;
$tokens = array_map('trim', $tokens);

return $tokens;

}

[commi]

could you make that a github-pull-request,so lazy me could test that without typing?:)