codeathon/Tokenizer
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
1. code_check.pl
This Perl program performs a check of 'C Programs'. Here I have written a code that checks for an assignment
statement made by mistake inside an if statement.
example
if (x = y) {
}
is an erroneous if statement.
Such a statement is then redirected to the output_codecheck file. You could check that file to see how the
erroneous if statement is identified and outputted.
2. token.pl
This Perl program is a sample Tokenizer built using Perl's built-in function - split().
I have create the sample inuput file - Perl.
Its basically an ASCII form of the Wikipedia page which talks about perl :
http://en.wikipedia.org/wiki/Perl
I have applied tokenization to this file and have redirected the output to
output_token.txt
I have also attached the output_*.txt for reference.
Have included different methods to perform parsing on the text file.
3. parsewords.pl
This Perl program is a better way of performing tokenization based on whitespaces.
I have used the library and the shellwords () function to do whitespacing tokenization.
4. datastructures.pl
This Perl program includes a simple implementation of the Stack Data structure using Perl.
5. http.pl
This Perl program connects to the specified URL and the response header is stored in a string along with
the html content