Tokenizing source code in Java?

Don't fear the regex, get yourself a regex editor such as the following eclipse plugin brosinski.com/regex/update and you'll be able to test the expressions without compiling or even before writing your program.

Don't fear the regex, get yourself a regex editor such as the following eclipse plugin, brosinski.com/regex/update and you'll be able to test the expressions without compiling or even before writing your program. If you need more reference, here are some very useful sites : regular-expressions.info/ regexlib.com/ Although I think the suggestion above of using JavaCC sound like the right approach. Another option would be ANTLR.

Heres a post comparing the experience of ANTLR vs JavaCC.

I second this. It won't take you more than 30 minutes to learn enough about regex to effectively use String. Split or Scanner.

For a programmer learning to write basic regexes is easy and takes very little time. Becoming a master will take you the rest of your career. – Mike Deck Oct 5 '10 at 21:34 1 Although I do still chuckle at the 1997 quote from Jamie Zawinski one of the founders of Netscape and Mozilla.Org "Some people, when confronted with a problem, think 'I know, I'll use regular expressions.

' Now they have two problems. " – crowne Oct 5 '10 at 22:07 That quote is a favorite of mine as well. – Greg Case Oct 6 '10 at 0:00.

I believe that the java.util. Scanner class has replaced StringTokenizer. Scanner let's you handle tokens one at a time, whereas String.split() will split the entire string (which could be large, if you're parsing a source code file).

Using Scanner, you can examine each token, decide what action to take, then discard that token.

Generally, you shouldn't be parsing an entire source file at once, but a single source line at a time. It's easier on memory, and it makes it easier to keep track of line numbers for issuing error messages. – Loadmaster Oct 5 '10 at 19:33.

From the documentation: StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util. Regex package instead.

The following example illustrates how the String. Split method can be used to break up a string into its basic tokens: String result = "this is a test". Split("\\s"); for (int x=0; xLength; x++) System.out.

Println(resultx); prints the following output: this is a test.

1 for documentation – eiefai Oct 5 '10 at 19:30 Right, I came across this as well. It would have been good if I noted that more clearly, but this is what I was referring to when I said "essentially deprecated". – Ryan Oct 5 '10 at 19:41.

If what you're building is an assembler, I would use JavaCC for building the parser/compiler.

This would have been an extremely helpful tool, but we were explicitly forbidden to use tools like this. Thank you, though - this is pretty cool! – Ryan Oct 5 '10 at 19:58.

Something is deprecated when there is a better alternative, or those methods are dangerous in some situations. So the answer is - Yep, you can use it, but there is a better way to achieve what you need. Btw, what is complicate about split?

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.