I try to implement a simple C/C++ parser, which try to partially parsing C++ language. So I need to create a Lexer, a Preprocessor and a Parser class.

I'm considering what is the data type I need to pass information from those three layers. Normally, a Token class is need here, for right now, my Token class looks like below:

struct Token 
    TokenKind id; 
    std::string lexeme; 
    int fileIndex;
    int line; 
    int column;

I think the most important part is the TokenKind(it could be IDENTIFIER or CLASS_KEYWORD or any other punctuation like LPAREN), and some times, the lexeme is also important, because it usually contains the type name or variable name information.

I looked at some implementations about how the Token is passed to Parsers.

1, I see the Clang has some functions in it's Preprocessor class like Preprocessor.cpp:739

void Preprocessor::Lex(Token &Result) 

You see, a reference is passed as a the function argument, and the function fill the object with the result, see another reference here on a Clang's tutorial here:Clang-tutorial/CItutorial3.cpp at master · loarabia/Clang-tutorial, here the instance tok is reused in a loop.

Token tok;
do {
    if( ci.getDiagnostics().hasErrorOccurred())
    std::cerr << std::endl;

2, For some lexer generator, I see function yylex() just return an int type, which is actually a TokenKind, and the other information such as the actual lexeme string is stored in a global variables like yylval.

3, For a tiny language for GCC A tiny GCC front end – Part 3 | Think In Geek, I see the Lexer return a std::shared_ptr<Token>, that is:

  static TokenPtr
  make_identifier (location_t locus, const std::string& str)
    return TokenPtr(new Token (IDENTIFIER, locus, str));

The Lexer return a TokenPtr which is a smart pointer of the Token object to the Parser, so the whole Token is returned to the Parser.

4, GCC's cpp library has some interface of the cpp_get_token() function like below:

const cpp_token *token = cpp_get_token (pfile);

Then token->type is just like the TokenKind field.

So, my question is: what are the advantages and disadvantages of those kinds of implementations. Some of the mentioned methods above do not even have a preprocess layer, for me, I do need three layers(the lexer, the preprocessor and the parser).

Note that my parser won't be big enough as clang or GCC's parser. My main idea is that my parser can only parse very limited part of C++ language, and I would like to make them all hand written.

EDIT A similar question is here What should be the datatype of the tokens a lexer returns to its parser?, I also post some comments there several days ago, but that question does not involve the three layers.

  • Your struct token appears to me fine.For a simpler language I would have added a numeric field for scalars, but you can do that in the parser. However you've really got to lex again after preprocessing (assuming cpp is integrated) – Malcolm McLean Feb 4 '17 at 16:54
  • @MalcolmMcLean Thanks, it looks like my question dose not get answered for nearly one month. My main question is that how the wholel token information is passed from the low layer(lexer) to the high layer(parser). Roger Ferrer gives me some comments in his blog here: A tiny GCC front end – Part 3 | Think In Geek. – ollydbg23 Feb 26 '17 at 6:08

Your Answer


Browse other questions tagged or ask your own question.