23 std::vector<LexerRule> rules;
33 std::vector<std::string> readFile(std::string fileName);
44 std::vector<Token> tokenize(std::string input);
55 bool matchAnyRule(std::string input);
66 int matchToken(std::string input);
90 std::vector<Token>
getTokens(std::string fileName);
The Lexer class is responsible for reading input, applying lexer rules, and generating tokens from th...
Definition lexer.h:21
Lexer()
Constructs a Lexer object and loads the configuration rules.
Definition lexer.cpp:6
std::vector< Token > getTokens(std::string fileName)
Gets the tokens from a given file.
Definition lexer.cpp:142
std::vector< ProductionRule > getProductionRules(std::string fileName)
Extracts production rules from a given file.
Definition lexer.cpp:152
void loadConfiguration()
Loads lexer configuration rules.
Definition lexer.cpp:85