token.hpp File Reference

#include "atom.hpp"
#include "util.hpp"
#include <algorithm>
#include <optional>
#include <sstream>
#include <string>
#include <string_view>
#include <utility>
#include <vector>

Include dependency graph for token.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
struct	yume::Loc
	Represents a location in source code, as a range starting at a line and column and ending at some other line and column of some file. More...
struct	yume::Token
	A categorized token in source code, created by the tokenizer. These tokens are consumed by the lexer. More...

Namespaces
namespace	llvm
namespace	yume

Functions
auto	yume::tokenize_preserve_skipped (std::istream &in, const string &source_file) -> vector< Token >
	Consume the contents of the input stream and create corresponding tokens, preserving every token, including whitespace. This is usually undesired. More...
auto	yume::tokenize (std::istream &in, const string &source_file) -> vector< Token >
	Consume the contents of the input stream and create corresponding tokens, ignoring insignificant whitespace. More...