A categorized token in source code, created by the tokenizer. These tokens are consumed by the lexer. More...

#include <token.hpp>

Collaboration diagram for yume::Token:

[legend]

Public Types
enum struct	Type { Word , Skip , Symbol , Literal , Number , Char , Separator , EndOfFile }

using	Payload = optional< Atom >

Public Member Functions
auto	is_a (const std::pair< Type, Atom > &type_atom) const -> bool

constexpr	Token (Type type)

constexpr	Token (Type type, Payload payload) noexcept

constexpr	Token (Type type, Payload payload, int i, Loc loc) noexcept

Static Public Member Functions
static auto constexpr	type_name (Type type) -> const char *

Public Attributes
Type	type

Payload	payload

int	index = -1

Loc	loc {}

Friends
auto	operator<< (llvm::raw_ostream &os, const Token &token) -> llvm::raw_ostream &

Detailed Description

A categorized token in source code, created by the tokenizer. These tokens are consumed by the lexer.

Each token has a type, an associated payload (usually the text the token was created from) and a location Loc

Definition at line 80 of file token.hpp.

Member Typedef Documentation

◆ Payload

using yume::Token::Payload = optional<Atom>

Definition at line 105 of file token.hpp.

Member Enumeration Documentation

◆ Type

enum struct yume::Token::Type

strong

Enumerator
Word	Any form of keyword or identifier, essentially the "default" token type.
Skip	Tokens which should be ignored, i.e. insignificant whitespace.
Symbol	Special characters, such as those representing operators.
Literal	A string literal, enclosed in quotes.
Number	A number literal.
Char	A character literal, beginning with `?`
Separator	A newline or a semicolon `;`
EndOfFile	A token added at the very end of the file.

Definition at line 81 of file token.hpp.

Constructor & Destructor Documentation

◆ Token() [1/3]

constexpr yume::Token::Token ( Type type )

inlineexplicitconstexpr

Definition at line 116 of file token.hpp.

◆ Token() [2/3]

constexpr yume::Token::Token	(	Type	type,
		Payload	payload
	)

inlineconstexprnoexcept

Definition at line 117 of file token.hpp.

◆ Token() [3/3]

constexpr yume::Token::Token	(	Type	type,
		Payload	payload,
		int	i,
		Loc	loc
	)

inlineconstexprnoexcept

Definition at line 118 of file token.hpp.

Member Function Documentation

◆ is_a()

auto yume::Token::is_a ( const std::pair< Type, Atom > & type_atom ) const -> bool

inline

Definition at line 112 of file token.hpp.

References payload, and type.

◆ type_name()

static auto constexpr yume::Token::type_name ( Type type ) -> const char*

inlinestaticconstexpr

Definition at line 91 of file token.hpp.

References type.

Referenced by yume::ast::parser::Parser::consume(), yume::ast::parser::Parser::expect(), and yume::ast::parser::Parser::try_peek().

Friends And Related Function Documentation

◆ operator<<

auto operator<<	(	llvm::raw_ostream &	os,
		const Token &	token
	)		-> llvm::raw_ostream&

friend

Definition at line 291 of file token.cpp.

Member Data Documentation

◆ index

int yume::Token::index = -1

Definition at line 109 of file token.hpp.

◆ loc

Loc yume::Token::loc {}

Definition at line 110 of file token.hpp.

◆ payload

Payload yume::Token::payload

Definition at line 108 of file token.hpp.

Referenced by is_a().

◆ type

Type yume::Token::type

Definition at line 107 of file token.hpp.

Referenced by is_a(), and type_name().

The documentation for this struct was generated from the following file:

src/yume/token.hpp

Public Types

Public Member Functions

Static Public Member Functions

Public Attributes

Friends