page49
30
Tokenization
Within the context of a query optimizer, tokenization is the process of breaking down the
SQL statement into smaller units called tokens. Tokenization breaks down a query into
keywords, identifiers, literals, operators, and punctuation symbols. The tokens are used
to identify the structure and meaning of the query constituent parts in machine-
usable format.
Tokenization is used by many RDBMS query optimizers but is not explicitly called
out as a discrete component within Snowflake query optimization. It may be an inferred
component within the Snowflake query optimization process and is mentioned to
provide context for those migrating from legacy RDBMSs.
An alternative use of the word tokenization relates to both cybersecurity and
substitution of data content with undecipherable tokens.
Parsing
Parsing is also referred to as syntactic analysis and is the action of analyzing the
SQL statement structure or tokens. It validates the integrity of the query to ensure
completeness and correctness prior to producing a parse tree from which intermediate
code can be generated.
The parse tree provides the hierarchy and structure of the query along
with all internal relationships and converts the form to Query Block Internal
Representation (QBIR).
Semantic Analysis
Semantic analysis receives the QBIR and validates the structure matches both available
and accessible Snowflake objects. We assume a lookup to FoundationDB is performed
at this time. FoundationDB holds our Snowflake account metadata, that is, information
about every object, relationship, and security feature. This is the catalog that documents
and articulates your account.
Semantic analysis involves resolving object and attribute names, performs tag
checking, expands referenced views, expands user-defined functions (UDFs), and
expands common table expression (CTEs).
We anticipate additional checks are performed during semantic analysis including
optional steps, which may be skipped.
Chapter 2
the Query Optimizer