page186
272
Information is derived from data via consolidation, cleansing, transformation,
and consumption processes. Knowledge—the intellectual property—is derived from
information, and wisdom is gained from applying knowledge. Figure 8-2 illustrates the
relationship between each layer of the DIKW pyramid, demonstrating the value chain.
Figure 8-2.
DIKW pyramid
Data products are the resultant “value-add” enabling organizations to monetize their
intellectual property without disclosing their internal methodology, or, if you prefer, their
“secret sauce.”
With intellectual property embedded within the data processing pipelines, curating
data products involves maintaining core table data with content from the raw or staging
tables. As a rule of thumb and discussed at length in Chapter 6, you should plan for a
Medium warehouse for data curation. However, as you will see later within this chapter,
parallelization may allow a smaller warehouse to be used in a more efficient manner.
Without igniting a debate regarding the data modeling style implemented within an
application, you should understand the profile of your core data and the cluster keys and
match your ingestion process to the core data model and structures. The efficient processing
of data is the core theme of this chapter, and I will unpack this theme in detail later.
Produce
Your objective is to deliver curated data products to your clients. Traditional data
distribution mechanisms such as secure file transfer via SFTP and on-premise dedicated
server provisioning are being replaced with automated, seamless approaches. Along
with supporting traditional data distribution mechanisms, Snowflake offers several new
and innovative data distribution approaches, as shown in Figure 8-3.
Chapter 8
parallelization