page185
271
Ingest
Apart from the initial seeding of an application, steady-state data ingestion typically
involves modest data volumes at known frequency. You expect your load testing will
have informed your warehouse strategy and provide an indicative maximum velocity per
feed or consistent data set ingestion approach.
Ingesting data into application raw or staging tables is a prerequisite before merging
ingested data into a suite of core tables where you curate your data products. As a rule of
thumb and discussed at length in Chapter 6, you should plan for X-Small warehouses for
data ingestion.
Raw or staged data contains the following:
•
New data that does not exist in the core tables; new data is usually
inserted into core tables.
•
Old data marked for removal from the core tables; old data is usually
physically deleted or logically deleted from core tables.
•
Changed data for existing records in the core tables; changed data is
usually updated, or new data is inserted, and then old data is logically
deleted from core tables.
You should know both the frequency of data ingestion (the velocity) and the type of
DML (the volume) for
INSERT
,
UPDATE
, and
DELETE
operations to be performed on our
core tables as this information will prove essential later.
All applications ingest data. For the purposes of our investigation, you will use the
supplied TPC data set as our data source.
Curate
Data products are created from the combination of intellectual property usually in the
form of bespoke logic and the ingestion of data. When data meets business process,
value results.
As I identified in my earlier book Building the Snowflake Data Cloud, data in
its correct context provides information. This is more clearly stated using the data,
information, knowledge, wisdom (DIKW) pyramid; see
https://en.wikipedia.org/
wiki/DIKW_pyramid
.
Chapter 8
parallelization