page197
283
does not show feed runtimes or breach the SLAs, but historical performance monitoring
does show a worrying trend: feed runtimes are increasing for a steady workload profile.
You should also check costs where you find warehouse runtimes are increasing and
therefore increasing consumption costs. Then you notice something surprising: data
replication costs are higher than the cost of curating data.
Ingest Factors
Let’s continue the investigation by identifying factors internal to the feed process that
may affect data ingestion.
The inbound data lands in the raw or staging table. The actual delivery mechanism
is unimportant, but to add flavor, let’s assume the bulk load operation occurs via a
COPY
command from file held in external CSP storage.
Raw or staged data contains new records for
INSERT
, changed records for
UPDATE
,
and old data marked for
DELETE
. You assume either unique records are loaded into our
staging table or a mechanism exists to de-duplicate data prior to
MERGE
into the target
core table.
The ingest process must also identify a primary key, unique composite key, or hash
for merging into the target core table. Your real-world application will already have
solved these challenges, but new applications will need to consider how to do the same.
Not shown are validation routines and setting
NULL
values to known defaults, these
may be edge cases and either positively excluded from your application design or
explicitly included within your application design.
Figure 8-10 shows the steps involved in preparing the feed for ingestion into a
core table.
Figure 8-10.
Ingest data preparation
Chapter 8
parallelization