page141
124
In some cases where data ingestion is a bottleneck, you have a few design options to
consider.
•
Adopt an insert-only model, i.e., Data Vault 2.0.
•
Parallelize loads where discrete data isolation boundaries can be
enforced.
•
Reduce batch size and increase frequency noting the likely increase
in micro-partition churn.
•
Increase warehouse size.
•
When sourcing data from external CSP storage, implement file
caching or faster storage devices.
We will return to parallelization in a later chapter as the subject is worthy of wider
consideration.
In the next section, you will investigate storage costs, but for now, you might consider
optimizing your data ingestion costs by doing the following:
•
Setting Time Travel to 0 for your staging tables where persistence is
not required; note Fail-Safe is retained at 7 days.
•
Use transient tables for your staging tables with Time Travel set to 0
as transient tables do not utilize Fail-Safe.
Both of these approaches imply staged data can be reloaded from source. Where
external stages are used, ensure the CSP storage is set to retain files according to the
requirements.
Data Processing
With your data loaded into staging tables and all feed dependencies resolved, core
application processing occurs.
in this section, you will investigate “how” to identify both performance timings and
storage costs.
ChaPTer 4
MiCro-ParTiTionS