page229
315
Summary
This chapter by setting the scene for wider data product distribution and explored an
example application profile calling out the typical sections along with capabilities. I
assigned nominal warehouse sizes to each section as a starting point.
After establishing an example application profile, I covered a typical end-user
complaint and provided information and tools to analyze the root cause, capturing
useful information along the way. You investigated how to segregate a data load by DML
operation and learned why you cannot run these three operations concurrently.
You then investigated how to parallelize an existing data feed by creating segmented
data sets according to a known business key derived from a core table clustering key.
This work led to the creation of five separate automated data pipelines along with test
cases for each step along the way. I called out some side effects of DML operations
relating to streams along with mitigating actions to support rerun capability.
Temporal data loads were also discussed to expose an alternative date-based
segmentation strategy along with identifying the annual maintenance overhead.
I concluded by assessing the real-world impact of implementing parallelization,
noting the technique outlined in this chapter is not a silver bullet and should not be
applied ubiquitously.
Having discussed parallelization in depth, we will now move on to investigate
entitlements.
Chapter 8
parallelization