page226
312
Concluding Steps
In concluding the test case, you carry out these steps:
1.
Create a stored procedure to load segments.
2.
Test a single data load.
3.
Create and run tasks and then suspend tasks.
4.
Clean up to get ready for the next load.
While the number of segments may vary along with the orchestration tooling, the
technique is sound and delivers measurable performance benefits in real-world use.
Temporal Loads
This test case considered a fictitious scenario that you developed into parallelizing a
single load into five separate region segments. The feed may not conform to the same
pattern, and furthermore, our data may skew over time.
Let’s assume you have a temporal feed where the bulk of data changes over time.
Imagine a feed where the majority of the data is for a sliding three-year window. The
segmentation strategy must adapt to cater to the feed, so the steps involved include the
following:
•
Identify the date key and create segments for date ranges.
•
Set the segment date ranges as follows:
•
Large for low-volume changes
•
Small for high-volume changes
•
Implement annual segment maintenance for the feed sliding window
content.
Figure 8-23 illustrates a sample date range with relative data volumes.
Chapter 8
parallelization