page228
314
While the sample data does not contain records for 2030, it is prudent to extend the
highest date range partition into the future to capture any outliers. For the same reason,
the earliest date range partition should start well before 2018.
The sharp-eyed reader will note the suggested number of partitions matches
a two-cluster X-Small warehouse. Where the opportunity exists, you should
maximize warehouse processor unit consumption and favor multiples of eight when
partitioning tables.
Real-World Impact
The practical application of the parallelization technique outlined in this chapter in a
real-world production environment delivers significant business benefit. When applied
selectively, you may see between 40 percent and 70 percent reduction in query response
times in like-for-like scenarios along with a 25 percent reduction in ingestion and
curation time leading to more rapid data product delivery to clients.
You also experienced an unexpected benefit: parallelizing processing may reduce
the number of micro-partitions churned. When replicating tables to other accounts, the
cost of replication can exceed the cost of data product curation. Reducing micro-
partition churn significantly reduces replication costs.
Improving overall system performance, throughput, and cost reduction using
this technique may expose a hidden issue: the increase in concurrent processing
can lead to an increase in the volume of logging into a single table. The inadvertent
serialization of logging causes queueing as each process locks the log table. The solution
is to implement
EVENT
-based logging as described in Chapter 6. This is the Law of
Unintended Consequences in action again.
I must point out the selective adoption of this technique; I strongly suggest this
technique is not implemented ubiquitously. There are no silver bullets, and this
technique is no exception. If poorly implemented, this technique may experience higher
overall warehouse cost when parallelizing our data pipelines. However, increased data
curation costs may be offset by a reduction in runtimes. Every change you make requires
testing, and implementing parallelization is no different.
Chapter 8
parallelization