page183
269
© Andrew Carruthers 2024
A. Carruthers, Tuning the Snowflake Data Cloud, https://doi.org/10.1007/979-8-8688-0379-6_8
CHAPTER 8
Parallelization
This chapter marks a change of focus for the remainder of this book as we will now look
to solving real-world performance issues. My experience is derived from developing and
implementing pragmatic solutions to seemingly intractable problems.
Throughout this chapter, you will investigate a performance issue step-by-step
using an example data ingestion process based upon my real-world experience. I will
offer a diagnostic approach to educate and inform how to approach problems with the
expectation that the implementation of proactive monitoring will expose future risks.
Snowflake is the latest and greatest data warehouse to hit the market and has rightly
attracted a lot of positive attention. However, for some, Snowflake is expensive to run;
this criticism is mostly ill-founded and arises due to misunderstanding and misapplying
best practices when implementing Snowflake. There is also a general reluctance to tune
existing applications when porting to Snowflake, a bad mistake to make.
Operating in a cloud-based global marketplace presents different challenges for both
data distribution to closed local applications operating within a corporate network and
service delivery via dedicated on-prem hardware in fixed data centers. Your approach
must adapt, because what has served you well in the past will not serve you well into
the future.
Curation of data incurs both cost and time. In our new global marketplace paradigm,
you must also replicate data seamlessly and accept that replicating data incurs both cost
and time.
While you adopt a Snowflake-centric view of this data marketplace strategy, you
must also consider how to act as a “data master” and enable distribution to other vendor
offerings.
There is plenty to investigate, so let’s look at some foundational information to
expose where problems occur in some existing applications. Then later you’ll see how to
remediate them.