page192
278
I offered a solution to serialized logging in Chapter 6 by implementing an
EVENT
table
and noted that only a single
EVENT
table can be active at any given time.
EVENT
information must be periodically collated into a separate log table for long-
term audit trail preservation.
Optimizing Data Processing
Every application has an optimal or target processing time from data landing in raw or
staging tables through to content appearing within the client-consumed data product.
In most cases, clients will pay a premium for faster data product updates, change
propagation, and availability.
You goal should be to both reduce cost and improve timeliness across the
whole application life cycle from the initial point of data ingestion through to client
consumption. For this to become reality, you must adopt a holistic approach to
identifying the root causes of both cost consumption and latency and then apply
effective remediation.
I assume you have checked your code for Cartesian joins, long compilation times,
long execution times, and long table scans as part of the user acceptance tests and
commissioning into production.
Let’s look at an example system as viewed from the client perspective.
Problem Statement
Several clients have observed that the time it takes for data to appear in their licensed
product is getting progressively slower month over month. The first step is to validate the
client’s claim by checking the telemetry information logged for the feed. Let’s assume the
feed contains normal data volumes, and the logged information corroborates the client
claim. You confirm the feed runtimes have marginally degraded over time.
From the information supplied and analysis conducted by your product support
team, you deduce one single inbound data feed, and onward ingestion into the core data
set is affected. Figure 8-8 illustrates the left-to-right data flows for our sample feed.
Chapter 8
parallelization