page263
350
Clients are rightly protective of their intellectual property and in the event of a
performance issue, the data product provider will prefer access to the exact queries
issued by the consumer. However, sharing SQL is problematic; SQL often contains
bespoke logic, and consumers must be prevented from accessing a provider’s intellectual
property.
Historized Data
Many data products are offered as point-in-time, current view only. That is, ingested data
overwrites earlier data without retaining a history. Ingested data may not contain every
intermediate transaction recorded by a source system; therefore, each uploaded file
contains a snapshot.
An easy-to-implement value-add is to retain the full history of all captured data.
By adding temporal attributes to record valid-from and valid-to date stamps, you can
provide the ability to reconstruct data at any point in time for the retained data.
You can find more information on slowly changing dimensions at
https://
en.wikipedia.org/wiki/Slowly_changing_dimension
.
Implementing temporal attributes offers an inexpensive approach to adding value to
your data particularly when generating your codebase.
Data Model
At some point your clients may want your data product to be modeled in a particular
form so that it’s compatible with their existing implementation. The first step toward
integrating your data product with client data model is to publish an entity-relationship
diagram.
From a data provider’s perspective, offering consistent and interlinked data models
across all data product offerings is a worthy ambition. But this is hard to achieve in
practice. I do not prescribe any particular data modeling technique except to say 3NF,
DV2.0, and the dimensional approach each offers advantages and disadvantages.
Adopting an
INSERT
-only data model provides the fastest method of delivering data
into shared objects.
Chapter 9
Client expeCtations