High-Water Mark Loading is a way to optimize your load strategy. Some data sources will have a column such as “update” or “effective” which can be relied on to track when data in the table has been inserted or changed. It is possible to track the latest entry (aka High-Water Mark) of a given load by storing the value of such a column. Later, when loading that same source, that stored value can be used to only ingest the most recent data. Doing so makes your loads smaller, faster, and (for all of us cloud-users) cheaper.
infoVia has developed a lightweight and easy-to-implement solution to implement High-Water Mark loads using WhereScape Red. We hope you find it as useful as our clients have!
Recent Comments