Key takeaway
A result can be timely and incomplete. Publish its cutoff and update meaning instead of quietly changing history.
Separate the event from its arrival
A repair may finish before midnight but reach the reporting system the next morning. Beam distinguishes event time from processing time and treats watermarks as estimates of completeness. Triggers determine when window results are emitted. Those concepts help a buyer ask a precise question: which events should this output describe, and how long will the recipient wait for them?
A timestamp interpretation review must come first. Once instants are resolved, agree the window boundary, receiving cutoff and treatment of late records. The cutoff is a business and engineering choice for this feed. The sources do not establish a universal number of hours that makes operational history complete.
Label provisional and final outputs
In this hypothetical UTC daily window, 97 permitted job completions arrive by the first publication. Three more events belonging to that same day arrive before an agreed 36-hour closing cutoff. The closing output contains 100 events. An additional event appears after closure and goes to an exception review.
| Output | Meaning | Receiving action |
|---|---|---|
| Initial result: 97 | Provisional full snapshot | Replace that window’s previous snapshot |
| Closing result: 100 | Final under the agreed cutoff | Replace 97; do not add 100 to it |
| Delta result: +3 | Only new events, if explicitly agreed | Apply once to the prior 97 |
| After-cutoff event | Excluded pending review | Record exception; follow correction procedure |
Distinguish a replacement from a delta
Beam’s accumulating panes retain prior contents, while discarding panes represent data since the preceding emission. That distinction matters downstream: combining 97 and 100 as if both were new yields 197, which is wrong for the illustrated full snapshots. A delta of three can be correct only when the recipient knows its base and applies it once.
Give each output a window identifier, revision, phase, event-time interval, receiving cutoff and update mode. For a delta, also identify the base revision and a unique application key. A filename that merely says latest.csv does not explain whether its rows replace or supplement what arrived yesterday. Document whether deleted or corrected events are represented too.
Measure the unresolved tail
During a bounded internal review, compare the first publication with later arrivals for the same event window. Report arrival delays and excluded counts separately from source completeness. A record never entered by the source cannot appear in that delay histogram, so the observed tail cannot prove that all real-world events are present.
Choose a cutoff by comparing the receiving decision’s need for timeliness with the demonstrated correction burden. A same-day scheduling decision and a quarterly trend review can reasonably require different policies. If a late correction changes an already-issued conclusion, retain the previous output and publish a linked correction rather than obscuring the change.
- Test an arrival exactly at the agreed cutoff and record the boundary convention.
- Test an out-of-order correction and a repeated delta.
- Verify that a reopened window receives a new revision and an identifiable downstream notice.
Make the completeness claim narrow
A final result is final under the agreed procedure, not necessarily a complete account of reality. Its statement should name the timezone, event definition, observed sources, cutoff, exclusions and correction route. Ask the recipient to retain that context alongside any model or business analysis.
The inventory builder can capture the relevant system and history limits; the readiness planner can assign late-record and correction tasks. This technical agreement does not authorize wider record access, resolve privacy rights or establish market value. A receiving route can be explored with approved metadata while the actual sample decision remains separate.