#30 [Design] No incremental sync on D365 reference streams (+ no backoff)

[Anonymous]

Originally created by: pyy3

Type: design / scalability · Label fit: enhancement

Summary

Several D365 reference streams are full-refresh only; they re-pull the entire entity on every sync rather than fetching changes incrementally.

Location

source-d365-fno/source_d365_fno/streams.py — incremental cursor logic exists for transactional streams (e.g. GL/journal entries), but reference streams such as main accounts, legal entities, financial dimensions, fiscal calendars are full-refresh.

Why it matters

For large D365 environments (the roadmap targets 50–500 legal entities) full-refresh of reference data on every run is wasteful and increases sync time and D365 API load. Compounds with the lack of rate-limiting/backoff in pagination.

Suggested directions

• Where the D365 OData entity exposes a ModifiedDateTime (or equivalent), add an incremental cursor.
• Add exponential backoff on HTTP 429 in the pagination loop.
• Document which streams are incremental vs. full-refresh and why.

[Anonymous]

Originally posted by: pyy3

Fixed in [#34] (merged to main).

[Anonymous]

Ticket changed by: pyy3

• status: open --> closed