It’s technically possible for a node that is an authority over some DHT region to try and run “get” for some entry in this region while not yet having this entry available because data propagation hasn’t completed yet. But according to the documentation, both of get options guarantee that in such case node wouldn’t try to run a network query. How to make sure that this get goes to the network in such case?
Here’s an example where I think that would be useful.
- Suppose we have a DNA that stores 90 entries with nodes A and B. Partition size = 1, so every node is the single authority on a certain DHT shard.
- Node C joins the network and now the rebalancing is happening because C should be responsible for some DHT shard with 30 entries (I assume DHT contents are split equally for simplicity). Data propagation begins at time T1 and node C is supposed to have received all 30 entries by some future time T2.
- Out of those 30 entries, there are 2 that node C needs ASAP to render the UI properly. So before T2 happens, node C makes the get request to retrieve those 2 entries. (Let’s assume they’re available through a hard-coded path so C already knows where to look.) But since the data propagation hasn’t finished and since get options guarantee that node wouldn’t go to the network for entries it’s an authority for, node C wouldn’t get these entries before T2.
In this example, p3 presents a problem that would only get worse when scaling because:
- there’s no guarantee about the order of events due to eventual consistency;
- there’s no priority for different DHT entries;
- there’s no way for a developer to manually intervene in the data propagation process to prioritize certain data.
So in the worst case scenario we will be waiting (T2 - T1) time for the data we need ASAP without having any control over it. Delays like that would make for a poor UX.
I see two potential ways out of here, but both require changes in Holochain:
- allow developers to manually intervene in the data propagation process, providing an option to enforce gets to go to the network. I do realize that the whole point of these smart get options is to reduce the network load hApps would create on devices, and that allowing developers to intervene would go against those efforts. But I also feel that developers should be held responsible for the choices they make, and if used wisely this mechanism would help to improve the UX.
- allow developers to define a priority level for their entries for the data propagation purposes. That would still require to sit still and receive the network gossip, but at least that would increase the chances for the right data to arrive sooner than later. So developers get more control while not being allowed to mess with the data propagation directly, which may be a win/win. But, it also feels like a pretty fundamental change in the entire Holochain data model, so I would expect that to be difficult and time-consuming.
Does anyone see any solutions that can be implemented right now? Do presented options make sense?