Holochain Forum

How to bidirectionally replicate data between DNAs?

I notice you get Cyclic dependency in bridge configuration when you attempt to have DNAs talking to each other. Are there any plans to work towards lifting this limitation?

There are some use-cases where it feels useful to be able to send messages bidirectionally between two different DNAs, and currently this limitation prevents such an arrangement.

Some more context on this: what I’m actually trying to achieve here is bidirectional data propagation between DNAs, so perhaps the signalling API can be used to implement this. More in holo-rea#57.

Short version: I think it could be a good solution, provided signals emitted in one DNA are received by all connected DNAs running in the conductor; and no bridging is needed to enable this data to propagate.

Do you sacrifice data validity doing it that way?

You mean with signals vs with call? I don’t think so- either way, the payload has to validate against a Serde schema in order to be parseable; and the incoming data goes through the same validation steps regardless of how it enters the zome (that’s just up to how you architect your code).

1 Like

@pospi I can’t help but notice you created a cyclic forum / GH issue dependency there with cross-linking :smiley: life imitates art!

I don’t know the reason behind us preventing cyclical bridge dependencies. Ashanti might know about this; I know she was working on bridge config sanitisation a couple months ago. She’s not on the forum though; will ask on MM.

So far signals are only something you explicitly trigger in a DNA for consumption by the client app. I don’t know what the long-range plan for signals is, but I believe we’re also looking at having DHT events trigger signals to subscribers, and perhaps signal propagation between DNAs as well. Don’t know why you couldn’t; you can have calling dependencies between zomes in a DNA.

(It occurs to me that the client could also do ad-hoc bridging between DNAs, and signals could help facilitate this.)

I got more information on this from @zippy:

@lucksus did most of the work on this. I think the cyclic dependency check has to do with the order of instantiating the DNAs and how things are called to during that instantiation to establish the bridging. Technically two DNA’s should be able to depend on eachother because the bridging is just about a capability token having been requested and granted. It maybe that the current implementation of the conductor can’t handle doing that in two phases during installation. Let’s check in with @lucksus when he gets back.

I’m also going to tag @zippy to draw him in to the emerging broader discussion here.

Is developing a “signal passing” system design worth pursuing as the preferred way to manage inter-DNA communication? On face value it feels as though it may simplify a lot of the complex link management logic that I’ve had to implement between records kept in different network spaces. And there are still limits to what I can do there- so far all the event flows have been in one direction but that won’t hold for every feature I need.

A programming paradigm based on signal processing also feels like more of a “Ceptr-like” workflow :wink:

Jamison Day would be another good guy to bring into this conversation, but sadly he’s not on the forum (yet). When we were discussing this in the internal MM, he was concerned that allowing bidirectional bridges would create circular dependencies that (if not managed correctly) could trigger an infinite loop of new entries being created in response to other entries being created. To my mind, though, that risk already exists between zomes and even within a zome. It does require discipline to prevent this sort of thing.

I’m also pretty fond of the programming paradigm based on signals, and YES, it does feel very Ceptr-like. Very much ‘receptive capacity’/yin/etc and all that. Also makes me think of functional reactive programming, which is touted as a way to get a grip on knock-on effects because it’s all about one-way data binding (although you can definitely create infinite causative loops there too).

Hmmmm, signal-passing as a way of propagating data across bridges. This is an interesting line of inquiry. Here’s what popped up for me:

A client and a DNA have a conversation with each other with a pair of channels. One of them, zome calls, is active/direct/yang: the client directs the DNA to do something.

The other, signals, are passive/diffuse/yin: the DNA is informing rather than directing. There’s no dependency on the client; in fact, the DNA doesn’t even know/care if a client is listening.

So now we’ve got this nice duality, yin/yang, passive/active, diffuse/direct, a nice normalisation that creates a control flow without circular dependencies.

Hmmmmm… what other thing acts as a ‘client’ that can make zome calls?

Well how about other DNAs via bridging?

We probably want to avoid explicit circular dependencies among DNAs too. So what could we use to get two-way communication between DNAs and still keep the dependency graph clean?

:thinking:

I don’t think there is a way to do it with the dependency graph clean- as you said, these things require discipline.

The way I would implement it for bidirectional functionality is to have a control flow as follows (indentation shows caller ⇒ callee relationships):

UI call to DNA A
    DNA A gateway
        DNA A handler
        update signal
            DNA B receiver
                DNA B handler

UI call to DNA B
    DNA B gateway
        DNA B handler
        update signal
            DNA A receiver
                DNA A handler

Essentially, separating out the handler logic from the various interpreters ensures you avoid any infinite loops.

When processing operations within the same DNA, different combinations are possible since you can create zome A records from within another zome B and vice versa:

UI call to zome A
    zome A gateway
        zome A handler
        zome B handler
        update signal
            (for 3rd-party use only, nothing listening)

UI call to zome B
    zome B gateway
        zome A handler
        zome B handler
        update signal
            (for 3rd-party use only, nothing listening)

Would be interested in others thoughts on these approaches and how the logic might differ when intra-DHT vs inter-DHT; specifically with regard to validation. (I think this might be where bidirectional call is needed, if receiving networks care about the integrity of external data linking to them.)

I think there is parity between inter-DNA messaging and client messaging- could it just be the same API, with explicit grants between zomes to filter the message types they are listening for?

Yeah, isn’t this risk also there when connecting say three DNAs, cyclic entry creation that could spin out lots of entries by dependency with poor code? So not a viable route anyway?

There’s been some really good indepth discussion on this between @pauldaoust & myself for those who are watching along. Requirements and use-cases crystallizing nicely.

@ViktorZaunders yeah, that’s exactly the concern that one of our team members raised. Reasoning about two mutually interdependent DNAs is moderately easy, and so is reasoning about three if you wrote them all. But once you have an ecosystem of developers and their DNAs, you could run into circles with unintended consequences.

@pospi

you can?! Wow, I didn’t realise that — it feels like leaky encapsulation to me. Wonder if that was just an oversight and it’s gonna get closed in the future…

Back in my programming days, I used to fret and obsess about circular dependencies. In fact, they were impossible cuz I was writing in C#. Made for some convoluted ways of getting the dependency graph untangled; dependency injection was usually my go-to.

Thinking about it, ambient signal emission doesn’t prevent this either. DNA X could emit signal A that causes DNA Y to take action and emit signal B that DNA X receives, causing it to emit signal A again. It was just my proposal for a way to un-circular-ize hard DNA dependencies in a tidy way. Ambient/declarative/passive signals allow a DNA to be ignorant of its consumers, whereas direct/imperative/active calling/message-passing introduces tight coupling because it requires the caller to know how its callees work.

I’m chewing over your thoughts on signalling, message passing, and validation @pospi. Raises interesting thoughts. Re: validation, I see the value. It could be ensured just as easily with emit as with call, I think, because my conductor can provide assurances that the right DNA emitted that signal.

Re:

Definitely see what you’re saying there. Looking for the base abstraction, call could certainly be seen as a special case of send with a method name and tuple of arguments as its message.

And within one conductor, that’s pretty much how it works — the conductor creates a special public grant type for intra-DNA zome calls, inter-DNA zome calls, and UI calls, then gives the token to the callers. It is nice to have a special affordance for function calls layered on top of message passing though, wouldn’t ya say? :wink:

There’s even a pattern for doing it between agents — Alice shares the cap token with Bob, which he then passes back to her whenever he wants to “call a zome function in her running instance” (which actually just looks like him sending her a message consisting of function name, parameters, and cap token). Alice checks the function he wants to call against the conditions of the grant represented by his token, then calls the function for him if it all checks out. Eventually I think this might have its own convenience function in the HDK.

I hope not, because I think it’s a really good time-saver for zome mixins. Think of them as stateful additions to custom business logic… it’s way easier if they can be plugged in to define all the record types, and can be driven by that business logic. I think it’s actually necessary to have that feature to be able to use zomes as functional mixins correctly- otherwise you need to expose all zome functionality over the RPC gateway, which means that external clients would be able to manipulate the zome state without restriction. Unsure if I’m explaining that properly but hope it makes sense…

Oh, hmmm, I see your point — you’re saying that if you want to use a zome to add functionality to another zome’s stuff (e.g., zome B could hang links on zome A’s entries, and you want to make it generic so that zome A could be anything) either they need to be able to privately talk to each other without exposing their guts to the UI (that is, some sort of exposure level other than hc_public), or they need to be able to access each other’s entries directly. Is that about right?

I’ve always seen zomes as basic units of encapsulation that shouldn’t be able to access each other’s data. This is less important when one developer creates all the zomes in a DNA, but it becomes a big deal when devs start plugging third-party zomes into their own DNAs. You’ve got this issue of zomes serving two related but distinct purposes: encapsulation and modularity.

I only half know what I’m saying; it’s hard to talk about it with concrete examples. Sounds like you’ve got some though; what sorts of mixin-style zomes have you built that depend on access to other zomes’ entry types?

Interesting that you say ‘functional’ mixins, since data hiding isn’t really a thing in functional programming; that’s more of a leftover from OOP days.

@freesig I guess this answers our question about whether entry types are namespaced by zome name or hash! Looks like all data lives in a common pool.