Best practises and patterns for app modularity?

A big conversation has opened up around modularisation and splitting functionality into separate DNAs, vs having stuff in different zomes within the same DNA.

Core framing: DNAs are like microservices. Zomes are not like microservices at all, they’re more like libraries. They cannot be remixed without developer know-how.

So there is a big accessibility concern here— non-technical users should be able to create their own arrangements of apps out of existing building blocks, to service their particular needs.

When it comes to groups- ideally I don’t want to have to have a group membership zome in every DNA in an application suite - awkward to synchronise and maintain the code during updates. We also don’t want to end up with “one DNA per app” for the same maintenance reasons, but also to facilitate diversity of features (we want to empower users to swap microservices out for others that are more appropriate to their context). The ideal for me is that groups act like “lobby networks”, and one group agent DNA is the point of authority for several other cooperating DNAs within an application suite - it’s not just one big network, but a collection of networks in a “neighbourhood”.

Potential disadvantages of this degree of granularity were raised (many thanks to @guillemcordoba for his deep insights), to which I have some potential approaches, but there are gaps. These points are discussed in the context of a Reddit-like app (with membership, subreddits and voting modules) to give some tangibility to the discussion-

Membrane inconsistency

ie. one DNA for all of Reddit, vs separate DNAs for every subreddit.

If every subreddit is its own DHT, and someone wants to create a new subreddit but maintaining the membrane with the same people, they basically have to clone the DNA and start the membrane vouching problem all over again. If instead, I can have multiple subreddits in the same DNA, I can just create another one in that same app.

I’m not sure that this is a correct interpretation of the design I’m aiming for. What I would like is for there to be a “delegated authority” zome within each subreddit DNA, and for all those zomes to delegate to the same group membership / registration DNA for access control. So, the membership would be carried across.

The alternative is building non-trivial membership functionality into the monolithic DNA which manages who is in which subreddit.

In order to do this the modular way, each subreddit network needs to be forced to authenticate against the specific group membership DNA (otherwise users can swap in any random group and authenticate against it). There are some thoughts on how to do this, but it’s still an unsolved problem that needs to be discussed separately. Most likely, every zome API method in the subreddit needs to manually check against the membership data on each request. Capabilities do not work for this- they manage user-to-user permissioning, not groupwise permissioning.

Validation rules / warrant inconsistency

Holochain warrants don’t cross DNAs. So, if I hack a separate voting DNA and do something like give myself a heap of upvotes and then get banned, I don’t also get banned from the membership DNA.

Possible ways to address this:

  • We have the same validation rules in both DNAs and we bring invalid data from one DNA to the other (we need to trust the agent that is actually doing the bridging, somehow), and holochain develops a custom “emit warrant” function (which I wouldn’t bet on). This doesn’t seem feasible at first glance because it would defeat separation of concerns between the modules.
  • We have the validation rules of the voting DNA depend on the membership DNA through bridging (and even then we are not preventing new invalid votes to be cast). This Guillem is very wary of since it includes its own group of requirements problems:
    • The voting DNA has a hard dependency on the membership DNA: all nodes that have the voting DNA installed need to also have installed the membership DNA. What happens when we want to update only the membership DNA? What governance should be coded around changing the allowable set of membership DNAs we authenticate against? A curly problem, but not an unsolvable one.
    • The voting DNA has to trust the membership DNA to be deterministic on the function calls inside the validation rules. Maybe OK with standard zome traits. Does it really matter though? You can always retry operations after the membership DNA has synchronised.
    • We don’t think holochain internals can catch “DHT call determinism”. For example, if in a validation rule there is a get_entry to the DHT, holochain subconscious will try to retrieve the entry and if there is a timeout and it does not find it, it will pause the validation of the entry and resume it later. This way, we achieve determinism in validation rules while allowing to query the DHT. BUT, if this is done accross DNAs, I don’t think holochain subconscious can do the same tricks, so if there is some error in retrieving the entry in the other DNA, that validation rule will fail and this entry marked invalid (when it was not). Unclear whether this feature will be possible in future.

Access control inconsistency

Imagine that, in trust networks, I only want to share my voting data with the people that are already inside the membrane (my neighbors). I don’t want any other agent to be able to see the data. If voting and content DNAs are separated, how do I achieve that? Possible solutions:

  • The voting DNA needs to include the same membrane as the content one. Issue was raised as “have to keep updating both membranes when a new agent is vouched for, for example”, but I don’t think this applies if implemented as delegated authority. The whole point is to have group membership auth in one single location, shared by all DNAs in the neighbourhood.
  • Personas way + global shared voting DNA: every agent has its own little “votes DNA”. This will allow any DNA to use the personal DNA for every agent to store their voting information. Disadvantage is, if other agents want to read my votes, I can grant them permissions to do so with capability tokens, through the shared voting DNA and then bridging to my personal DNA. But I pretty much have to keep the capability tokens in sync with the membranes of the shared spaces in which I’m in, which have different rules… Seems hard.

@guillemcordoba please provide any corrections / additions to these comments, I have butchered your original posts quite a bit in this distillation heh

2 Likes

I think this is basically right.

About the membrane inconsistency point, huuuuuh right. So for clarification, you are proposing a global DNA with nothing more than “who is allowed to enter spaces for this group”, and smaller DNAs making requests to that one.

In the case of reddit this may be overkill, but if we find a way for any application to call that DNA, that same group can basically install new apps that call that global one, this could work well. Every one of those apps will have a hard dependency on the membrane one thought, but maybe that’s okey.

Interesting pattern, wondering how the “DHT call inconsistencies” can come into place here too (membranes are just validation rules on agents), since we need determinism.

In the case of the latest iteration of Acorn app, we have prototyped a way of bounding DNA spaces for the app overall.

There is one DNA called “profiles” (forgive me, we can integrate this with personas/profiles, or other ID management systems in future, and would intend to!) which handles and hosts peoples profile data, and all agents using the app are registered there. The UI interacts first with that DNA. Then, when you create a project, it uses Conductor Admin calls to create a new DNA by cloning an as-yet-uninstantiated “projects” DNA, and creating an instance of it.

By entering a “secret phrase”, which is built-in as the UUID of the Project, other users can join that DHT/DNA.

So in this photo, each Project represents a distinct DNA that the UI is interacting with. All Goals/Edges/Comments/Votes/etc occur within those DHTs.

So for our acorn-hc releases we went for releasing one DNA, to two, per release.

5 Likes

Omg I didn’t know that cloning (or its poor man version install from file) was already possible.

Curious. How do you manage membranes in those little dnas?

Yes, that’s right.
Here’s that…
link to code

How do you manage membranes in those little dnas?

what is it that you mean?

Or perhaps my answer is that the membrane management happens via the uuid which contains a secret key such as uuid-special-tree-dance-happen-monster randomly generated with extremely low collision likelihoods. Once you are “in” to the DNA, everything is shared public amongst members. When people join, they list themselves as having joined.

Here was the weird part… testing whether a DNA already exists for joining. I had to…

  • join the DNA and instantiate it
  • fetch a particular expected entry type, called “Project Meta” which should be hanging off an expected anchor if someone actually created this project and invited you to it
  • if that entry does not exist, back out of the DNA, and uninstantiate it

link to code

1 Like

@guillemcordoba we chatted about this in today’s Asia-Pacific Hackalong. Another way to address this could be some sort of a flagging system that ripples across related DNAs due to reputation staking. Basically if I flag an agent, or receive a flag about an agent in one DNA, I should be able to see their malicious behaviour in other DNAs as well. Something like this could be implemented using Reputation staking protocols like this.

However, it may make way more sense for HC Core to address this, rather than patching on solutions for such critical functions.

@sidsthalekar @pospi yeah I might be wrong here, but I doubt that holochain core will support propagating warrants, since there is no builtin way from outer DNAs to verify that those warrants are good, and not fabricated by some agent. The only reason nodes rely and propagate warrants in a single DNA is because they can verify if the warranted entry was invalid or not by running the validation rules of that DNA.

So maybe the way to go is having a way to include a “flag” module in all DNAs accross which you want to be able to propagate bad behaviour, and include some mechanism to verify that external entries to the current DNA are indeed invalid. We have to be smart about this, I don’t see a clear definitive pattern but different options with tradeoffs.

Have you considered the “compile every combination of zomes you want users to have available option”? This will work out-of-the-box, and I think it will give a pretty good UX as well. If you want I can expand on the flow here. Maybe this can work as the proof-of-concept / MVP pattern.

That same group can basically install new apps that call that global one, this could work well. Every one of those apps will have a hard dependency on the membrane one thought, but maybe that’s okey.

Yeah, I think when you look at this as “module-first”, rather than “library-first”, a lot of interesting possibilities open up. @sid calls this architectural pattern “specific culture, generic tools”; which is essentially what happens if you build a bunch of really useful modules as microservices which all interoperate with each other readily and over common protocols. It means that groups using the software pick the pieces that resonate with them, rather than having to be lumped with a single “upvotes / downvotes” quality indicator of the sea of data beneath them.

UIs need to be built to support this, too… so that’s why building general-purpose UI libraries is a key goal that has to complement this one. Otherwise, again, you end up with a situation where human/interface experimentation is the domain of computer geeks, and geeks alone.

there is no builtin way from outer DNAs to verify that those warrants are good, and not fabricated by some agent

include a “flag” module in all DNAs accross which you want to be able to propagate bad behaviour

Back-of-the-napkin thoughts: treat warrants like zome traits. In order for a warrant-passing effect to occur, affected DNAs must opt in via zome.json configuration to allow warrants incoming from a particular source, with a particular DNA hash.

As an MVP, write those DNA hashes within the genesis callback of the zome. Use pre-known entries that are defined in “core Holochain packages” on crates.io, so that there are constants and structs available to pull into code in order to manipulate the allowable origin of warrants. This leaves app developers to plug in any governance behaviour around that, or leave it to forking out of the network to deal with any disagreement in changing the mutually agreed-upon value over time.

2 Likes

Hi @pospi, having a sense of following you wherever you go in the forum because you are touching all the issues that excite me haha :slight_smile:

About the warrants… Doubt this can work, because the DNA that is “receiving” warrants from another one has to verify that the warrant was not counterfeited or fabricated by the agent that is propagating it. In one DNA, this is done by running the validation rules and verifying things yourself, but outside of that DNA you have basically no way of doing that.

The approach that I would try to take whenever possible is develop data structures that are “self-validating”, in the sense that you don’t need to look at other DNAs to be able to be sure if the entry is valid or not. In the case of the vouches in the social triangulation zome, the vouches are self validating because they can include the signature of the emitter of the voucher, so I don’t have to go to the original DNA to verify whether the vouch is there or not, I know for a fact that the emitter did create the vouch. This makes cross-DNA behaviours and validation orders of magnitude easier.

About composing dynamic UIs… I’m so excited about this :DDDDD, it has been my work the past year basically. My dream (which talking with @sidsthalekar the other day he seems to share and be very aligned with his Neighborhoods) was something like this:

  1. A user wants to create a new community/group of work/whatever flow to connect to other people.
  2. They go in UI “building blocks” mode, and they choose what their group needs. “Let me see… I’m going to include a chat, and also a kanban board, and a shared folder space, and a calendar”. You can imagine a building blocks library with hundreds of modules.
  3. After they choose what they want, the UI is dynamically composed and works out of the box.
  4. They invite other people to the space, which is already set up for them.

I think you know about the _Prtcl micro-frontend infrastructure (code, documentation on how to build MicroModules ), which is what I’ve been building to reach this goal. I love your approach, and I think the choice of Svelte is great (Custom Elements FTW!), and I think it could be perfectly integrated with our MicroModules architecture. We can talk about that anytime you want if you’re interested, I think it’s a perfect fit.

3 Likes

If that is the case then the way you have of doing it is to only allow for inter-DNA warrant delegation on the basis of an exact DNA hash; in that way you can use such hash to guarantee that you’re running the apprpriate calling code for consistency. Otherwise, you can just fail validation?

We can talk about [MicroModules] anytime you want if you’re interested

Yes please, would love to! I was going to handle backend composition via GraphQL- since you have to declare dependencies all the way to the root of your queries for each component connected to the “backend” then it’s easy to parse the schemas covered by those modules and use them to determine the set of required DNAs to service the needed functionality.

2 Likes

Please loop me in for the micromodules conversation if possible @pospi @guillemcordoba
Maybe in the coming week?

I would like be there in conversation as well. If possible please add me. @guillemcordoba @sidsthalekar @pospi

+1 me please, if ok with all!

1 Like

Okey yeah, seems like it’d be great to have an in depth session about these topics. How about next week? This one is a little packed and by next week I’ll maybe have some more things prepared to show. When works for you?

@hedayat @pospi @Qubeo @sidsthalekar

1 Like

Me too please, if it’s okay with the group. Thanks!

1 Like

For those watching, @sidsthalekar has pulled a Telegram channel together to coordinate a time and one of us will publish it here and to the Holochain events calendar when ready (:slight_smile: