Link Registry (or, replacing search engines)

Technical Discussion Libraries
#1

There have been a few conversations ongoing around how we manage linking data structures in Holochain apps. @lucksus has some interesting ideas on the benefits afforded by a public link broadcasting network, which I have tried to expand upon here. IIRC @artbrock, @zippy, @lucksus and myself are all fans of XDI Link Contracts for link definitions; and much of the metadata needed to create these records is provided natively by Holochain.

It occurs to me that this is a larger conversation than simply “let’s create a DNA for storing links”. If we do this right, it becomes a large-scale project to build a search engine for the commons and an active act of resistance against surveillance capitalism. Think about it- what is the value-add that a search engine service creates? Surely the index of links and associations between content that their crawlers provide are a core part of that service. What happens if we co-create that data in a shared DHT and provide further tools to extend and analyse it, rather than defaulting to private ownership?

Having iterated from an ‘address only’ pattern for linking to external content (see holo-threaded-comments and a previous iteration of HoloREA), we have now moved to an ‘address + metadata’ approach. What this means for HoloREA is that we create Satisfactions in one DNA, then trigger creation of a mirroring Satisfaction record in the paired DNA. This ensures the metadata defining the relationship is available to participants of both networks.

I imagine there is an equivalent pattern here for more abstractly recording links between data in different DNAs. It would probably look much the same, except that instead of replicating Satisfaction entries we can replicate LinkMetadata entries. At some point I would like to revisit the way we have implemented links between records in HoloREA and look at making it more generic, so that more abstract metadata can be provided to lower-level tooling (and basically, Holoverse link browser UIs would be able to follow REA data more sensibly).

I think there might be a need for several related standards here- not just for zome API traits but for inter-DNA signalling as well— see How to bidirectionally replicate data between DNAs? and holo-rea#57 but consider in the more abstract context of link broadcasting rather than replicating Satisfaction records.

2 Likes
#2

I was thinking of a distributed spider UI as well, that would go out and find external resources and submit them into a shared zome.

#3

I think for the delete option with help of a new DNA that’s like a registry of deleted entries links in order to make the entries actually remove from the storage.

#4

This would be cool. A part of the infrastructure I think is plug-in zomes that add indexing capabilities. Kinda like the Go prototype Holodex.
(not to be confused with the current HoloDEX decentralised exchange)

#5

Yeah my idea is different UIs may share common or use different zomes, plugins on either end. And some UIs may let the user configure or disable certain features which may use additional zomes or according to what zomes are found running.

#6

I think with that kind of functionality it may actually need to be coded into the source DNAs. For example, to broadcast links into a separate “link indexing” module you would need to code your zome to look for bridges connecting to networks that implement an agreed-upon org.xdi.link_indexing trait; and call those methods appropriately.

At least, that’s one way. There might be better ways. It may be a case that you emit a signal when your zome links to other things; and in response to an agreed-upon payload format any other DNA listening for org.xdi.link_created can record & track its version of the link.

Either way, I think you provide this functionality as a thin wrapper around hdk::emit_signal or hdk::call which those wishing to integrate with this zome (or others) can import and call easily.

1 Like
#7

It’s like thunking between privilege levels, the DNA ensures syntactical correctness and the UI side does external stuff.