A quick question: does Holochain depend on any external networking libraries? After a (very) quick look, it seems most advanced stuff is done in-house (kitsune_p2p
)?
(I’m not totally certain what I’m looking for myself, but not just QUIC/quinn (?) - if you want to help me out, keep reading - if you can, this might prove beneficial for Holochain too, once other internet architectures break through )
Reason I’m asking (comments and suggestion appreciated!):
Next (school)year I’m going to do a thesis around Ouroboros (very similar to RINA) and a supervisor sent me a proposal, which is about using this network architecture in serverless environments, where TCP is too much overhead and Redis is suboptimal.
Proposal for more context:
Inter-process communication for serverless application workflows
Serverless computing as possible with AWS Lambda, Google Cloud Functions or Microsoft
Azure Functions enables application developers to write their application logic without
needing to worry about the type and configuration of underlying physical or virtual
infrastructure (VMs or containers). Developers can write an entire cloud application or data
processing pipeline as a combination of functions with short-lived execution time and
limited memory requirements. The underlying serverless platform will then enable the
application to seamlessly scale without requiring any additional input of the developer.
This new cloud computing model has been adopted massively due to these advantages
for developers, but also because it enables the cloud providers to better multiplex their
infrastructure. Despite the tremendous adoption, certain applications still might require
longer completion times due to the suboptimal data communication between function
execution instances. Use of the regular TCP/IP networking stack and associated socket
APIs are not used in serverless environments, because the overhead required to set up
such connectivity for function execution environments which are very short-lived.
Therefore, existing serverless platforms rely on data storage platforms such as Redis to
pipe the outcome of one function execution instance to the input of another. This leads to
suboptimal execution times and inefficient network usage, especially in applications which
involve one-to-many communication patterns.
Objective
The goal of this thesis is to design and evaluate alternative mechanisms for direct
communication between serverless execution instances environments. Ideally, network
communication should be possible between temporary endpoints (execution
environments) using identifiers which can be resolved in ultra-short times, supporting both
uni- and multicast communication patterns between them with negligible connection
setup times.In a first phase, the student will do a preliminary study of scientific literature and network
protocol stack implementations in addition to analysing base workflows in serverless
platforms (e.g., OpenWhisk, Kubeless, etc.).
Next, the student will design novel communication mechanisms supporting rapid setup of
network communication between such short-lived function executions, for example
building further on the concepts of Recursive Internetwork Architecture protocol stacks.
Finally, the student will compare the resulting performance and scalability against
common practice of intermediate storage solutions as well as against a regular network
socket-based approach. Co-design with a serverless application scheduler able to exploit
the new communication paradigm might be part of the thesis. The local data center and
testlab infrastructure of IDLab will be used to experiment at larger scales.
This made me think about distributed environments in general (first Unison, then its influencer Erlang) ending up at thinking, can Rust do this too?
I'm still in the middle of research, but I found already many interesting sources/libraries:
- Bastion
- Reddit: fearless distributed computing
- timely-dataflow
- citybound with its back-end kay, also usable in wasm
- Reddit: Distributed computing
-
Amadeus with the constellation framework, which uses
quinn
, which I think is also used by Holochain? - Ballista
- libp2p (?)
So, if possible, I would nudge my subject to write something in the Rust (distributed computing) ecosystem.
Now I was thinking if there are libraries/tools/… (present of future) Holochain uses which might benefit from such a thesis (which is 720 hours nominally).
The subjects are mostly in the domain of networking for the research group I’m in contact with now.
Of course, in the end the supervisors decide if there is enough novelty etc. to allow a subject. (A ‘simple’ refactor of Holochain networking to support Ouroboros isn’t novel enough )