I am writing a paper on the performance of holochain for mass communications, in comparison to other systems like blockchain.
Is there any dataset or testing application that can generate data for such comparison? I have tried some benchmarking codes from GitHub but most of them are archived or deprecated.
1 Like
I’m not sure about datasets that are available, but I do remember seeing a paper on it in IOT. I’ll try and find it, but feel free to search as well.
1 Like
Hi! great to hear you’re looking at Holochain in your research paper. There are two projects I can think of that are really focused on performance metrics – Elemental Chat and RedGrid.
Elemental Chat is a toy application meant to let people try Holochain and probe it for performance bottlenecks – we’re currently rolling it out to a bunch of people who bought the HoloPort hardware. We wanted it to feel fast, so there are some ‘scenario tests’ that try to push it to its limit. These tests generate all the test fixtures and data (number of nodes, number of messages over a certain timespan) on the fly. Modifying and running the tests for yourself might have a bit of a learning curve, so let us know if you hit some roadblocks. They use a JavaScript framework called Tryorama, which is orchestrates tests involving multiple agents on multiple devices.
The RedGrid team and supporters – @mikeg, @simwilso, @guillemcordoba, and @Connoropolous – are doing lots of benchmarks for their internet-of-energy project, where lots of smart devices need to be talking to each other and to grid energy providers with lots of short bursts of high activity. You might want to talk to them too.
Comparing Holochain and blockchain is a bit like comparing apples to onions; they’re quite different. With blockchain you’ve got a consistent global state with reasonable finality guarantees (IOW “I know this transaction has completed it because it got into block X which was N blocks ago”), but you’ve also got that bottleneck on writes, because each block can only be a certain size and it’s gotta be propagated and accepted by all nodes in the network. There are two kinds of latency at work: write latency (time for your message to get accepted into a block) and confirmation latency (time until the block containing your message reaches a reasonable level of finality).
Whereas with Holochain, there’s never a guaranteed global system state, but because there’s no global ledger there’s also no write bottlenecks. Each participant writes to their own ledger, so they’re really only limited by the performance of their own hardware. Then they just share a copy of their ledger entry with a few validators. Once a quorum of n validators accepts it and haven’t found a problem (e.g., author has tried to ‘fork’ their own ledger or commit data that breaks the application’s rules) it’s considered ‘final’. So there’s no write latency but there is latency between when you write an entry and it becomes available to others as trustworthy data.
Right now in our testing of Elemental Chat with a 600-node network in which 135 nodes are selected to each send one message, all within a one-minute timespan, we’re seeing a long-tail distribution of finality times. 92% of messages reach network finality within 30 seconds of being written locally, 98% within one minute, and a few outliers at 10 minutes. This is using a relay server to help peer-to-peer nodes punch through their home firewalls; if we remove that server and test on a local network, we see all those numbers improve by about 10×. I misinterpreted the test; just checking with the devs to see what is actually true 
One nice thing is that, as a network grows, both the validation burden and the throughput available to each individual node stays roughly constant because Holochain ‘shards’ the validation responsibility for the global data set (where node = actual human or IoT device; we don’t make distinctions between full nodes, validator nodes, and light clients because the validation burden is so light).
Any more framing questions, just ask; happy to help!
2 Likes