Conductor Library Roadmap

realfritz · April 9, 2021, 1:45pm

Thanks @Connoropolous… Looks like there are a number of hurdles here. This helps answer my question of whether I would want to invest time on Holochain right now. I’ll check back in six months and see if the team has communicated a clear plan for mobile. I hope they do. As someone looking for a platform to build on, it would help much.

guillemcordoba · April 9, 2021, 3:51pm

Hey

For mobile, I will be targeting holo for sure, as even if holochain can compile down to android/iOS, the conductor may violate the performance measures of that OS and kill (eg too much background noise when gossiping).

Kizuna is the happ that comes first to mind, they are targeting holo to deliver a mobile happ.

jakintosh · July 25, 2021, 9:38pm

Hi! I’m trying to attempt calling a zome function from a C# (Unity) app, and from my understanding, what I want to be doing is using a websocket connection to the port that the conductor is running on, formatted with MessagePack(?), using this ZomeCall structure.

My question is if I’m wrong about any of those parts, and then if not, I’m looking for more information about how to properly construct that “ZomeCall”. I’ve got WebSockets and MessagePack set up in my test project, but I’m at a loss as to how to actually create the correct data to send over to the conductor. The documentation page linked to above has 404s for the specific types in that call (like CellId or ExternIO) so I’m not sure what I should be sending.

Any help would be appreciated!

Connoropolous · July 26, 2021, 1:36am

Hey,
I can almost def help.

The recent “documentation drive” produced some good results, including this, which can give you a generalized reference and some good descriptions of what you’re trying to do:

It gives you a jumping off point for an alternative implementation as well.
there’s lower level stuff to get at here… are you JS familiar?

github.com

holochain/holochain-conductor-api/blob/f3f01b4d0d53e4ba9fc5d01fffa2efbc3db20b20/src/websocket/client.ts#L12-L106

    
      
          export class WsClient {
            socket: Websocket;
            pendingRequests: Record<number, { fulfill: Function; reject: Function }>;
            index: number;
            alreadyWarnedNoSignalCb: boolean;
          
          
  constructor(socket: any, signalCb?: AppSignalCb) {
              this.socket = socket;
              this.pendingRequests = {};
              this.index = 0;
              // TODO: allow adding signal handlers later
              this.alreadyWarnedNoSignalCb = false;
              socket.onmessage = async (encodedMsg: any) => {
                let data = encodedMsg.data;
          
          
      // If data is not a buffer (nodejs), it will be a blob (browser)
                if (typeof Buffer === "undefined" || !Buffer.isBuffer(data)) {
                  data = await data.arrayBuffer();
                }

This file has been truncated. show original

There is @dellams been trying to do it in a Unity friendly language I think, but he’s on windows and has some some big roadblocks. However, I think his code may be close to hitting the mark:

github.com

NextGenSoftwareUK/Our-World-OASIS-API-HoloNET-HoloUnity-And-.NET-HDK/blob/master/NextGenSoftware.Holochain.HoloNET.Client.Core/HoloNETClientBase.cs

using System;
using System.Text;
using System.Threading;
using System.Net.WebSockets;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.IO;
using System.Diagnostics;
using MessagePack;

namespace NextGenSoftware.Holochain.HoloNET.Client.Core
{
    public abstract class HoloNETClientBase
    {
        private const int ReceiveChunkSizeDefault = 1024;
        private const int SendChunkSizeDefault = 1024;
        private const int KeepAliveSecondsDefault = 30;
        private const int TimeOutSecondsDefault = 30;

This file has been truncated. show original

This is the base layer of the spec they are both implementing for…

github.com

holochain/holochain/blob/2d9401a5e0f74934195a0bf02ca198679b53089d/crates/holochain_websocket/src/lib.rs#L138-L167

    
      
          #[derive(Debug, serde::Serialize, serde::Deserialize, SerializedBytes)]
          #[serde(tag = "type")]
          /// The messages actually sent over the wire by this library.
          /// If you want to impliment your own server or client you
          /// will need this type or be able to serialize / deserialize it.
          pub enum WireMessage {
              /// A message without a response.
              Signal {
                  #[serde(with = "serde_bytes")]
                  /// Actual bytes of the message serialized as [message pack](https://msgpack.org/).
                  data: Vec<u8>,
              },
              /// A request that requires a response.
              Request {
                  /// The id of this request.
                  /// Note ids are recycled once they are used.
                  id: u64,
                  #[serde(with = "serde_bytes")]
                  /// Actual bytes of the message serialized as [message pack](https://msgpack.org/).
                  data: Vec<u8>,

This file has been truncated. show original

Connoropolous · July 26, 2021, 1:40am

Are you in a process of publishing your source code anywhere, I could help a lot if so, or you can paste snippets?

jakintosh · July 26, 2021, 4:16am

I think this (WireMessage) is what I was missing!

pub enum WireMessage {
    ...
    Request {
        id: u64,
        #[serde(with = "serde_bytes")]
        data: Vec<u8>,
    }
    ...
}

So, the holochain conductor is receiving messages via WebSocket, expecting them to be formatted as the Rust enum WireMessage encoded via MessagePack?

And yes, that makes sense given @dellams request structure here.

github.com

NextGenSoftwareUK/Our-World-OASIS-API-HoloNET-HoloUnity-And-.NET-HDK/blob/758e4ca133dfb9a9fef16d482c64f967b5eae470/NextGenSoftware.Holochain.HoloNET.Client.Core/HoloNETClientBase.cs#L378-L385

    
      
          HoloNETRequest request = new HoloNETRequest()
          {
              id = id,
              type = "Request",
              data = MessagePackSerializer.Serialize(holoNETData)
          };
          
          
await SendRawDataAsync(MessagePackSerializer.Serialize(request));

I think I’m not 100% sure about how Rust enums get serialized, and/or how MessagePack might be doing the serialization. To clarify, would a Rust enum serialize out a Case with the type field like this?

pub enum Enum {
    Case
}

→

{
    type: "Case"
}

So, if I have all of that straight, I would assume that I should be deserializing any messages coming from the WebSocket connection with the WireMessage::Response schema, and that I should be assigning my own (arbitrary, but unique) id’s with the requests to track which response is which?

It’s on my list to get the whole project tidied up and onto GitHub soon! But I’ll take another swing at this in the morning using the things I just learned and post a snippet then.

Connoropolous · July 26, 2021, 5:11am

You are clearly onto it now, well done.

rust enums can serialize any number of ways, and so it is per configuration. In this case you see #[serde(tag = “type”)] above the WireMessage enum which is how you end up with ‘type = “Request”’ in Dellams code.

jakintosh · July 27, 2021, 3:23pm

Not sure if this is still the best thread for this, but:

Yesterday I fleshed out my test a bit more, but when I sent a request to the conductor, I got no response. There are many things that could be going wrong here, but I don’t know how to confirm any of them.

Here is a gist that contains the C# code along with a tiny test zome (taken from the gym) with the function I’m trying to call:

gist.github.com

https://gist.github.com/Jakintosh/8578f39a66e50c4beb24bc34275e6ce5

Holochain.cs

using NativeWebSocket;
using SouthPointe.Serialization.MessagePack;
using System;
using UnityEngine;

namespace Client {

	public class App : MonoBehaviour {
		
		[System.Serializable]

This file has been truncated. show original

lib.rs

use hdk::prelude::*;

#[derive(Serialize, Deserialize, Debug)]
pub struct SomeExternalInput {
    first_name: String,
    last_name: String,
}

#[derive(Serialize, Deserialize, Debug)]
pub struct SomeExternalOutput(String);

This file has been truncated. show original

One thought that came to mind though was around Nix, since I don’t really know how it works. If I’m running an hApp within the Nix shell, will I still be able to reach ws://localhost:8888 from outside that Nix shell?

I have very little background in web/networking (mostly work with “native API” type stuff), so I’m also not really sure how to check what kind of ports might be open, and to see what kind of traffic is occurring. And then on top of that, I’m also not really sure how to log things from inside the zome code to see if anything is happening on the holochain side.

EDIT: Worth noting that I’m working from macOS 11.4 (big sur)

jakintosh · July 27, 2021, 3:37pm

Taking the time to write this out this morning with fresh eyes, I realized I’ve made the eternal mistake of “not calling the function”. Turns out I needed to call “websocket.Receive()”

I’ve got response bytes coming back from the conductor now, I’ll keep digging to see if I run into any more issues.

jakintosh · July 27, 2021, 7:41pm

So as an update, I’m communicating with the holochain conductor now, but I’m getting back some errors it seems.

This is a response/request that I’m sending/receiving, in what I believe should be the WireMessage structure. These strings are just me taking the raw bytes and decoding as UTF8 for some readability.

Holochain.Conductor sent msgpack bytes:
��type�Request�id�data�	��cell_id��5uhC0koENtar-jdhP2X5BhC4U5WmmpJWl9GpbvbjGfVRdnv9gmDRZ9�5uhCAkgG-79Oz3TZw7gJ1CXwv7F4Ky-mBWEceGa2KR86Zs_YaPvtfL�zome_name�simple�fn_name�say_my_name�payload� ��first_name�Jak�last_name�Tiano�provenance�5uhCAkgG-79Oz3TZw7gJ1CXwv7F4Ky-mBWEceGa2KR86Zs_YaPvtfL

Holochain.AppConductor received msgpack bytes: 
��type�Response�id�data�X��type�error�data��type�deserialization�data�*Bytes(Deserialize("missing field `type`"))

The gist from earlier is updated with my latest implementation.

The request seems to be including the “type” field in its WireMessage payload, so I’m wondering if the internal “data” field also has another schema that requires a “type”? I’m not really sure where else to dig in the holochain source code to follow along; I’m getting used to Rust from a hApp standpoint, but digging through the holochain github repo still mostly looks greek to me.

jakintosh · July 27, 2021, 9:38pm

At this point, I’ve hijacked the thread (sorry) so I might as well continue documenting my process.

So, once again, after writing out my last post I figured out my problem. The holochain_conductor_api::AppRequest struct inside the WireMessage also needs to be formatter with type and data. Once I got that working, I was getting a new error, which was Bytes(Deserialize("HoloHash error: BadPrefix(\"DnaHash\", [117, 104, 67])")).

After searching through the holochain source, I found that the issue was that my bytes in my byte array were not lining up with the expected byte prefix:

github.com

holochain/holochain/blob/45e524167615e7751a12ada73c8e35e6f9cf33ae/crates/holo_hash/src/hash_type/primitive.rs#L56-L59

    
      
          Err(HoloHashError::BadPrefix(
              PrimitiveHashType::hash_name(P::new()).to_string(),
              prefix.try_into().expect("3 byte prefix"),
          ))

This leads me to believe that my encoding of my dna/agentpubkey hashes is not correct; currently I’m just encoding to UTF8, which I’m assuming is wrong:

	static class AgentPubKey {
		private static string _agentPubKey = "uhCAkt_cNGyYJZIp08b2ZzxoE6EqPndRPb_WwjVkM_mOBcFyq7zCw";
		public static byte[] Get () => System.Text.Encoding.UTF8.GetBytes( _agentPubKey );
	}
	static class DNAHash {
		private static string _dnaHash = "uhC0kTMixTG0lNZCF4SZfQMGozf2WfjQht7E06_wy3h29-zPpWxPQ";
		public static byte[] Get () => System.Text.Encoding.UTF8.GetBytes( _dnaHash );
	}

So I guess my next blocker is, how should I be encoding my C# strings so that they are the correct format for the byte array being sent into rust?

Connoropolous · July 27, 2021, 10:12pm

Right on @jakintosh!

great progress. Let me gather what I can from your posted details.

Connoropolous · July 27, 2021, 10:23pm

So it looks like you have some strings representing the agentpubkey and dna_hash and those appear to be in the base64 encoding. Get a buffer/bytes for that b64 encoded string and you should be able to move forward.

jakintosh · July 27, 2021, 11:14pm

That was it! I overcomplicated it, the strings were fine as is: no byte array conversion needed. That was the last piece! I’m calling a zome function in a live cell from the unity app, and correctly deserializing the response back in C#. Thanks for your help! I should write some of this up, especially regarding pain points and lessons learned. Very exciting to actually prove out this Unity → Holochain bridge for myself. Now for some cleanup and refactoring

Connoropolous · July 27, 2021, 11:41pm

Ah, Im surprised and delighted lol. But whatever works has proved itself! I’d love to be able to produce some documentation about this api out of this!

dellams · July 30, 2021, 9:23am

REALLY excited to see this! Finally after a long 2 years being the only .net holochain dev I could find, I find this yay! It has been really tough when I had noone else to turn to when I got stuck, but @Connoropolous has helped a lot lately thanks.

I got unity and .net talking to holochain back in 2019 with HoloNET but after it was refactored into RSM I had a lot of issues even getting the conductor to connect through WSL2. I still get connection refused so not sure how you got around that issue?

I reversed engineered their node client with MessagePack etc. I noticed you are using a different MessagePack and web socket library to me so that’s interesting and something to note…

Be great to compare notes and be happy to help write all this up with all the experience I have gained over the years working with holochain and .net…

Also, I don’t suppose you wanted a job? I’m currently hiring and been looking for devs with good c# and unity experience so with holochain experience on top that would be a bonus!

We can hopefully work together and would love your help with HoloNET and the. Net HDK I have been working on…

I checked your code and was pleased with the quality and impressed how far you have got, hence the offer… I have seen enough of your skills and experience…

Cheers
D.

Connoropolous · July 30, 2021, 3:36pm

@dellams just fyi. so no wsl2 or anything like that would be going on.

dellams · July 30, 2021, 7:03pm

Ok great thanks.

Not sure if you guys have seen this yet?

Update on the .NET HDK (now evolved into STAR ODK)…

Would love help with this… thanks guys.

Connoropolous · July 30, 2021, 10:24pm

Hey @dellams looks like big stuff you’re building! Pretty wild.

Just wanted to make a correction since I’m mentioned
" UPDATE: We have just bene informed by a dev working for Holochain that Windows support has now been added again!"

I dont work for Holochain I have a big history with Holochain, including working for Holo at one time, but as you may know I run my own business now, Sprillow. We build many projects on Holochain and are primarily involved in happ development, but also low level library development when it helps us achieve our goals.

dellams · July 31, 2021, 9:42am

Oh ok I’ll reword that bit, then thanks…