Conductor Library Roadmap

Technical Discussion Concepts & Architecture
#1

Good evening, congrats on your recent acclaim. I am currently evaluating the holochain project for use as a persistence layer below a generalized structured information application.

I understand that you are currently doing a transition of sorts. I’ve watched the recent AMA and scanned through the forums and docs. I’m curious if you can comment on conductor libraries for the major OSes.

Where are they on the roadmap?
Have you done any proof of concept work?

Thanks

1 Like
Welcome! Introduce Yourself
#2

Bumping this up. Is there someone on the team that could speak to the question of conductor libraries?

#3

Hi @realfritz … I’m peripheral to the core team… I used to work for Holo and still work as a professional Holochain developer, I can perhaps help.

Your comment about “OSes”… would you like to be able to run the main holochain binary holochain (which is sometimes / contexts called the Conductor) on different operating systems?
It is currently possible to run it on Windows, Linux, or Mac, but Windows ONLY with wsl2 which is essentially because its like Linux on Windows.

There is separately something we call the conductor library, which is a way of interfacing with, and controlling, the Holochain conductor while it’s running.
Three such ways to do this are

  • raw websocket requests via any library or language of your choice
  • from Javascript via the “holochain-conductor-api” (which should be renamed to indicate JS) https://github.com/holochain/holochain-conductor-api
  • from the command line via the hc dev tools binary hc sandbox call ...

Does that help? Further questions?

1 Like
#4

Thanks for the response @Connoropolous. This helps to round out the picture.

Specifically though, I’m curious about conductor libraries for mobile devices (iOS, Android). This is a must-have for most applications and I’m having a hard time seeing how this is done with the current toolset or how this will be supported in the future.

Based on this support article, it does seem that the concept of a conductor on mobile has been conceived.

https://developer.holochain.org/docs/concepts/2_application_architecture/

#5

Interesting, I see. That’s actually something I’ve been quite interested in myself…

The pathway to IOS (not sure about Android, but possibly not much different) is, I believe, upgrading the wasm runtime (wasmer https://github.com/wasmerio/wasmer/) to version 1 at least, up from 0.7.1 or something that Holochain currently uses, in this sub package https://github.com/holochain/holochain-wasmer. The team has suggested that they may be able to get to this after they get the deepkey happ up and running.

There could possibly be ways to get it even sooner, based on what I just heard from @ddd-mtl that he got wasmer to build for an arm64 chip on a Raspberry Pi in the last couple of days, after it originally didn’t work and wouldn’t compile.

wasmer upgrade or configuration is the first hurdle, and beyond that there may not be many others, or there might be, I’m not totally sure past that.

#6

You can see an issue I opened in November on the subject: https://github.com/wasmerio/wasmer/issues/1832

But I think v1 actually does compile now

#7

Thanks @Connoropolous… Looks like there are a number of hurdles here. This helps answer my question of whether I would want to invest time on Holochain right now. I’ll check back in six months and see if the team has communicated a clear plan for mobile. I hope they do. As someone looking for a platform to build on, it would help much.

1 Like
#8

Hey :slight_smile:

For mobile, I will be targeting holo for sure, as even if holochain can compile down to android/iOS, the conductor may violate the performance measures of that OS and kill (eg too much background noise when gossiping).

Kizuna is the happ that comes first to mind, they are targeting holo to deliver a mobile happ.

#9

Hi! I’m trying to attempt calling a zome function from a C# (Unity) app, and from my understanding, what I want to be doing is using a websocket connection to the port that the conductor is running on, formatted with MessagePack(?), using this ZomeCall structure.

My question is if I’m wrong about any of those parts, and then if not, I’m looking for more information about how to properly construct that “ZomeCall”. I’ve got WebSockets and MessagePack set up in my test project, but I’m at a loss as to how to actually create the correct data to send over to the conductor. The documentation page linked to above has 404s for the specific types in that call (like CellId or ExternIO) so I’m not sure what I should be sending.

Any help would be appreciated!

1 Like
#10

Hey,
I can almost def help.

The recent “documentation drive” produced some good results, including this, which can give you a generalized reference and some good descriptions of what you’re trying to do:

It gives you a jumping off point for an alternative implementation as well.
there’s lower level stuff to get at here… are you JS familiar?

There is @dellams been trying to do it in a Unity friendly language I think, but he’s on windows and has some some big roadblocks. However, I think his code may be close to hitting the mark:

This is the base layer of the spec they are both implementing for…

1 Like
#11

Are you in a process of publishing your source code anywhere, I could help a lot if so, or you can paste snippets?

#12

I think this (WireMessage) is what I was missing!

pub enum WireMessage {
    ...
    Request {
        id: u64,
        #[serde(with = "serde_bytes")]
        data: Vec<u8>,
    }
    ...
}

So, the holochain conductor is receiving messages via WebSocket, expecting them to be formatted as the Rust enum WireMessage encoded via MessagePack?

And yes, that makes sense given @dellams request structure here.

I think I’m not 100% sure about how Rust enums get serialized, and/or how MessagePack might be doing the serialization. To clarify, would a Rust enum serialize out a Case with the type field like this?

pub enum Enum {
    Case
}

→ 

{
    type: "Case"
}

So, if I have all of that straight, I would assume that I should be deserializing any messages coming from the WebSocket connection with the WireMessage::Response schema, and that I should be assigning my own (arbitrary, but unique) id’s with the requests to track which response is which?

It’s on my list to get the whole project tidied up and onto GitHub soon! But I’ll take another swing at this in the morning using the things I just learned and post a snippet then.

2 Likes
#13

You are clearly onto it now, well done.

rust enums can serialize any number of ways, and so it is per configuration. In this case you see #[serde(tag = “type”)] above the WireMessage enum which is how you end up with ‘type = “Request”’ in Dellams code.

1 Like
#14

Not sure if this is still the best thread for this, but:

Yesterday I fleshed out my test a bit more, but when I sent a request to the conductor, I got no response. There are many things that could be going wrong here, but I don’t know how to confirm any of them.

Here is a gist that contains the C# code along with a tiny test zome (taken from the gym) with the function I’m trying to call:

One thought that came to mind though was around Nix, since I don’t really know how it works. If I’m running an hApp within the Nix shell, will I still be able to reach ws://localhost:8888 from outside that Nix shell?

I have very little background in web/networking (mostly work with “native API” type stuff), so I’m also not really sure how to check what kind of ports might be open, and to see what kind of traffic is occurring. And then on top of that, I’m also not really sure how to log things from inside the zome code to see if anything is happening on the holochain side.

EDIT: Worth noting that I’m working from macOS 11.4 (big sur)

1 Like
#15

Taking the time to write this out this morning with fresh eyes, I realized I’ve made the eternal mistake of “not calling the function”. Turns out I needed to call “websocket.Receive()” :sweat_smile:

I’ve got response bytes coming back from the conductor now, I’ll keep digging to see if I run into any more issues.

2 Likes
#16

So as an update, I’m communicating with the holochain conductor now, but I’m getting back some errors it seems.

This is a response/request that I’m sending/receiving, in what I believe should be the WireMessage structure. These strings are just me taking the raw bytes and decoding as UTF8 for some readability.

Holochain.Conductor sent msgpack bytes:
��type�Request�id�data�	��cell_id��5uhC0koENtar-jdhP2X5BhC4U5WmmpJWl9GpbvbjGfVRdnv9gmDRZ9�5uhCAkgG-79Oz3TZw7gJ1CXwv7F4Ky-mBWEceGa2KR86Zs_YaPvtfL�zome_name�simple�fn_name�say_my_name�payload� ��first_name�Jak�last_name�Tiano�provenance�5uhCAkgG-79Oz3TZw7gJ1CXwv7F4Ky-mBWEceGa2KR86Zs_YaPvtfL

Holochain.AppConductor received msgpack bytes: 
��type�Response�id�data�X��type�error�data��type�deserialization�data�*Bytes(Deserialize("missing field `type`"))

The gist from earlier is updated with my latest implementation.

The request seems to be including the “type” field in its WireMessage payload, so I’m wondering if the internal “data” field also has another schema that requires a “type”? I’m not really sure where else to dig in the holochain source code to follow along; I’m getting used to Rust from a hApp standpoint, but digging through the holochain github repo still mostly looks greek to me.

#17

At this point, I’ve hijacked the thread (sorry) so I might as well continue documenting my process.

So, once again, after writing out my last post I figured out my problem. The holochain_conductor_api::AppRequest struct inside the WireMessage also needs to be formatter with type and data. Once I got that working, I was getting a new error, which was Bytes(Deserialize("HoloHash error: BadPrefix(\"DnaHash\", [117, 104, 67])")).

After searching through the holochain source, I found that the issue was that my bytes in my byte array were not lining up with the expected byte prefix:

This leads me to believe that my encoding of my dna/agentpubkey hashes is not correct; currently I’m just encoding to UTF8, which I’m assuming is wrong:

	static class AgentPubKey {
		private static string _agentPubKey = "uhCAkt_cNGyYJZIp08b2ZzxoE6EqPndRPb_WwjVkM_mOBcFyq7zCw";
		public static byte[] Get () => System.Text.Encoding.UTF8.GetBytes( _agentPubKey );
	}
	static class DNAHash {
		private static string _dnaHash = "uhC0kTMixTG0lNZCF4SZfQMGozf2WfjQht7E06_wy3h29-zPpWxPQ";
		public static byte[] Get () => System.Text.Encoding.UTF8.GetBytes( _dnaHash );
	}

So I guess my next blocker is, how should I be encoding my C# strings so that they are the correct format for the byte array being sent into rust?

Wanted: A C++ Client
#18

Right on @jakintosh!

great progress. Let me gather what I can from your posted details.

3 Likes
#19

So it looks like you have some strings representing the agentpubkey and dna_hash and those appear to be in the base64 encoding. Get a buffer/bytes for that b64 encoded string and you should be able to move forward.

1 Like
#20

That was it! I overcomplicated it, the strings were fine as is: no byte array conversion needed. That was the last piece! I’m calling a zome function in a live cell from the unity app, and correctly deserializing the response back in C#. Thanks for your help! I should write some of this up, especially regarding pain points and lessons learned. Very exciting to actually prove out this Unity → Holochain bridge for myself. Now for some cleanup and refactoring :sweat_smile:

4 Likes
Wanted: A C++ Client