Best approach for deserializing websocket holochain call results in RUST

simwilso · November 15, 2019, 9:09pm

Hi team.
Having some great fun testing on sim2h guys. Fantastic job, very nice for a newb like me!
Battling with rust though and looking for some help on the best/right way to deserialize some output from the DHT from a websocket call by my RUST program.

My setup has two agents bob and alice. Bob is able to read alices chain and gets the following standard output from the dht:

Text(
    "{\"jsonrpc\":\"2.0\",
    \"result\":\"{
        \\\"Ok\\\":[{
            \\\"price\\\":\\\"62.4563\\\",
            \\\"author_id\\\":\\\"HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja\\\"
                    },
                    {
            \\\"price\\\":\\\"62.4563\\\",
            \\\"author_id\\\":\\\"HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja\\\"
                    }
            ]}\",
            \"id\":\"bob
    \"}",

I want to organize this output so have created the following structures:

#[derive(Default, Debug, Clone, PartialEq, serde_derive::Serialize, serde_derive::Deserialize)]
struct Root {
    jsonrpc: String,
    result: Result,
    id: String,
    }

#[derive(Default, Debug, Clone, PartialEq, serde_derive::Serialize, serde_derive::Deserialize)]
struct Result {
    #[serde(rename = "Ok")]
    ok: Vec<Ok>,
    }

#[derive(Default, Debug, Clone, PartialEq, serde_derive::Serialize, serde_derive::Deserialize)]
struct Ok {
    price: String,
    author_id: String,
    }

I want my program to capture in a variable only the last ‘price:’ string published on alices chain but I’m struggling to work out the syntax to even just println! the price variable let alone find and capture the last one in the dynamic object.

Can someone help with a clean way to do that?

What have I tried:
I’ve tried to call even the first ‘price’ variable above using the following which compiles but panics on the last println! when I run it:

let res = get_from_dht("HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja".to_string());
println!("{:#?}", res);  //i see all of alices entries.. success
let dhtjson: Root = serde_json::from_str(&res).unwrap();
println!("{:#?}", dhtjson.result.ok[0].price);

It gives me the following error:

"{\"jsonrpc\":\"2.0\",\"result\":\"{\\\"Ok\\\":[{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja\\\"}]}\",\"id\":\"bob\"}"
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("invalid type: string \"{\\\"Ok\\\":[{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja\\\"}]}\", expected struct Result", line: 1, column: 141)', src/libcore/result.rs:1084:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

here also is my dht call which works:

fn get_from_dht(_address: String) -> String {
        let json = serde_json::json!(
            {"id": "bob",
             "jsonrpc": "2.0",
             "method": "call",
             "params": {"instance_id": "test-instance",
             "zome": "spot_signal",
             "function": "get_price",
             "args": {"agent_address": _address}}
         });
        let (tx, rx) = channel();
        let tx1 = &tx;
        connect("ws://localhost:3402", |out| {
            // call an RPC method with parameters
            out.send(json.to_string()).unwrap();
            move |msg| {
                println!("Got message: {:#?}", msg);
                tx1.send(msg).ok();
                out.close(CloseCode::Normal)
            }
        }).unwrap();
        rx.recv().unwrap().to_string() // get the value in the dht via websocket
    }

Can you suggest a syntax and approach above to take the return and extract only a particular variable from the multilayered result (i.e. in this case the last price entry included on alices chain)?

simwilso · November 16, 2019, 9:48pm

my problem is with this line:
let dhtjson: Root = serde_json::from_str(&res.to_string()).unwrap();
println!("{:#?}", dhtjson.jsonrpc);
when I try above to parse the result into my own struct ‘Root’ I get the error:
expected struct Result", line: 1, column: 244)', src/libcore/result.rs:1084:5

But it works when I parse to Value i.e.:
let dhtjson: Value = serde_json::from_str(&res.to_string()).unwrap()
println!("{:#?}", dhtjson["jsonrpc"]);
I get:
String("2.0",)

Even as a temp workaround is there a syntax I can use under in the ‘Value’ structure to access my price response?
<and apologies I know this is not a holochain question it is a Rust question so probably shouldn’t clog up this channel asking>
i.e this fails (I get Null) but I was expecting I’d be able to use something like this:
println!("{:#?}", dhtjson["result"]["Ok"][0]["price"]);

thanks:pray:

simwilso · November 18, 2019, 6:19am

Not the neatest solution but managed to parse my dht data from websocket.
I had to extract it twice to unwrap the ‘result’ which holds a series of objects in an array.
With the code below I was able to extract the first ‘price’ value from the response data. I’m sure there’s probably a neater solution in the code examples but if you’re playing this works:

    let recieve = rx.recv().unwrap().to_string();
    let res = serde_json::from_str(&recieve.to_owned());
    let mut end_price = json!({ "price": "notset" });
        if res.is_ok() {
            let p: Value = res.unwrap();
            let q: Value = serde_json::from_str(p["result"].as_str().unwrap()).unwrap();
            println!("{}", q["Ok"][0]["price"]);
            end_price = q["Ok"][0]["price"].clone();

        } else {
            println!("didnt work");
        }
    end_price

My next problem is how to alter this code so only the last ‘price’ value from the array of objects is printed… i.e.

            println!("{}", q["Ok"][THISARRAY.len - 1]["price"]);

pauldaoust · November 19, 2019, 5:33pm

Hey @simwilso it’s too bad that @freesig is not here (prob in an airport somewhere) because he could probably drill right into a solution for you. I’ll do my best to muddle through my limited knowledge of Rust in the meantime.

In the first example:

let dhtjson: Root = serde_json::from_str(&res).unwrap();

it looks like, in panicking, your code is doing what it ought to. That’s because serde_json::from_str() gives a Result type (which is either Ok(your_deserialised_value) or Err(some_error)). Unwrap is an all-or-nothing thing which means “get the value or die”. As to why it’s not deserialising, though, it turns out that the return value of the JSON-RPC call is actually a plain ol’ string that has to be deserialised to your native type on its own. This would probably work:

struct Root {
    jsonrpc: String,
    result: String,
    id: String,
}

// ...

let dhtjson: Root = serde_json::from_str(&res).unwrap().result;
let result: Result = serde_json::from_str(&dhtjson.result).unwrap();

// ... Now you can access it as a vector of prices.

But I can see some ways to make things easier.

First of all, there is a JSON-RPC library from Parity that implements a client that’ll give you a typed return value. With this you can skip all the JSON-RPC boilerplate and focus on just your domain types. I don’t know if this is the one Tom was using, but maybe he can set you straight once he gets back to Melbourne.

In this case you’d probably use it something like this (forgive my abominable Rust):

// Note: rather than re-implementing the zome's structs in this client as I've shown,
// you might just want to take all the domain types from your spot_signal zome
// and put them into a crate of their own, so you can import their type defs
// into both your zome and this client. DRY FTW!
#[derive(Default, Debug, Clone, PartialEq, serde_derive::Serialize, serde_derive::Deserialize)]
struct SpotPrice {
    price: String,
    author_id: String,
}

// It's easier to use Rust's built in Result type than roll your own; Serde
// supports it for free. This lets you support both possible return values
// from a zome function: Ok(some_value) or Error(some_error).
alias SpotPricesResult = std::result::Result<Vec<SpotPrice>>;

// [...] skip to the body of your get_from_dht function that gets the last spot price...
let get_price_params = serde_json::json!(
    {
        "instance_id": "test-instance",
        "zome": "spot_signal",
        "function": "get_price",
        "args": { "agent_address": _address }
    }
);
// call_method() returns a Future, which is the same as a JS promise. It's a
// bit weird because the error condition of this future could be an error in
// serialising the args, or it could be a WebSocket timeout, or it could be an
// error in deserialising the zome function's return value. But at any rate, we
// wait for the future to resolve, then use the ? operator to either unwrap it
// to a successfully deserialised vector of prices, or return an error using the
// ? operator.
let prices = client.call_method("call", "SpotPricesResult", get_price_params).wait()?;
let maybe_last_price = prices.last();
match maybe_last_price {
    Ok(last_price) => println!("{:#?}", last_price.price),
    None => println!("No prices"),
};

simwilso · November 20, 2019, 10:29am

ah thanks Paul.
I managed to get it going using ‘Serde_json::Value’ type by doing the following:

let res = serde_json::from_str(&recieve.to_owned());
        let mut end_price = ();
            if res.is_ok() {
                let p: Value = res.unwrap();
                let q: Value = serde_json::from_str(p["result"].as_str().unwrap()).unwrap();
                //println!("{}", q["Ok"][0]["price"]);
                let end_price = &q["Ok"].as_array().unwrap().last().unwrap()["price"];
                println!("{:#?}", end_price);
            } else {
                println!("didnt work");
            }
        end_price

This gives me the figure I am after in the ‘end_price’:

String("Low",)
My next problem is though that I want my function here to return a String and in current form above it gives me an error I’m thinking because it is returning a Value::String type:

end_price
^^^^^^^^^ expected struct std::string::String , found ()

To finish my function I need to work out how to turn the above into a String here which I’m sure is probably pretty simple but is stretching the limits of my lamo Rust skills atm )

Anyway once I can work that out I might check if it with Tom and see if we can get a sample added to the rust examples in github.

I still think though the structured model is a much neater model than using the serde_json::Value structure so will go back to that and give your suggestions above a try for sure in the next few days.

Thanks Paul!

freesig · November 23, 2019, 3:14am

Just catching up on this stuff now.
One thing to note is that what is returned is actually a Result<Vec<Price, Error>> not a Vec<Ok>.
I’m away from the computer right now but I’ll try and give a code example soon.
You probably want something like:

struct Price {
  price: f64,
  author_id: Address,
}

freesig · November 23, 2019, 3:18am

I would probably use the Value method to get down to the result part of the object then parse that as a Vec<Result<Price, Error>>.

simwilso · November 24, 2019, 4:16am

Thanks @freesig. I managed to get it going using the looser serde_json Value approach. Will share my code snip with my approach in this thread tonight.
Still will be keen to get my code a bit cleaner by using with this structure though so will have a play a bit later and get this going and share as well.

freesig · November 24, 2019, 11:08pm

If you get a chance paste your code here and I’ll have a look at how we can make it cleaner more structured

simwilso · November 25, 2019, 12:15am

No worries. Here’s the snip:

   fn get_from_dht(_address: String) -> Value {
        let json = serde_json::json!(
            {"id": "device",
             "jsonrpc": "2.0",
             "method": "call",
             "params": {"instance_id": "test-instance",
             "zome": "signal_agent",
             "function": "get_price",
             "args": {"agent_address": _address}}
         });
        let (tx, rx) = channel();
        let tx1 = &tx;
        connect("ws://localhost:3401", |out| {
            // call an RPC method with parameters
            out.send(json.to_string()).unwrap();
            move |msg| {
                tx1.send(msg).ok();
                out.close(CloseCode::Normal)
            }
        }).unwrap();
        let recieve = rx.recv().unwrap().to_string();
        let res = serde_json::from_str(&recieve.to_owned());
        let end_price: Value;
            if res.is_ok() {
                let p: Value = res.unwrap();
                let q: Value = serde_json::from_str(p["result"].as_str().unwrap()).unwrap();
                end_price = q["Ok"].as_array().unwrap().last().unwrap()["price"].clone();//q["Ok"][0]["price"].clone();
                println!("{:#?}", end_price);
            } else {
                end_price = json!(["didnt work"]);
                println!("didnt work");
            }
        end_price
    }

pauldaoust · November 25, 2019, 6:45pm

(one quick note: I think the price should be a string or integer, because floating point math can cause wacky errors. Looks like Rust has no native decimal type, but there’s a neat Rust lib that emulates decimals though.)

freesig · November 25, 2019, 8:39pm

Hey I have been trying to get this to work for days but I just can’t get the output you posted to parse. Are you sure this is what the zome is returning?

"{\"jsonrpc\":\"2.0\",\"result\":\"{\\\"Ok\\\":[{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSciaDXrXkqxwv7gukMChvazIuscvcrk457t8n88cmo95en4sPCEdDKvj4ucja\\\"}]}\",\"id\":\"bob\"}"

The backslashes seem wrong.
I made a simple playground to test:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=35bf0bf43d3e396c056efc36ff30ab51

freesig · November 25, 2019, 11:08pm

Also could you post the zome function that returns this. I think maybe that could be simplified.

simwilso · November 27, 2019, 5:41am

Hi @freesig
Here’s the output I get:

"{\"jsonrpc\":\"2.0\",\"result\":\"{\\\"Ok\\\":[{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"},{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"},{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"},{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"},{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"},{\\\"price\\\":\\\"Norm\\\",\\\"author_id\\\":\\\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\\\"}]}\",\"id\":\"signal\"}",

pauldaoust · November 27, 2019, 4:06pm

I’m seeing double-escaping in some of my debug output too (if that’s what you’re noticing @freesig). Removing one layer of escaping, the following seems correct, because Holochain doesn’t know if the result should be JSON so it just defaults to a string that has to be deserialised by itself.

{
  "jsonrpc": "2.0",
  "result": "{\"Ok\":[{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"},{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"},{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"},{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"},{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"},{\"price\":\"Norm\",\"author_id\":\"HcSCJjCxUn8Jv4fv7x4BYcqY5TmWvvg3ytrb35FYKMtyc6qwBXsae4w84qhkyjz\"}]}",
  "id": "signal"
}

simwilso · November 28, 2019, 10:02am

no worries Tom. here’s the set and get functions from my zome:

#[zome_fn("hc_public")]
pub fn set_price(price: String) -> ZomeApiResult<Address> {
    let signal = PriceRange {
    price,
    author_id: hdk::AGENT_ADDRESS.clone(),
    };
    let agent_address = hdk::AGENT_ADDRESS.clone().into();
    let entry = Entry::App("price".into(), signal.into());
    let address = hdk::commit_entry(&entry)?;
    hdk::link_entries(&agent_address, &address, "author_price", "")?;
    Ok(address)
}

// this is the function that sets the spot price for each state every 5 minutes
#[zome_fn("hc_public")]
fn get_price(agent_address: Address) -> ZomeApiResult<Vec<PriceRange>> {
    hdk::utils::get_links_and_load_type(
        &agent_address,
        LinkMatch::Exactly("author_price"),
        LinkMatch::Any,
    )
}