You are here

Rust implementation of GNUnet with GSoC - Mid-term progress

The GSoC mid-term evaluation is happening now, so this is a good time to write up the work I've done so far. My contributions are threefold. The files mentioned below can be found in the async branch (https://github.com/kc1212/gnunet-rs/tree/async).

1. Packed message structs
GNUnet messages are represented by packed C structures. Serialisation and deserialisation are thus straightforward, messages are simply sent byte for byte over the network. gnunet-rs did not do this initially. So I ported a couple of message structures from the C code into Rust and implemented ways to easily construct and send those messages. In the future we hope to automatically generate message structures from the C code using bindgen when it supports packed structures (https://github.com/crabtw/rust-bindgen/issues/351).

2. Removed dynamic links with the GNUnet C library
Previously gnunet-rs links with the C library to use the functions for converting between GNUnet's ASCII encoding and binary value. Linking with the C library only for a couple of functions creates unnecessary overhead. The Java library also doesn't link against the C library. Thus I ported those functions to Rust and wrote some simple test cases for them, those can be found in src/util/string.rs. This removes the need for linking with the C library.

3. Asynchronous peerinfo library with gjio
Finally, the most important contribution is asynchronous support using gjio (https://github.com/dwrensha/gjio). gjio is a high-level asynchronous IO library based on event loop and promises. It supports most of the features in the GNUnet Scheduler such as timeout, cancellation and so on.

Currently three public functions for peerinfo are provided.

pub fn iterate_peers(cfg: &Cfg, network: &Network)
-> Promise<Peers, IteratePeersError>
pub fn get_peer(cfg: &Cfg, network: &Network, pk_string: String)
-> Promise<(Option<PeerIdentity>, Option<Hello>), NextPeerError>
pub fn self_id(cfg: &Cfg, network: &Network)
-> Promise<PeerIdentity, TransportServiceInitError>

Since they all return Promises<T, E>, we can perform asynchronous operations by creating multiple promises, chain them together if necessary, finally wait for all of them to evaluate. For instance, to check whether two specific peers are online, we can write the following.

EventLoop::top_level(|wait_scope| -> Result<(), ::std::io::Error> {
// setup config, network, etc.
let promise1 = get_peer(&config, &network, pk_string1)
let promise2 = get_peer(&config, &network, pk_string2)
let result = Promise::all(vec![promise1, promise2].into_iter()).wait(wait_scope, &mut event_port)
// process result
}

More usage examples can be found under examples/example-list-peers.rs.

The work so far presents a proof of concept of accessing the peerinfo service asynchronously. That is exactly what I aimed to achieve by mid-term evaluation in my original proposal.

However, the work is far from over. iterate_peers doesn't return an Iterator, the user must manually evaluate the promise on every iteration and check whether it has reached the end, I'm hoping to make it into Iterator trait. The current API was designed without paying much attention to the existing C or Java API, I'd like to introduce some consistency with the existing API both in terms of naming and functionality. The tests and documentation I have at the moment are minimal, so some work needs to be done to improve on those fronts. Finally, I'd like to implement a couple of other subsystems such as GNS and/or statistics.

Fortunately, summer holidays are finally starting in a couple of weeks, which substantially increases the amount of time that I can spend on GNUnet. I'm looking forward to the next 8 weeks!

Kelong

Comments

Hi Kelong,
One issue that has haunted previous attempts to write parts of GNUnet in other languages is API divergence. We do still sometimes make (often very minor) changes to the IPC protocols, and devs hacking on the C code are unlikely to be aware that they might be breaking stuff for the Java/Rust/etc. bindings.

So assuming you have peerinfo working, one thing we should try to make sure of is that we setup some automation to monitor that compatibility is preserved, and if not actively alert devs. The buildbot would seem like the right place in general, however, we also need to make that system more visible and alert us more strongly about failures as they are right now too often ignored for too long.

Anyway, please do worry about C-integration testcases, and maybe discuss with Jeff on how to run them automatically when either implementation changes.

Happy hacking for the 2nd half & keep it up!