NGI Assure project: Layer-2-Overlay

This project was funded through the NGI Assure Fund, a fund established by NLnet.

The GNUnet transport subsystem

The current GNUnet TRANSPORT architecture with its pluggable transport mechanism (TCP, UDP, HTTP(S) and other protocols) together with the ATS subsystem for bandwidth allocation and choosing plugins has several issues with its design. With the Layer-2-Overlay project we like to implement the design goals of the future GNUnet TRANSPORT Next Generation (TNG) subsystem. One major change in the design is to separate the protocol plugins into processes (now called communicators) detached from the main transport service. Three communicators were already implemented (TCP, UDP and UNIX sockets). The old transport code is hard to maintain, because it is cluttered with "manipulation" support code for TESTBED (the actual testing framework). Testing TRANSPORT is a hard task, especially with TESTBED which has its own design flaws, and test code which is very hard to read to get an idea what the test code is doing. Therefore the first task (milestone 1) is to implement a new testing framework which uses network namespaces to make testing of TNG much easier. Have a look into what is planed for Layer-2-Overlay in the milestones.

The new style of writing tests in GNUnet is borrowed from the GNU Taler testing library. In milestone 1 we implemented commands to setup the netjails, the test environment for each peer, to start a single peers and sending a simple test message. Because some commands depend on other commands to be finished, and those commands are asynchronous, we needed additional functionality in the command interpreter library to block execution until some commands are finished (e.g. all peers needs to be running, before peers starting to send messages). For a detailed description have a look into the testing ng documentation.

To do extensive testing of the new transport implementation one needs to simulate various network topologies to enable faking network characteristics like lossy connections or firewalls. To achieve this we are working with network namespace. We have commands for starting and stopping network namespaces. Those commands are scripts, which are using several shell commands to setup the network namespace. A third script then is responsible for start a GNUnet helper. This helper can load plugins. Each plugin represents some test case. Per node in the network namespaces one helper is started, which means on each node is a local interpreter loop running. For a detailed description have a look into the testing ng documentation.

The first subtask consists of implementing a framework for setting up VLANs between network namespaces and a framework to test communication between peers which are running in those VLANs. Finally a minimal Test Case will be implemented. Despite the fact that the transport service is already able to use several communicators (transport protocol implementations), it will only use the tcp communicator, not the unix socket nor the udp communicator, which already are in place and working.

• SUID helpers to setup network namespace and starting peers with network namespace.
• Basic transport-level operations (get address, send, receive, connect).
• Peers connected through test and transfer data.

Deliverable

First MVP which uses the TCP communicator to send messages between peers. The deliverable can be verified through out the specific test cases running in the GNUnet CI. Details

To test more complex functionality we need to enhance the capabilities of the testing framework. Hooks for performance measurement will be implemented.

• Enhancing transport-level operations.
• Block execution of commands at a peer. (Barriers).

Deliverable

Outcome of this deliverable are advanced test cases (again verifiable in the GNUnet continuous integration (CI)). Details

With this subtask I will implement enhanced L2O features like using unidirectional transport protocols with backchannels. Addresses by which a peer can be reached can be delivered on handshake or by UDP broadcast. With this milestone the transport service will be able to use more than one communicator (pluggable transport).

• unidirectional communication and backchannels.
• UDP broadcast.

Deliverable

The CI contains test cases which uses the UDP protocol to message between peers and to learn about “foreign” peers. Details

In this subtask I will enhance connectivity to peers not directly connected. Therefore peers have to act as relay. To achieve this there is the distance vector protocol. The DV algorithm sends out so called learn messages to other peers. If those learn messages are coming back to the initiating peer via some other peer and the path does not return to any other peer we have a circle path. If there are bidirectional connections between peers somewhere in the DV path and the learn message comes back to a peer we call it inverse path.

Deliverable

The CI contains test cases with a setup of peers not connected directly. The test cases proof that each peer can reach any other peer. We have test cases for the circle path and for the inverse path. Details

This subtask will make peers behind NAT reachable. Two simple traversal methods will be implemented.

• NAT traversal via UpnPC.
• Autonomous NAT Traversal using fake ICMP messages.

Deliverable

The CI contains test case with a peer setup containing peer behind a NAT. The test cases proof that each peer can be reached, even if that peer is behind a NAT. The test case are measuring the performance. This measurement is used to compare with the outcome of the next milestone. This result of this milestone will be a first stable release. Details

In this subtask I will implement algorithms (flow and congestion control, quality of service optimizations) which will select the optimal transport protocol for a given situation. I will analyze potential performance gains by integrating libraries of the interpeer project. If the effort of integrating interpeer presumably would lead to better performance than other optimizations of the same amount of work, this integration is done. I will finish the project with a performance analysis to optimize the selection logic.

• Queue management.
• Interpeer project synergy
  1. Analysis of the interpeer project in regard to integrate it into GNUnet.
  2. Optional integration into GNUnet, if it can be done at all and in a reasonable amount of work.
• Commands for performance measurement(s).
• Performance analysis.

Deliverable

The test cases in the CI are measuring the performance gains. This measurement can be compared with the measurement of milestone 5. Again one outcome of this milestone will be a stable release.
Documentation of the interpeer project analysis. More Details.

The code of the old transport API and of the ATS API will be removed from the code base. This is mainly the use of the API in the Core layer, and suggesting addresses to connect to other peers in all kinds of other GNUnet layers. Additionally all tests which rely on the removed code will be set inactive. All usages of the old APIs will be removed. If it is not much work to replace the old with the new API this will be done directly, otherwise there will be a TODO comment what needs to be doneto make that part of the code working again with the new code. Documentation of code that will be deleted without replacement, because the functionality is missing in the new API.

Deliverable

Removed APIs from code base, deactivated dependent tests and either documentation of what is necessary for the integration of the new API, or if the effort for this is as high as the documentation directly the implementation. More Details.

All missing implementations of the new API will be finished, necessary to make GNUnet start and being usable for its basic functionality (DHT, GNS, CADET, FS, Messenger …).

Deliverable

GNUnet running with new API. More Details.

Because all integration tests are deactivated, and there are no resources to rewrite them at the time being, extensive manual testing and certainly bug fixing is necessary.

Deliverable

GNUnet release running with new API. More Details.