RVN Improvement for Slow Networks

This update to RVN improves RVN behavior, reducing lag between graphics displayed on the rvn_sender and endstations, and improving the transition between SSC and non-SSC mode in a RVN session. To achieve optimal behavior, some tuning may be required. This page describes some system and environment settings that can be used to tune RVN performance.

Rationale

RVN uses TCPIP sockets to communicate with endstations. In order to obtain maximum network throughput, the output buffer on the rvn_sender node needs to be large enough that TCPIP can be placing data on the network continuously. If the buffer is too small, then TCPIP stalls when the buffer is full, waiting for acknowledgement of the data that was already sent, where the minimum time to receive the acknowledgement is the network round trip time.

To minimize latency, the TCPIP output buffer size should not be too large, since an application that generates images rapidly can fill the buffer with so much data that it appears that the latency of the RVN connection to the endstation exceeds the network latency as additional data gets enqueued for transmission.

Tuning TCPIP Buffer Size

The System TCPIP buffer size should be set to handle its worst case upper bound as determined by the maximum bandwidth-delay product. To determine buffer size, you should do the following

  1. Compute the bandwidth-delay product, in bytes, for representative end-to-end connections. This is the product of the network bandwidth and round trip latency. You may use whatever units are convenient as long as the units, with conversion factors, cancel to yield bytes. The measurements and calculations can be approximate. Choose the largest result.
    1. Determine network bandwidth: Frequently, the "last mile" bandwidth to the endstation is known and can be used directly (for instance, a 1.5 megabit/sec DSL or 5 megabit/sec cable modem). Alternatively, a flood ping (ping -f) may be useful in determining the network capacity over a given path, especially if the bottleneck is not the final hop to the endstation, but it should be used with care and for limited durations because of its potential for disrupting other users whose traffic goes through the same bottleneck. Another caveat is that in cases where the network capacity far exceeds the application's ability to generate images, you may choose to use the bandwidth that you require rather than the available bandwidth. In this case you would substitute the product of maximum image size, frame rate and compression ratio for the network capacity. For example, on a 1 gigabit/sec network you might use a fifth of that , 200 megabits/sec, based upon a maximum update capacity of 2 million pixels/image * 100 images/sec * 1 bit/pixel .
    2. Determine the network round trip time. Issue "ping remotenode" and note the round trip time and the units.
    3. Compute the maximum bandwidth-delay product. For example, using "scientific notation" where E is ten-to-the-power-of , and where you had the following network connections available:
      • For a 100 Megabit network and 1 millisecond round-trip-time: 1E8 bits/sec * 1E-3 sec * 1 byte/8 bits = 0.125E5 bytes = 12500 bytes .
      • For a 5 Megabit network and 50 millisecond round-trip-time: 5E6 bits/sec * 5E-2 sec * 1 byte/8 bits = 0.3125E5 bytes = 31250 bytes
      • For a 100 megabit network and 80 millisecond round-trip time, 1E8 bits/sec *8E-2 sec * 1 byte/8 bits = 1000000 (i.e. one million) bytes.
    4. For this set of connections, you would choose the highest product, i.e. 1000000 -- one million.
  2. Verify that the system setting is at least as large as the largest buffer you will need. The system setting can be viewed by "cat /proc/sys/net/core/wmem_max" . If your computed value is higher than the system's setting, increase it in the following way. As root, set the value of /proc/sys/net/core/wmem_max to the value calculated, for instance echo 1000000 > /proc/sys/net/core/wmem_max

Note that this setting will not persist across system reboots. You will probably want to make this setting permanent, in which case you should issue the command to set this variable in a system startup script such as /etc/rc.d/rc.local

Tuning RVN Sessions

In addition to setting TCPIP buffer sizes, you may need to tune RVN for best behavior in a specific RVN session. To do this, you can use several environment variables as described below. These environment variables should be set before invoking RVN sender. While a calculation might give a particular value, please regard this as a starting point for tuning and evaluate higher or lower settings for their effect on latency, fluidity, and image quality.

RVN_SO_SNDBUF

The system TCPIP buffer size should be set for worst case requirements as described above. For a specific RVN session, you can specify a smaller buffer size to reduce latency. You cannot obtain more than the system's buffer size as specified by /proc/sys/net/core/wmem_max. To use a smaller buffer size, set the RVN_SO_SNDBUF environment variable to the desired value, computed using the same method as for computing the system TCPIP buffer size.

For instance, for a 5 megabit, 50 millisecond connection, 5E6 bits/sec * 0.050 sec *1/8 byte/bit = 31250 bytes, and you might use "export RVN_SO_SNDBUF=40000". Very low values may give unsatisfactory results. A lower bound might be considered. For instance, you might choose a buffer adequate to hold an entire outgoing compressed image (at approximately 1 bit per pixel). For one megapixel images, this would be about 1 megabit, or 125000 bytes.

If this environment variable is not set, RVN attempts to use the default value of 1000000, subject to the limit set for the system TCPIP buffer. The value of 1000000 bytes should be reasonable for a high-bandwidth, high-latency corporate network.

RVN_DISABLE_SKIP

If an application is generating images faster than they can be sent to an endstation, then the TCPIP buffers holding data to be sent to that endstation will fill up. The default behavior of RVN with this update is to drop image buffers (frames) intended for that endstation rather than enqueing them for transmission and increasing latency. This approach reduces latency and allows all endstations to be nominally in synchronization, even with differences in network performance for each endstation connection.

You can override the default behavior by setting the RVN_DISABLE_SKIP environment variable to 1. If you do this, then as TCPIP buffers for an endstation connection fill, RVN will enqueue additional images for transmission to that endstation, and increasing latency. You can still affect latency with this environment variable set by changing the quality setting in the RVN dashboard, or by changing the image size. These changes would apply to all endstations connected to the application.

RVN_THROTTLE_MS

Dropping frames reduces latency, but this may be wasteful of system resources because the application is generating images which are subsequently discarded. You can improve system resource utilization by limiting the rate at which an application generates images. This is done by setting the RVN_THROTTLE_MS environment variable to the minimum interval (maximum frame rate) that images should be generated. After each image is generated, RVN will suspend the image generation thread for the number of milliseconds specified by this environment variable. Setting RVN_THROTTLE_MS to 10, for instance, would limit the image generation to a maximum of 100 images per second, which should be reasonable, since most monitors cannot display images at that frame rate.

RVN_SKIP_VERBOSE

If the RVN_SKIP_VERBOSE environment variable is set to 1, then rvn_sender will issue messages as it discards image buffers. These messages may be useful in tuning RVN to achieve best behavior and best use of system resources. The default RVN behavior is to silently discard images.

Ideally, you will observe these messages without affecting the frame-dropping behavior you are attempting to monitor. Be aware that there may be significant added cost of transmitting these console messages, particularly if the desktop on which the messages appear is being transported by VNC over the same network link that is carrying the image data. In that case the network congestion may increase and more frames may get dropped. If the sender must be observed remotely, ssh may provide a more efficient channel for receiving this textual information. You can also redirect stdout and stderr to a file and monitor the file periodically for messages, for example, using the tail or tail -f commands.

RVN_SUBSAMPLING

With this update, RVN allows the subsampling to be user-selected via the environment variable RVN_SUBSAMPLING. The default setting is 2, which yields 422 subsampling, i.e. every other pixel in x is used. Setting to 4 gives 444 sampling, which uses all pixels. Setting to 1 gives 411 sampling, i.e. every other x and every other y pixel is used. 422 is usually justified based upon human perception. "1" will (approximately) halve the bits for a compressed image, likely with a perceptible impact on image quality. "4" will (approximately) double the data requirements

RVN_IMAGE_QUALITY

As a reminder, in RVN, compressed-image quality is selectable by the dashboard slider. Its default of 60 can be changed to between 4 and 100 by setting the environment variable RVN_IMAGE_QUALITY.

Older RVN versions

Endstations running an earlier version of DCV 1.2 than this update can still be used with RVN at this update level. RVN will be unable to drop frames for an endstation running an older DCV level. Older endstations will still encounter the same latency and performance problems, and can affect performance of all endstations in the RVN session. When rvn_sender detects an older endstation connecting to the session, it will issue a warning message, then allow that endstation to connect.