SCTP performance test - MP3 audio over RTP

Scenario: COP as server, SOLDIER as client. Network bandwidth varied between 200kbps and 300kbps, 30ms network latency, packet loss varied between 0% and 10%.

Application adapted to SCTP: poc version 0.3.7.

The test determined the ability of some network combinations to deliver a MP3 audio stream with reasonable quality, through a limited bandwidth network with packet losses. This test is completely different from all others because definition of "reasonable" audio is highly subjective. Still, the following guidelines were followed:

• An isolated fail in audio reproduction does not invalidate a scenario;
• Fail frequency should be low enough, so it will not angry an average listener;
• But if audio gets "lost", or stops, and can not resyncronize, the scenario is invalidated;
• Periodic failures also invalidates a scenario since are caused by resyncronization problems and would angry an average listener;
• An audio stop will not invalidate a scenario, but it must recover fast and it must not stop again for a reasonably long time.

Another difference from the other tests, is that we compare SCTP with UDP here. SCTP was allowed to deliver messages unordered, as well as discard messages past lifetime. Also, PR-SCTP could finally be used and compared with standard SCTP.

Prebuffering at client was 200ms, except for FEC protocol since it does not allow buffer dimensioning in terms of time.

Even though MP3 has fixed bitrate and is somewhat capable of recovering from losses, it is not really appropriate for transmission over lossy channels. (RTP is not lossy by itself but is normally transmitted over UDP, so the RTP+UDP combination makes a lossy channel). Some protocols were designed to adapt MP3 to lossy RTP: RFC 2250, RFC 3119 ana FEC (Forward Error Correction), the last one proposed by poc's author.

In the graphics below, X axis is the packet loss, and Y axis is the bandwitdh. The higher and more at left points represent better network scenarios. There is one line for every protocol bundle (RTP+UDP, RTP+SCTP, RTP+PR-SCTP) that divides the cartesian plane in two portions:

• Left/upper portion represents the set of scenarios where audio content was delivered acceptably.
• Right/lower portion is the set of scenarios where audio was not satisfactory.

So we can say that, the bigger the area at left/over the protocol line, the stronger is the protocol bundle when handling network problems.

MP3 via RFC 2250 over RTP

RFC 2250 is a poor adaptation of MP3 over RTP since it is very sensible to packet losses, as shows the bad performance of UDP. Any loss introduces periodic glitches in audio, and it tends not to recover.

Of course, using a confirmed protocol improves a lot the loss resistance. Standard SCTP renders RFC 2250 perfectly usable in several adverse scenarios. Another interesting side effect of SCTP was recovery of failures: after the fail, the periodic glitch disappears after some seconds.

PR-SCTP performance was less tolerant than standard SCTP, and it did not work with losses above 2%. Possibly the message discard pattern of PR-SCTP was less tolerated by RFC 2250.

MP3 via RFC 3119 over RTP

RFC 3119 is more efficient than RFC 2250 if used over lossy networks, and SCTP further improved packet loss resistance, as shown by the above graphic.

PR-SCTP did somewhat better than standard SCTP when losses were below 2%. Message lifetime was configured to 200ms. We believe that PR-SCTP could perform even better if application changed the lifetime accordingly to network conditions.

MP3 via FEC (Forward Error Correction) over RTP

The FEC (Forward Error Correction) algorithm creates redundant data for error correction. The 192kbps MP3 content fattens up to 260kbps, but this coded content is resistant to data loss, because receiver can recreate lost data.

FEC is roughly the same mechanism used in audio CDs, where 25% of raw data is redundancy, which allows correction of errors caused by scratches and dust. Data CDs have a second level of redundancy of 25%, which make them even more scratch-resistant.

FEC is tailored for lossy transports as UDP, and graphic shows that UDP performance was excellent. Note that audio content could be delivered even when network bandwith was narrower than 260kbps (with few packet losses, that said). This shows that FEC is really rebuilding lost data.

Standard SCTP performed badly, worse than UDP in every scenario. The main reason is that it cannot cope with a network slower than 260kbps data flow. This is acceptable since SCTP is a confirmed protocol, that strives to deliver all data to destination...

PR-SCTP was a lot more adequate in this teste; it could handle with a network slower than data flow (althrough not as well as UDP since SCTP overhead is bigger). And PR-SCTP was better than UDP in severe packet loss sceneries.

The better performance of PR-SCTP was not achieved for free. It was necessary to lower the message lifetime from 200ms to 80-100ms, when packet loss was 5% or higher. Without this, the flow stalls completely.

PR-SCTP could do even better if transmitter calibrated message lifetime accordingly to actual network conditions, and minimum RTO were lowered to an value adequate to network latency (30ms in this test). The manual adjust to 80ms had in mind the network RTT of 60ms (2x30ms), with a small margin for variancy, which leaves room for only one retransmission. In a 10% loss network, the possibility of one retransmission will reduce the actual losses in reception to 1% (10% of 10%), which FEC can handle easily.