Overview
Introduction
- Multiplexing and pipelining
- Shared streams
Using the default global HTTP streams hub
Defining a local HTTP streams hub

Overview

The HTTP streams plugin manages concurrent shared persistent http(s) connections ("streams") to accelerate client threads.

supports http and also https when all source code is compiled with the WITH_OPENSSL or WITH_GNUTLS or WITH_WOLFSSL or WITH_SYSTEMSSL flag
supports http/s gzip compression when all source code is compiled with the WITH_GZIP flag
supports all transports over http/s such as SOAP/XML, SOAP/XML with attachments (MTOM, MIME, DIME), JSON, XML-RPC
client-side contexts should register the plugin as explained in this document, but no other client-side source code changes are necessary
no limit on the number of client threads
configurable parameters for the maximum number of host:port servers to share among client threads and the maximum number of open connections permitted to each server (maximum level of concurrent connections permitted per server)
uses HTTP/1.1 without upgrading to HTTP/2 or HTTP/3, supporting legacy systems without introducing additional overhead and complexity while effectively mitigating the dreaded TIME_WAIT state and head-of-line-blocking among concurrent client threads

Introduction

When client applications repeatedly and frequently connect to the same server, problems with TCP/IP sockets become apparent when connections are starting to fail due to server connection overload. Even when the server can handle hundreds of simultaneous connections, connections that are closed afterwards must be recycled for reuse. When this happens, the TCP socket lingers in TIME_WAIT state before it accepts a new connection. This effectively reduces the number of connections that can be established to a server in a given time. This can be mitigated by tinkering with the TCP socket linger time (time in TIME_WAIT state). In gSOAP this can be done using the soap.linger_time variable and by setting soap.connect_flags = SO_LINGER. Setting the linger time to zero can bypass the TIME_WAIT state. However, this approach is not always possible or even disarable because of other effects this has on the server.

Multiplexing and pipelining

One solution is to use multiplexing or pipelining. In this scenario with multiplexing/pipelining, multiple client threads talk to the server over the same TCP/IP connection that persists during a session. With multiplexing, clients send requests independently and recieve responses independently over the same connection. Multiple requests may be sent first before the responses are received. Pipelining (e.g. with the HTTP pipelining plugin) means that all responses are received in the same order that requests were send. Multiplexing means that responses may be received out-of-order, but are reordered by the control layer. These mechanisms require inclusion of a request message ID to match the ID of the response message (i.e. IDs are carried in HTTP headers).

Multiplexing eliminates the "head-of-line blocking" problem, where a single transaction can prevent other transactions from taking place by allowing for multiple requests and responses to be in transit "simultaneously" (to be more clear: message transport is not concurrent or simultaneous, but message ordering is independent so that a slow response on a request message does not block another request message on the same connection). While HTTP/2 multiplexing, has merits, especially for browsers that typically fetch many images and other static data from a server with a burst of HTTP GET requests, it has some drawbacks that researchers and developers have reported on, including application coding requirements and network knowledge to exploit the mechanism fully.

The HTTP/2 and HTTP/3 protocols support multiplexing. This is done by negotiating the HTTP/1.1 request by a client to upgrade the protocol to HTTP/2 when agreed to by the server in a roundtrip handshake. Besides the additional roundtrip, additional complexity is involved to establish secure HTTPS transfers for clients over a multiplexed connection, using ALPN to extend TLS avoiding additional roundtrips.

Multiplexing and pipelining require a managing layer to keep track of IDs and to store the messages received, buffering the messages to deliver them to clients in the right order. When serving time is low (short response times) or when client requests are more spaced apart in time (i.e. no bulk sent requests), then multiplexing/pipelining becomes less effective. Message exchanges over the same connection no longer overlap (do not compete for network access).

Related subjects:

Server-side HTTP pipelining on the server side using the http_pipe plugin in the gSOAP documentation.

Message queueing using the inbound message queue mq plugin in the gSOAP documentation.

See also the gSOAP documentation of plugins and plugin registry functions

Shared streams

Is multiplexing worth it? Yes, but insofar "burst-mode" web request are made to the same server potentially causing "head-of-line" requests with slow responses that may block subsequent traffic on the same connection. When implementing SOAP/XML and JSON web services, these concerns are still present, but the message exchange behavior differs from the typical bursty web browser traffic. Furthermore, web services may not be stateless, meaning that a sequence of message exchanges must be performed in a pre-determined order and each must be completed before the next can commence. Such web service client threads won't benefit from multiplexing.

To optimize web services connectivity and message exchanges without overloaidng the server and risking TIME_WAIT states, the HTTP streams plugin establishes one or more persistent connections (bidirectional "streams") to a server for up to a given maximum number of active open connections. The number of active persistent connections depends on the concurrent demands by the client threads sharing these connections. Message exchanges are guaranteed to be sequenced as before without the plugin. Rather than multiplexing, another available connection is automatically selected to prevent "head-of-line-blocking". In this way, a blocking client-server exchange is bypassed effectively by another connection to the server. A limited number of shared connections is established on demand. Each connection is closed when inactive (by the server when timing out), preventing a server overload while maximizing throughput with concurrency. This allows for hundreds of client threads to share a limited number of connections to a server without overloading the server and avoid the dreaded TIME_WAIT state.

Another benefit of the HTTP streams plugin is that the overhead to set up and tear down a secure SSL/TLS connection (for HTTPS) by each client thread that wants to connect securely to a server can be amortized, since this is done only once to establish the secure persistent connection that is shared among multiple client threads. This keeps the complexity and overhead of shared HTTP and HTTPS persistent keep-alive connections low.

The lifetime of a persistent connection to a server is determined by the maximum number of HTTP keep-alive roundtrips permitted by the server and by its connection timeout policies. This may vary across server implementations. The default maximum number of HTTP keep-alive roundtrips are 100 (SOAP_MAXKEEPALIVE) for gSOAP multi-threaded services enabled with the SOAP_IO_KEEPALIVE flag. The server-side roundtrip max count can be modified by setting soap.max_keep_alive to the desired value. Timeouts on the client and server sides can be set to soap.send_timeout and soap.recv_timeout measured in seconds, or in microseconds (10^-6 sec) that are expressed as a negative value. A zero value disables timeouts. A connection is closed by the server when the max keep-alive rountrips are reached or sooner when a connection timeout occurs (which means that nothing was sent to the server within a timeout time span.)

Connections shared by client threads are managed by a HTTP streams plugin "shared streams hub" that optimally activates and manages connectivity to one or more servers used by the clients. Each distinct host:port is considered a server (i.e. the path part in a URL does not matter). To use the HTTP streams plugin, the maximum number of distinct host:port servers must be specified in advance (the specific host and port info comes from the client threads). The maximum number of open connections to each server must also be specified in advance, which is used by the hub to manage the shared connections.

Both HTTP and HTTPS traffic is supported, assuming that distinct server ports support HTTP and HTTPS (such as the default HTTP and HTTPS ports 80 and 443, respectively). The HTTP streams plugin blocks HTTPS requests attempted over shared HTTP connections, when an application's client thread switches from HTTPS to HTTP by accident (but only when HTTPS was used before to connect to a server, since the hub has no knowledge of the server HTTP versus HTTPS capabilities.)

Using the default global HTTP streams hub

By default when registering client threads with the HTTP streams plugin, a single global static shared http streams hub is created that manages persistent connections. This supports most scenarios in which all client application threads share connections to one or more servers.

When a shared streams hub is initialized, the maximum number of host:port servers must be specified in advance by the application. When not specified, this defaults to eight (8) distinct host:port servers that can be reached by an application's client threads. The server host:port addresses do not need to be specified in advance as these are picked up from the client thread requests automatically.

In addition, when a shared streams hub is initialized, the maximum number of persisten connections opened to each server must be specified in advance. When not specified, this default to up to four (4) open connections per host:port server. The maximum number of open connections should be less than or equal to the number of active client threads. A low number (1 to 3) increases connection contention, but may not necessarily decrease overall performance when server responses are quick enough on average and when client threads connect less frequently (not in bursts). In the best case, only one of the client threads connects and exchanges a message at a time, which means that only one persistent connection is activated and maintained by the hub (no connection contention means no connection concurrency is required). However, if many client threads are created (tens to hundreds or more) that all make frequent requests where server responses to client requests are slow, then the maximum number of open connections may have to be increased to increase connection concurrency. This increases the overall performance, but up to the point when network traffic bandwidth is exhausted.

To specify the maximum number of host:port servers and maximum number of open connections per server, invoke soap_http_streams_max_host() and soap_http_streams_max_open() functions, respectively:

#include "plugin/httpstreams.h"
...
soap_http_streams_max_host(2); // max 2 distinct http/s host:port servers
soap_http_streams_max_open(4); // max 4 concurrent open socket streams per server

As a simple example we define a function to start a new client thread that makes one or more web service requests:

void *start_client(void *arg)
{
  struct soap *soap = soap_new1(SOAP_IO_CHUNK | SOAP_IO_KEEPALIVE);
  soap_register_plugin(soap, http_streams);
  // THREAD_DETACH(THREAD_ID); // use if we don't THREAD_JOIN()
  ... // optional: soap_ssl_client_context(soap, ...) for HTTPS ...
  ... // ... or use soap_ssl_client_setup() from ssl_setup.c
  ... // do client stuff, i.e. requesting services...
  soap_destroy(soap);
  soap_end(soap);
  soap_free(soap);
  return NULL;
}

The run-time flags SOAP_IO_CHUNK and SOAP_IO_KEEPALIVE are specified for optimal HTTP communications and other non-SOAP_IO_ flags are supported, such as SOAP_ENC_ZLIB compression, but the HTTP streams plugin always sets SOAP_IO_CHUNK and SOAP_IO_KEEPALIVE anyway when registering.

The usual client-side settings for timeouts are also supported and can be used, for example after creating the context we may set some or all of these:

soap->connect_retry = 3;        // retry 3 times to connect with exponential backoff
soap->connect_timeout = 5;      // connecting times out after 5s (not on Linux)
soap->send_timeout = 5;         // each send should not block longer than 5s
soap->recv_timeout = 5;         // each recv should not block longer than 5s
soap->recv_maxlength = 1048576; // allow up to 1MB received, fail if too large
soap->transfer_timeout = 10;    // allow up to 10s message transfer time

Note that a connection retry uses exponential backoff, so that the first retry takes 1 second delay after 5 second connect timed out, the second retry takes 2 seconds delay after connect timed out again, then 4, 8, 16, 32, 32, 32 seconds delay with 32 seconds delay max. For example, with the settings above we get 5+(5+1)+(5+2)+(5+4)=27 seconds delay (at least) for each failed client thread request to the server when the server is non-responsive or does not run at all, causing failed connection attempts.

Client threads are created with:

THREAD_TYPE tid[N];
for (i = 0; i < N; i++)
  THREAD_CREATE(&tid[i], &start_client, NULL);

and joined with:

for (i = 0; i < N; i++)

THREAD_JOIN(tid[i]);

The plugin/httpstreams.c and plugin/threads.c code must be compiled and linked with the application. An example application plugin/httpstreamstest.c is included.

To support HTTPS, all source code (application, gSOAP source code and the HTTP streams plugin sources) must be compiled with one of the following four compile-time flags:

WITH_OPENSSL to compile and link with OpenSSL, or
WITH_GNUTLS to compile and link with GNU TLS, or
WITH_WOLFSSL to compile and link with WolfSSL, or
WITH_SYSTEMSSL to compile and link with System SSL.

Note: A SOAP_TCP_ERROR "http_streams plugin cerror" with detail "Out of host:post server slots" is generated for a client call when the number of distinct host:port server combinations exceed soap_streams_max_host(); please make sure to specify number of distinct host:port that clients use to connect.; A SOAP_SSL_ERROR is generated when attempting to connect with HTTPS over a prior plain HTTP connection or when HTTPS is not supported.; A shared streams array is dynamically allocated once for the static shared streams hub, which is never deallocated (may be reported as a memory leak). By contrast, a user-defined managing hub (see below) does not use the static hub and can clean everything up afterwards.

Defining a local HTTP streams hub

Instead of using the default single global static shared HTTP streams hub, you can define one or more local hubs. Each local hub is responsible to manage its streams independently of the other hubs.

For example, to create a new hub to support two host:port servers used by client threads with up to four persisten connections each:

#include "plugin/httpstreams.h"
...
// a new hub with max 2 distinct http/s host:port servers with
// up to 4 concurrent open socket streams per server
struct soap_http_shared_streams *hub = soap_new_http_streams(2, 4);
...
THREAD_TYPE tid[N];
for (i = 0; i < N; i++)
  THREAD_CREATE(&tid[i], &start_client, hub);
...
for (i = 0; i < N; i++)
  THREAD_JOIN(tid[i]);
...
// release hub and free all streams (use only after client threads terminated!)
soap_free_http_streams(hub);

Note that the local hub is passed on to start_client to register the hub with the client thread. Otherwise, the client thread would use the global hub. A local hub is registered with the client thread's context using soap_register_plugin_arg(soap, http_streams, hub) as follows:

void *start_client(void *arg)
{
  struct soap_http_shared_streams *hub = arg; // use the local hub
  struct soap *soap = soap_new1(SOAP_IO_CHUNK | SOAP_IO_KEEPALIVE);
  soap_register_plugin_arg(soap, http_streams, hub);
  // THREAD_DETACH(THREAD_ID); // use if we don't THREAD_JOIN()
  ... // optional: soap_ssl_client_context(soap, ...) for HTTPS ...
  ... // ... or use soap_ssl_client_setup() from ssl_setup.c
  ... // do client stuff, i.e. requesting services...
  soap_destroy(soap);
  soap_end(soap);
  soap_free(soap);
  return NULL;
}