Performance question

Hi,

we are thinking of migrating our transport layer to MQ Series.
one issue we need to understand is the following:

- sender is in Asia
- receiver is in Europe
- a ping between these 2 locations takes approximately 350 ms
- we intend to use distributed point to point queueing, and have the sender send messages to a server located in Asia, which will automatically forward the message to the server located in Europe, where the receiver will listen for messages
- according to beliefs in our neighbourhood, messages in a specific queue are forwarded sequentially, that is MQ Server waits for a message to be acknowledged before sending the next one
- we could not find in the docs any element to invalidate this belief
- with the above latency, this would result in a maximum throughput per queue of 3 messages per second which is far from our needs

Questions:
- are we missing something ?
- is there a way to tell MQ to parallelize message forwarding ?
- if there is, does MQ preserve message sequence ?
- can anyone share experience on this subject ?

Regards,

Eric

[email]ext-eric.vergnaud@hsbc.fr[/email] wrote:[color=blue]
> Hi,
>
> we are thinking of migrating our transport layer to MQ Series.
> one issue we need to understand is the following:
>
> - sender is in Asia
> - receiver is in Europe
> - a ping between these 2 locations takes approximately 350 ms
> - we intend to use distributed point to point queueing, and have the sender send messages to a server located in Asia, which will automatically forward the message to the server located in Europe, where the receiver will listen for messages
> - according to beliefs in our neighbourhood, messages in a specific queue are forwarded sequentially, that is MQ Server waits for a message to be acknowledged before sending the next one
> - we could not find in the docs any element to invalidate this belief
> - with the above latency, this would result in a maximum throughput per queue of 3 messages per second which is far from our needs
>
> Questions:
> - are we missing something ?[/color]

You can send messages in batches, the whole batch will be committed or
rolled-back as one unit, so you will only incur the turnaround cost at
the end of each batch. The whole batch must be resent if any of it is
missing or corrupted however, so don't make your batches too big.

For non-persistent messages you can use NPMSPEED(FAST) on the channel to
significantly reduce the latency of non-persistent messages at the cost
of losing them if something goes wrong while a non-persistent message is
in transit. Persistent messages will still be sent in syncpoint and
committed at the batchinterval (and fast non-persistent messages do
count against the batch interval).
[color=blue]
> - is there a way to tell MQ to parallelize message forwarding ?[/color]

You can define more than one channel from one queue manager to the
other, all channels can run in parallel and thus one will be able to use
the available network bandwidth while the other is waiting for a
line-turnaround.
[color=blue]
> - if there is, does MQ preserve message sequence ?[/color]

Message ordering is in general not guaranteed across channels; if you
need to get messages in the order in which they were put you should use
message groups.

Regards,

Phil Willoughby
--
Staff Software Engineer - IBM WebSphere MQ for z/OS
Senior Inventor

IBM Certified System Administrator - WebSphere MQ V6.0
IBM Certified Solution Designer - WebSphere MQ V6.0
IBM Certified SOA Solution Designer

MQ "batches" messages for transmission across channels. Read the WebSphere MQ Intercommunication manual.

Hi, thanks for paying attention.

Here is what I read in the manual:

The batch size is the maximum number of messages to be sent before a syncpoint is taken. The batch size does not affect the way the channel transfers messages; messages are always transferred individually, but are committed or backed out as a batch.

I understand that the batch improves the overall transmission time by reducing the number of commits, but it seem sto still require 1 transmission per message.

350ms turn-around latency does not occur on every single message.

If you need better response, get a better network.

350 ms is NOT a bad performance.

Due to the wave transmission speed limit, the minimum theorical time for a turn-around between Hong-Kong and Paris (via London in our case) is roughly 100 ms ( 2 * 15.000km / 300.000 km/s ).

In a perfect world (that is without any electronical equipment between the 2 points), an MQ transmission queue would have a maximum throughput of 10 messages per second.

I can hardly believe MQ does not provide any means to improve this throughput.

MQ can operate at full link speed to sequentially transmit messages, no matter how far away the destination is. It only experiences the turn-around latency after each "batch" of messages, to perform the commit handshake. Non-persisent messages don't do this, so they are faster, at the higher risk of being lost. See the NPMSPEED attribute of MQ channels.

Non-persistent messages are not acceptable in our solution.

So, correct me if I'm wrong:
- MQ forwards message sequentially
- the channel runs over TCP
- when MQ sends a packet it takes 350ms to receive the TCP ack
- meanwhile the sender is blocked by the TCP stack
- so MQ cannot send the next message until the TCP ack is received

In this context, can you explain:

"MQ can operate at full link speed to sequentially transmit messages, no matter how far away the destination is"

Eric

[email]ext-eric.vergnaud@hsbc.fr[/email] wrote:[color=blue]
> Non-persistent messages are not acceptable in our solution.
>
> So, correct me if I'm wrong:
> - MQ forwards message sequentially[/color]

Yes.
[color=blue]
> - the channel runs over TCP[/color]

Yes.
[color=blue]
> - when MQ sends a packet it takes 350ms to receive the TCP ack[/color]

At least; probably more as the TCP stack in the receiver will delay it
waiting for a data packet on which it can piggy-back.
[color=blue]
> - meanwhile the sender is blocked by the TCP stack[/color]

No. That's not how TCP works. The sender can send more bytes up to the
receiver's TCP receive window size, which is anything up to 1GiByte on
modern systems.

The required windows size to use the available bandwidth is the product
of the bandwidth in bytes/second and the latency in seconds; if your
link is 2MBit/sec then the window size required to achieve that with MQ
is 87.5KBytes. Some systems and routing hardware restrict the window
size to 64KiByte; if you find yourself in such a situation the MQ
solution I would recommend is to put one or more forwarding queue
managers between the two endpoints: this will improve throughput at the
cost of some small additional latency.

Regards,

Phil Willoughby
--
Staff Software Engineer, WebSphere MQ for z/OS