CMCC PERFORMANCE MEASUREMENT MESSAGE FORMATS IEN 157 David Flood Page Bolt, Beranek and Newman Inc. 50 Moulton Street Cambridge, Massachusetts 02238 (617)491-1850 26 September 1980 IEN 157 Bolt, Beranek and Newman Inc. TABLE OF CONTENTS Page 1. PREFACE 1 2. INTRODUCTION 2 3. MESSAGE FORMATS 3 3.1 CPU Idle Time - report type 8 4 3.2 Packet delay - report type 9 4 3.3 Gateway to gateway delay - report type 10 5 3.4 Bit Throughput - report type 11 7 3.5 Queue occupancy - trap type 3 8 i IEN 157 Bolt, Beranek and Newman Inc. 1. PREFACE This document describes the message formats for the performance measurement reports and traps which are to be added to the ARPA CMCC gateway monitoring messages. It is an addendum to the Gateway Monitoring Protocol described in IEN 131, which should be consulted first. Processing for the new reports and traps will be added to the CMCC, and a document describing their use will be issued later. 1 Bolt, Beranek and Newman Inc. IEN 157 2. INTRODUCTION The message types described here will be used in two ways: either experimentally, in conjunction with message generators used to load the Catenet until something breaks, or regularly as are the other report types, to show how the Catenet is behaving at any time. In evaluating Catenet performance, message generators will be required to load the gateways with traffic until packets are dropped, the delays start to rise steeply, or the throughput saturates. These conditions indicate that the limit of some resource has been reached. These message generators, whose implementation is yet to be defined, can be located in either the gateways, the CMCC program, or in other internet hosts. When both the CMCC program and the gateways implement the new message types, and message generators are defined and implemented, the CMCC will be able to find out how much traffic the gateways were processing, where the bottlenecks lie in the Catenet, and what the accompanying delays were. All the measures described here are cumulative from the time the gateway started. The CMCC will, by keeping suitable histories of the measures, be able to show shorter term values as required. 2 IEN 157 Bolt, Beranek and Newman Inc. 3. MESSAGE FORMATS All gateway monitoring messages consist of an Internet header followed by a monitoring header, and then the message data. A monitoring header, as described in IEN 131, has the following format: Bits Contents 0 0 - Report or trap. 1 - Negotiation message. 1 0 - Report. 1 - Trap. 2-3 For a negotiation message: 0 - DO 1 - DONT 2 - WILL 3 - WONT For a report or trap: zero. 4-7 Reserved. 8-15 Report or trap type. 16-31 For a negotiation message: report identifier. For a regular report: Sequence number. For a trap: data depending on trap type, i.e. this field is not part of the header for a trap message. The five new message types are: o CPU idle time (which gives a measure of how heavily the gateway is loaded). o Packet delay across a gateway. o Gateway to gateway delay (round trip time). o Throughput (bits). 3 Bolt, Beranek and Newman Inc. IEN 157 o Queue traps (which signal when the occupancy of a queue goes above or below a certain threshold value). 3.1 CPU Idle Time - report type 8 CPU idle time gives an idea of the amount of time the gateway machine is not doing useful processing. The purpose of this is to find out when the CPU becomes saturated, which will be the case if the proportion of idle time becomes very small. The report consists of two 32-bit counts following the monitoring header: 1. The amount of CPU idle time since the gateway started, in milliseconds. 2. The time since the gateway started, in seconds. 3.2 Packet delay - report type 9 Packet delay refers to the length of time a packet stays in the gateway. The measurement of this delay and of gateway to gateway delay are related: measurement of one begins where the other ends. The model used here assumes that gateway processing takes place in three parts: network I/O, queuing and routing. Implementation considerations will affect just where the packets can be timestamped on their way through the gateway, so that for some gateways it may be possible to stamp a packet at the network I/O level, while for others it may not be possible until the 4 IEN 157 Bolt, Beranek and Newman Inc. packet enters the routing processing. This specification therefore does not define where the boundary should lie, but it is important that together the measures account for all the delay that a packet will experience as far as the gateway is concerned. It is recommended that the packet delay be made to refer to as large a fraction as possible of the time the packet spends in the gateway. The report consists of two 32-bit counts and two 16-bit counts. All delays are in milliseconds. The format is: 1. The total number of packets processed since the gateway started (32 bits). 2. The total delay for all packets processed (32 bits). 3. The minimum delay experienced by a single packet (16 bits). 4. The maximum delay experienced by a single packet (16 bits). 3.3 Gateway to gateway delay - report type 10 The measurement of this delay and of packet delay are related: see the first paragraph in the previous section. This report could be something of an interim measure, provided the gateways obtain radio synchronizing equipment for measuring the one-way delay directly. However, currently only the round trip delay can be determined. The report assumes that gateways will use some kind of tagged echo packets to find the 5 Bolt, Beranek and Newman Inc. IEN 157 round trip delay to each of their neighbors, the tagging being used to identify specific packets. The report format is a table ordered by internet address considered as a 32-bit unsigned integer. Each table entry consists of an internet address followed by two 32-bit counts and two 16-bit counts. The internet address is the neighbor address for which this delay applies. Of the 32-bit counts, the first is the cumulative total of the echo packets returned by the neighbor since this gateway started, and the second is the total delay experienced by those returned packets, in milliseconds. The two 16-bit counts are the minimum and maximum delays, in milliseconds, for a single packet sent to the neighbor. There will be one table entry for each neighbor address, so that if a gateway is a neighbor on two networks then it will have two table entries. There will be an entry for each such address for each neighbor that replies to the echoes, whether or not that neighbor is a routing gateway. The table size may grow as new neighbors come up while a gateway is running, but it may not shrink; the entry for a gateway that stops replying will simply remain unchanged. 6 IEN 157 Bolt, Beranek and Newman Inc. The report format is therefore: Internet address of first neighbor, lowest network number Total of echo packets returned by this neighbor (32 bits) Total delay experienced (32 bits) Minimum delay to this neighbor (16 bits) Maximum delay to this neighbor (16 bits) . . Internet address of last neighbor, highest network number Total echo packets returned (32 bits) Total delay (32 bits) Minimum delay for this neighbor (16 bits) Maximum delay for this neighbor (16 bits) 3.4 Bit Throughput - report type 11 In contrast with the packet throughput report type, which has its emphasis on the number of packets a gateway can process, the bit throughput report type focuses on how fast a gateway's network connections can accept or deliver data. The report is a table of pairs of 32-bit counts ordered by interface; the first count in each pair is the cumulative total of bits processed coming in at that interface, and the second is the output count. Interfaces are ordered as in the gateway description message, report type 0. There are two extra 32-bit counts at the end of the message: the first is the total of bits dropped, and the second is the time since the gateway started, in seconds. The counts for the interfaces include all traffic at that interface, including control traffic and messages originating at the 7 Bolt, Beranek and Newman Inc. IEN 157 gateway. The format of the message is therefore: Input count for first interface Output count for first interface . . Input count for nth interface Output count for nth interface Dropped count Time since gateway up 3.5 Queue occupancy - trap type 3 This is a trap message which is sent by the gateway whenever a queue length exceeds a threshold percentage specified in the trap request message, or when the occupancy falls below that threshold after having been above it for some time. If a queue is loaded such that the threshold occupancy is continually being passed in each direction, a large number of these traps would be generated in a short time. To avoid this, there should be some minimum time interval between successive trap messages. It is left up to the individual gateway implementors to decide what this time interval should be; experience with using this trap type will probably suggest a reasonable value. Note that this replaces the earlier Queue full trap described in IEN 131. I believe that a percentage occupancy trap is more useful because if a queue becomes full, the gateway is 8 IEN 157 Bolt, Beranek and Newman Inc. probably already dropping packets and the message is not useful as an early warning. In any case, a queue full trap is just a 100% percentage occupancy trap. The DO TRAP message for this trap type requires an extra piece of information: the percentage occupancy of the queue which is to trigger the trap. This is expressed as an integer in a single byte following the report id field in the DO TRAP message. A gateway should only use one value of this threshold at a time, so that a second DO TRAP message will supersede the previous one if the threshold value is different. The DO TRAP message for this trap type has the format: Bits Contents 0 1 1 1 2-3 0 4-7 0 8-15 3 16-31 report identifier 32-39 occupancy threshold The trap message has the following format: Bits Contents 0-7 Interface number of Queue. 8-11 Input(0) or output(1) queue. 12-15 Above(0) or below(1) the specified occupancy. 16-23 The occupancy percentage used as a trigger. 9