ConvertBot: Remote Midas InterFace handles communication with remote processors

2020-04-27T22:05:14Z

Remote Midas InterFace handles communication with remote processors

New page

Remote Midas InterFace handles communication with remote processors
<pre>
<PORT> - UDP port number for local node (may be output) use PORT=-1
to use any available local port (NeXtMidas only).
<PROPS> - Property list to export _:result|q:message|_pipe
<REMOTE> - Host address and port of remote connection...
NeXtMidas uses: {ID=<remoteId>,HP="<host>:<port>"}
X-Midas uses: <host>:<port>
<ITEMS> - List of remote properties to connect to ("|" separated list)
<REMOTEN> - Remote connection #2,...
<ITEMSN> - List of remote properties #2,...

Since there are X-Midas (C/FORTRAN) and NeXtMidas (Java) versions of this
primitive, (N) and (X) are used to indicate NeXtMidas or X-Midas only
capabilities, respectively.

This command, when put in a macro, allows another X-Midas or NeXtMidas macro
access to any of the items listed. These items may be messages, results
parameters, pipes, or files. The remote connections will always have read
access to any parameter available through the results table.

The /HTTP switch (NeXtMidas only) starts an Http server on the same port for
application status, debug, control, and file serving through a web browser.
For example, on the local node, the URL might be: http://localhost:9000

The protocol is modeled after Remote Method Invocation (RMI) - a distributed
object model for the Java programming language. The underlying socket transport
is UDP. A reliable data protocol (RDP) is implemented by RMIF to "guarantee"
delivery of messages (configurable with the RETRY attribute).

The RMIF protocol definition allows for data packet formats: RAW, BINARY, ASCII,
or XML. Currently only RAW is used.

The port number, along with the local node name, serve to identify this
interface within a subsystem. When this routine sends packets to remote nodes,
the local address is attached to the packet, allowing the recipient to respond
directly to the sender.

If the specified port is already in use and the /PRANGE switch is specified,
RMIF will search for a free port. In this case the actual port number will be
written to the first results parameter (assuming that an ASCII input is
specified, if the input is numeric, this step is skipped).

All results table entries are available to the local interface upon request
using the SET and GET messages. The properties argument is a "|" separated
list of results, messages, pipes, or files that this interface can serve to
remote application interfaces without queries. Once a remote interface issues
an OPEN on one of these properties, updates will be sent whenever new data is
available, until a CLOSE message is issued. The list of properties is made
available through the GETKEYNAMES message allowing a macro to publish
properties available to be displayed or manipulated by client GUI's.

The pairs of remote node:port and "|" separated items list define client
connections that should be automatic and permanent. Remote properties that are
specified here are automatically reOPENed if the remote node is temporarily
unavailable for any reason (death, network, reboot, ...)

Each remote connection can have differently tuned parameters using the Port
Parameter Table. The /PPT={table} switch defines the default for all new
connections. See the switches section for specifics.

Clients that have OPEN channels, regularly ping the remote server to verify
health and status. The servers also ping the clients. This ping interval can
be set with the /TIMEPING switch. The /TIMEOUT switch specifies how many
seconds to wait for a response before closing the remote channels. The PINGs
are still issued to poll for life, but channels are CLOSED. When the remote
node starts responding again, the listed channels are reOPENED.

For X-Midas clients, the /RECONN switch allows RMIF to try to reconnect to
servers that disconnect and return. All pipes must be the same format between
the two instances since X-Midas can't change pipe headers on the fly. The client
decided that the server has disconnected if it doesn't respond to a PING packet
within the timeout or if no pipe data has been transferred during this time. The
latter case could happen if the server were recycled between two PINGs. If the
connection has been dropped and you don't want to wait for the timeouts, you can
send an OPEN message (X-Midas client only) to force a reconnect.

LIMITATIONS
===========
Packet Size:
Note that RMIF has an internal 32 KiB buffer and, while it can transmit
piped data with frame sizes or record lengths greater than this (by
splitting the frames into multiple packets), they are subject to misalignment
if any packets are lost. There is no internal means to realign the data, so
use of RMIF with large buffer size is not recommended.

The 32 KiB limit comes from the max size of a IP packet used for UDP. The IP
packet includes IP header, UDP header, RMIF header (fixed), RMIF adjunct
header (internal use only). The remaining space is available for RMIF data.
Buffer sizes slightly larger than 32KiB will often fit, but this should not
be counted on.

RMIF uses RDP (over UDP) for sending control messages (this includes any
messages sent via the macro). Consequently, it is subject to the same 32 KiB
limitation.

RMIF was designed to transmit plot data and small control messages associated
with the same data. RMIF is not intended as a means of reliable transport for
other types of data (such as database queries, imagery, HTML, etc); there are
many other protocols (TCP, HTTP, FTP, SCTP, etc) that are designed for this
type of application.

Network Bandwidth:
The network bandwidth along with the max read/write rate limits the actual
transmission rate of the UDP packets. If too many packets are sent at once,
some of them will be lost.

RMIF uses RDP to counteract this by retransmitting the lost packets, but this
can only do so much (a saturated network is likely to drop many of the packets
on retransmit).

The WINDOW= (in the /PPT= table) controls the maximum number of RDP packets
outstanding (sent, but not yet receipted). RMIF will buffer outgoing packets
when the window fills up. This prevents RMIF from over-saturating the network
with its own packets. Making the window too large (even sizes >32 can be "too
large" on many networks) will result in RMIF saturating the network which
then causes the RMIF packets to be lost.

The RETRY= parameter in the /PPT= table controls the number of retries that
will be attempted. If /WINDOW= is too large, increasing RETRY= only makes a
bad problem worse.

Network Latency:
Increasing WINDOW= (in the /PPT= table) on networks with high latency will
improve performance since it allows more outstanding packets to be in transit
at the same time.

Faulty Network:
Increasing RETRY= (in the /PPT= table) on networks with a high rate of packet
loss or corruption will improve performance since it permits more transmission
attempts in the event of a lost/corrupted packet. (Remember, if loss/error
rate is 10%, the odds of a loss/error on retransmit are also 10%.)

Faulty Networks With High Latency:
This is what RMIF was designed for... the WINDOW= and RETRY= give users the
control they need to "tune" RMIF so that it will run well on these networks.
Just be careful not to oversaturate the network.

COMPRESSION (X-Midas or NeXtMidas server, NeXtMidas clients)
============================================================
Compression is meant to be used on pipes destined for plotting. Compression
only works on pipes with FIXED-POINT SB data. When using the NeXtMidas PLOT
command, the plot's rmif ID may be set to allow information to be sent so that
RMIF will compress the data to fill the plot screen.

There are two types of compression, ONLY ONE of which is currently supported:

1. Data - TBD
There are hooks in place for this type of compression but it it not yet
implemented. This is completely recoverable non-lossy data compression
(like zip).

2. Plot - This is commonly used for plotting.
In this mode, RMIF automatically chooses one of two modes based on the
number of points per pixel.
a.) BAND ( >= 3pts/pixel )
Computes the upper and lower bounds giving a compression of >=1.5 times.
For example, a 2k length frame would compress ~7 times in a plot window
300 pixels wide.
b.) PACK ( < 3pts/pixel )
Computes the MAX of first point, and then 4bit deltas giving a compression
of ~2x.
When zoomed, the data not visible will also be thrown out.

COPYING FILES from an RMIF server(N)
====================================

Files may be transfered from an RMIF server to a client using HTTP or MFTP, the
Midas File Transfer Protocol. For example,

nM> noop/gpw http://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp
or
nM> noop/gpw mftp://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp

RMIF can find and transfer any MIDAS file in its' current AUX path. To specify
the AUX, use the AUX qualifier or use the syntax:

nM> noop/gpw http://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp

the MFTP protocol has additional qualifiers to tune the performance.
PKTRATE = target bytes per second to transfer over the link
PKTLEN = length in bytes of individual UDP data packets (typically 1K - 8K)
PKTTO = timeout in seconds for request responses

For example:

nM> noop/gpw mftp://remotename:9000/Files/DAT/world.prm{PKTRATE=1e5,PKTTO=3}
localfile.tmp

See library help on MFTP for more information.

Using an embedded server with the /HTTP switch
==============================================
If RMIF is run with the /HTTP switch, controls, files, and Queries can be
accessed from a remote node with the following syntax:
* http://<server:port>/Controls --> Returns HTML
* http://<server:port>/Files/<AUX>/remotefile.prm
* http://<server:port>/Query --> Returns HTML

For instance, to access a file on the server's DAT aux from a client,
nM> noop http://<server:port>/Files/DAT/remotefile.prm localfile.prm

To see an example of how this can be used run:

nM> sd360/server

then point your browser to http://localhost:9000/ and from there you will have
links to files, controls and more.

Messages: (X) = X-Midas Only, (N) = NeXtMidas Only, otherwise valid for both
In X-Midas, SETKEY, ACKKEY, GETKEY, and RETKEY are homogeneous messages
consisting of one 40 char address followed by pairs of 40 char keynames and
40 char keyvalues. The KEYMSGSTRUCT struct in rmif.inc maps this message
data. The NDATA field is dynamic depending on the number of keys per
message. Namely NDATA=1+2*NKEY. In NeXtMidas, these 4 messages are replaced
by sending a SET, ACK, GET, or RET message to the ID of the remote
itself. The system knows to route it through the RMIF connection.
In NeXtMidas, when RMIF receives a SET, ACK, GET, or RET message that has a
MSGNAME key in the data table, it's value is used as the message name to
send the data (with the MSGNAME key/value removed from the table) to the
upper level (i.e. the macro) instead of the SET, ACK, GET, or RET name.
In X-Midas, the ADDR, KEYS, PROPS, and PING messages uses the INFO field to
represent the index of the remote to send the message to. Valid input values
are 1 to MAXREMOTE(32).

NOTE: Any messages to a remote must be with FUNC=SEND (not SENDW). Messages to
the RMIF primitive can be with either SEND or SENDW. (N)

When sending messages to RMIF, the only why to know that it has been
fully processed (client-side and server-side) is via the corresponding
acknowledgments sent back to the macro. For example, an OPENED message
is the acknowledge for the OPEN message. See below for details.

ACK - Acknowledges a SET message. An ACK message sent to RMIF with a null
remote value will notify all remotes of the attribute change.
(X-Midas uses "ACKKEY" instead of "ACK")

ADDC - Add a channel to a property on a remote. (N)
ID=<remote>
NAME="ADDC"
INFO=<channel number, -1 for next available number>
DATA=<property to monitor or "Q:ACK">
A "channel" is a real-time attribute (a pipe or server-side change
to a result). If "Q:ACK" is specified, this setups a channel to
receive all ACKs from the remote (server) that are sent directly to
this RMIF or to all it's remotes (clients). Even though a table-like
syntax maybe be used when specifying a pipe to map (e.g.
"{_clientPipe=_serverPipe}"), RMIF treats the message data as a
string. Examples:

nM> message send ID=RID NAME="ADDC" INFO=1 &
DATA="{_clientPipe=_serverPipe}"
nM> message send ID=RID NAME="ADDC" INFO=2 DATA="SFREQ"

Note: This message should only be sent after an OPENED message is
received from the remote in the processMessage procedure
(i.e. it should not be send between PIPE ON ... PIPE OFF).

ADDC/M - Add a channel in multi mode to a property on a remote. This allows
multiple remotes writing into a single pipe. (N)

ADDR - Create a reference to an RMIF process on a remote machine.
ID=<RMIF ID>
NAME="ADDR"
INFO=<desired remote index number; 1-MAXREMOTE> (X)
DATA=<Table with properties for the remote to address> (N)
ID = ID to name created remote (good idea to keep unique).
HP = Host:Port string of remote, e.g. the Internet
address (IP/hostname and PORT) of the remote machine.
<ARGS>=1 (X)
<TYPE>=S[40] (X)
<VALUES=remote address IP/hostname:port string or hexadecimal
string of IP address/port (in network byte order). (X)

Note: The connection is not established until an OPEN message is
processed.
In X-Midas, care should be taken for the specified remote index,
otherwise an existing remote at that index may be overridden.

CLOSE - Close the connection to a remote. (N)
A CLOSED message acknowledges this request.
Note: An OPEN message to the same remote <Host:Port> should not be
sent until the CLOSED message is received, otherwise the new
remote may receive the server's acknowledgment to this remote
(i.e. it gets a CLOSED message that was still in transit on
the network to RMIF).

CLOSE/R - Close the connection to a remote and remove it from the table. (N)

DELC - Delete a channel to a property on a remote. (N)
ID=<RMIF ID>
INFO=<if > 0, channel number>
DATA=<else, name of exported property>

GET - Is used to request one or more key=value pairs. (X-Midas uses
"GETKEY" instead of "GET"). A RET message acknowledges this request.

KEYS - Requests a list of properties to be returned for specified remote.
ID=<remote> (N)
or
ID=<RMIF ID> (X)
INFO=<desired remote index number; 1-MAXREMOTE> (X)

MODIFY - Modifies parameters of an open channel (usually used for plot
compression). (N)
NOTE: Compression only works on pipes with fixed-point SB data.
nM> message FUNC=send ID=RID NAME=MODIFY &
DATA={PROPERTY=_serverPipe,COMP=2,PLOTWIDTH=200}
xM> rmsg send <addr> MODIFY,,
{PROPERTY=_serverPipe,COMP=2,PLOTWIDTH=200}
sets compression on the pipe for a 200 pixel wide plotter

nM> message FUNC=send ID=RID NAME=MODIFY &
DATA={PROPERTY=_serverPipe,TRIM1=plot.x1i,TRIM2=plot.x2i}
xM> rmsg send <addr> MODIFY,,
{PROPERTY=_serverPipe,TRIM1=plot.x1i,TRIM2=plot.x2i}
sets compression for a plot zoom window from frame index x1i to x2i

OPEN - Open a connection to a remote given the SERVER host and port.
An OPENED message acknowledges this request.
nM> message send RMIFID ,, OPEN ,, {ID=<rid>,HP=<host>:<port>}

Note: Since <rid> is not passed to the server side, the RMIF server
defaults the remote's message ID to "REMOTE". In most cases
it is not important to uniquely identify a client since the
server typically treats all clients the same (given that they
have the same channels open). Since NeXtMidas 2.5.3 it is
possible to specify an alternate ID for each remote client.
This is done by setting the remote's id to desired value when
processing the OPEN message in the server macro, for example:
nM> set msg.data.id "MY_REMOTE"

RADDR - Remove an address from RMIFs table of remotes. (N)

RET - Acknowledges a GET message. (X-Midas uses "RETKEY" instead of "RET")

RPKT - Is a more direct means to SEND or RECV an RMIF packet with less
restrictions (see nxm.sys.exp.rmsg.exp). This is for X-Midas only.
If the address field is blank, the packet is sent to all open
remotes ONLY if the /RECONN switch is set (this was done to
preserve the legacy, pre-/RECONN switch behavior).

PAUSE - Pause, specify the time (seconds) in the info entry. (X)

PING - Request a PING to be sent to specified remote.
ID=<remote> (N)
or
ID=<RMIF ID> (X)
INFO=<desired remote index number; 1-MAXREMOTE> (X)

PROPS - Same as KEYS message.

SET - Is used to set one or more key=value pairs. (X-Midas uses "SETKEY"
instead of "SET"). An ACK message acknowledges this request.

Examples:
1. X-Midas server setup:
rmif/pktmsg port _mypipe|myresult

2. NeXtMidas server setup with web server:
rmif/http port sfreq|_waveb /ppt={RETRY=2,TIMEPING=3}

3. NeXtMidas client setup with web server:
rmif/http/prange=5 port

4. NeXtMidas opening a connection to a remote server:
set remote "<host>:<port>"
message send RMIF ,, "OPEN" ,, {ID=XM,HP="^remote"}
message send XM ,, "ADDC" 1 {_WAVEB=_WAVEB}
message send XM ,, "ADDC" 2 "Q:ACK"

.... later in processMessage handler - sync up
if msg.name eqs "OPENED" then
message send XM ,, "GET" ,, {SFREQ=?,CFREQ=?}
endif

5. NeXtMidas sending a message to an un-opened remote:
message send "RMIF" ,, "ADDR" ,, {ID=RID,HP="^rhost:^rport"}
message send "RID" ,, "SET" ,, {FREQ=123,RATE=2345}

6. NeXtMidas replying to a GET message from a remote:
if msg.name eqs "GET" then

... fill in values for entries in the msg.data table
message send "REPLY" msg "RET"

7. NeXtMidas replying to a SET message from a remote:
if msg.name eqs "SET" then

... validate entries in the msg.data table and perform necessary actions
message send "REPLY" msg "ACK"

8. NeXtMidas client to monitor all Q:ACK (channel) on remote host:port
rmif -1 ,, {ID=REMOTE,HP="^host:^port"} "Q:ACK"

Many examples exists as part of the RMIF test cases, see:
$NMROOT/nxm/sys/test/rmif/test_rmif_*.mm

Switches: (X) = X-Midas Only, (N) = NeXtMidas Only, otherwise valid for both
/AUXLIST=l - List of AUXs to make available under
"http://host:port/Files/<AUX>/<filename>". By default all
AUXs in AUX.READ are listed (same as /AUXLIST="*"). Use
/AUXLIST=null to disable. [Note that for to backwards-
compatibility with older apps, the defaults for this
switch differ between RMIF and HTTPSERV.] (N)
/DEVICE=dev - Performs a network interface lookup to resolve server IP address
to bind on based on a network device name (e.g. eth0, eth1.101).
This switch takes precedence over the /HOST switch. (N)
Since NeXtMidas 3.1.2.
/HOMEPAGE=p - Homepage location for HTTP web server (p = path). [This
matches HTTPSERV.] (N)
/HOST=ip - Sets the host IP address to bind on.
/HTTP - Start HTTP server for application status, debug, and file
serving. Note that /AUXLIST= and /HOMEPAGE= have no affect
if this switch is not specified. (N)
/KEEPADJ - Keep adjunct header with the data. (X)
/KEYMSG - Convert SET,GET,RET,ACK packets to SETKEY, GETKEY, ... messages.
This allows the macro to take special action when a remote
interface sends or requests information instead of letting
RMIF handle this 'behind the scenes'. (X)
/LOG=fname - Logs all control activity to text file
/MSGID=id - The ID to process messages from the remote nodes (usually 1 in
X-Midas, usually MAIN in NeXtMidas). [DEF=MAIN in NeXtMidas]
/PAUSE=s - Time (sec) to pause at the end of idle loop (DEF=Mc.pause) (X)
/PKTMSG - Convert SET,GET,RET,ACK packets to RPKT messages. This is the
default if /MSGID is specified. (See nxm.sys.exp.rmsg.exp.) (X)
/PPT={tbl} - Port Parameter Table for tuning connections. (N)
Most setable attribute may be included but some typical
parameters are: RETRY,TIMEPING,TIMEOUT
TIMEDROP is not yet implemented in NeXtMidas.
/PRANGE=n - Range of port numbers above <port> to use if already allocated.
/RECONN - Reopens pipes after server recycle. (X)
/REOPEN=NO - Don't reopen same named pipes on client side when reconnecting.
This keeps plotters from resetting but should only be used if
the application is reconnecting to pipes with similar headers.
(N)
/RETRY=n - Set maximum number of retries (default=5). (X)
Use the /PPT switch for NeXtMidas.
/STATUS=lbl - Writes the current number of remote connections to the result
/TIMEDROP=s - Timeout to drop efforts to reconnect (default=600). (X)
Use the /PPT switch for NeXtMidas.
/TIMEOUT=s - Timeout (in seconds) to declare link down (default=15). (X)
Use the /PPT switch for NeXtMidas.
/TIMEPING=s - Timeout (in seconds) between pings (default=4). (X)
Use the /PPT switch for NeXtMidas.
/VERBOSE - Set verbose option.
/WINDOW=n - Set maximum window of messages to a given remote before RDP will
not send. (X) Use the /PPT switch for NeXtMidas.

See Also: nxm.sys.net.Rmif, SD360, nxm.sys.test.test_rmif.mm, HTTPSERV

</pre>

[[Category:NXM_Explain]]

RMIF - Revision history

ConvertBot: Remote Midas InterFace handles communication with remote processors