RMIF
Jump to navigation
Jump to search
Remote Midas InterFace handles communication with remote processors
<PORT> - UDP port number for local node (may be output) use PORT=-1 to use any available local port (NeXtMidas only). <PROPS> - Property list to export _:result|q:message|_pipe <REMOTE> - Host address and port of remote connection... NeXtMidas uses: {ID=<remoteId>,HP="<host>:<port>"} X-Midas uses: <host>:<port> <ITEMS> - List of remote properties to connect to ("|" separated list) <REMOTEN> - Remote connection #2,... <ITEMSN> - List of remote properties #2,... Since there are X-Midas (C/FORTRAN) and NeXtMidas (Java) versions of this primitive, (N) and (X) are used to indicate NeXtMidas or X-Midas only capabilities, respectively. This command, when put in a macro, allows another X-Midas or NeXtMidas macro access to any of the items listed. These items may be messages, results parameters, pipes, or files. The remote connections will always have read access to any parameter available through the results table. The /HTTP switch (NeXtMidas only) starts an Http server on the same port for application status, debug, control, and file serving through a web browser. For example, on the local node, the URL might be: http://localhost:9000 The protocol is modeled after Remote Method Invocation (RMI) - a distributed object model for the Java programming language. The underlying socket transport is UDP. A reliable data protocol (RDP) is implemented by RMIF to "guarantee" delivery of messages (configurable with the RETRY attribute). The RMIF protocol definition allows for data packet formats: RAW, BINARY, ASCII, or XML. Currently only RAW is used. The port number, along with the local node name, serve to identify this interface within a subsystem. When this routine sends packets to remote nodes, the local address is attached to the packet, allowing the recipient to respond directly to the sender. If the specified port is already in use and the /PRANGE switch is specified, RMIF will search for a free port. In this case the actual port number will be written to the first results parameter (assuming that an ASCII input is specified, if the input is numeric, this step is skipped). All results table entries are available to the local interface upon request using the SET and GET messages. The properties argument is a "|" separated list of results, messages, pipes, or files that this interface can serve to remote application interfaces without queries. Once a remote interface issues an OPEN on one of these properties, updates will be sent whenever new data is available, until a CLOSE message is issued. The list of properties is made available through the GETKEYNAMES message allowing a macro to publish properties available to be displayed or manipulated by client GUI's. The pairs of remote node:port and "|" separated items list define client connections that should be automatic and permanent. Remote properties that are specified here are automatically reOPENed if the remote node is temporarily unavailable for any reason (death, network, reboot, ...) Each remote connection can have differently tuned parameters using the Port Parameter Table. The /PPT={table} switch defines the default for all new connections. See the switches section for specifics. Clients that have OPEN channels, regularly ping the remote server to verify health and status. The servers also ping the clients. This ping interval can be set with the /TIMEPING switch. The /TIMEOUT switch specifies how many seconds to wait for a response before closing the remote channels. The PINGs are still issued to poll for life, but channels are CLOSED. When the remote node starts responding again, the listed channels are reOPENED. For X-Midas clients, the /RECONN switch allows RMIF to try to reconnect to servers that disconnect and return. All pipes must be the same format between the two instances since X-Midas can't change pipe headers on the fly. The client decided that the server has disconnected if it doesn't respond to a PING packet within the timeout or if no pipe data has been transferred during this time. The latter case could happen if the server were recycled between two PINGs. If the connection has been dropped and you don't want to wait for the timeouts, you can send an OPEN message (X-Midas client only) to force a reconnect. LIMITATIONS =========== Packet Size: Note that RMIF has an internal 32 KiB buffer and, while it can transmit piped data with frame sizes or record lengths greater than this (by splitting the frames into multiple packets), they are subject to misalignment if any packets are lost. There is no internal means to realign the data, so use of RMIF with large buffer size is not recommended. The 32 KiB limit comes from the max size of a IP packet used for UDP. The IP packet includes IP header, UDP header, RMIF header (fixed), RMIF adjunct header (internal use only). The remaining space is available for RMIF data. Buffer sizes slightly larger than 32KiB will often fit, but this should not be counted on. RMIF uses RDP (over UDP) for sending control messages (this includes any messages sent via the macro). Consequently, it is subject to the same 32 KiB limitation. RMIF was designed to transmit plot data and small control messages associated with the same data. RMIF is not intended as a means of reliable transport for other types of data (such as database queries, imagery, HTML, etc); there are many other protocols (TCP, HTTP, FTP, SCTP, etc) that are designed for this type of application. Network Bandwidth: The network bandwidth along with the max read/write rate limits the actual transmission rate of the UDP packets. If too many packets are sent at once, some of them will be lost. RMIF uses RDP to counteract this by retransmitting the lost packets, but this can only do so much (a saturated network is likely to drop many of the packets on retransmit). The WINDOW= (in the /PPT= table) controls the maximum number of RDP packets outstanding (sent, but not yet receipted). RMIF will buffer outgoing packets when the window fills up. This prevents RMIF from over-saturating the network with its own packets. Making the window too large (even sizes >32 can be "too large" on many networks) will result in RMIF saturating the network which then causes the RMIF packets to be lost. The RETRY= parameter in the /PPT= table controls the number of retries that will be attempted. If /WINDOW= is too large, increasing RETRY= only makes a bad problem worse. Network Latency: Increasing WINDOW= (in the /PPT= table) on networks with high latency will improve performance since it allows more outstanding packets to be in transit at the same time. Faulty Network: Increasing RETRY= (in the /PPT= table) on networks with a high rate of packet loss or corruption will improve performance since it permits more transmission attempts in the event of a lost/corrupted packet. (Remember, if loss/error rate is 10%, the odds of a loss/error on retransmit are also 10%.) Faulty Networks With High Latency: This is what RMIF was designed for... the WINDOW= and RETRY= give users the control they need to "tune" RMIF so that it will run well on these networks. Just be careful not to oversaturate the network. COMPRESSION (X-Midas or NeXtMidas server, NeXtMidas clients) ============================================================ Compression is meant to be used on pipes destined for plotting. Compression only works on pipes with FIXED-POINT SB data. When using the NeXtMidas PLOT command, the plot's rmif ID may be set to allow information to be sent so that RMIF will compress the data to fill the plot screen. There are two types of compression, ONLY ONE of which is currently supported: 1. Data - TBD There are hooks in place for this type of compression but it it not yet implemented. This is completely recoverable non-lossy data compression (like zip). 2. Plot - This is commonly used for plotting. In this mode, RMIF automatically chooses one of two modes based on the number of points per pixel. a.) BAND ( >= 3pts/pixel ) Computes the upper and lower bounds giving a compression of >=1.5 times. For example, a 2k length frame would compress ~7 times in a plot window 300 pixels wide. b.) PACK ( < 3pts/pixel ) Computes the MAX of first point, and then 4bit deltas giving a compression of ~2x. When zoomed, the data not visible will also be thrown out. COPYING FILES from an RMIF server(N) ==================================== Files may be transfered from an RMIF server to a client using HTTP or MFTP, the Midas File Transfer Protocol. For example, nM> noop/gpw http://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp or nM> noop/gpw mftp://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp RMIF can find and transfer any MIDAS file in its' current AUX path. To specify the AUX, use the AUX qualifier or use the syntax: nM> noop/gpw http://<host>:<port>/Files/<aux>/remotefile.tmp localfile.tmp the MFTP protocol has additional qualifiers to tune the performance. PKTRATE = target bytes per second to transfer over the link PKTLEN = length in bytes of individual UDP data packets (typically 1K - 8K) PKTTO = timeout in seconds for request responses For example: nM> noop/gpw mftp://remotename:9000/Files/DAT/world.prm{PKTRATE=1e5,PKTTO=3} localfile.tmp See library help on MFTP for more information. Using an embedded server with the /HTTP switch ============================================== If RMIF is run with the /HTTP switch, controls, files, and Queries can be accessed from a remote node with the following syntax: * http://<server:port>/Controls --> Returns HTML * http://<server:port>/Files/<AUX>/remotefile.prm * http://<server:port>/Query --> Returns HTML For instance, to access a file on the server's DAT aux from a client, nM> noop http://<server:port>/Files/DAT/remotefile.prm localfile.prm To see an example of how this can be used run: nM> sd360/server then point your browser to http://localhost:9000/ and from there you will have links to files, controls and more. Messages: (X) = X-Midas Only, (N) = NeXtMidas Only, otherwise valid for both In X-Midas, SETKEY, ACKKEY, GETKEY, and RETKEY are homogeneous messages consisting of one 40 char address followed by pairs of 40 char keynames and 40 char keyvalues. The KEYMSGSTRUCT struct in rmif.inc maps this message data. The NDATA field is dynamic depending on the number of keys per message. Namely NDATA=1+2*NKEY. In NeXtMidas, these 4 messages are replaced by sending a SET, ACK, GET, or RET message to the ID of the remote itself. The system knows to route it through the RMIF connection. In NeXtMidas, when RMIF receives a SET, ACK, GET, or RET message that has a MSGNAME key in the data table, it's value is used as the message name to send the data (with the MSGNAME key/value removed from the table) to the upper level (i.e. the macro) instead of the SET, ACK, GET, or RET name. In X-Midas, the ADDR, KEYS, PROPS, and PING messages uses the INFO field to represent the index of the remote to send the message to. Valid input values are 1 to MAXREMOTE(32). NOTE: Any messages to a remote must be with FUNC=SEND (not SENDW). Messages to the RMIF primitive can be with either SEND or SENDW. (N) When sending messages to RMIF, the only why to know that it has been fully processed (client-side and server-side) is via the corresponding acknowledgments sent back to the macro. For example, an OPENED message is the acknowledge for the OPEN message. See below for details. ACK - Acknowledges a SET message. An ACK message sent to RMIF with a null remote value will notify all remotes of the attribute change. (X-Midas uses "ACKKEY" instead of "ACK") ADDC - Add a channel to a property on a remote. (N) ID=<remote> NAME="ADDC" INFO=<channel number, -1 for next available number> DATA=<property to monitor or "Q:ACK"> A "channel" is a real-time attribute (a pipe or server-side change to a result). If "Q:ACK" is specified, this setups a channel to receive all ACKs from the remote (server) that are sent directly to this RMIF or to all it's remotes (clients). Even though a table-like syntax maybe be used when specifying a pipe to map (e.g. "{_clientPipe=_serverPipe}"), RMIF treats the message data as a string. Examples: nM> message send ID=RID NAME="ADDC" INFO=1 & DATA="{_clientPipe=_serverPipe}" nM> message send ID=RID NAME="ADDC" INFO=2 DATA="SFREQ" Note: This message should only be sent after an OPENED message is received from the remote in the processMessage procedure (i.e. it should not be send between PIPE ON ... PIPE OFF). ADDC/M - Add a channel in multi mode to a property on a remote. This allows multiple remotes writing into a single pipe. (N) ADDR - Create a reference to an RMIF process on a remote machine. ID=<RMIF ID> NAME="ADDR" INFO=<desired remote index number; 1-MAXREMOTE> (X) DATA=<Table with properties for the remote to address> (N) ID = ID to name created remote (good idea to keep unique). HP = Host:Port string of remote, e.g. the Internet address (IP/hostname and PORT) of the remote machine. <ARGS>=1 (X) <TYPE>=S[40] (X) <VALUES=remote address IP/hostname:port string or hexadecimal string of IP address/port (in network byte order). (X) Note: The connection is not established until an OPEN message is processed. In X-Midas, care should be taken for the specified remote index, otherwise an existing remote at that index may be overridden. CLOSE - Close the connection to a remote. (N) A CLOSED message acknowledges this request. Note: An OPEN message to the same remote <Host:Port> should not be sent until the CLOSED message is received, otherwise the new remote may receive the server's acknowledgment to this remote (i.e. it gets a CLOSED message that was still in transit on the network to RMIF). CLOSE/R - Close the connection to a remote and remove it from the table. (N) DELC - Delete a channel to a property on a remote. (N) ID=<RMIF ID> INFO=<if > 0, channel number> DATA=<else, name of exported property> GET - Is used to request one or more key=value pairs. (X-Midas uses "GETKEY" instead of "GET"). A RET message acknowledges this request. KEYS - Requests a list of properties to be returned for specified remote. ID=<remote> (N) or ID=<RMIF ID> (X) INFO=<desired remote index number; 1-MAXREMOTE> (X) MODIFY - Modifies parameters of an open channel (usually used for plot compression). (N) NOTE: Compression only works on pipes with fixed-point SB data. nM> message FUNC=send ID=RID NAME=MODIFY & DATA={PROPERTY=_serverPipe,COMP=2,PLOTWIDTH=200} xM> rmsg send <addr> MODIFY,, {PROPERTY=_serverPipe,COMP=2,PLOTWIDTH=200} sets compression on the pipe for a 200 pixel wide plotter nM> message FUNC=send ID=RID NAME=MODIFY & DATA={PROPERTY=_serverPipe,TRIM1=plot.x1i,TRIM2=plot.x2i} xM> rmsg send <addr> MODIFY,, {PROPERTY=_serverPipe,TRIM1=plot.x1i,TRIM2=plot.x2i} sets compression for a plot zoom window from frame index x1i to x2i OPEN - Open a connection to a remote given the SERVER host and port. An OPENED message acknowledges this request. nM> message send RMIFID ,, OPEN ,, {ID=<rid>,HP=<host>:<port>} Note: Since <rid> is not passed to the server side, the RMIF server defaults the remote's message ID to "REMOTE". In most cases it is not important to uniquely identify a client since the server typically treats all clients the same (given that they have the same channels open). Since NeXtMidas 2.5.3 it is possible to specify an alternate ID for each remote client. This is done by setting the remote's id to desired value when processing the OPEN message in the server macro, for example: nM> set msg.data.id "MY_REMOTE" RADDR - Remove an address from RMIFs table of remotes. (N) RET - Acknowledges a GET message. (X-Midas uses "RETKEY" instead of "RET") RPKT - Is a more direct means to SEND or RECV an RMIF packet with less restrictions (see nxm.sys.exp.rmsg.exp). This is for X-Midas only. If the address field is blank, the packet is sent to all open remotes ONLY if the /RECONN switch is set (this was done to preserve the legacy, pre-/RECONN switch behavior). PAUSE - Pause, specify the time (seconds) in the info entry. (X) PING - Request a PING to be sent to specified remote. ID=<remote> (N) or ID=<RMIF ID> (X) INFO=<desired remote index number; 1-MAXREMOTE> (X) PROPS - Same as KEYS message. SET - Is used to set one or more key=value pairs. (X-Midas uses "SETKEY" instead of "SET"). An ACK message acknowledges this request. Examples: 1. X-Midas server setup: rmif/pktmsg port _mypipe|myresult 2. NeXtMidas server setup with web server: rmif/http port sfreq|_waveb /ppt={RETRY=2,TIMEPING=3} 3. NeXtMidas client setup with web server: rmif/http/prange=5 port 4. NeXtMidas opening a connection to a remote server: set remote "<host>:<port>" message send RMIF ,, "OPEN" ,, {ID=XM,HP="^remote"} message send XM ,, "ADDC" 1 {_WAVEB=_WAVEB} message send XM ,, "ADDC" 2 "Q:ACK" .... later in processMessage handler - sync up if msg.name eqs "OPENED" then message send XM ,, "GET" ,, {SFREQ=?,CFREQ=?} endif 5. NeXtMidas sending a message to an un-opened remote: message send "RMIF" ,, "ADDR" ,, {ID=RID,HP="^rhost:^rport"} message send "RID" ,, "SET" ,, {FREQ=123,RATE=2345} 6. NeXtMidas replying to a GET message from a remote: if msg.name eqs "GET" then ... fill in values for entries in the msg.data table message send "REPLY" msg "RET" 7. NeXtMidas replying to a SET message from a remote: if msg.name eqs "SET" then ... validate entries in the msg.data table and perform necessary actions message send "REPLY" msg "ACK" 8. NeXtMidas client to monitor all Q:ACK (channel) on remote host:port rmif -1 ,, {ID=REMOTE,HP="^host:^port"} "Q:ACK" Many examples exists as part of the RMIF test cases, see: $NMROOT/nxm/sys/test/rmif/test_rmif_*.mm Switches: (X) = X-Midas Only, (N) = NeXtMidas Only, otherwise valid for both /AUXLIST=l - List of AUXs to make available under "http://host:port/Files/<AUX>/<filename>". By default all AUXs in AUX.READ are listed (same as /AUXLIST="*"). Use /AUXLIST=null to disable. [Note that for to backwards- compatibility with older apps, the defaults for this switch differ between RMIF and HTTPSERV.] (N) /DEVICE=dev - Performs a network interface lookup to resolve server IP address to bind on based on a network device name (e.g. eth0, eth1.101). This switch takes precedence over the /HOST switch. (N) Since NeXtMidas 3.1.2. /HOMEPAGE=p - Homepage location for HTTP web server (p = path). [This matches HTTPSERV.] (N) /HOST=ip - Sets the host IP address to bind on. /HTTP - Start HTTP server for application status, debug, and file serving. Note that /AUXLIST= and /HOMEPAGE= have no affect if this switch is not specified. (N) /KEEPADJ - Keep adjunct header with the data. (X) /KEYMSG - Convert SET,GET,RET,ACK packets to SETKEY, GETKEY, ... messages. This allows the macro to take special action when a remote interface sends or requests information instead of letting RMIF handle this 'behind the scenes'. (X) /LOG=fname - Logs all control activity to text file /MSGID=id - The ID to process messages from the remote nodes (usually 1 in X-Midas, usually MAIN in NeXtMidas). [DEF=MAIN in NeXtMidas] /PAUSE=s - Time (sec) to pause at the end of idle loop (DEF=Mc.pause) (X) /PKTMSG - Convert SET,GET,RET,ACK packets to RPKT messages. This is the default if /MSGID is specified. (See nxm.sys.exp.rmsg.exp.) (X) /PPT={tbl} - Port Parameter Table for tuning connections. (N) Most setable attribute may be included but some typical parameters are: RETRY,TIMEPING,TIMEOUT TIMEDROP is not yet implemented in NeXtMidas. /PRANGE=n - Range of port numbers above <port> to use if already allocated. /RECONN - Reopens pipes after server recycle. (X) /REOPEN=NO - Don't reopen same named pipes on client side when reconnecting. This keeps plotters from resetting but should only be used if the application is reconnecting to pipes with similar headers. (N) /RETRY=n - Set maximum number of retries (default=5). (X) Use the /PPT switch for NeXtMidas. /STATUS=lbl - Writes the current number of remote connections to the result /TIMEDROP=s - Timeout to drop efforts to reconnect (default=600). (X) Use the /PPT switch for NeXtMidas. /TIMEOUT=s - Timeout (in seconds) to declare link down (default=15). (X) Use the /PPT switch for NeXtMidas. /TIMEPING=s - Timeout (in seconds) between pings (default=4). (X) Use the /PPT switch for NeXtMidas. /VERBOSE - Set verbose option. /WINDOW=n - Set maximum window of messages to a given remote before RDP will not send. (X) Use the /PPT switch for NeXtMidas. See Also: nxm.sys.net.Rmif, SD360, nxm.sys.test.test_rmif.mm, HTTPSERV