OBS-LOCK-1

Interlocking for
Observing Systems

Guy Rixon

Issue 1.1; 5th May 1995

Royal Greenwich Observatory,
Madingley Road,
Cambridge CB3 0EZ

Telephone (01223) 374000
Fax (01223) 374700
Internet G.Rixon@ast.cam.ac.uk

1 Introduction

1.1 Purpose of this document

This is an interface-control document describing how programs in a distributed observing-system can take advantage of a system of interlocks on their actions. Any engineers intending to write or maintain such programs should be familiar with the protocol described below. There is no discussion of the locking of specific systems and observers will find little of interest here.

This document is a specification of the locking protocol. If the software implementing the protocol does not follow the description, then the software is at fault. It is valid to claim conformance of software to a particular issue of this document in respect of the protocol only.

1.2 Scope of the locking protocol

All the observing systems developed for ING are distributed multi-tasking systems. For some operations, the programs within these systems can run independently, but it is more normal to have interdependencies between programs and actions of programs. It is a general requirement that the system recognize these dependencies and react accordingly; `reacting accordingly' could cause a command to be temporarily rejected, or could mean that the user is warned and asked for authority to proceed.

When a program, such as a DRAMA task to command a CCD controller, is used with several instruments, it has to respect a different set of interlocks for each configuration. Thus, it is undesirable to build the interlocking into the programs to which it applies. The scheme described here concentrates the locking intelligence into a specialized program, the lock manager. Programs manipulate interlocks by exchanging messages with the lock manager.

When a client program intends to execute a command on a server program, the client contacts the lock manager and requests authority for the command. The lock manager checks for mandatory locks and warning locks on the command and sends the results of the search back to the client. Mandatory locks forbid the client from executing the command on the, usually on the grounds that the command cannot succeed. Warning locks suggest that the command may have undesirable side effects. If there are no locks, or if all the locks are warnings, the lock manager grants a lock to the client; that is, a mandatory lock is placed on the command to prevent any other client using it. This transaction is called requesting a lock.

When a lock is placed on one command, the lock manager consults a table of interlocks and typically places locks on many other commands. Locks are removed when the client holding the lock explicitly gives it up, or when that client exits.

By this scheme, clients can be coded without reference to the details of any instrument or telescope, but can still use the interlocks of a particular observing configuration.

1.3 Glossary

Client: a transient program, normally corresponding to a user-command, that sequences a set of operations.

Server: a permanently-resident program that executes actions at the request of clients.

RPC: Remote Procedure Call: the invocation by a client of an action on a server when the transaction appears as a single function in the client. See OBS-RPC-1.

Lock: in the context of this document, a condition discouraging a client from executing some action.

Lock manager: the central server that records the state of the locking.

Mandatory lock: a lock that the client must respect.

Warning lock: a lock the the client may choose to ignore.

Requesting a lock: the process in which a client asks the lock manager for permission to execute an operation.

Granting a lock: the act in which the lock manager permits a client to execute the requested operation and thereby imposes interlocks on associated operations.

Dialogue RPC: an RPC to a `talker' for the purposes of presenting text to the user. See OBS-TALK-1.

Talker: a UI-server that presents text to the user and prompts for input as required.

Condition-code: an integer status-code identifying some condition (usually an error). DRAMA condition codes have a severity bit field as described in [2].

ERS: DRAMA's Error-Reporting System. See [1].

1.4 References

[1] DRAMA Error Reporting System (ERS)
AAO document ERS_SPEC/4 by Tony Farrel.

[2] A portable (VMS-compatible) Message-Code System
AAO document MESS/6 by Tony Farrel.

[3] RPCs for DRAMA clients
RGO/ING document OBS-RPC-1 by Guy Rixon.

[4] Dialogues with the user: general principles
RGO/ING document OBS-TALK-1 by Guy Rixon.

2 The protocol for lock requests

Clients request locks by exchanging DRAMA messages with the lock manager. The lock manager is a DITS task and the named locks are its actions. Each lock request completes one instance of an action.

In any given observing system there is only one lock manager. The manager registers with DITS with the name LOCK (note the capitals).

To request the hypothetical lock named LK1, a client must send a DITS OBEY message to LOCK for the action LK1. In the argument list of this OBEY, the item Argument1 (note mixed case) must be a single character in the set R, I, F or Q. These define the request as follows.

R: request the lock. The lock manager may refuse to grant this lock.

I: impose the lock. The lock manager may not refuse to grant this lock.

F: free (i.e. remove, destroy) the lock.

Q: query the current locks without setting new ones.

The lock manager must return `immediately' a single message listing the current locks; the message should always be sent within 0.25 seconds of receiving the request.

In the argument list of the lock-manager's reply are arguments Lock1, Lock2 etc. (note mixed case) each of reports one lock on the command in question. If there are no locks, the message contains no argument list. If the option on the request was `R' or `I', the list of locks does not include a lock granted to the current client in the course of the request. For example, if no client holds LK1 and client A requests it, the reply to A lists no locks and the transaction ends with LK1 locked in favour of A. A request from client B shows the lock held by A. If A requests LK1 again withoutfirst freeing it, A sees its own lock from the previous request.

Each argument in the list is an SDS structure containing the elements

.holder name of the task holding the lock

.code condition-code describing the lock

The condition-code representing each lock indicates the reason for the lock. Each condition-code has severity bits, arranged as desribed in [1], indicating Warning or Error severity. Symbolic constants for these condition codes are defined in the file ciaLock.h. The table of messages corresponding to the condition codes is in ciaLock_msgt.h; this table can be used with DRAMA's message-translation facility as described in [2].

3 Protocol for monitoring locks

Normally, a client program run from a command line requests a lock and terminates if the lock is not granted: the nett effect of the locking is to abort the command. The client can only find out if a command is locked by itself requesting a lock. A graphical-interface server needs to know whether or not a command is locked before the command is invoked, in order to selectively enable the controls. The lock-manager makes this information available through DRAMA parameters, with one parameter for each lockable command.

To query the state of the hypothetical lock LK1, a client sends a DRAMA Get message for parameter LK1 to the task LOCK. The lock manager sends back a single reply with the state of the lock in the argument list.

To continuously monitor the state of LK1, a client sends a DRAMA MONITOR message for to the task LOCK with Argument1 set to START and Argument2 set to LK1. The lock manager sends back an initial response indicating that the parameter is monitored and then sends a single update each time the state of the lock changes.

Each lock-parameter is a single character, taking the values M (mandatory lock), W (warning lock) or F (free: no locks); upper case is always used. The parameter is set to indicate the most restrictive lock of any applying to the command.

4 The client's responsibilities

To avoid deadlocks in the system clients must not abuse the locking. The rules below should be followed.

Locks should be held for as short a time as possible. When writing a client that locks a particular command, bear in mind that a large number of associated commands may be interlocked.

Clients should request locks with the `R' option. The `I' option (to impose a lock) is reserved to servers that want to comunicate teh state of their associated hardware.

If a client reacts to warnings by abandoning a command, the lock on that command is still held. The client should free the lock immediately.

Clients can free a lock either by a request with the `F' option or by logging out of the DRAMA message net. Logging out is preferred, as it make less work for the lock manager. A client that exits without logging out of the net (by crashing, or in response to SIGKILL) does not free its locks. The lock manager may ultimately free the locks when it notices that the client is missing, but clients should not take advantage of this.

The response of clients to warning locks is left to the client's designer.

Ideally, the client will have code to abort on warning, ignore warnings or query warnings by a prompt to the user and these options will be selected by a command-line switch.

Occasionally, a client may be run when the lock manager is absent. The client should deal with tthis situation gracefully.

It is not sufficient for a client to assume the state of a lock by testing the corresponding parameter: operations on the parameters do not grant locks or set interlocks. Clients that monitor parameters must also make lock requests in the normal way.

5 The lock manager's responsibilities

The lock manager must reply to all lock requests promptly. The replies should be sent within 0.25 seconds of the requests and preferably within 50 ms.

Lock requests should be processed atomically and the results of each request must be calculated and recorded before attending to the next request. In particular, all interlocks resulting from the granting of a lock must be in place before processing the next request; all interlocks lifted when a lock is frred must be cleared away before the next request is processed. It must not be possible for two clients to be granted the same lock.

When initialized, the lock manager must clear all its locks and interlocks.

The parameters that report the state of each lock must be kept in step with the internal records of the lock's state. The lock manager should update the parameters as soon as a lock is set or removed, and before processing other lock requests.

6 Proposed library functions

Since lock requests are normal `Obey' transactions in DRAMA, it is reasonable to handle them in the RPC formalism described in OBS-RPC-1. This approach will be supported by library functions from the cia facility.

Before using the locking library functions, a client should call the initialization function

(void) = ciaLockInit( int *argc, char *argv[], StatusType *status );

where argc and argv[] are the argument list from the command line. This call detects optional arguments on the command line to control the client's reaction to warning locks.

The simplest way to make a lock request is by calling the function ciaLockReq(). This function, which has the signature

(void) = ciaLockReq( const char *lockName, StatusType *status );

makes a lock request for the lock named by lockName. If there are locks, they are reported to the user via the dialogue system described in OBS-TALK-1. On exit, status is good if there are no locks, or if warnings are being over-ridden, and bad if the command is locked out. and must be aborted.

At a lower level,

(void) = ciaLockRpc( const char *lockName, const char requestType,
	   ciaRpc_t *rpc, StatusType *status );

sets up an RPC-control structure to be executed by ciaExecuteRpc(). On exit from ciaExecuteRpc(), when the lock-request has been answered, the output arguments rpc.argsOut hold the list of locks as described in section 2. The arguments can be reported through ERS by calling

(StatusType) = ciaLockStatus( SdsIdType argsOut, 
                                           StatusType *status );

which adds the textual description of each lock into the ERS message stack and returns a condition code in which the status bits indicate success (no locks), warning or error (mandatory locks). The client can then call one of the flushing routines described in OBS-TALK-1 to report the locks. The function

(char) = ciaLockOption();

returns a code for the locking option chosen in ciaLockInit(): one of 0 for no locking, A for `abort on warning', Q for `query warnings' or I for `ignore warnings'.

An example of the un-packaged locking follows.

/* Check first whether locking has been disabled.                  */
    option = ciaLockOption()
    if( option == '0' ) goto NOLOCKING

/* Make the lock request.  If the request fails (no lock-manager?) */
/* treat this locking disabled and shut up about it.               */
    ciaLockRpc( "Run", "R", &rpc, &status );
    ciaExecuteRpc( &status, 1, &rpc );
    if( !StatusOk(status) )
    {
       ErsAnnel( &status );
       goto NOLOCKING;
    }

/* Get the ruling status of the locks.                            */
    lockStatus = ciaLockStatus( rpc.argsOut, &status );

/* Report mandatory locks as commenets.                            */
    if( lockStatus & STS_M_ERROR )
    {
       ErsRep( ERS_M_NOFMT, &status, "Command rejected." );
       ciaCommentErs( &status );
    }

/* Report warning locks.                                           */
    else if( lockStatus & STS_M_WARNING )
    {
       switch( option )
       {
/*        Ignore warnings; erase the lock reports.                 */
          case 'I' : 
             ErsAnnul( &lockStatus );
             break;

/*        Abort on warnings; report the locks as comments.         */
          case 'A' :
              ErsRep( ERS_M_NOFMT, &lockStatus. "Command rejected" );
              ciaCommentErs( &lockStatus )
              status = CIA__CMDLOCKED
              break;

/*       Query warnings; report the locks as a query.              */
          case 'Q' :
              ErsRep( ERS_M_NOFMT, &lockStatus,
                      "Do you want to go on with this command?" );
              if( !ciaGoNogoErs(&lockStatus) ) 
                 status = CIA__CMDLOCKED;
              break;
       }
    }
    NOLOCKING:

This sequence is the implementation of ciaLockReq(). A more-sophisticated client might choose to look at individual locks and choose a different result for each of them.

7 An example of locking

Consider a simple CCD camera with a shutter (driven by the CCD controller) and a filter wheel (driven seperately from the CCD controller). This configuration has only the user commands run and filter. The user commands can be given from a command line and there is also a GUI with buttons for the two commands; the CLI and GUI are independent of one another. There is also a local UI for the CCD controller which can order the CCD to clear, expose and read out.

There are seven DRAMA tasks in this system. The CCD controller and filter wheel each have a server task. There is one GUI task, and a transient client for each command. The lock manager is accompanied by a `relay' task that sets locks in response to changes of parameters in the servers.

The clients see only the locks RUN and FILTER

, corresponding to the two user commands. The relay task knows about three extra locks, CLEARING, EXPOSING and READING, which it sets to reflect events in the CCD controller.

A user pushes the `run' button on the GUI. The client run executes and requests a lock on action RUN. There are currently no locks on this action; the run client notes this from the lock manager's reply, deduces that it has authority to proceed and executes its command on the CCD server. The lock manager has set a mandatory lock on RUN (because this action is in use) and a warning interlock on FILTER (because it is unusual, but not entirely forbidden, to move the filter wheel during a run).

The user now types run at the CLI. The run client detects the mandatory lock lock on RUN, reports the situation and refuses to continue. The user tries the filter command instead. The filter client detects the warning interlock on FILTER, explains that a run is in progress, and asks the user whether or not to continue. Some time later, the CCD server reports to the run client that the run has finished. The client then exits with good status, disconnecting from the message net in the process. The lock manager detects the disconnection and clears all locks held by run and all the resulting interlocks.

This interlocking is not ideal: the warning interlock on FILTER lasts until the end of the run when it should be lifted as soon as the readout starts. An improved interlocking uses the relay task and the locks on CLEARING, EXPOSING and READING to achieve this. Locking RUN no longer interlocks FILTER, but executing RUN on the CCD server causes a CCD_STATE parameter to cycle through CLEARING...EXPOSING...READING...IDLE. The relay task monitors this parameter and imposes locks on CLEARING, EXPOSING, READING

as appropriate. Inside the lock manager, CLEARING and EXPOSING interlock FILTER but READING does not: as soon as the readout starts, the user can move the filter wheel without interference.

Appendix A. Document history

Issue 1.1 05/05/95 First formal issue as OBS-LOCK-1. Early drafts were classified as INS-LOCK-1.

Interlocking for Observing Systems