↑ Writing ↑

GEONius.com
28-May-2003
 E-mail 

TPOCC Configuration Monitor Design

1991

An informal document prepared in 1991 for a program design walkthrough. The contractor's software development methodology only required a structure chart and PDL, but I was getting tired of being invited to design reviews for programs whose function I had to deduce from reading the PDL. In addition, putting my thoughts down on paper helped me to map out my design.

CFGMON is the TPOCC configuration monitor, a program that periodically checks the values of a predefined set of telemetry parameters. The basic requirements for the configuration monitor, as set out in the SAMPEX and WIND/POLAR SRSs, are as follows:

The following pages provide a fairly high-level overview of the configuration monitor, after which you can dive right into the PDL. Among the topics discussed are:


External Interfaces


          State Manager  <----->          <----->  Event Logger
              TSTOL      <----->  cfgmon
             xcfgmon     <----->    ^     <----->  Data Server
                                    |
                                  Shared
                                  Memory

CFGMON will support interfaces with:

multiple clients - commands are sent to CFGMON as ASCII strings. An initial "[XQtag]" indicates an executable command. The optional tag is saved and used to tag responses returned to a client. The lack of a leading bracket expression implies "[XQ]" with no tag.

data server - to get the values of system variables. This interface is used if the data server is running on a remote host. If the data server is running on the same host as CFGMON, then values are retrieved from shared memory.

shared memory - to get the values of system variables if CFGMON can access them directly, i.e., CFGMON is running on the same processor as the data server.

event logger - to log monitoring events.

data server (optional) - to allow CFGMON to function as a synchronous data source of monitoring events for clients such as display.

Not shown on the diagram above are the disk files that CFGMON reads and writes:

 

Configuration Files (Input)

A configuration file defines the parameters that CFGMON should monitor. Configuration files are normal UNIX text files containing a line for each parameter to be checked. Parameter names are specified in a TSTOL-style format:

[P@]mnemonic[#process][@host]

The optional "P@" prefix specifies that the EU-converted value of the variable should be examined. A process name should be specified if the parameter's mnemonic is not unique in the database. Values can be fetched from remote systems by specifying the host name of the remote computer.

The remainder of a parameter's configuration entry specifies the tests that should be applied to the parameter's values:

parameter op operand [ and op operand ]
parameter op operand [ or op operand ]

The allowed operators include =, <>, <, <=, >, and >=. The optional AND- and OR-clauses provide for checking that a value falls in or out of a given range. For example, the following configuration entries for parameter P are interpreted as shown:

P >= X and <= Y
tests if X <= P <= Y; i.e., is P in the range [X..Y]?

P < X or > Y
tests if P < X or Y < P; i.e., is P not in the range [X..Y]?

Operands can be one of three types:

Signed Integer - if the first character of the operand is a digit and the number doesn't look like a floating-point number. The standard C conventions for integers are supported: N for decimal numbers, 0N for octal numbers, and 0xN for hexadecimal numbers.

Floating Point - if the first character of the operand is a digit and the number does look like a floating-point number.

Character String - if the first character of the operand is not a digit. If the first character is a double quote, the text of the string is delimited by double quotes; otherwise, the text is delimited by blanks and/or tabs.
 

Configuration Monitoring Reports (Output)

When a parameter fails the tests specified in a configuration file, the event is logged to the event logger as well as being recorded in a configuration monitoring report. A separate report file is generated for each configuration monitored. If a configuration is monitored for more than one satellite pass, then the reports for subsequent passes are appended to the first report file.

The proposed format of the configuration monitoring report looks somewhat as follows:

     Configuration:  fileName
     Start of Pass:  YY:DDD:HH:MM.SS

       Date/Time
       of Check     Parameter       Value           Check Expression
      ---------     ---------       -----           ----------------
   YY-DDD-HH:MM:SS  P@AD-RAMP      123.456       ((P@AD-RAMP > 250) and
                                                  (P@AD-RAMP < 300.5))
   YY-DDD-HH:MM:SS  BATT_VOLT     Unchecked
        ...

       End of Pass:  YY:DDD:HH:MM.SS

This is not the same as the format presented in the WIND/POLAR CDR, but it looks more readable, will be easier to generate, and will consume less paper.

 

Run-Time Environment

Environment variable, $mission_CFG_FILE, like TSTOL's procedure file variable, specifies a list of pathnames used to find configuration files. For example:

    % setenv mission_CFG_FILE "./.cfg, $PROJECT_DIR/runtime/config/"

The file name for a monitoring report is the configuration name, converted to lower-case. Environment variable, $mission_CFG_REPORT, provides default components for report file names. For example:

    % setenv mission_CFG_REPORT "$PROJECT_DIR/runtime/config/.rpt"

Additional TPOCC environment variables:

$mission_SYV_FILE
specifies the system variable database dump file.
$mission_CLASS_FILE
specifies the event class definition file.
$mission_EVENT_BASE
specifies the file of subsystems' event base numbers.
$mission_EVENT_TEXT
specifies the event message text file.
$LOG_SUBSYS
specifies the base server name for event logger subsystem.

Network server names:

mission_cfgmon
is the service name for connection requests from the state manager, TSTOL, or any other application.
mission_cfgmon_ds (Optional)
is the service name for connection requests from data servers. CFGMON serves as a synchronous data source and plays monitoring events out to clients.

System variables:

HOST (mission_EVENT_LOGGER)
contains the name of the host on which the event logger is running. CFGMON reads this name and passes it to the routine that initializes the interface with the event logger.
HOST (CFGMON)
is assigned the local host name by the configuration monitor when it starts up.
DEBUG (CFGMON)
if defined, is periodically read by CFGMON and used to update its internal debug switch. This allows CFGMON debug to be turned on and off in the middle of program execution.
 

How CFGMON Works

CFGMON comes up, creates its listening sockets (mission_cfgmon and mission_cfgmon_ds), and sits around waiting for connection requests. When a state manager-like client calls, CFGMON answers the connection request and adds the client to its list of clients. When and if the connection is broken, the client and its configurations are deleted.

CFGMON recognizes the following commands received from a client:

MONITOR configuration [interval]
asks CFGMON to monitor a particular configuration. A client can send multiple MONITOR commands if it is desired that CFGMON simultaneously monitor multiple configurations.

FORGET configuration|ALL
deactivates the specified configuration.

PASS INIT|TERM
signals the beginning or end of a satellite pass.

SHOW
lists the currently loaded configurations. (Not supported by state manager clients.)

LIST
lists the available configurations. CFGMON scans the directories specified in environment variable, $mission_CFG_FILE, for configuration files. (Not supported by state manager clients.)

DEBUG ON|OFF
turns debug output on and off.

The MONITOR command causes CFGMON to load the specified configuration file, build an expression list, and add the configuration to CFGMON's list of configurations. A configuration has the following attributes: the name of its configuration file, the expression list for the configuration, the monitoring interval, and a reference to the client who requested the configuration.

The configurations in CFGMON's configuration list are not actually monitored until the satellite pass begins. This is signalled to CFGMON by a PASS INIT command received from any of its clients. A PASS TERM command signals the end of a pass, at which time CFGMON ceases to monitor the configurations in its list.

 

Scheduling

The scheduling of configuration evaluations is accomplished using the timeout capability of the system select(2) call. Implementing the scheduling in this manner avoids the need to use timers and interrupt handlers, the code for which requires significant amounts of conditional compilation to work under both the UNIX and VxWorks operating systems. The scheduling scheme used by CFGMON, described in the following paragraphs, is sufficient for the non-critical timing of configuration monitoring.

Each configuration has associated with it the time of the next evaluation, which is initially the current time. Prior to calling select(2), CFGMON queries its configurations to determine the time of the next evaluation. The difference between this time and the current time is the timeout value passed to select(2). When the select(2) call returns, CFGMON first checks for and processes any connection requests or incoming commands. It then scans the list of configurations and evaluates any configurations whose interval has elapsed. After evaluation, a configuration's time of next evaluation is incremented to:

current_time + requested_interval

Upon completing the scan of the configuration list, CFGMON determines the time of the earliest upcoming evaluation and calls select(2) again. If select(2) returns because of an interrupt (ERRNO = EINTR), CFGMON can simply recompute the timeout and loop to call select(2) again.

 

Configuration Representation and Evaluation

A "configuration" is simply a list of expressions that must be evaluated periodically, where the expressions are comparisons of actual telemetry values against their expected values. In CFGMON, an expression is represented by a tree of expression nodes, where an expression node is a reference to a system variable, a literal value, or an operator. For example, a configuration file entry of

SC_ATT < 45 or > 90

is represented by the following expression tree:

                                  or
                              /         \
                            <             >
                          /   \         /   \
                     SC_ATT   45   SC_ATT   90

The evaluation of an expression utilizes a depth-first, recursive algorithm that evaluates each expression node in the tree, from the bottom up and from left to right. The evaluation of an expression node returns a value. A reference to a system variable returns its value retrieved out of shared memory or from a remote data server; a literal value returns itself. A relational operator evaluates each of its operands, compares them, and returns a true or false value. Logical operators (and and or) work in a similar fashion.

If the evaluation of an expression returns a final value of true, then the parameter passed its test. If false is returned, the parameter failed its configuration check. In that case, CFGMON logs an event message and records the failure in the report file.

(The preceding provides a basic understanding of how expressions are represented and evaluated in the configuration monitor. The actual implementation has 3 items for every parameter to be checked: information about the parameter, its current value, and its test expression. Information about the parameter [e.g., its mnemonic] is needed when logging configuration failures. The current value of the parameter is fetched before evaluating its test expression; it provides a single value for the multiple references in the test expression, as well as saving the parameter's value for reporting purposes. Finally, the test expression is as shown above, except that system variable references [e.g., "SC_ATT"] are linked to the parameter's current value.)

On first glance, the scheme above might seem overly elaborate. Wouldn't it be easier to just store the parameter, its 2 comparison operators, and its 2 operands in a simple record structure? However, the need to handle different types of telemetry data (integer, real, etc.) would necessitate pointers to other structures anyway. The small number of parameters involved (50 at most) and the low monitoring frequency (once every 10 seconds) make any performance penalty relatively insignificant. A benefit of this representation scheme is the ease with which CFGMON could be expanded to handle more complex configuration tests in the future.

 

The Design of CFGMON

CFGMON was designed in an object-oriented manner. Doing so broke the program up into small, functionally-independent modules. Once an understanding of CFGMON's purpose was achieved, the basic objects in the system were fairly obvious. The root object of the program is the configuration monitor itself. When created, the CFGMON object creates a system variable database object, an event logger object, an empty client list, an empty configuration list, etc.

The CFGMON object then sits in a loop, listening for connection requests from new clients and commands from existing clients. When a new client requests a connection, CFGMON creates a client object and adds it to the client list. When input is available on a network connection, CFGMON notifies the corresponding client object; the client object reads the message and sends it to the CFGMON object for interpretation.

When a MONITOR (CFGMON) directive is received, CFGMON creates a configuration object and adds it to the configuration list. When a PASS INIT or PASS TERM directive is received, the satellite pass object is notified. During a pass, an evaluate "message" is periodically sent to each of the configuration objects. Each configuration object, if ready, tests the parameters specified in its configuration file. If the test of a parameter fails, messages are sent to the event logger and report objects.

The objects in the CFGMON program are described in more detail further on. The program is implemented in an object-based fashion in C. A "message" is passed between objects via a function call to the selected "method" in the target object. For each type of object, there is a header file and a function file:

    TOY_object.h
    TOY_object.c

The header file defines the data structure for instances of the object and provides external definitions for functions that operate on these objects. The function file contains the source code for the functions; the first argument to each function is the object upon which the function operates. The functions belonging to a particular object class are stored in a single source file for two reasons:

  1. To keep related functions together.
  2. To provide a central location to define static (hidden) and non-static (visible) class-wide variables.

The object-based technique utilized in CFGMON is similar to the X Toolkit technique for simulating object-oriented programming in C, but it differs in some respects:

  1. Xt adds a separate header file, "objectP.h", to hide the representation of its objects. This is purely cosmetic, since nothing prevents the application programmer from including the private header file in his/her code.

  2. Xt stores pointers to an object's functions in each object instance. This allows for overriding functions in subclasses, but CFGMON's horizontal class hierarchy has no evident need for inheritance!
 

Object Hierarchy

This diagram illustrates the relationships between objects in CFGMON. Note that the diagram shows the logical relationships between instances of objects, not between classes of objects. For example, an expression list contains expressions. An expression, in turn, contains system variables and values (and operators).

              Configuration Monitor
                  Client List
                      Client
                  Message
                  Configuration List
                      Client
                      Configuration
                          Expression List
                              Expression
                                  System Variable
                                  Value
                              ...
                          Report
                      ...
                  Satellite Pass
                  System Variable Database
                  Event Logger
 

Objects

The types of objects in the CFGMON program are described in this section. For each object type, the following information is given:

object
The object type
Variables:
Variables found in each instance of an object, i.e., the contents of an object's data structure.
Methods:
The functions that operate upon objects of this type.

An object's methods are mapped to functions in the object's C source file using the following naming scheme:

    TOY_object_method ()

For the sake of simplicity, not all methods or objects are explicitly implemented; these methods and objects are marked by asterisks (*). The exceptions are as follows:

* Configuration Monitor
Variables:
Methods:
* create() - Performs program initialization and listens for connection requests and commands.
parse() - Parses and processes commands received from clients.
destroy() - Terminates program.
Client
Variables:
XNET I/O handle
Methods:
create() - Answers connection request and creates XDR stream for client.
receive() - Receives message from client.
send() - Sends message to client.
socket() - Returns client's socket number.
destroy() - Closes connection and destroys XDR stream.
* Client List
Variables:
Methods:
* create() - Creates an empty client list.
* add() - Adds a client object to the list of clients.
* get() - Returns I-th client in the list.
* length() - Returns number of clients in list.
* mask() - Returns SELECT(2) mask for clients.
* destroy()
Expression
Variables:
Expression type (system variable, literal value, operator)
Left child
Right child
Value
Methods:
create() - Returns a tree whose root may be a literal value or a system variable reference, a unary operator applied to a single subtree, or a binary operator applied to 2 subtrees. Variable and value nodes reference "Value" objects. (Variable-length argument list)
evaluate() - Evaluates expression and returns "Value" object.
display() - Returns string representation of expression.
value() - Returns previously-evaluated value of expression.
destroy()
* Expression List
Variables:
Methods:
* create() - Creates an empty expression list.
* add() - Adds an expression to the list of expressions.
* get() - Returns I-th expression in the list.
* length() - Returns number of elements in list.
* destroy()
Configuration
Variables:
File name
Client
List of parameters to test
List of current value expressions
List of test expressions
Scheduling interval (in seconds)
Time of next update (a TIMEVAL structure)
Methods:
create() - Load file and construct expression list.
evaluate() - Evaluates each expression in the list of expressions and reports failures.
next_update() - Returns time of next evaluation.
destroy() - Delete configuration.
* Configuration List
Variables:
Methods:
* create() - Initialize empty list.
* add() - Create configuration and add to list. Other parameters: interval, client.
* evaluate() - Scan list and evaluate configurations whose interval has elapsed.
* next_update() - Scan list and determine earliest time of next evaluation.
* delete() - Delete configuration from list.
* destroy() - Destroy configuration list.
System Variable Database
Variables:
System variable context
Data server host
Methods:
create() - Load system variable database and determine if data server is remote or local.
destroy() - Unload system variable database.
System Variable
Variables:
Tag structure (for access through data server)
Address (for direct access in shared memory)
Methods:
create() - Parses mnemonic specification ("[P@]mnemonic[#process][@host]"), creates a tag structure for the system variable, and looks up its address in shared memory.
display() - Returns name of system variable.
get_value() - Returns "Value" object.
set_value()
destroy()
Value
Variables:
Data type
Pointer to buffer for value
Methods:
create()
clone() - Returns duplicate of value.
compare() - Must handle different data types.
data() - Returns pointer to data buffer.
display() - Returns string version of value.
destroy()
Message
Variables:
Type
Tag
Text
Methods:
create()
tag_of() - Tag from state manager.
text_of() - Message text.
type_of() - [XQ] or [ST].
destroy()
Satellite Pass
Variables:
Pass active flag
Pass start time
Pass end time
Methods:
create()
init()
term()
start_time() - Returns time of pass INIT.
end_time() - Returns time of pass TERM.
in_progress()
destroy()
Report
Variables:
File name
(FILE *) descriptor
Methods:
create() - Open file for appending.
open() - Write header for pass.
write() - Write configuration event to file.
close() - Write trailer for pass.
destroy() - Close file.
Event Logger
Variables:
Methods:
* create() - Initialize interface with event logger.
* log() - Send message to logger.
* destroy()

C Source Files and Functions

        TOY_cfgmon.c
             main ()
             TOY_cfgmon_parse ()
             TOY_cfgmon_destroy ()
        TOY_client.c
            TOY_client_create ()
            TOY_client_receive ()
            TOY_client_send ()
            TOY_client_socket ()
            TOY_client_destroy ()
        TOY_expr.c
            TOY_expr_create ()
            TOY_expr_evaluate ()
            TOY_expr_display ()
            TOY_expr_value ()
            TOY_expr_destroy ()
        TOY_config.c
            TOY_config_create ()
            TOY_config_evaluate ()
            TOY_config_next_update ()
            TOY_config_destroy ()
        TOY_sysvar.c
            TOY_sysvar_create ()
            TOY_sysvar_display ()
            TOY_sysvar_get ()
            TOY_sysvar_set ()
            TOY_sysvar_destroy ()
        TOY_syvdb.c
            TOY_syvdb_create ()
            TOY_syvdb_destroy ()
        TOY_value.c
            TOY_value_create ()
            TOY_value_clone ()
            TOY_value_compare ()
            TOY_value_data ()
            TOY_value_display ()
            TOY_value_destroy ()
        TOY_message.c
            TOY_message_create ()
            TOY_message_tag ()
            TOY_message_text ()
            TOY_message_type ()
            TOY_message_destroy ()
        TOY_pass.c
            TOY_pass_create ()
            TOY_pass_init ()
            TOY_pass_term ()
            TOY_pass_start_time ()
            TOY_pass_end_time ()
            TOY_pass_in_progress ()
            TOY_pass_destroy ()
        TOY_report.c
            TOY_report_create ()
            TOY_report_open ()
            TOY_report_write ()
            TOY_report_close ()
            TOY_report_destroy ()
 

CFGMON Structure Chart

The following chart shows the calling hierarchy of the modules in the configuration monitor program:

  TOY_cfgmon
  .       TOY_syvdb_create
  .       TOY_pass_create
  .       TOY_client_socket
  .       TOY_config_next_update
  .       TOY_client_create
  .       TOY_client_receive
  .       .       TOY_client_destroy
  .       .       .       TOY_config_destroy
  .       .       .       .       TOY_sysvar_destroy
  .       .       .       .       TOY_expr_destroy
  .       .       .       .       .       TOY_value_destroy
  .       .       .       .       .       TOY_expr_destroy
  .       .       .       .       .       TOY_sysvar_destroy
  .       .       .       .       TOY_report_destroy
  .       .       TOY_message_create
  .       .       TOY_cfgmon_parse
  .       .       .       TOY_message_text
  .       .       .       TOY_config_create
  .       .       .       .       TOY_config_load
  .       .       .       .       .       TOY_sysvar_create
  .       .       .       .       .       TOY_expr_create
  .       .       .       .       TOY_report_create
  .       .       .       .       TOY_pass_in_progress
  .       .       .       .       TOY_config_pass
  .       .       .       .       .       TOY_pass_in_progress
  .       .       .       .       .       TOY_report_open
  .       .       .       .       .       .       TOY_pass_start_time
  .       .       .       .       .       TOY_report_close
  .       .       .       .       .       .       TOY_pass_end_time
  .       .       .       TOY_config_destroy
  .       .       .       TOY_pass_init
  .       .       .       TOY_pass_term
  .       .       .       TOY_config_pass
  .       .       .       TOY_client_send
  .       .       .       .       TOY_message_type
  .       .       .       .       TOY_message_tag
  .       .       .       .       TOY_message_text
  .       .       .       .       TOY_client_destroy
  .       .       .       TOY_message_create
  .       .       .       TOY_message_destroy
  .       TOY_config_destroy
  .       TOY_client_destroy
  .       TOY_pass_in_progress
  .       TOY_config_evaluate
  .       .       TOY_expr_evaluate
  .       .       .       TOY_expr_evaluate
  .       .       .       TOY_value_create
  .       .       .       TOY_value_compare
  .       .       .       TOY_sysvar_get
  .       .       TOY_expr_value
  .       .       .       TOY_value_clone
  .       .       .       .       TOY_value_create
  .       .       TOY_report_write
  .       .       .       TOY_sysvar_display
  .       .       .       TOY_value_display
  .       .       .       TOY_expr_display
  .       .       .       .       TOY_expr_display
  .       .       .       .       TOY_value_display


Exit Handler:

  TOY_cfgmon_destroy
  .       TOY_config_destroy
  .       TOY_client_destroy
  .       TOY_pass_destroy
  .       TOY_syvdb_destroy


Unused (but expected to be of use):

  TOY_sysvar_set
  .       TOY_value_display
  .       TOY_value_destroy

Alex Measday  /  E-mail