INtime Explorer may be used to monitor a system for unhandled hardware exceptions ("crashes") and to generate a report about the crashed thread. This monitoring is known as "crash watching".

Configuring crash watching for hardware faults

With the Watch page, found in the INtime Explorer Options, Watch hardware faults can be enabled and the action defined.

Crash watching is enabled for all nodes at the same time.

The Directory for watch log files contains files with the node name with an .RTC extension; when the directory name does not end with a \, / or : character, a \ is appended; for a Distributed RTOS node you should put a Distributed RTOS file system compatible directory name here that includes the separator character. Each crash log file acts as a circular buffer of the given the Maximum size of a report file in Kilobytes; when the file is full, the oldest crash record is overwritten.

The Program to view a crash report can be selected; this should be a Windows program capable of displaying plain text files, such as Notepad.

What to do to faulting process can be configured via the Add/Edit action dialog. Each line in the list defines the action for one process; one line can contain the filename * to indicate ‘all other processes’.

At the bottom of the screen is a button Export to rtintex.ini... to save INtime Explorer settings to a file. This file can be transported to a node for standalone fault watching.

Crash watching in connected mode

INtime Explorer can be used to log crashes (unhandled hardware exceptions) occurring on a node to which it is connected. To activate, start INtime Explorer with a connection to the required nodes and enable crash watching with the Options dialog. From then on, each of the servers will monitor its hardware fault mailboxes and when a fault message is received, a number of steps are taken:

the server collects all context information on the crashed process and thread and writes it into its crash log file in the crash log directory (the name of the file is taken from the node name with extension .RTC).
the server then sends a message to the INtime Explorer that a new crash record has been written.
INtime Explorer shows the number of new crash records in the heading of the node’s window.
if the crashed process’ name occurs in the action list, the belonging action is taken.
if the crashed process’ name is not in the list, but a ‘*’ process name is present, that action is taken.
if neither of the two preceding conditions is true, the process is left alone.

To create a readable crash report, use the Crash log... menu item or View crash log button.

The crash log file is filled as a circular buffer, so its size will always be within the limit set in the Watch page. If for some reason you want to get rid of old crash records, just delete the .RTC file while it is not in use.

When a crash occurs on a Distributed RTOS node, the crash log file must be written to the native file system; if you have some networking connection like FTP, INtime Explorer may be able to access the file directly; otherwise you must first transport it to the Windows PC.

Stand alone crash watching

To log crashes on an INtime Distributed RTOS node while the Windows PC is not connected to it, or on the Windows PC without showing INtime Explorer, you must run the server part of INtime Explorer in stand alone mode. This requires installing or modifying the following files on the INtime node:

RTINTEX.RTA

the stand alone version of the INtime Explorer server; it should be started with the option "-w optionsfile" (see STARTUP.RSC below); the default options file is RTINTEX.INI, which may or may not be suitable for the node. The server is prepared to perform its normal lookup actions for INtime Explorer, but in addition watches for crashes and logs these in a local crash log file.

options file

the options file (RTINTEX.INI if no name is specified) can be created by INtime Explorer (see setting watch options; make sure the file name is complete and compatible with the local host requirements. The filename in this section should refer to a file that can be accessed by the server.
An additional parameter defined for this mode is the Priority parameter in the Options section. This parameter may be set to the desired priority of the internal threads for RTINTEX.RTA instead of the default 252. This parameter might be used in the case where an application is running threads at an unusually high priority and is pre-empting the RTINTEX.RTA threads.

startup.rsc

The standard loader script for an INtime node on Distributed RTOS; add a line to load rtintex.rta with the –w option, for example:

launch rtintex.rta -a "-w /config/nodea/rtintex.ini"

You can use the web based Distributed RTOS configuration tool to edit the startup.rsc file.

When operating in stand alone mode, the server does not terminate itself when INtime Explorer terminates. It does not require an INtime Explorer connection for the crash watching.

After a fault message is received, the server takes a number of steps:

it collects all context information on the crashed process and thread and writes it into its local crash log file.
if the crashed process’ name occurs in the action list, the belonging action is taken.
if the crashed process’ name is not in the list, but a ‘*’ process name is present, that action is taken.
if neither of the two preceding conditions is true, the process is left alone.

To create a readable crash report in this case, you must first copy the crash log file from the INtime node to a Windows PC and then use the View>Crash log.. menu item or button.

The crash log file is filled as a circular buffer, so its size will always be within the limit set in the options file. If for some reason you want to get rid of old crash records, just delete the .RTC file while it is not in use.

Sample rtintex.ini file for stand-alone crash watching on INtime Distributed RTOS

; This section configures the watch parameters
[Watch]
; Log: path to the log file (default intex.log). 
Log=/config/NodeA/intex.rtc
; Max: maximum size of log file, in KBytes
Max=256
; This section configures watch options
[Options]
; Priority: set the monitoring thread priority to this value (default 252)
Priority=200
; this section configures actions to take when a crash occurs. Use INtime Explorer to configure this section
[Actions]
; number of actions configured
Count=0

Creating a crash report

After a crash has occurred on an INtime node, a crash record is written in a crash log file. If you are watching crashes in stand alone mode, transport this file to a Windows PC. Then start INtime Explorer, if it is not running yet; there is no need to make any INtime node connection and INtime need not be running on the local node.

Select the View>Crash log… menu item or button and open the appropriate crash log file (it is normally called nodename.RTC). If the file contains more than one crash record, you can select one from a list, showing the date and time of each crash; if there is only one crash record, you do not have to select it.

INtime Explorer reads the crash record and in order to provide source information, it needs access to the file from which the process was loaded. You are prompted to locate the file. You should not locate a file that differs from the actual process file.

When the process file is found and contains debug information, INtime Explorer uses that information to show a trace of all active functions with their parameters and local variables. If there is no file or it does not contain debug information, only hexadecimal information is printed. We strongly recommend that you include debug information, which you could add to a release version of the program.

Program locations found during a stack trace are displayed like this:

Stackframe: frameptr=00408f28 cs:eip=432b:0040109c
Location = D:\USER\JOE\gp\gp.c:Test1 + 0x5c line = 58 + 0x3

meaning that the location is at 0x5c bytes from the beginning of function Test1 in the given source file, and at 0x3 bytes from the start of the code corresponding to line 58 in that source file.

Parameters and local variables are displayed with their stack address; if the variable is a scalar or a pointer, its value is also displayed. In all cases the full value is shown in the stack dump. Static variables and data pointed to by a pointer is never displayed.

A crash report is written to a text file with the name nodename_n.TXT; here n is a number starting at 1, each report uses the next available number; if all numbers from one to 100 are in use, report 1 is overwritten. The crash report is written to the same directory as the crash log file.

After a crash report has been written, the INtime Explorer offers to open it in the viewer program that you configured on the Watch page.

Utilities

INtime Explorer (intex.exe)