Capturing a system-level trace

It is sometimes useful to run a system-level trace, to get details of what is happening to the signaling bits on the telephone channels.

If there is a lot of activity on the system, the system trace will be large and more difficult to interpret. If possible, take the trace on a system with low activity (if you can reproduce the problem there).

Before taking a trace, check that there is enough free space in the file system for the trace file. The default trace file size is 1280 KB, but this can be increased, for example to over 10 MB. The trace file is stored in the /var/adm/ras directory, which must have enough space for the size of trace file required. The default value is big enough for a short trace, but 8 MB is a typical value for a system that has a lot of activity.

If you need to take a longer trace, perhaps to investigate an infrequently occurring problem, you must allocate a larger amount of space to the trace file. For example, 50 minutes of trace information for a system at moderate load might require over 40 MB.  

  1. To log on before starting the trace: If no AIX window is open, open one by selecting an area on the screen background and clicking AIX Login.
  2. Log onto the Blueworx Voice Response AIX account (usually as dtuser).

    The system displays the User Login menu.

  3. Type 2 and press Enter.

    The system displays the system prompt - usually a dollar sign ($).

  4. To start the trace: Type the following command and press Enter:
    trace -1 -L 8000000 -T 1000000
    • For a short trace, the -L and -T options can be omitted.
    • The -L option gives the trace file size in bytes. The default is 1280 KB. In the example above it is set to 8 MB.
    • The -T option gives the size of the trace file buffer in memory in bytes. The default is 128 KB, and the maximum size is 1 MB, as used in the example above.

    The system displays the > prompt.

    Note: By default, the trace command creates the trace file in the directory /var/adm/ras, with a filename of trcfile.1. You can use the additional option -o to change the location and filename of the trace file. For example:
    trace -1 -L8000000 -T1000000 -o /home/dtuser/tracefile
    Remember to ensure that there is sufficient space in whichever location you decide to use.
  5. Perform the action on Blueworx Voice Response that causes the problem you want to trace.
  6. To stop the trace: Wait a short time to allow the actions to complete.

    The system displays the > prompt.

    Note: Although the default is for the trace command to run in an interactive mode, for long traces you might find it useful to run tracing in the background, thereby returning the command line prompt immediately and allowing you to perform other tasks. To do this, use the -a option on the trace command, as in the following example:
    trace -1 -a -L8000000 -T1000000
    In order to stop the trace using this asynchronous mode, enter the command trcstop -1.
  7. Type the following command and press Enter:
    q

    By default, the system writes the unformatted trace into a file in the /var/adm/ras directory. These files are named trcfile.*, and they are in binary format.

  8. To save the trace in a file: Type the following command and press Enter:
    print_trace > <filename>
    The system writes the formatted trace, in ASCII format, into the file name given in the command. The command assumes that you are using the default file trcfile.1 in /var/adm/ras. If you created a different trace file in /home/dtuser as described in step 4 above, then the new syntax would be:
    print_trace /home/dtuser/tracefile > formatted.trace
What files and information do I need to send to Blueworx Support if I have taken a system trace?
Send the file or files that have been formatted by using the print_trace command (see the procedure described in, Capturing a system-level trace). These files are in ASCII format, so take care to keep this format if they are transferred to another type of operating system.

Alternatively, you might be asked to send the files that were created in the /var/adm/ras directory. These files are named trcfile.*. Look at the time and date of the files to determine which file (or files) you want. These files are in binary format, so take care to keep this format if you transfer them to another type of operating system.

You should include details of the version of Blueworx Voice Response you are using, and details of any patches or PTFs that have been applied.

Also include a description of the actions Blueworx Voice Response was performing when you took the trace, and what happened when the problem occurred. For example, you might say:

  • Called into channel 5 (number = 408 555 1234)
  • Tried to transfer to a busy phone (number = 408 555 5678)
  • Application hung up after 30 seconds because it did not detect that the phone was busy, and timed out

If there is more than one failure mode, try to take a separate trace for each mode.

For more information about taking system traces see AIX: Technical Reference: Base Operating System and Extensions Volume 2.

Using Blueworx Voice Response AIX system trace event Ids

Table 1 lists the AIX system trace event ID that Blueworx Voice Response uses. If you want to trace only specific events, there is a -j option on the AIX trace command which, when used with the event IDs listed below, allows you to restrict tracing to only those events.

Table 1. AIX system trace event IDs used by Blueworx Voice Response

Event ID

Subsystem ID

Location

Users

ID string

34F

0

User

Servers

 

450

0

Kernel

Device driver

 

451

0

Kernel

Signaling driver

 

456

0

User

CHP

 

456

1

User

CHP manager

 

456

2

User

Custom Server API

CA_LIB

456

3

User

VP library

VPLIB

456

4

User

Prompt manager

PROMPTM

456

5

User

Cache

CACHEM

456

6

User

State table

STATEM

457

0

User

OAM

 

457

1

User

SDI

 

457

2

User

ERROR_ENROLL

ERROR_ENROLL

457

3

User

Buffer pool

BufPool

457

4

User

Hardware configuration

ADAPTUPDT

458

0

User

Performance — file system

PLT

458

1

User

Performance — databases

MBS

459

0

User

Signaling interface

SIGLIB

459

1

User

SMSI signaling process

 

459

2

User

ACL signaling process

 

459

3

User

Signaling daemon

SL_DAEMON