PMFAULT(3)                Library Functions Manual                PMFAULT(3)

NAME         top

       __pmFaultInject,  PM_FAULT_POINT,  PM_FAULT_CHECK, __pmFaultSummary -
       Fault Injection Infrastracture for QA

C SYNOPSIS         top

       #include <pcp/pmapi.h>
       #include <pcp/impl.h>
       #include <pcp/fault.h>

       void __pmFaultInject(const char *ident, int class);
       void __pmFaultSummary(FILE *f);

       PM_FAULT_POINT(ident, class);

       cc -DPM_FAULT_INJECTION=1 ... -lpcp_fault

DESCRIPTION         top

       As part of the coverage-driven changes to QA in PCP 3.6, it became
       apparent that we needed someway to exercise the ``uncommon'' code
       paths associated with error detection and recovery.

       The facilities described below provide a basic fault injection
       infrastructure (for libpcp only at this stage, alhough the mechanism
       is far more general and could easily be extended).

       A special build is required to create libpcp_fault and the associated
       <pcp/fault.h> header file.  Once this has been done, new QA
       applications may be built with -DPM_FAULT_INJECTION=1 and/or existing
       applications can be exercised in presence of fault injection by
       forcing libpcp_fault to be used in preference to libpcp as described

       In the code to be tested, __pmFaultInject defines a fault point at
       which a fault of type class may be injected.  ident is a string to
       uniquely identify the fault point across all of the PCP source code,
       so something like "libpcp/" __FILE__ ":<number>" works just fine.
       The ident string also determines if a fault will be injected at run-
       time or not - refer to the RUN-TIME CONTROL section below.  class
       selects a failure type, using one of the following defined values
       (this list may well grow over time):

              Will cause the next call to malloc(3), realloc(3) or strdup(3)
              to fail, returning NULL and setting errno to ENOMEM.  We could
              extend the coverage to all of the malloc-related routines, but
              these three are sufficient to cover the vast majority of the
              uses within libpcp.

              Will cause the next call to a PMAPI routine to fail by
              returning the (new) PCP error code PM_ERR_FAULT.  At the this
              stage, only __pmRegisterAnon(3) has been instrumented as a
              proof of concept for this part of the facility.

              Will cause the next call to an instrumented routine to return
              the PCP error code PM_ERR_TIMEOUT.  At this stage, only
              __pmGetPDU(3) has been instrumented to check for this class of
              fault injection.

       To allow fault injection to co-exist within the production source
       code, PM_FAULT_POINT is a macro that emits no code by default, but
       when PM_FAULT_INJECTION is defined this becomes a call to
       __pmFaultInject.  Throughout libpcp we use PM_FAULT_POINT and not
       __pmFaultInject so that both libpcp and libpcp_fault can be built
       from the same source code.

       Similarly, the macro PM_FAULT_CHECK emits no code unless
       PM_FAULT_INJECTION is defined, in which case if a fault of type class
       has been armed with __pmFaultInject then, the enclosing routine will
       trigger the associated error behaviour.  For the moment, this only
       works for the following class types:

              The enclosing routine will return immediately with the value
              PM_ERR_FAULT - this assumes the enclosing routine is of type
              int foo(...)  like all of the PMAPI routines.

              The enclosing routine will return immediately with the value
              PM_ERR_TIMEOUT - this assumes the enclosing routine is of type
              int foo(...)  like all of the PMAPI routines.

       A summary of fault points seen and faults injected is produced on
       stdio stream f by __pmFaultSummary.

       Additional tracing (via -Dfault and DBG_TRACE_FAULT) and a new PMAPI
       error code (PM_ERR_FAULT) are also defined, although these will only
       ever be seen or used in libpcp_fault.  If DBG_TRACE_FAULT is set the
       first time __pmFaultInject is called, then __pmFaultSummary will be
       called automatically to report on stderr when the application exits
       (via atexit(3)).

       Fault injection cannot be nested.  Each call to __pmFaultInject
       clears any previous fault injection that has been armed, but not yet

       The fault injection infrastructure is not thread-safe and should only
       be used with applications that are known to be single-threaded.

RUN-TIME CONTROL         top

       By default, no fault injection is enabled at run-time, even when
       __pmFaultInject is called.

       Faults are selectively enabled using a control file, identified by
       the environment variable $PM_FAULT_CONTROL; if this is not set, no
       faults are enabled.

       The control file (if it exists) is read the first time
       __pmFaultInject is called, and contains lines of the form:
               ident op number
       that define fault injection guards.

       ident is a fault point string (as defined by a call to
       __pmFaultInject, or more usually the PM_FAULT_POINT macro).  So one
       needs access to the libpcp source code to determine the available
       ident strings and their semantics.

       op is one of the C-style operators >=, >, ==, <, <=, != or % and
       number is an unsigned integer.  op number is optional and the default
       is >0

       The semantics of the fault injection guards are that each time
       __pmFaultInject is called for a particular ident, a trip count is
       incremented (the first trip is 1); if the C-style expression
       tripcount op number has the value 1 (so true for most ops, or the
       remainder equals 1 for the % op), then a fault of the class defined
       for the fault point associated with ident will be armed, and executed
       as soon as possible.

       Within the control file, blank lines are ignored and lines beginning
       with # are treated as comments.

       For an existing application linked with libpcp fault injection may
       still be used by forcing libpcp_fault to be used in the place of
       libpcp.  The following example shows how this might be done.
       $ export PM_FAULT_CONTROL=/tmp/control
       $ cat $PM_FAULT_CONTROL
       # ok for 2 trips, then inject errors
       libpcp/events.c:1  >2

       $ export LD_PRELOAD=/usr/lib/
       $ pmevent -Dfault -s 3 sample.event.records
       host:      localhost
       samples:   3
       interval:  1.00 sec
       sample.event.records[fungus]: 0 event records
       __pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP
       sample.event.records[bogus]: 2 event records
         10:46:12.413 --- event record [0] flags 0x1 (point) ---
           sample.event.param_string "fetch #0"
         10:46:12.413 --- event record [1] flags 0x1 (point) ---
           sample.event.param_string "bingo!"
       __pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP
       sample.event.records[fungus]: 1 event records
         10:46:03.416 --- event record [0] flags 0x1 (point) ---
       __pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT
       sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
       __pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT
       sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory
       __pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT
       sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
       === Fault Injection Summary Report ===
       libpcp/events.c:1: guard trip>2, 5 trips, 3 faults

EXAMPLES         top

       Refer to the PCP and PCP QA source code.

       src/libpcp/src/derive.c uses PM_FAULT_CHECK.

       src/libpcp/src/err.c and src/libpcp/src/events.c use PM_FAULT_POINT.

       src/libpcp/src/fault.c contains all of the the underlying

       src/libpcp_fault contains the recipe and Makefile for creating and
       installing libpcp_fault and <pcp/fault.h>.

       QA/477 and QA/478 show examples of control file use.

ENVIRONMENT         top

              Full path to the fault injection control file.

              Force libpcp_fault to be used in preference to libpcp.

SEE ALSO         top


DIAGNOSTICS         top

       Some non-recoverable errors are reported on stderr.

COLOPHON         top

       This page is part of the PCP (Performance Co-Pilot) project.
       Information about the project can be found at ⟨⟩.
       If you have a bug report for this manual page, send it to  This page was obtained from the project's upstream
       Git repository ⟨git://⟩ on 2017-03-13.  If you discover
       any rendering problems in this HTML version of the page, or you
       believe there is a better or more up-to-date source for the page, or
       you have corrections or improvements to the information in this
       COLOPHON (which is not part of the original manual page), send a mail

Performance Co-Pilot                                              PMFAULT(3)