|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|
|
NAME | C SYNOPSIS | DESCRIPTION | RUN-TIME CONTROL | EXAMPLES | DIAGNOSTICS | ENVIRONMENT | SEE ALSO | COLOPHON |
|
|
|
PMFAULT(3) Library Functions Manual PMFAULT(3)
__pmFaultInject, __pmFaultSummary, PM_FAULT_POINT,
PM_FAULT_RETURN, PM_FAULT_CHECK, PM_FAULT_CLEAR - Fault Injection
Infrastructure for QA
#include <pcp/pmapi.h>
#include <pcp/fault.h>
void __pmFaultInject(const char *ident, int class);
void __pmFaultSummary(FILE *f);
PM_FAULT_POINT(ident, class);
PM_FAULT_RETURN(retvalue);
PM_FAULT_CHECK;
PM_FAULT_CLEAR;
cc -DPM_FAULT_INJECTION=1 ... -lpcp_fault
As part of the coverage-driven changes to QA in PCP 3.6, it became
apparent that we needed someway to exercise the ``uncommon'' code
paths associated with error detection and recovery.
The facilities described below provide a basic fault injection in‐
frastructure (for libpcp only at this stage, although the mecha‐
nism is far more general and could easily be extended).
A special build is required to create libpcp_fault and the associ‐
ated <pcp/fault.h> header file. Once this has been done, new QA
applications may be built with -DPM_FAULT_INJECTION=1 and/or ex‐
isting applications can be exercised in presence of fault injec‐
tion by forcing libpcp_fault to be used in preference to libpcp as
described below.
In the code to be tested, __pmFaultInject defines a fault point at
which a fault of type class may be injected. ident is a string to
uniquely identify the fault point across all of the PCP source
code, so something like "libpcp/" __FILE__ ":<number>" works just
fine. The ident string also determines if a fault will be inject‐
ed at run-time or not - refer to the RUN-TIME CONTROL section be‐
low. class selects a failure type, using one of the following de‐
fined values (this list may well grow over time):
PM_FAULT_ALLOC
Will cause the next call to malloc(3), realloc(3) or
strdup(3) to fail, returning NULL and setting errno to
ENOMEM. We could extend the coverage to all of the malloc-
related routines, but these three are sufficient to cover
the vast majority of the uses within libpcp.
PM_FAULT_CALL
Will cause the next call to an instrumented routine to fail
by returning an error code (possibly the new PM_ERR_FAULT
code). The actual error code is defined in the
PM_FAULT_RETURN macro at the head of an instrumented rou‐
tine. Initially, only __pmRegisterAnon(3) (returns
PM_ERR_FAULT), __pmGetPDU(3) (returns PM_ERR_TIMEOUT) and
__pmAllocResult(3) (returns NULL) were instrumented as a
proof of concept for this part of the facility, however
other routines may have this fault injection capability
added over time.
PM_FAULT_MISC
The ``other'' class, currently used with PM_FAULT_CHECK as
described below.
To allow fault injection to co-exist within the production source
code, PM_FAULT_POINT is a macro that emits no code by default, but
when PM_FAULT_INJECTION is defined this becomes a call to __pm‐
FaultInject. Throughout libpcp we use PM_FAULT_POINT and not
__pmFaultInject so that both libpcp and libpcp_fault can be built
from the same source code.
Similarly, the macro PM_FAULT_RETURN emits no code unless
PM_FAULT_INJECTION is defined, in which case if a fault of type
PM_FAULT_CALL has been armed with __pmFaultInject then, the en‐
closing routine return with the function value retvalue.
The PM_FAULT_CHECK macro returns a value that may be 0 or 1. If
PM_FAULT_INJECTION is defined then if a fault of type
PM_FAULT_MISC has been armed with __pmFaultInject then the value
is 1 else it is 0.
PM_FAULT_CHECK is most often used in concert with the
PM_FAULT_POINT macro with the PM_FAULT_MISC class to potentially
arm a trigger, then test PM_FAULT_CHECK and if this has the value
1, then the PM_FAULT_CLEAR macro is used to clear any armed
faults, and the fault injection code is executed.
This is illustrated in the example below from src/libpcp/src/ex‐
ec.c:
pid = fork();
/* begin fault-injection block */
PM_FAULT_POINT("libpcp/" __FILE__ ":4", PM_FAULT_MISC);
if (PM_FAULT_CHECK) {
PM_FAULT_CLEAR;
if (pid > (pid_t)0)
kill(pid, SIGKILL);
setoserror(EAGAIN);
pid = -1;
}
/* end fault-injection block */
A summary of fault points seen and faults injected is produced on
stdio stream f by __pmFaultSummary.
Additional tracing (via -Dfault or pmDebugOptions.fault) and a new
PMAPI error code (PM_ERR_FAULT) are also defined, although these
will only ever be seen or used in libpcp_fault. If pmDebugOp‐
tions.fault is set the first time __pmFaultInject is called, then
__pmFaultSummary will be called automatically to report on stderr
when the application exits (via atexit(3)).
Fault injection cannot be nested. Each call to __pmFaultInject
clears any previous fault injection that has been armed, but not
yet executed.
The fault injection infrastructure is not thread-safe and should
only be used with applications that are known to be single-thread‐
ed.
By default, no fault injection is enabled at run-time, even when
__pmFaultInject is called.
Faults are selectively enabled using a control file, identified by
the environment variable $PM_FAULT_CONTROL; if this is not set, no
faults are enabled.
The control file (if it exists) is read the first time __pmFault‐
Inject is called, and contains lines of the form:
ident op number
that define fault injection guards.
ident is a fault point string (as defined by a call to __pmFault‐
Inject, or more usually the PM_FAULT_POINT macro). So one needs
access to the libpcp source code to determine the available ident
strings and their semantics.
op is one of the C-style operators >=, >, ==, <, <=, != or % and
number is an unsigned integer. op number is optional and the de‐
fault is >0
The semantics of the fault injection guards are that each time
__pmFaultInject is called for a particular ident, a trip count is
incremented (the first trip is 1); if the C-style expression trip‐
count op number has the value 1 (so true for most ops, or the re‐
mainder equals 1 for the % op), then a fault of the class defined
for the fault point associated with ident will be armed, and exe‐
cuted as soon as possible.
Within the control file, blank lines are ignored and lines begin‐
ning with # are treated as comments.
For an existing application linked with libpcp fault injection may
still be used by forcing libpcp_fault to be used in the place of
libpcp. The following example shows how this might be done.
$ export PM_FAULT_CONTROL=/tmp/control
$ cat $PM_FAULT_CONTROL
# ok for 2 trips, then inject errors
libpcp/events.c:1 >2
$ export LD_PRELOAD=/usr/lib/libpcp_fault.so
$ pmevent -Dfault -s 3 sample.event.records
host: localhost
samples: 3
interval: 1.00 sec
sample.event.records[fungus]: 0 event records
__pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP
sample.event.records[bogus]: 2 event records
10:46:12.413 --- event record [0] flags 0x1 (point) ---
sample.event.param_string "fetch #0"
10:46:12.413 --- event record [1] flags 0x1 (point) ---
sample.event.param_string "bingo!"
__pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP
sample.event.records[fungus]: 1 event records
10:46:03.416 --- event record [0] flags 0x1 (point) ---
__pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT
sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
=== Fault Injection Summary Report ===
libpcp/events.c:1: guard trip>2, 5 trips, 3 faults
Refer to the PCP and PCP QA source code.
The macro definitions are in src/include/pcp/fault.h.
src/libpcp/src/fault.c contains all of the the underlying imple‐
mentation.
src/libpcp_fault and src/libpcp_fault/src contains the recipe and
Makefiles for creating and installing libpcp_fault.so and
<pcp/fault.h>.
PM_FAULT_RETURN was initiallly used in the following libpcp source
files: derive_parser.y.in, pdu.c and result.c.
PM_FAULT_POINT. was initiallly used in the following libpcp
source files: derive_parser.y.in, desc.c, e_indom.c, e_labels.c,
err.c, events.c, exec.c, fetch.c, help.c, instance.c, interp.c,
labels.c, logmeta.c, pmns.c, p_profile.c and store.c.
The ``fault'' group of QA tests show examples of control file use.
To see which tests are involved
$ cd qa
$ check -n -g fault
Some non-recoverable errors are reported on stderr.
PM_FAULT_CONTROL
Full path to the fault injection control file.
LD_PRELOAD
Force libpcp_fault to be used in preference to libpcp.
PMAPI(3)
This page is part of the PCP (Performance Co-Pilot) project. In‐
formation about the project can be found at ⟨http://www.pcp.io/⟩.
If you have a bug report for this manual page, send it to
pcp@groups.io. This page was obtained from the project's upstream
Git repository ⟨https://github.com/performancecopilot/pcp.git⟩ on
2025-08-11. (At that time, the date of the most recent commit
that was found in the repository was 2025-08-11.) If you discover
any rendering problems in this HTML version of the page, or you
believe there is a better or more up-to-date source for the page,
or you have corrections or improvements to the information in this
COLOPHON (which is not part of the original manual page), send a
mail to man-pages@man7.org
Performance Co-Pilot PMFAULT(3)
|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|