|
HTML rendering created 2025-02-02 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|
PMSERIES(1) General Commands Manual PMSERIES(1)
pmseries - display information about performance metric timeseries
pmseries [-adFiIlLmMnqsStvV?] [-c config] [-g pattern] [-h host]
[-p port] [-w window] [-Z timezone] [query | labels ... | series
... | source ... ]
pmseries displays various types of information about performance
metrics available through the scalable timeseries facilities of
the Performance Co-Pilot (PCP) and a distributed key-value data
store such as https://valkey/io/ ..
By default pmseries communicates with a local key-value server
however the -h and -p options can be used to specify an alternate
key-value server. If this instance is a node within a cluster,
all other instances in the cluster will be discovered and used
automatically.
pmseries runs in several different modes - either querying
timeseries identifiers, metadata or values (already stored in the
key-value store), or manually loading timeseries. The latter mode
is generally only used for loading previously collected (inactive)
archives, since pmproxy(1) automatically performs this function
for "live" (actively growing) local pmlogger(1) instances, when
running in its default time series mode. See the TIMESERIES
LOADING section below and the -L option for further details.
Without command line options specifying otherwise, pmseries will
issue a timeseries query to find matching timeseries and values.
All timeseries are identified using a unique SHA-1 hash which is
always displayed in a 40-hexdigit human readable form. These
hashes are formed using the metadata associated with every metric.
Importantly, this includes all metric metadata (labels, names,
descriptors). Metric labels in particular are (as far as
possible) unique for every machine - on Linux for example the
labels associated with every metric include the unique
/etc/machine-id, the hostname, domainname, and other automatically
generated machine labels, as well as any administrator-defined
labels from /etc/pcp/labels. These labels can be reported with
pminfo(1) and the pmcd.labels metric.
See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
pmLookupDesc(3) for detailed information about metric labels and
other metric metadata used in each timeseries identifier hash
calculation.
The timeseries identifiers provide a higher level (and machine
independent) identifier than the traditional PCP performance
metric identifiers (pmID), instance domain identifiers (pmInDom)
and metric names. See PCPIntro(1) for more details about these
traditional identifiers. However, pmseries uses timeseries
identifiers in much the same way that pminfo(1) uses the lower
level indom, metric identifiers and metric names.
The default mode of pmseries operation (i.e. with no command line
options) depends on the arguments it is presented. If all non-
option arguments appear to be timeseries identifiers (in 40 hex
digit form) pmseries will report metadata for these timeseries -
refer to the -a option for details. Otherwise, the parameters
will be treated as a timeseries query.
Query expressions are formed using the pmseries query language
described below, but can be as simple as a metric name.
The following is an example of querying timeseries from all hosts
that match a metric name pattern (globbed):
$ pmseries kernel.all.cpu*
1d7b0bb3f6aec0f49c54f5210885464a53629b60
379db729afd63fb9eff436625bd6c55a7adc5cfd
3dd3b45bb05f96636043e5d58b52b441ce542285
[...]
ed2bf325ff6dc7589ec966698e5404b67252306a
dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
To identify timeseries expression operands, the query language us‐
es the general syntax:
[metric.name] '{metadata qualifiers}' '[time specification]'
The metric.name component restricts the timeseries query to any
matching PCP metric name (the list of metric names for a PCP
archive or live host is reported by pminfo(1) with no arguments
beyond --host or --archive). The pmseries syntax extends on that
of pminfo and allows for glob(7) based pattern matching within the
metric name. The above describes operands available as the leaves
of pmseries expressions, which may include functions, arithmetic
operators and other features. See the EXPRESSIONS section below
for further details.
Metadata qualifiers are enclosed by ``curly'' braces ({}), and
further restrict the query results to timeseries operands with
various metadata properties. These qualifiers are based on metric
or instance names, and metric label values, and take the general
form metadata.name OPERATOR value, such as:
instance.name == "cpu0"
metric.name != "kernel.all.pswitch"
When using label names, the metadata qualifier is optional and can
be dropped, such as:
label.hostname == "www.acme.com"
hostname == "www.acme.com"
For metric and instance names only the string operators apply, but
for metric label values all operators are available. The set of
available operators is:
Boolean operators
All string (label, metrics and instances) and numeric (label) val‐
ues can be tested for equality ("==") or inequality ("!=").
String operators
Strings can be subject to pattern matching in the form of glob
matching ("~~"), regular expression matching ("=~"), and regular
expression non-matching ("!~"). The ":" operator is equivalent to
"~~" - i.e., glob matching.
Relational operators (numeric label values only)
Numeric label values can be subject to the less than ("<"),
greater than (">"), less than or equal ("<="), greater than or
equal (">="), equal ("==") and not equal ("!=") operators.
Logical operators
Multiple metadata qualifiers can be combined with the logical op‐
erators for AND ("&&") and OR ("||") as in many programming lan‐
guages. The comma (",") character is equivalent to logical AND
("&&").
The final (optional) component of a query allows the user to spec‐
ify a specific time window of interest. Any time specification
will result in values being returned for all matching timeseries
only for the time window specified.
The specification is ``square'' bracket ([]) enclosed, and con‐
sists of one or more comma-separated components. Each component
specifies some aspect related to time, taking the general form:
keyword: value, such as:
samples:10
Sample count
The number of samples to return, specified via either the samples
or (equivalent) count keyword. The value provided must be a posi‐
tive integer. If no end time is explicitly set (see ``Time win‐
dow'' later) then the most recent samples will be returned.
Sample interval
An interval between successive samples can be requested using the
interval or (equivalent) delta keyword. The value provided should
be either a numeric or string value that will be parsed by
pmParseInterval(3), such as 5 (seconds) or 2min (minutes).
Time window
Start and end times, and alignments, affecting the returned val‐
ues. The keywords match the parameters to the
pmParseTimeWindow(3) function which will be used to parse them,
and are: start or (equivalent) begin, finish or (equivalent) end,
align and offset.
Time zones
The resulting timestamps can be returned having been evaluated for
a specific timezone, using the timezone or hostzone keywords. The
value associated with timezone will be interpreted by
pmNewZone(3). A true or false value should be associated with
hostzone, and when set to true this has the same effect as de‐
scribed by pmNewContextZone(3).
As described above, operands are the leaves of a query expression
tree.
[metric.name] '{metadata qualifiers}' '[time specification]'
Note in most of the query expression examples below, the metadata
qualifiers have been omitted for brevity. In all cases, multiple
time series may qualify, particularly for the hostname label.
In the simple case, a query expression consists of a single
operand and may just be a metric name. In the more general case,
a query expression is either an operand or the argument to a func‐
tion, or two operands in a binary arithmetic or logical expres‐
sion. Most functions take a single argument (an expression),
though some require additional arguments, e.g. rescale.
operand | expr operator expr | func(expr[, arg])
This grammar shows expressions may be nested, e.g. using the addi‐
tion (+) operator as an example,
func1(func2(expr))
func1(expr) + func2(expr)
expr + func(expr)
func(expr) + expr
expr + expr
Rules governing compatibility of operands in an expression gener‐
ally depend on the function and/or operators and are described be‐
low individually. An important rule is that if any time windows
are specified, then all operands must cover the same number of
samples, though the time windows may differ individually. If no
time windows or sample counts are given, then pmseries will return
a series identifier (SID) instead of a series of timestamps and
values. This SID may be used in subsequent /series/values?se‐
ries=SID RESTAPI calls, along with a specific time window.
Arithmetic Operators
pmseries support addition, subtraction, division and multiplica‐
tion on each value in the time series of a binary pair of
operands. No unary or ternary operators are supported (yet). In
all cases, the instance domain and the number of samples of time
series operands must be the same. The metadata (units and dimen‐
sions) must also be compatible. Depending on the function, the
result will usually have the same instance domain and (unless not‐
ed otherwise), the same units as the operands. The metadata di‐
mensions (space, time, count) of the result may differ (see be‐
low).
Expression operands may have different qualifiers, e.g. you can
perform binary arithmetic on metrics qualified by different labels
(such as hostname), or metric names. For example, to add the two
most recent samples of the process context switch (pswitch)
counter metric for hosts node88 and node89, and then perform rate
conversion:
$ pmseries 'rate(kernel.all.pswitch{hostname:node88}[count:2] +
kernel.all.pswitch{hostname:node89}[count:2])'
1cf1a85d5978640ef94c68264d3ae8866cc11f7c
[Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924
Note the resulting time series of values has one less sample than
the expression operand passed to the rate function.
Other rules for arithmetic expressions:
1. if both operands have the semantics of a counter, then only ad‐
dition and subtraction are allowed
2. if the left operand is a counter and the right operand is not,
then only multiplication or division are allowed
3. if the left operand is not a counter and the right operand is a
counter, then only multiplication is allowed.
4. addition and subtraction - the dimensions of the result are the
same as the dimensions of the operands.
5. multiplication - the dimensions of the result are the sum of
the dimensions of the operands.
6. division - the dimensions of the result are the difference of
the dimensions of the operands.
Functions
Expression functions operate on vectors of time series values, and
may be nested with other functions or expressions as described
above. When an operand has multiple instances, there will gener‐
ally be one result for each series of instances. For example, the
result for
$ pmseries 'min_sample(kernel.all.load[count:100])'
will be the smallest value of the 100 most recent samples, treat‐
ing each of the three load average instances as a separate time
series. As an example, for the two most recent samples for each
of the three instances of the load average metric:
$ pmseries 'kernel.all.load[count:2]'
726a325c4c1ba4339ecffcdebd240f441ea77848
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
[Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
Using the min_sample function :
$ pmseries 'min_sample(kernel.all.load[count:2])'
11b965bc5f9598034ed9139fb3a78c6c0b7065ba
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
For singular metrics (with no instance domain), a single value
will result, e.g. for the five most recent samples of the context
switching metric:
$ pmseries 'kernel.all.pswitch[count:5]'
d7832c4fba33bcc980b1a1b614e0508043288480
[Tue Nov 10 12:44:59.380666000 2020] 460774294
[Tue Nov 10 12:44:49.382070000 2020] 460747232
[Tue Nov 10 12:44:39.378545000 2020] 460722370
[Tue Nov 10 12:44:29.379029000 2020] 460697388
[Tue Nov 10 12:44:19.379096000 2020] 460657412
$ pmseries 'min_sample(kernel.all.pswitch[count:5])'
1b6e92fb5bc012372f54452734dd03f0f131fa06
[Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480
Some pmseries functions provide operations across both time and
instances domain. For example, max_inst(expr) finds the maximum
value across instances while max_sample(expr) finds the maximum
value across time.
Future versions of pmseries may provide functions that perform ag‐
gregation, interpolation, filtering or transforms in other ways.
Function Reference
max_inst(expr) the maximum value in the time series for each in‐
stance of expr. For backwards compatibility, the synonym max is
equivalent to max_inst.
max_sample(expr) the maximum value in the time series for each
sample of expr across time.
min_inst(expr) the minimum value in the time series for each in‐
stance of expr. For backwards compatibility, the synonym min is
equivalent to min_inst.
min_sample(expr) the minimum value in the time series for each
sample of expr across time.
sum_inst(expr) sum of the values in the time series for each in‐
stance of expr. For backwards compatibility, the synonym sum is
equivalent to sum_inst.
sum_sample(expr) sum of the values in the time series for each
sample of expr across time.
avg_inst(expr) average of the values in the time series for each
instance of expr. For backwards compatibility, the synonym avg is
equivalent to avg_inst.
avg_sample(expr) average of the values in the time series for each
sample of expr across time.
rate(expr) the rate with respect to time of each sample. The giv‐
en expr must have counter semantics and the result will have in‐
stant semantics (the time dimension reduced by one). In addition,
the result will have one less sample than the operand - this is
because the first sample cannot be rate converted (two samples are
required).
rescale(expr,scale) rescale the values in the time series for each
instance of expr to scale (units). Note that expr should have in‐
stant or discrete semantics (not counter - rate conversion should
be done first if needed). The time, space and count dimensions
between expr and scale must be compatible. Example: rate convert
the read throughput counter for each disk instance and then
rescale to mbytes per second. Note the native units of
disk.dev.read_bytes is a counter of kbytes read from each device
instance since boot.
$ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'
abs(expr) the absolute value of each value in the time series for
each instance of expr. This has no effect if the type of expr is
unsigned.
floor(expr) rounded down to the nearest integer value of the time
series for each instance of expr.
round(expr) rounded up or down to the nearest integer for each
value in the time series for each instance of expr.
log(expr) logarithm of the values in the time series for each in‐
stance of expr
sqrt(expr) square root of the values in the time series for each
instance of expr
stdev_inst(expr) standard deviation of the values in the time se‐
ries for each instance of expr.
stdev_sample(expr) standard deviation of the values in the time
series for each sample of expr across time.
topk_inst(expr,k) the top k values in the time series for each in‐
stance of expr.
topk_sample(expr,k) the top k values in the time series for each
sample of expr across time.
nth_percentile_inst(expr,percentile_value) the nth percentile of
the values in the time series for each instance of expr. Note that
percentile_value has value in the range 0 to 100.
nth_percentile_sample(expr,percentile_value) the nth percentile of
the values in the time series for each sample of expr across time.
Note that percentile_value has value in the range 0 to 100.
Compatibility
All operands in an expression must have the same number of sam‐
ples, but not necessarily the same time window. e.g. you could
subtract some metric time series from today from that of yesterday
by giving different time windows and different metrics or quali‐
fiers, ensuring the same number of samples are given as the
operands.
Operands in an expression must either all have a time window, or
none. If no operands have a time window, then instead of a series
of time stamps and values, the result will be a time series iden‐
tifier (SID) that may be passed to the /series/values?series=SID
REST API function, along with a time window. For further details,
see PMWEBAPI(3).
If the semantics of both operands in an arithmetic expression are
not counter (i.e. PM_SEM_INSTANT or PM_SEM_DISCRETE) then the re‐
sult will have semantics PM_SEM_INSTANT unless both operands are
PM_SEM_DISCRETE in which case the result is also PM_SEM_DISCRETE.
Using command line options, pmseries can be requested to provide
metadata (metric names, instance names, labels, descriptors) asso‐
ciated with either individual timeseries or a group of timeseries,
for example:
$ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
PMID: 60.0.21
Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
Semantics: counter Units: millisec
Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
Metric: kernel.all.cpu.nice
labels {"agent":"linux","domainname":"localdomain",\
"groupid":1000,"hostname":"shard",\
"latitude":-25.28496,"longitude":152.87886,\
"machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
"userid":1000}
The complete set of pmseries metadata reporting options are:
-a, --all
Convenience option to report all metadata for the given time‐
series, equivalent to -deilms.
-d, --desc
Metric descriptions detailing the PMID, data type, data se‐
mantics, units, scale and associated instance domain. This
option has a direct pminfo(1) equivalent.
-F, --fast
Query or load series metadata only, not values.
-g pattern, --glob=pattern
Provide a glob(7) pattern to restrict the report provided by
the -i, -l, -m, and -S.
-i, --instances
Metric descriptions detailing the PMID, data type, data se‐
mantics, units, scale and associated instance domain.
-I, --fullindom
Print the InDom in verbose mode. This option has a direct
pminfo(1) equivalent.
-l, --labels
Print label sets associated with metrics and instances. La‐
bels are optional metric metadata described in detail in
pmLookupLabels(3). This option has a direct pminfo(1) equiv‐
alent.
-m, --metrics
Print metric names.
-M, --fullpmid
Print the PMID in verbose mode. This option has a direct
pminfo(1) equivalent.
-n, --names
Print comma-separated label names only (not values) for the
labels associated with metrics and instances.
-s, --series
Print timeseries identifiers associated with metrics, in‐
stances and sources. These unique identifiers are calculated
from intrinsic (non-optional) labels and other metric metada‐
ta associated with each PMAPI context (sources), metrics and
instances. Archive, local context or pmcd(1) connections for
the same host all produce the same source identifier. This
option has a direct pminfo(1) equivalent. See also
pmLookupLabels(3) and the -l/--labels option.
A source is a unique identifier (represented externally as a
40-byte hexadecimal SHA-1 hash) that represents both the live host
and/or archives from which each timeseries originated. The con‐
text for a source identifier (obtained with -s) can be reported
with:
-S, --sources
Print names for timeseries sources. These names are either
hostnames or fully qualified archive paths.
It is important to note that live and archive sources can and will
generate the same SHA-1 source identifier hash, provided that the
context labels remain the same for that host (labels are stored in
PCP archives and can also be fetched live from pmcd(1)).
Timeseries metadata and data are loaded either automatically by a
local pmproxy(1), or manually using a specially crafted pmseries
query and the -L/--load option:
$ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
pmseries: [Info] processed 2275 archive records from [...]
This query must specify a source archive path, but can also re‐
strict the import to specific timeseries (using metric names, la‐
bels, etc) and to a specific time window using the time specifica‐
tion component of the query language.
As a convenience, if the argument to load is a valid file path as
determined by access(2), then a short-hand form can be used:
$ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
NOTE: Timeseries loading is append-only (timestamp-wise) and if
more than stream.maxlen entries (defined in $PCP_SYSCONF_DIR/pm‐
series/pmseries.conf) are loaded for a given metric, the oldest
entries are dropped.
The available command line options, in addition to timeseries
metadata and sources options described above, are:
-c config, --config=config
Specify the config file to use.
-h host, --host=host
Connect to the key-value server at host, rather than the one
the localhost.
-L, --load
Load timeseries metadata and data into the key-value cluster.
-p port, --port=port
Connect to the key-value server at port, rather than the de‐
fault 6379.
-q, --query
Perform a timeseries query. This is the default action.
-t, --times
Report time stamps numerically (in milliseconds) instead of
the default human readable form.
-v, --values
Report all of the known values for given label name(s), or
report values for the given series identifiers.
-w, --window
Provide a time specification that will be applied to values
being returned when returning values via use of series iden‐
tifiers (i.e. when not using a query string). The time spec‐
ification uses the same square-bracket enclosed form de‐
scribed earlier in the ``TIME SPECIFICATION'' section.
-V, --version
Display version number and exit.
-Z timezone, --timezone=timezone
Use timezone for the date and time. Timezone is in the for‐
mat of the environment variable TZ as described in
environ(7).
-?, --help
Display usage message and exit.
The following sample query shows several fundamental aspects of
the pmseries query language:
$ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'
eb713a9cf472f775aa59ae90c43cd7f960f7870f
[Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
[Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
[Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
[Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
[Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
[Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
This query returns the two most recent values for all instances of
the kernel.all.load metric with a label.hostname matching the reg‐
ular expression "toium". This is a set-valued metric (i.e., a
metric with an ``instance domain'' which in this case consists of
three instances: 1, 5 and 15 minute averages). The first column
returned is a timestamp, then a floating point value, and finally
an instance identifier timeseries hash (two values returned for
three instances, so six rows are returned). The metadata for
these timeseries can then be further examined:
$ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
eb713a9cf472f775aa59ae90c43cd7f960f7870f
PMID: 60.2.0
Data Type: float InDom: 60.2 0xf000002
Semantics: instant Units: none
Source: 0e89c1192db79326900d82131c31399524f0b3ee
Metric: kernel.all.load
inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
Environment variables with the prefix PCP_ are used to parameter‐
ize the file and directory names used by PCP. On each installa‐
tion, the file /etc/pcp.conf contains the local values for these
variables. The $PCP_CONF variable may be used to specify an al‐
ternative configuration file, as described in pcp.conf(5).
For environment variables affecting PCP tools, see
pmGetOptions(3).
PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), access(2), PMAPI(3),
PMWEBAPI(3), pmLookupDesc(3), pmLookupInDom(3), pmLookupLabels(3),
pmLookupName(3), pmNewContextZone(3), pmNewZone(3),
pmParseInterval(3), pmParseTimeWindow(3), pcp.conf(5), environ(7),
glob(7) and regex(7).
This page is part of the PCP (Performance Co-Pilot) project. In‐
formation about the project can be found at ⟨http://www.pcp.io/⟩.
If you have a bug report for this manual page, send it to
pcp@groups.io. This page was obtained from the project's upstream
Git repository ⟨https://github.com/performancecopilot/pcp.git⟩ on
2025-02-02. (At that time, the date of the most recent commit
that was found in the repository was 2025-01-30.) If you discover
any rendering problems in this HTML version of the page, or you
believe there is a better or more up-to-date source for the page,
or you have corrections or improvements to the information in this
COLOPHON (which is not part of the original manual page), send a
mail to man-pages@man7.org
Performance Co-Pilot PCP PMSERIES(1)
Pages that refer to this page: pcpcompat(1), pcp-geolocate(1), pmfind(1), pmfind_check(1), pmproxy(1), pmdiscoversetup(3), pmseriesdescs(3), pmseriesquery(3), pmseriessetup(3), pmwebapi(3), labels.conf(5)
|
HTML rendering created 2025-02-02 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|