pmseries(1) — Linux manual page


PMSERIES(1)              General Commands Manual             PMSERIES(1)

NAME         top

       pmseries - display information about performance metric

SYNOPSIS         top

       pmseries [-adFiIlLmMnqsStvV?]  [-c config] [-g pattern] [-h host]
       [-p port] [-Z timezone] [query | labels ... | series ... | source
       ... ]

DESCRIPTION         top

       pmseries displays various types of information about performance
       metrics available through the scalable timeseries facilities of
       the Performance Co-Pilot (PCP) and the Redis distributed data

       By default pmseries communicates with a local redis-server(1),
       however the -h and -p options can be used to specify an alternate
       Redis instance.  If this instance is a node of a Redis cluster,
       all other instances in the cluster will be discovered and used

       pmseries runs in several different modes - either querying
       timeseries identifiers, metadata or values (already stored in
       Redis), or manually loading timeseries into Redis.  The latter
       mode is seldom used, however, since pmproxy(1) will automatically
       perform this function for local pmlogger(1) instances, when
       running in its default time series mode.

       Without command line options specifying otherwise, pmseries will
       issue a timeseries query to find matching timeseries and values.
       All timeseries are identified using a unique SHA-1 hash which is
       always displayed in a 40-hexdigit human readable form.  These
       hashes are formed using the metadata associated with every

       Importantly, this includes all metric metadata (labels, names,
       descriptors).  Metric labels in particular are (as far as
       possible) unique for every machine - on Linux for example the
       labels associated with every metric include the unique
       /etc/machine-id, the hostname, domainname, and other
       automatically generated machine labels, as well as any
       administrator-defined labels from /etc/pcp/labels.  These labels
       can be reported with pminfo(1) and the pmcd.labels metric.

       See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
       pmLookupDesc(3) for detailed information about metric labels and
       other metric metadata used in each timeseries identifier hash

       The timeseries identifiers provide a higher level (and machine
       independent) identifier than the traditional PCP performance
       metric identifiers (pmID), instance domain identifiers (pmInDom)
       and metric names.  See PCPIntro(1) for more details about these
       traditional identifiers.  However, pmseries uses timeseries
       identifiers in much the same way that pminfo(1) uses the lower
       level indom, metric identifiers and metric names.

       The default mode of pmseries operation (i.e. with no command line
       options) depends on the arguments it is presented.  If all non-
       option arguments appear to be timeseries identifiers (in 40 hex
       digit form) pmseries will report metadata for these timeseries -
       refer to the -a option for details.  Otherwise, the parameters
       will be treated as a timeseries query.


       Query expressions are formed using the pmseries query language
       described below, but can be as simple as a metric name.

       The following is an example of querying timeseries from all hosts
       that match a metric name pattern (globbed):

         $ pmseries kernel.all.cpu*

       To identify timeseries expression operands, the query language
       uses the general syntax:

         [] '{metadata qualifiers}' '[time specification]'

       The component restricts the timeseries query to any
       matching PCP metric name (the list of metric names for a PCP
       archive or live host is reported by pminfo(1) with no arguments
       beyond --host or --archive).  The pmseries syntax extends on that
       of pminfo and allows for glob(7) based pattern matching within
       the metric name.  The above describes operands available as the
       leaves of pmseries expressions, which may include functions,
       arithmetic operators and other features.  See the EXPRESSIONS
       section below for further details.


       Metadata qualifiers are enclosed by ``curly'' braces ({}), and
       further restrict the query results to timeseries operands with
       various metadata properties.  These qualifiers are based on
       metric or instance names, and metric label values, and take the
       general form OPERATOR value, such as:
 == "cpu0" != "kernel.all.pswitch"

       When using label names, the metadata qualifier is optional and
       can be dropped, such as:

         label.hostname == ""
         hostname == ""

       For metric and instance names only the string operators apply,
       but for metric label values all operators are available.  The set
       of available operators is:

   Boolean operators
       All string (label, metrics and instances) and numeric (label)
       values can be tested for equality ("==") or inequality ("!=").

   String operators
       Strings can be subject to pattern matching in the form of glob
       matching ("~~"), regular expression matching ("=~"), and regular
       expression non-matching ("!~").  The ":" operator is equivalent
       to "~~" - i.e., glob matching.

   Relational operators (numeric label values only)
       Numeric label values can be subject to the less than ("<"),
       greater than (">"), less than or equal ("<="), greater than or
       equal (">="), equal ("==") and not equal ("!=") operators.

   Logical operators
       Multiple metadata qualifiers can be combined with the logical
       operators for AND ("&&") and OR ("||") as in many programming
       languages.  The comma (",") character is equivalent to logical
       AND ("&&").


       The final (optional) component of a query allows the user to
       specify a specific time window of interest.  Any time
       specification will result in values being returned for all
       matching timeseries only for the time window specified.

       The specification is ``square'' bracket ([]) enclosed, and
       consists of one or more comma-separated components.  Each
       component specifies some aspect related to time, taking the
       general form: keyword: value, such as:


   Sample count
       The number of samples to return, specified via either the samples
       or (equivalent) count keyword.  The value provided must be a
       positive integer.  If no end time is explicitly set (see ``Time
       window'' later) then the most recent samples will be returned.

   Sample interval
       An interval between successive samples can be requested using the
       interval or (equivalent) delta keyword.  The value provided
       should be either a numeric or string value that will be parsed by
       pmParseInterval(3), such as 5 (seconds) or 2min (minutes).

   Time window
       Start and end times, and alignments, affecting the returned
       values.  The keywords match the parameters to the
       pmParseTimeWindow(3) function which will be used to parse them,
       and are: start or (equivalent) begin, finish or (equivalent) end,
       align and offset.

   Time zones
       The resulting timestamps can be returned having been evaluated
       for a specific timezone, using the timezone or hostzone keywords.
       The value associated with timezone will be interpreted by
       pmNewZone(3).  A true or false value should be associated with
       hostzone, and when set to true this has the same effect as
       described by pmNewContextZone(3).

EXPRESSIONS         top

       As described above, operands are the leaves of a query expression

         [] '{metadata qualifiers}' '[time specification]'
       Note in most of the query expression examples below, the metadata
       qualifiers have been omitted for brevity.  In all cases, multiple
       time series may qualify, particularly for the hostname label.

       In the simple case, a query expression consists of a single
       operand and may just be a metric name.  In the more general case,
       a query expression is either an operand or the argument to a
       function, or two operands in a binary arithmetic or logical
       expression.  Most functions take a single argument (an
       expression), though some require additional arguments, e.g.

         operand | expr operator expr | func(expr[, arg])

       This grammar shows expressions may be nested, e.g. using the
       addition (+) operator as an example,

         func1(expr) + func2(expr)
         expr + func(expr)
         func(expr) + expr
         expr + expr

       Rules governing compatibility of operands in an expression
       generally depend on the function and/or operators and are
       described below individually.  An important rule is that if any
       time windows are specified, then all operands must cover the same
       number of samples, though the time windows may differ
       individually.  If no time windows or sample counts are given,
       then pmseries will return a series identifier (SID) instead of a
       series of timestamps and values.  This SID may be used in
       subsequent /series/values?series=SID RESTAPI calls, along with a
       specific time window.

   Arithmetic Operators
       pmseries support addition, subtraction, division and
       multiplication on each value in the time series of a binary pair
       of operands.  No unary or ternary operators are supported (yet).
       In all cases, the instance domain and the number of samples of
       time series operands must be the same.  The metadata (units and
       dimensions) must also be compatible.  Depending on the function,
       the result will usually have the same instance domain and (unless
       noted otherwise), the same units as the operands.  The metadata
       dimensions (space, time, count) of the result may differ (see

       Expression operands may have different qualifiers, e.g. you can
       perform binary arithmetic on metrics qualified by different
       labels (such as hostname), or metric names.  For example, to add
       the two most recent samples of the process context switch
       (pswitch) counter metric for hosts node88 and node89, and then
       perform rate conversion:

         $ pmseries 'rate(kernel.all.pswitch{hostname:node88}[count:2] +
             [Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924

       Note the resulting time series of values has one less sample than
       the expression operand passed to the rate function.

       Other rules for arithmetic expressions:

       1. if both operands have the semantics of a counter, then only
       addition and subtraction are allowed

       2. if the left operand is a counter and the right operand is not,
       then only multiplication or division are allowed

       3. if the left operand is not a counter and the right operand is
       a counter, then only multiplication is allowed.

       4. addition and subtraction - the dimensions of the result are
       the same as the dimensions of the operands.

       5. multiplication - the dimensions of the result are the sum of
       the dimensions of the operands.

       6. division - the dimensions of the result are the difference of
       the dimensions of the operands.

       Expression functions operate on vectors of time series values,
       and may be nested with other functions or expressions as
       described above.  When an operand has multiple instances, there
       will generally be one result for each series of instances.  For
       example, the result for

         $ pmseries 'min(kernel.all.load[count:100])'

       will be the smallest value of the 100 most recent samples,
       treating each of the three load average instances as a separate
       time series.  As an example, for the two most recent samples for
       each of the three instances of the load average metric:

         $ pmseries 'kernel.all.load[count:2]'
             [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
             [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
             [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
             [Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
             [Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
             [Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133

       Using the min function :

         $ pmseries 'min(kernel.all.load[count:2])'
             [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
             [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
             [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133

       For singular metrics (with no instance domain), a single value
       will result, e.g. for the five most recent samples of the context
       switching metric:

         $ pmseries 'kernel.all.pswitch[count:5]'
             [Tue Nov 10 12:44:59.380666000 2020] 460774294
             [Tue Nov 10 12:44:49.382070000 2020] 460747232
             [Tue Nov 10 12:44:39.378545000 2020] 460722370
             [Tue Nov 10 12:44:29.379029000 2020] 460697388
             [Tue Nov 10 12:44:19.379096000 2020] 460657412

         $ pmseries 'min(kernel.all.pswitch[count:5])'
             [Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480

       Future versions of pmseries may provide functions that perform
       aggregation, interpolation, filtering or transforms in other
       ways, e.g. across instances instead of time.

   Function Reference
       max(expr) the maximum value in the time series for each instance
       of expr

       min(expr) the minimum value in the time series for each instance
       of expr

       rate(expr) the rate with respect to time of each sample.  The
       given expr must have counter semantics and the result will have
       instant semantics (the time dimension reduced by one).  In
       addition, the result will have one less sample than the operand -
       this is because the first sample cannot be rate converted (two
       samples are required).

       rescale(expr,scale) rescale the values in the time series for
       each instance of expr to scale (units).  Note that expr should
       have instant or discrete semantics (not counter - rate conversion
       should be done first if needed).  The time, space and count
       dimensions between expr and scale must be compatible.  Example:
       rate convert the read throughput counter for each disk instance
       and then rescale to mbytes per second.  Note the native units of is a counter of kbytes read from each device
       instance since boot.

         $ pmseries 'rescale(rate([count:4]), "mbytes/s")'

       abs(expr) the absolute value of each value in the time series for
       each instance of expr.  This has no effect if the type of expr is

       floor(expr) rounded down to the nearest integer value of the time
       series for each instance of expr.

       round(expr) rounded up or down to the nearest integer for each
       value in the time series for each instance of expr.

       log(expr) logarithm of the values in the time series for each
       instance of expr

       sqrt(expr) square root of the values in the time series for each
       instance of expr

       All operands in an expression must have the same number of
       samples, but not necessarily the same time window. e.g. you could
       subtract some metric time series from today from that of
       yesterday by giving different time windows and different metrics
       or qualifiers, ensuring the same number of samples are given as
       the operands.

       Operands in an expression must either all have a time window, or
       none.  If no operands have a time window, then instead of a
       series of time stamps and values, the result will be a time
       series identifier (SID) that may be passed to the
       /series/values?series=SID REST API function, along with a time
       window.  For further details, see PMWEBAPI(3).

       If the semantics of both operands in an arithmetic expression are
       not counter (i.e. PM_SEM_INSTANT or PM_SEM_DISCRETE) then the
       result will have semantics PM_SEM_INSTANT unless both operands
       are PM_SEM_DISCRETE in which case the result is also


       Using command line options, pmseries can be requested to provide
       metadata (metric names, instance names, labels, descriptors)
       associated with either individual timeseries or a group of
       timeseries, for example:

         $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19

             PMID: 60.0.21
             Data Type: 64-bit unsigned int  InDom: PM_INDOM_NULL 0xffffffff
             Semantics: counter  Units: millisec
             Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
             Metric: kernel.all.cpu.nice
             labels {"agent":"linux","domainname":"localdomain",\

       The complete set of pmseries metadata reporting options are:

       -a, --all
            Convenience option to report all metadata for the given
            timeseries, equivalent to -deilms.

       -d, --desc
            Metric descriptions detailing the PMID, data type, data
            semantics, units, scale and associated instance domain.
            This option has a direct pminfo(1) equivalent.

       -F, --fast
            Query or load series metadata only, not values.

       -g pattern, --glob=pattern
            Provide a glob(7) pattern to restrict the report provided by
            the -i, -l, -m, and -S.

       -i, --instances
            Metric descriptions detailing the PMID, data type, data
            semantics, units, scale and associated instance domain.

       -I, --fullindom
            Print the InDom in verbose mode.  This option has a direct
            pminfo(1) equivalent.

       -l, --labels
            Print label sets associated with metrics and instances.
            Labels are optional metric metadata described in detail in
            pmLookupLabels(3).  This option has a direct pminfo(1)

       -m, --metrics
            Print metric names.

       -M, --fullpmid
            Print the PMID in verbose mode.  This option has a direct
            pminfo(1) equivalent.

       -n, --names
            Print comma-separated label names only (not values) for the
            labels associated with metrics and instances.

       -s, --series
            Print timeseries identifiers associated with metrics,
            instances and sources.  These unique identifiers are
            calculated from intrinsic (non-optional) labels and other
            metric metadata associated with each PMAPI context
            (sources), metrics and instances.  Archive, local context or
            pmcd(1) connections for the same host all produce the same
            source identifier.  This option has a direct pminfo(1)
            equivalent.  See also pmLookupLabels(3) and the -l/--labels


       A source is a unique identifier (represented externally as a
       40-byte hexadecimal SHA-1 hash) that represents both the live
       host and/or archives from which each timeseries originated.  The
       context for a source identifier (obtained with -s) can be
       reported with:

       -S, --sources
            Print names for timeseries sources.  These names are either
            hostnames or fully qualified archive paths.

       It is important to note that live and archived sources can and
       will generate the same SHA-1 source identifier hash, provided
       that the context labels remain the same for that host (labels are
       stored in PCP archives and can also be fetched live from


       Timeseries metadata and data are loaded either automatically by a
       local pmproxy(1), or manually using a specially crafted pmseries
       query and the -L/--load option:

         $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
         pmseries: [Info] processed 2275 archive records from [...]

       This query must specify a source archive path, but can also
       restrict the import to specific timeseries (using metric names,
       labels, etc) and to a specific time window using the time
       specification component of the query language.

       As a convenience, if the argument to load is a valid file path as
       determined by access(2), then a short-hand form can be used:

         $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0

OPTIONS         top

       The available command line options, in addition to timeseries
       metadata and sources options described above, are:

       -c config, --config=config
            Specify the config file to use.

       -h host, --host=host
            Connect Redis server at host, rather than the one the

       -L, --load
            Load timeseries metadata and data into the Redis cluster.

       -p port, --port=port
            Connect Redis server at port, rather than the default 6379.

       -q, --query
            Perform a timeseries query.  This is the default action.

       -t, --times
            Report time stamps numerically (in milliseconds) instead of
            the default human readable form.

       -v, --values
            Report all of the known values for given label name(s).

       -V, --version
            Display version number and exit.

       -Z timezone, --timezone=timezone
            Use timezone for the date and time.  Timezone is in the
            format of the environment variable TZ as described in

       -?, --help
            Display usage message and exit.

EXAMPLES         top

       The following sample query shows several fundamental aspects of
       the pmseries query language:

         $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'

             [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
             [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480

       This query returns the two most recent values for all instances
       of the kernel.all.load metric with a label.hostname matching the
       regular expression "toium".  This is a set-valued metric (i.e., a
       metric with an ``instance domain'' which in this case consists of
       three instances: 1, 5 and 15 minute averages).  The first column
       returned is a timestamp, then a floating point value, and finally
       an instance identifier timeseries hash (two values returned for
       three instances, so six rows are returned).  The metadata for
       these timeseries can then be further examined:

         $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f

             PMID: 60.2.0
             Data Type: float  InDom: 60.2 0xf000002
             Semantics: instant  Units: none
             Source: 0e89c1192db79326900d82131c31399524f0b3ee
             Metric: kernel.all.load
             inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
             inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
             inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
             inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}


       Environment variables with the prefix PCP_ are used to
       parameterize the file and directory names used by PCP.  On each
       installation, the file /etc/pcp.conf contains the local values
       for these variables.  The $PCP_CONF variable may be used to
       specify an alternative configuration file, as described in

       For environment variables affecting PCP tools, see

SEE ALSO         top

       PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), redis-server(1),
       access(2), PMAPI(3), PMWEBAPI(3), pmLookupDesc(3),
       pmLookupInDom(3), pmLookupLabels(3), pmLookupName(3),
       pmNewContextZone(3), pmNewZone(3), pmParseInterval(3),
       pmParseTimeWindow(3), pcp.conf(5), environ(7), glob(7) and

COLOPHON         top

       This page is part of the PCP (Performance Co-Pilot) project.
       Information about the project can be found at 
       ⟨⟩.  If you have a bug report for this manual
       page, send it to  This page was obtained from the
       project's upstream Git repository
       ⟨⟩ on 2021-08-27.
       (At that time, the date of the most recent commit that was found
       in the repository was 2021-08-27.)  If you discover any rendering
       problems in this HTML version of the page, or you believe there
       is a better or more up-to-date source for the page, or you have
       corrections or improvements to the information in this COLOPHON
       (which is not part of the original manual page), send a mail to

Performance Co-Pilot               PCP                       PMSERIES(1)

Pages that refer to this page: pcpcompat(1)pmfind(1)pmfind_check(1)pmproxy(1)pmdiscoversetup(3)pmseriesdescs(3)pmseriesquery(3)pmseriessetup(3)pmwebapi(3)labels.conf(5)