pmseries(1) — Linux manual page


PMSERIES(1)                General Commands Manual               PMSERIES(1)

NAME         top

       pmseries - display information about performance metric timeseries

SYNOPSIS         top

       pmseries [-adFiIlLmMnqsStvV?]  [-c config] [-g pattern] [-h host] [-p
       port] [-Z timezone] [query | labels ... | series ... | source ... ]

DESCRIPTION         top

       pmseries displays various types of information about performance
       metrics available through the scalable timeseries facilities of the
       Performance Co-Pilot (PCP) and the Redis distributed data store.

       By default pmseries communicates with a local redis-server(1),
       however the -h and -p options can be used to specify an alternate
       Redis instance.  If this instance is a node of a Redis cluster, all
       other instances in the cluster will be discovered and used

       pmseries runs in several different modes - either querying timeseries
       identifiers, metadata or values (already stored in Redis), or
       manually loading timeseries into Redis.  The latter mode is seldom
       used, however, since pmproxy(1) will automatically perform this
       function for local pmlogger(1) instances, when running in its default
       time series mode.

       Without command line options specifying otherwise, pmseries will
       issue a timeseries query to find matching timeseries and values.  All
       timeseries are identified using a unique SHA-1 hash which is always
       displayed in a 40-hexdigit human readable form.  These hashes are
       formed using the metadata associated with every metric.

       Importantly, this includes all metric metadata (labels, names,
       descriptors).  Metric labels in particular are (as far as possible)
       unique for every machine - on Linux for example the labels associated
       with every metric include the unique /etc/machine-id, the hostname,
       domainname, and other automatically generated machine labels, as well
       as any administrator-defined labels from /etc/pcp/labels.  These
       labels can be reported with pminfo(1) and the pmcd.labels metric.

       See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
       pmLookupDesc(3) for detailed information about metric labels and
       other metric metadata used in each timeseries identifier hash

       The timeseries identifiers provide a higher level (and machine
       independent) identifier than the traditional PCP performance metric
       identifiers (pmID), instance domain identifiers (pmInDom) and metric
       names.  See PCPIntro(1) for more details about these traditional
       identifiers.  However, pmseries uses timeseries identifiers in much
       the same way that pminfo(1) uses the lower level indom, metric
       identifiers and metric names.

       The default mode of pmseries operation (i.e. with no command line
       options) depends on the arguments it is presented.  If all non-option
       arguments appear to be timeseries identifiers (in 40 hex digit form)
       pmseries will report metadata for these timeseries - refer to the -a
       option for details.  Otherwise, the parameters will be treated as a
       timeseries query.


       Query expressions are formed using the pmseries query language
       described below, but can be as simple as a metric name.

       The following is an example of querying timeseries from all hosts
       that match a metric name pattern (globbed):

         $ pmseries kernel.all.cpu*

       To identify timeseries, the query language uses the general syntax:

         [] '{metadata qualifiers}' '[time specification]'

       The component restricts the timeseries query to any
       matching PCP metric name (the list of metric names for a PCP archive
       or live host is reported by pminfo(1) with no arguments beyond --host
       or --archive).  The pmseries syntax extends on that of pminfo and al‐
       lows for glob(7) based pattern matching within the metric name.


       Metadata qualifiers are enclosed by ``curly'' braces ({}), and
       further restrict the query results to timeseries with various
       metadata properties.  These qualifiers are based on metric or
       instance names, and metric label values, and take the general form OPERATOR value, such as:
 == "cpu0" != "kernel.all.pswitch"

       When using label names, the metadata qualifier is optional and can be
       dropped, such as:

         label.hostname == ""
         hostname == ""

       For metric and instance names only the string operators apply, but
       for metric label values all operators are available.  The set of
       available operators is:

   Boolean operators
       All string (label, metrics and instances) and numeric (label) values
       can be tested for equality ("==") or inequality ("!=").

   String operators
       Strings can be subject to pattern matching in the form of glob match‐
       ing ("~~"), regular expression matching ("=~"), and regular expres‐
       sion non-matching ("!~").  The ":" operator is equivalent to "~~" -
       i.e., regular expression matching.

   Relational operators (numeric label values only)
       Numeric label values can be subject to the less than ("<"), greater
       than (">"), less than or equal ("<="), greater than or equal (">="),
       equal ("==") and not equal ("!=") operators.

   Logical operators
       Multiple metadata qualifiers can be combined with the logical opera‐
       tors for AND ("&&") and OR ("||") as in many programming languages.
       The comma (",") character is equivalent to logical AND ("&&").


       The final (optional) component of a query allows the user to specify
       a specific time window of interest.  Any time specification will
       result in values being returned for all matching timeseries only for
       the time window specified.

       The specification is ``square'' bracket ([]) enclosed, and consists
       of one or more comma-separated components.  Each component specifies
       some aspect related to time, taking the general form: keyword: value,
       such as:

         samples: 10

   Sample count
       The number of samples to return, specified via either the samples or
       (equivalent) count keyword.  The value provided must be a positive
       integer.  If no end time is explicitly set (see ``Time window'' lat‐
       er) then the most recent samples will be returned.

   Sample interval
       An interval between successive samples can be requested using the in‐
       terval or (equivalent) delta keyword.  The value provided should be
       either a numeric or string value that will be parsed by
       pmParseInterval(3), such as 5 (seconds) or 2min (minutes).

   Time window
       Start and end times, and alignments, affecting the returned values.
       The keywords match the parameters to the pmParseTimeWindow(3) func‐
       tion which will be used to parse them, and are: start or (equivalent)
       begin, finish or (equivalent) end, align and offset.

   Time zones
       The resulting timestamps can be returned having been evaluated for a
       specific timezone, using the timezone or hostzone keywords.  The val‐
       ue associated with timezone will be interpreted by pmNewZone(3).  A
       true or false value should be associated with hostzone, and when set
       to true this has the same effect as described by pmNewContextZone(3).


       Using command line options, pmseries can be requested to provide
       metadata (metric names, instance names, labels, descriptors)
       associated with either individual timeseries or a group of
       timeseries, for example:

         $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19

             PMID: 60.0.21
             Data Type: 64-bit unsigned int  InDom: PM_INDOM_NULL 0xffffffff
             Semantics: counter  Units: millisec
             Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
             Metric: kernel.all.cpu.nice
             labels {"agent":"linux","domainname":"localdomain",\

       The complete set of pmseries metadata reporting options are:

       -a, --all
            Convenience option to report all metadata for the given time‐
            series, equivalent to -dilms.

       -d, --desc
            Metric descriptions detailing the PMID, data type, data seman‐
            tics, units, scale and associated instance domain.  This option
            has a direct pminfo(1) equivalent.

       -g pattern, --glob=pattern
            Provide a glob(7) pattern to restrict the report provided by the
            -i, -l, -m, and -S.

       -i, --instances
            Metric descriptions detailing the PMID, data type, data seman‐
            tics, units, scale and associated instance domain.

       -I, --fullindom
            Print the InDom in verbose mode.  This option has a direct
            pminfo(1) equivalent.

       -l, --labels
            Print label sets associated with metrics and instances.  Labels
            are optional metric metadata described in detail in
            pmLookupLabels(3).  This option has a direct pminfo(1) equiva‐

       -m, --metrics
            Print metric names.

       -M, --fullpmid
            Print the PMID in verbose mode.  This option has a direct
            pminfo(1) equivalent.

       -n, --names
            Print comma-separated label names only (not values) for the la‐
            bels associated with metrics and instances.

       -s, --series
            Print timeseries identifiers associated with metrics, instances
            and sources.  These unique identifiers are calculated from in‐
            trinsic (non-optional) labels and other metric metadata associ‐
            ated with each PMAPI context (sources), metrics and instances.
            Archive, local context or pmcd(1) connections for the same host
            all produce the same source identifier.  This option has a di‐
            rect pminfo(1) equivalent.  See also pmLookupLabels(3) and the
            -l/--labels option.


       A source is a unique identifier (represented externally as a 40-byte
       hexadecimal SHA-1 hash) that represents both the live host and/or
       archives from which each timeseries originated.  The context for a
       source identifier (obtained with -s) can be reported with:

       -S, --sources
            Print names for timeseries sources.  These names are either
            hostnames or fully qualified archive paths.

       It is important to note that live and archived sources can and will
       generate the same SHA-1 source identifier hash, provided that the
       context labels remain the same for that host (labels are stored in
       PCP archives and can also be fetched live from pmcd(1)).


       Timeseries metadata and data are loaded either automatically by a
       local pmproxy(1), or manually using a specially crafted pmseries
       query and the -L/--load option:

         $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
         pmseries: [Info] processed 2275 archive records from [...]

       This query must specify a source archive path, but can also restrict
       the import to specific timeseries (using metric names, labels, etc)
       and to a specific time window using the time specification component
       of the query language.

       As a convenience, if the argument to load is a valid file path as de‐
       termined by access(2), then a short-hand form can be used:

         $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0

OPTIONS         top

       The available command line options, in addition to timeseries
       metadata and sources options described above, are:

       -c config, --config=config
            Specify the config file to use.

       -h host, --host=host
            Connect Redis server at host, rather than the one the localhost.

       -L, --load
            Load timeseries metadata and data into the Redis cluster.

       -p port, --port=port
            Connect Redis server at port, rather than the default 6379.

       -q, --query
            Perform a timeseries query.  This is the default action.

       -t, --times
            Report time stamps numerically (in milliseconds) instead of the
            default human readable form.

       -v, --values
            Report all of the known values for given label name(s).

       -V, --version
            Display version number and exit.

       -Z timezone, --timezone=timezone
            Use timezone for the date and time.  Timezone is in the format
            of the environment variable TZ as described in environ(7).

       -?, --help
            Display usage message and exit.

EXAMPLES         top

       The following sample query shows several fundamental aspects of the
       pmseries query language:

         $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'

             [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
             [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480

       This query returns the two most recent values for all instances of
       the kernel.all.load metric with a label.hostname matching the regular
       expression "toium".  This is a set-valued metric (i.e., a metric with
       an ``instance domain'' which in this case consists of three in‐
       stances: 1, 5 and 15 minute averages).  The first column returned is
       a timestamp, then a floating point value, and finally an instance
       identifier timeseries hash (two values returned for three instances,
       so six rows are returned).  The metadata for these timeseries can
       then be further examined:

         $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f

             PMID: 60.2.0
             Data Type: float  InDom: 60.2 0xf000002
             Semantics: instant  Units: none
             Source: 0e89c1192db79326900d82131c31399524f0b3ee
             Metric: kernel.all.load
             inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
             inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
             inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
             inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}


       Environment variables with the prefix PCP_ are used to parameterize
       the file and directory names used by PCP.  On each installation, the
       file /etc/pcp.conf contains the local values for these variables.
       The $PCP_CONF variable may be used to specify an alternative
       configuration file, as described in pcp.conf(5).

       For environment variables affecting PCP tools, see pmGetOptions(3).

SEE ALSO         top

       PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), redis-server(1),
       access(2), PMAPI(3), PMWEBAPI(3), pmLookupDesc(3), pmLookupInDom(3),
       pmLookupLabels(3), pmLookupName(3), pmNewContextZone(3),
       pmNewZone(3), pmParseInterval(3), pmParseTimeWindow(3), pcp.conf(5),
       environ(7), glob(7) and regex(7).

COLOPHON         top

       This page is part of the PCP (Performance Co-Pilot) project.
       Information about the project can be found at ⟨⟩.
       If you have a bug report for this manual page, send it to  This page was obtained from the project's upstream
       Git repository ⟨⟩ on
       2020-11-01.  (At that time, the date of the most recent commit that
       was found in the repository was 2020-10-30.)  If you discover any
       rendering problems in this HTML version of the page, or you believe
       there is a better or more up-to-date source for the page, or you have
       corrections or improvements to the information in this COLOPHON
       (which is not part of the original manual page), send a mail to

Performance Co-Pilot                 PCP                         PMSERIES(1)

Pages that refer to this page: pcp-collectl(1)pcpcompat(1)PCPCompat(1)pmfind(1)pmfind_check(1)pmmgr(1)pmproxy(1)pmwebd(1)pmDiscoverClose(3)pmDiscoverSetConfiguration(3)pmDiscoverSetEventLoop(3)pmDiscoverSetMetricRegistry(3)pmDiscoverSetSlots(3)pmdiscoversetup(3)pmDiscoverSetup(3)pmSeriesClose(3)pmseriesdescs(3)pmSeriesDescs(3)pmSeriesInstances(3)pmSeriesLabels(3)pmSeriesLabelValues(3)pmSeriesLoad(3)pmSeriesMetrics(3)pmseriesquery(3)pmSeriesQuery(3)pmSeriesSetConfiguration(3)pmSeriesSetEventLoop(3)pmSeriesSetMetricRegistry(3)pmSeriesSetSlots(3)pmseriessetup(3)pmSeriesSetup(3)pmSeriesSources(3)pmSeriesValues(3)pmwebapi(3)PMWEBAPI(3)