NAME | SYNOPSIS | DESCRIPTION | CONFIGURATION FILE SYNTAX | EXAMPLES | FILES | ENVIRONMENT | PCP ENVIRONMENT | SEE ALSO | DIAGNOSTICS | COLOPHON

PMLOGGER(1)                General Commands Manual               PMLOGGER(1)

NAME         top

       pmlogger - create archive log for performance metrics

SYNOPSIS         top

       pmlogger [-c configfile] [-h host] [-H hostname] [-K spec] [-l
       logfile] [-L] [-m note] [-n pmnsfile] [-o] [-p pid] [-P] [-r] [-s
       endsize] [-t interval] [-T endtime] [-u] [-U username] [-v volsize]
       [-V version] [-x fd] [-y] archive

DESCRIPTION         top

       pmlogger creates the archive logs of performance metric values that
       may be ``played back'' by other Performance Co-Pilot (see
       PCPIntro(1)) tools.  These logs form the basis of the VCR paradigm
       and retrospective performance analysis services common to the PCP
       toolkit.

       The mandatory argument archive is the base name for the physical
       files that constitute an archive log.

       The -V option specifies the version for the archive that is
       generated.  By default a version 2 archive is generated, and the only
       value currently supported for version is 2.

       Unless directed to another host by the -h option or when directly
       using PMDAs via the -o option, pmlogger will contact the Performance
       Metrics Collector Daemon (PMCD) on the local host and use that as the
       source of the metric values to be logged.

       To support the required flexibility and control over what is logged
       and when, pmlogger maintains an independent two level logging state
       for each instance of each performance metric.  At the first
       (mandatory) level, logging is allowed to be on (with an associated
       interval between samples), or off or maybe.  In the latter case, the
       second (advisory) level logging is allowed to be on (with an
       associated interval between samples), or off.

       The mandatory level allows universal specification that some metrics
       must be logged, or must not be logged.  The default state for all
       instances of all metrics when pmlogger starts is mandatory maybe and
       advisory off.

       Use pmlc(1) to interrogate and change the logging state once pmlogger
       is running.

       If a metric's state is mandatory (on or off) and a request is made to
       change it to mandatory maybe, the new state is mandatory maybe and
       advisory off.  If a metric's state is already advisory (on or off)
       and a request is made to change it to mandatory maybe, the current
       state is retained.

       It is not possible for pmlogger to log specific instances of a metric
       and all instances of the same metric concurrently.  If specific
       instances are being logged and a request to log all instances is
       made, then all instances of the metric will be logged according to
       the new request, superseding any prior logging request for the
       metric.  A request to log all instances of a metric will supersede
       any previous request to log all instances.  A request to log specific
       instances of a metric when all instances are already being logged is
       refused.  To do this one must turn off logging for all instances of
       the metric first.  In each case, the validity of the request is
       checked first; for example a request to change a metric's logging
       state to advisory on when it is currently mandatory off is never
       permitted (it is necessary to change the state to mandatory maybe
       first).

       Optionally, each system running pmcd(1) may also be configured to run
       a ``primary'' pmlogger instance.  This pmlogger instance is launched
       by $PCP_RC_DIR/pmlogger, and is affected by the files
       $PCP_SYSCONF_DIR/pmlogger/control,
       $PCP_SYSCONF_DIR/pmlogger/control.d (use chkconfig(8) or similar
       platform-specific commands to activate or disable the primary
       pmlogger instance), $PCP_SYSCONFIG_DIR/pmlogger (environment variable
       settings for the primary pmlogger)
       $PCP_SYSCONF_DIR/pmlogger/pmlogger.options (command line options
       passed to the primary pmlogger) and
       $PCP_VAR_DIR/config/pmlogger/config.default (the default initial
       configuration file for the primary pmlogger).

       The primary pmlogger instance is identified by the -P option.  There
       may be at most one ``primary'' pmlogger instance on each system.  The
       primary pmlogger instance (if any) must be running on the same host
       as the pmcd(1) to which it connects (if any), so the -h and -P
       options are mutually exclusive.

       Logging of some metrics is possible even in the absence of a local
       pmcd(1), using the "local context" mode of operation.  This is
       activated using the -o option, and causes pmlogger to make use of
       local DSO PMDAs instead of communicating with pmcd(1).  When
       operating using a local context, the -K option may be used to control
       the DSO PMDAs that should be made accessible.  The spec argument
       conforms to the syntax described in __pmSpecLocalPMDA(3).  More than
       one -K option may be used.

       When launched as a non-primary instance, pmlogger will exit
       immediately if the configuration file causes no metric logging to be
       scheduled.  The -L option overrides this behavior, and causes a non-
       primary pmlogger instance to ``linger'', presumably pending some
       future dynamic re-configuration and state change via pmlc(1).
       pmlogger will also linger without the -L option being used if all the
       metrics to be logged are logged as once only metrics. When the once
       only metrics have been logged, a warning message will be generated
       stating that the event queue is empty and no more events will be
       scheduled.

       By default all diagnostics and errors from pmlogger are written to
       the file pmlogger.log in the directory where pmlogger is launched.
       The -l option may be used to override the default behavior.  If the
       log file cannot be created or is not writable, output is written to
       standard error instead.

       If specified, the -s option instructs pmlogger to terminate after a
       certain size in records, bytes or time units has been accumulated.
       If endsize is an integer then endsize records will be written to the
       log.  If endsize is an integer suffixed by b or bytes then endsize
       bytes of the archive data will be written out (note, however, that
       archive log record boundaries will not be broken and so this limit
       may be slightly surpassed).  Other viable file size units include: K,
       Kb, Kbyte, Kilobyte for kilobytes and M, Mb, Mbyte, Megabyte for
       megabytes and G, Gb, Gbyte, Gigabyte for gigabytes.  These units may
       be optionally suffixed by an s and may be of mixed case.
       Alternatively endsize may be an integer or a floating point number
       suffixed using a time unit as described in PCPIntro(1) for the
       interval argument (to the standard PCP -t command line option).
       Some examples of different formats:
          -s 100
          -s 100bytes
          -s 100K
          -s 100Mb
          -s 10Gbyte
          -s 10mins
          -s 1.5hours
       The default is for pmlogger to run forever.

       The -r option causes the size of the physical record(s) for each
       group of metrics and the expected contribution of the group to the
       size of the PCP archive for one full day of collection to be reported
       in the log file.  This information is reported the first time each
       group is successfully written to the archive.

       The -U option specifies the user account under which to run pmlogger.
       The default is the current user account for interactive use.  When
       run as a daemon, the unprivileged "pcp" account is used in current
       versions of PCP, but in older versions the superuser account ("root")
       was used by default.

       The log file is potentially a multi-volume data set, and the -v
       option causes pmlogger to start a new volume after a certain size in
       records, bytes, or time units has been accumulated for the current
       volume.  The format of this size specification is identical to that
       of the -s option (see above).  The default is for pmlogger to create
       a single volume log.  Additional volume switches can also be forced
       asynchronously by either using pmlc(1) or sending pmlogger a SIGHUP
       signal (see below). Note, if a scheduled volume switch is in
       operation due to the -v option, then its counters will be reset after
       an asynchronous switch.

       Independent of any -v option, each volume of an archive is limited to
       no more than 2^31 bytes, so pmlogger will automatically create a new
       volume for the archive before this limit is reached.

       Normally pmlogger operates on the distributed Performance Metrics
       Name Space (PMNS), however if the -n option is specified an
       alternative local PMNS is loaded from the file pmnsfile.

       Under normal circumstances, pmlogger will run forever (except for a
       -s option or a termination signal).  The -T option may be used to
       limit the execution time using the format of time as prescribed by
       PCPIntro(1).  The time is interpreted within the time zone of the
       PMCD server, unless the -y option is given, within which case the
       time zone at this logger host is used.
       Some examples of different formats:
          -T 10mins
          -T '@ 11:30'
       From this it can be seen that -T 10mins and -s 10mins perform
       identical actions.

       Alternatively, pmlogger runtime may be limited to the lifetime of
       another process by using the -p or --PID option to nominate the PID
       of the process of interest.  In this case the pmlogger will exit when
       the other process no longer exists.

       When pmlogger receives a SIGHUP signal, the current volume of the log
       is closed, and a new volume is opened.  This mechanism (or the
       alternative mechanism via pmlc(1)) may be used to manage the growth
       of the log files - once a log volume is closed, that file may be
       archived without ill-effect on the continued operation of pmlogger.
       See also the -v option above.

       Historically the buffers for the current log may be flushed to disk
       using the flush command of pmlc(1), or by sending pmlogger a SIGUSR1
       signal or by using the -u option.  The current version of pmlogger
       and the libpcp routines that underpin pmlogger unconditionally use
       unbuffered writes and a single fwrite(3) for each logical record
       written, and so ``flushing'' does not force any additional data to be
       written to the file system.  The -u option, the SIGUSR1 handling and
       the pmlc(1) flush command are retained for backwards compatibility.

       When launched with the -x option, pmlogger will accept asynchronous
       control requests on the file descriptor fd.  This option is only
       expected to be used internally by PCP applications that support
       ``live record mode''.

       The -m option allows the string note to be appended to the map file
       for this instance of pmlogger in the $PCP_TMP_DIR/pmlogger directory.
       This is currently used internally to document the file descriptor
       (fd) when the -x option is used, or to indicate that this pmlogger
       instance was started under the control of pmlogger_check(1).

       The -H option allows the hostname written into the archive label to
       be overridden.  This mirrors the -H option of pmcd(1) , but allows it
       to be specified on the pmlogger process. Without this option, the
       value returned from the logged pmcd(1) is used.

CONFIGURATION FILE SYNTAX         top

       The configuration file may be specified with the -c option.  If it is
       not, configuration specifications are read from standard input.

       If configfile does not exist, then a search is made in the directory
       $PCP_VAR_DIR/config/pmlogger for a file of the same name, and if
       found that file is used, e.g. if config.mumble does not exist in the
       current directory and the file
       $PCP_VAR_DIR/config/pmlogger/config.mumble does exist, then -c
       config.mumble and -c $PCP_VAR_DIR/config/pmlogger/config.mumble are
       equivalent.

       The syntax for the configuration file is as follows.

       1.   Words are separated by white space (space, tab or newline).

       2.   The symbol ``#'' (hash) introduces a comment, and all text up to
            the next newline is ignored.

       3.   Keywords (shown in bold below) must appear literally (i.e. in
            lower case).

       4.   Each specification begins with the optional keyword log,
            followed by one of the states mandatory on, mandatory off,
            mandatory maybe, advisory on or advisory off.

       5.   For the on states, a logging interval must follow using the
            syntax ``once'', or ``default'', or ``every N timeunits'', or
            simply ``N timeunits'' - N is an unsigned integer, and timeunits
            is one of the keywords msec, millisecond, sec, second, min,
            minute, hour or the plural form of one of the above.
            Internal limitations require the interval to be smaller than
            (approximately) 74 hours.  An interval value of zero is a
            synonym for once.  An interval of default means to use the
            default logging interval of 60 seconds; this default value may
            be changed to interval with the -t command line option.

            The interval argument follows the syntax described in
            PCPIntro(1), and in the simplest form may be an unsigned integer
            (the implied units in this case are seconds).

       6.   Following the state and possible interval specifications comes a
            ``{'', followed by a list of one or more metric specifications
            and a closing ``}''.  The list is white space (or comma)
            separated.  If there is only one metric specification in the
            list, the braces are optional.

       7.   A metric specification consists of a metric name optionally
            followed by a set of instance names.  The metric name follows
            the standard PCP naming conventions, see pmns(5), and if the
            metric name is a non-leaf node in the PMNS (see pmns(5)), then
            pmlogger will recursively descend the PMNS and apply the logging
            specification to all descendent metric names that are leaf nodes
            in the PMNS.  The set of instance names is a ``['', followed by
            a list of one or more space (or comma) separated names, numbers
            or strings, and a closing ``]''.  Elements in the list that are
            numbers are assumed to be internal instance identifiers, other
            elements are assumed to be external instance identifiers - see
            pmGetInDom(3) for more information.

            If no instances are given, then the logging specification is
            applied to all instances of the associated metric.

       8.   There may be an arbitrary number of logging specifications.

       9.   Following all of the logging specifications, there may be an
            optional access control section, introduced by the literal token
            [access].  Thereafter come access control rules that allow or
            disallow operations from particular hosts or groups of hosts.

            The operations may be used to interrogate or control a running
            pmlogger using pmlc(1) and fall into the following classes:

            enquire        interrogate the status of pmlogger and the
                           metrics it is logging
            advisory       Change advisory logging.
            mandatory      Change mandatory logging.
            all            All of the above.

            Access control rules are of the form ``allow hostlist :
            operationlist ;'' and ``disallow hostlist : operationlist ;''.

            The hostlist follows the syntax and semantics for the access
            control mechanisms used by PMCD and are fully documented in
            pmcd(1).  An operationslist is a comma separated list of the
            operations advisory, mandatory, enquire and all.

            A missing [access] section allows all access and is equivalent
            to allow * : all;.

       The configuration (either from standard input or configfile) is
       initially scanned by pmcpp(1) with the options -rs and -I
       $PCP_VAR_DIR/config/pmlogger.  This extends the configuration file
       syntax with include file processing (%include), a common location to
       search for include files ($PCP_VAR_DIR/config/pmlogger), macro
       definitions (%define), macro expansion (%name and %{name}) and
       conditional inclusion of lines (%ifdef name ... %else ... %endif and
       %ifndef name ... %else ... %endif).

EXAMPLES         top

       For each PCP utility, there is a sample pmlogger configuration file
       that could be used to create an archive log suitable for replaying
       with that tool (i.e. includes all of the performance metrics used by
       the tool).  For a tool named foo this configuration file is located
       in $PCP_VAR_DIR/config/pmlogger/config.foo.

       The following is a simple default configuration file for a primary
       pmlogger instance, and demonstrates most of the capabilities of the
       configuration specification language.

            log mandatory on once { hinv.ncpu hinv.ndisk }
            log mandatory on every 10 minutes {
                disk.all.write
                disk.all.read
                network.interface.in.packets [ "et0" ]
                network.interface.out.packets [ "et0" ]
                nfs.server.reqs [ "lookup" "getattr" "read" "write" ]
            }

            log advisory on every 30 minutes {
                environ.temp
                pmcd.pdu_in.total
                pmcd.pdu_out.total
            }

            %include "macros.default"

            %ifdef %disk_detail
            log mandatory on %disk_detail_freq {
                disk.dev
            }
            %endif

            [access]
            disallow * : all except enquire;
            allow localhost : mandatory, advisory;

FILES         top

       archive.meta
                 metadata (metric descriptions, instance domains, etc.) for
                 the archive log
       archive.0 initial volume of metrics values (subsequent volumes have
                 suffixes 1, 2, ...)
       archive.index
                 temporal index to support rapid random access to the other
                 files in the archive log
       $PCP_TMP_DIR/pmlogger
                 pmlogger maintains the files in this directory as the map
                 between the process id of the pmlogger instance and the IPC
                 port that may be used to control each pmlogger instance (as
                 used by pmlc(1))
       $PCP_VAR_DIR/config/pmlogger/config.default
                 default configuration file for the primary logger instance
                 launched from $PCP_RC_DIR/pmlogger
       $PCP_VAR_DIR/config/pmlogger/config.*
                 assorted configuration files suitable for creating logs
                 that may be subsequently replayed with the PCP
                 visualization and monitoring tools
       $PCP_LOG_DIR/pmlogger/hostname
                 Default directory for PCP archive files for performance
                 metric values collected from the host hostname.
       $PCP_SYSCONFIG_DIR/pmlogger
                 additional environment variables that will be set when the
                 primary pmlogger instance executes.  Only settings of the
                 form "PMLOGGER_VARIABLE=value" will be honoured.
       ./pmlogger.log
                 (or $PCP_LOG_DIR/pmlogger/hostname/pmlogger.log when
                 started automatically by either $PCP_RC_DIR/pmlogger or one
                 of the pmlogger(1) monitoring scripts such as
                 pmlogger_check(1))
                 all messages and diagnostics are directed here

ENVIRONMENT         top

       Normally pmlogger creates a socket to receive control messages from
       pmlc(1) on the first available TCP/IP port numbered 4330 or higher.
       The environment variable PMLOGGER_PORT may be used to specify an
       alternative starting port number.

       If set to the value 1, the PMLOGGER_LOCAL environment variable will
       cause pmlogger to run in a localhost-only mode of operation, where it
       binds only to the loopback interface.

       The PMLOGGER_MAXPENDING variable can be set to indicate the maximum
       length to which the queue of pending pmlc connections may grow.

       The default sampling interval used by pmlogger can be set using the
       PMLOGGER_INTERVAL variable (if not set, 60 seconds will be used).
       Both the command line and directives in the configuration file will
       override this value.  It is an integer in units of seconds.

PCP ENVIRONMENT         top

       Environment variables with the prefix PCP_ are used to parameterize
       the file and directory names used by PCP.  On each installation, the
       file /etc/pcp.conf contains the local values for these variables.
       The $PCP_CONF variable may be used to specify an alternative
       configuration file, as described in pcp.conf(5).

SEE ALSO         top

       PCPIntro(1), pmcd(1), pmdumplog(1), pmlc(1), pmlogger_check(1),
       __pmSpecLocalPMDA(3), pcp.conf(5), pcp.env(5), pmns(5) and
       chkconfig(8).

DIAGNOSTICS         top

       The archive logs are sufficiently precious that pmlogger will not
       truncate an existing physical file.  A message of the form
        __pmLogNewFile: "foo.index" already exists, not over-written
        __pmLogCreate: File exists
       indicates this situation has arisen.  You must explicitly remove the
       files and launch pmlogger again.

       There may be at most one primary pmlogger instance per monitored
       host; attempting to bend this rule produces the error:
        pmlogger: there is already a primary pmlogger running

       Various other messages relating to the creation and/or deletion of
       files in $PCP_TMP_DIR/pmlogger suggest a permission problem on this
       directory, or some feral files have appeared therein.

COLOPHON         top

       This page is part of the PCP (Performance Co-Pilot) project.
       Information about the project can be found at ⟨http://www.pcp.io/⟩.
       If you have a bug report for this manual page, send it to
       pcp@oss.sgi.com.  This page was obtained from the project's upstream
       Git repository ⟨https://github.com/performancecopilot/pcp.git⟩ on
       2017-09-15.  If you discover any rendering problems in this HTML ver‐
       sion of the page, or you believe there is a better or more up-to-date
       source for the page, or you have corrections or improvements to the
       information in this COLOPHON (which is not part of the original man‐
       ual page), send a mail to man-pages@man7.org

Performance Co-Pilot                 PCP                         PMLOGGER(1)

Pages that refer to this page: ganglia2pcp(1)iostat2pcp(1)mrtg2pcp(1)pcp(1)pcp-atop(1)pcp-atopsar(1)pcp-collectl(1)pcpintro(1)pcp-iostat(1)pcp-mpstat(1)pcp-pidstat(1)pcp-tapestat(1)pmcd(1)pmchart(1)pmcpp(1)pmdapapi(1)pmdatrace(1)pmdiff(1)pmdumplog(1)pmdumptext(1)pmie(1)pmie_check(1)pmlc(1)pmlogcheck(1)pmlogconf(1)pmlogextract(1)pmlogger(1)pmlogger_check(1)pmloglabel(1)pmlogmv(1)pmlogreduce(1)pmlogrewrite(1)pmlogsummary(1)pmmgr(1)pmnewlog(1)pmsnap(1)pmsocks(1)pmstat(1)pmval(1)pmview(1)sar2pcp(1)sheet2pcp(1)logimport(3)pcpintro(3)pmaf(3)pmafm(3)pmapi(3)pmconnectlogger(3)pmcontrollog(3)pmdestroycontext(3)pmdupcontext(3)pmgetarchiveend(3)pmgetarchivelabel(3)pmnewcontext(3)pmtrimnamespace(3)pmusecontext(3)pmwhichcontext(3)pcp-archive(5)