lex(1p) — Linux manual page

PROLOG | NAME | SYNOPSIS | DESCRIPTION | OPTIONS | OPERANDS | STDIN | INPUT FILES | ENVIRONMENT VARIABLES | ASYNCHRONOUS EVENTS | STDOUT | STDERR | OUTPUT FILES | EXTENDED DESCRIPTION | EXIT STATUS | CONSEQUENCES OF ERRORS | APPLICATION USAGE | EXAMPLES | RATIONALE | FUTURE DIRECTIONS | SEE ALSO | COPYRIGHT

LEX(1P)                 POSIX Programmer's Manual                LEX(1P)

PROLOG         top

       This manual page is part of the POSIX Programmer's Manual.  The
       Linux implementation of this interface may differ (consult the
       corresponding Linux manual page for details of Linux behavior),
       or the interface may not be implemented on Linux.

NAME         top

       lex — generate programs for lexical tasks (DEVELOPMENT)

SYNOPSIS         top

       lex [-t] [-n|-v] [file...]

DESCRIPTION         top

       The lex utility shall generate C programs to be used in lexical
       processing of character input, and that can be used as an
       interface to yacc.  The C programs shall be generated from lex
       source code and conform to the ISO C standard, without depending
       on any undefined, unspecified, or implementation-defined
       behavior, except in cases where the code is copied directly from
       the supplied source, or in cases that are documented by the
       implementation. Usually, the lex utility shall write the program
       it generates to the file lex.yy.c; the state of this file is
       unspecified if lex exits with a non-zero exit status. See the
       EXTENDED DESCRIPTION section for a complete description of the
       lex input language.

OPTIONS         top

       The lex utility shall conform to the Base Definitions volume of
       POSIX.1‐2017, Section 12.2, Utility Syntax Guidelines, except for
       Guideline 9.

       The following options shall be supported:

       -n        Suppress the summary of statistics usually written with
                 the -v option. If no table sizes are specified in the
                 lex source code and the -v option is not specified,
                 then -n is implied.

       -t        Write the resulting program to standard output instead
                 of lex.yy.c.

       -v        Write a summary of lex statistics to the standard
                 output. (See the discussion of lex table sizes in
                 Definitions in lex.)  If the -t option is specified and
                 -n is not specified, this report shall be written to
                 standard error. If table sizes are specified in the lex
                 source code, and if the -n option is not specified, the
                 -v option may be enabled.

OPERANDS         top

       The following operand shall be supported:

       file      A pathname of an input file. If more than one such file
                 is specified, all files shall be concatenated to
                 produce a single lex program. If no file operands are
                 specified, or if a file operand is '-', the standard
                 input shall be used.

STDIN         top

       The standard input shall be used if no file operands are
       specified, or if a file operand is '-'.  See INPUT FILES.

INPUT FILES         top

       The input files shall be text files containing lex source code,
       as described in the EXTENDED DESCRIPTION section.

ENVIRONMENT VARIABLES         top

       The following environment variables shall affect the execution of
       lex:

       LANG      Provide a default value for the internationalization
                 variables that are unset or null. (See the Base
                 Definitions volume of POSIX.1‐2017, Section 8.2,
                 Internationalization Variables for the precedence of
                 internationalization variables used to determine the
                 values of locale categories.)

       LC_ALL    If set to a non-empty string value, override the values
                 of all the other internationalization variables.

       LC_COLLATE
                 Determine the locale for the behavior of ranges,
                 equivalence classes, and multi-character collating
                 elements within regular expressions. If this variable
                 is not set to the POSIX locale, the results are
                 unspecified.

       LC_CTYPE  Determine the locale for the interpretation of
                 sequences of bytes of text data as characters (for
                 example, single-byte as opposed to multi-byte
                 characters in arguments and input files), and the
                 behavior of character classes within regular
                 expressions. If this variable is not set to the POSIX
                 locale, the results are unspecified.

       LC_MESSAGES
                 Determine the locale that should be used to affect the
                 format and contents of diagnostic messages written to
                 standard error.

       NLSPATH   Determine the location of message catalogs for the
                 processing of LC_MESSAGES.

ASYNCHRONOUS EVENTS         top

       Default.

STDOUT         top

       If the -t option is specified, the text file of C source code
       output of lex shall be written to standard output.

       If the -t option is not specified:

        *  Implementation-defined informational, error, and warning
           messages concerning the contents of lex source code input
           shall be written to either the standard output or standard
           error.

        *  If the -v option is specified and the -n option is not
           specified, lex statistics shall also be written to either the
           standard output or standard error, in an implementation-
           defined format. These statistics may also be generated if
           table sizes are specified with a '%' operator in the
           Definitions section, as long as the -n option is not
           specified.

STDERR         top

       If the -t option is specified, implementation-defined
       informational, error, and warning messages concerning the
       contents of lex source code input shall be written to the
       standard error.

       If the -t option is not specified:

        1. Implementation-defined informational, error, and warning
           messages concerning the contents of lex source code input
           shall be written to either the standard output or standard
           error.

        2. If the -v option is specified and the -n option is not
           specified, lex statistics shall also be written to either the
           standard output or standard error, in an implementation-
           defined format. These statistics may also be generated if
           table sizes are specified with a '%' operator in the
           Definitions section, as long as the -n option is not
           specified.

OUTPUT FILES         top

       A text file containing C source code shall be written to
       lex.yy.c, or to the standard output if the -t option is present.

EXTENDED DESCRIPTION         top

       Each input file shall contain lex source code, which is a table
       of regular expressions with corresponding actions in the form of
       C program fragments.

       When lex.yy.c is compiled and linked with the lex library (using
       the -l l operand with c99), the resulting program shall read
       character input from the standard input and shall partition it
       into strings that match the given expressions.

       When an expression is matched, these actions shall occur:

        *  The input string that was matched shall be left in yytext as
           a null-terminated string; yytext shall either be an external
           character array or a pointer to a character string. As
           explained in Definitions in lex, the type can be explicitly
           selected using the %array or %pointer declarations, but the
           default is implementation-defined.

        *  The external int yyleng shall be set to the length of the
           matching string.

        *  The expression's corresponding program fragment, or action,
           shall be executed.

       During pattern matching, lex shall search the set of patterns for
       the single longest possible match. Among rules that match the
       same number of characters, the rule given first shall be chosen.

       The general format of lex source shall be:

              Definitions %% Rules %% UserSubroutines

       The first "%%" is required to mark the beginning of the rules
       (regular expressions and actions); the second "%%" is required
       only if user subroutines follow.

       Any line in the Definitions section beginning with a <blank>
       shall be assumed to be a C program fragment and shall be copied
       to the external definition area of the lex.yy.c file. Similarly,
       anything in the Definitions section included between delimiter
       lines containing only "%{" and "%}" shall also be copied
       unchanged to the external definition area of the lex.yy.c file.

       Any such input (beginning with a <blank> or within "%{" and "%}"
       delimiter lines) appearing at the beginning of the Rules section
       before any rules are specified shall be written to lex.yy.c after
       the declarations of variables for the yylex() function and before
       the first line of code in yylex().  Thus, user variables local to
       yylex() can be declared here, as well as application code to
       execute upon entry to yylex().

       The action taken by lex when encountering any input beginning
       with a <blank> or within "%{" and "%}" delimiter lines appearing
       in the Rules section but coming after one or more rules is
       undefined. The presence of such input may result in an erroneous
       definition of the yylex() function.

       C-language code in the input shall not contain C-language
       trigraphs.  The C-language code within "%{" and "%}" delimiter
       lines shall not contain any lines consisting only of "%}", or
       only of "%%".

   Definitions in lex
       Definitions appear before the first "%%" delimiter. Any line in
       this section not contained between "%{" and "%}" lines and not
       beginning with a <blank> shall be assumed to define a lex
       substitution string. The format of these lines shall be:

           name substitute

       If a name does not meet the requirements for identifiers in the
       ISO C standard, the result is undefined. The string substitute
       shall replace the string {name} when it is used in a rule. The
       name string shall be recognized in this context only when the
       braces are provided and when it does not appear within a bracket
       expression or within double-quotes.

       In the Definitions section, any line beginning with a <percent-
       sign> ('%') character and followed by an alphanumeric word
       beginning with either 's' or 'S' shall define a set of start
       conditions. Any line beginning with a '%' followed by a word
       beginning with either 'x' or 'X' shall define a set of exclusive
       start conditions. When the generated scanner is in a %s state,
       patterns with no state specified shall be also active; in a %x
       state, such patterns shall not be active. The rest of the line,
       after the first word, shall be considered to be one or more
       <blank>-separated names of start conditions. Start condition
       names shall be constructed in the same way as definition names.
       Start conditions can be used to restrict the matching of regular
       expressions to one or more states as described in Regular
       Expressions in lex.

       Implementations shall accept either of the following two
       mutually-exclusive declarations in the Definitions section:

       %array    Declare the type of yytext to be a null-terminated
                 character array.

       %pointer  Declare the type of yytext to be a pointer to a null-
                 terminated character string.

       The default type of yytext is implementation-defined. If an
       application refers to yytext outside of the scanner source file
       (that is, via an extern), the application shall include the
       appropriate %array or %pointer declaration in the scanner source
       file.

       Implementations shall accept declarations in the Definitions
       section for setting certain internal table sizes. The
       declarations are shown in the following table.

                     Table: Table Size Declarations in lex

       ┌────────────┬────────────────────────────────────┬───────────────┐
       │Declaration Description             Minimum Value │
       ├────────────┼────────────────────────────────────┼───────────────┤
       │%p n        │ Number of positions                │     2500      │
       │%n n        │ Number of states                   │      500      │
       │%a n        │ Number of transitions              │     2000      │
       │%e n        │ Number of parse tree nodes         │     1000      │
       │%k n        │ Number of packed character classes │     1000      │
       │%o n        │ Size of the output array           │     3000      │
       └────────────┴────────────────────────────────────┴───────────────┘
       In the table, n represents a positive decimal integer, preceded
       by one or more <blank> characters. The exact meaning of these
       table size numbers is implementation-defined. The implementation
       shall document how these numbers affect the lex utility and how
       they are related to any output that may be generated by the
       implementation should limitations be encountered during the
       execution of lex.  It shall be possible to determine from this
       output which of the table size values needs to be modified to
       permit lex to successfully generate tables for the input
       language. The values in the column Minimum Value represent the
       lowest values conforming implementations shall provide.

   Rules in lex
       The rules in lex source files are a table in which the left
       column contains regular expressions and the right column contains
       actions (C program fragments) to be executed when the expressions
       are recognized.

           ERE action
           ERE action
           ...

       The extended regular expression (ERE) portion of a row shall be
       separated from action by one or more <blank> characters. A
       regular expression containing <blank> characters shall be
       recognized under one of the following conditions:

        *  The entire expression appears within double-quotes.

        *  The <blank> characters appear within double-quotes or square
           brackets.

        *  Each <blank> is preceded by a <backslash> character.

   User Subroutines in lex
       Anything in the user subroutines section shall be copied to
       lex.yy.c following yylex().

   Regular Expressions in lex
       The lex utility shall support the set of extended regular
       expressions (see the Base Definitions volume of POSIX.1‐2017,
       Section 9.4, Extended Regular Expressions), with the following
       additions and exceptions to the syntax:

       "..."     Any string enclosed in double-quotes shall represent
                 the characters within the double-quotes as themselves,
                 except that <backslash>-escapes (which appear in the
                 following table) shall be recognized. Any
                 <backslash>-escape sequence shall be terminated by the
                 closing quote. For example, "\01""1" represents a
                 single string: the octal value 1 followed by the
                 character '1'.

       <state>r, <state1,state2,...>r
                 The regular expression r shall be matched only when the
                 program is in one of the start conditions indicated by
                 state, state1, and so on; see Actions in lex.  (As an
                 exception to the typographical conventions of the rest
                 of this volume of POSIX.1‐2017, in this case <state>
                 does not represent a metavariable, but the literal
                 angle-bracket characters surrounding a symbol.) The
                 start condition shall be recognized as such only at the
                 beginning of a regular expression.

       r/x       The regular expression r shall be matched only if it is
                 followed by an occurrence of regular expression x (x is
                 the instance of trailing context, further defined
                 below). The token returned in yytext shall only match
                 r.  If the trailing portion of r matches the beginning
                 of x, the result is unspecified. The r expression
                 cannot include further trailing context or the '$'
                 (match-end-of-line) operator; x cannot include the '^'
                 (match-beginning-of-line) operator, nor trailing
                 context, nor the '$' operator. That is, only one
                 occurrence of trailing context is allowed in a lex
                 regular expression, and the '^' operator only can be
                 used at the beginning of such an expression.

       {name}    When name is one of the substitution symbols from the
                 Definitions section, the string, including the
                 enclosing braces, shall be replaced by the substitute
                 value. The substitute value shall be treated in the
                 extended regular expression as if it were enclosed in
                 parentheses. No substitution shall occur if {name}
                 occurs within a bracket expression or within double-
                 quotes.

       Within an ERE, a <backslash> character shall be considered to
       begin an escape sequence as specified in the table in the Base
       Definitions volume of POSIX.1‐2017, Chapter 5, File Format
       Notation ('\\', '\a', '\b', '\f', '\n', '\r', '\t', '\v').  In
       addition, the escape sequences in the following table shall be
       recognized.

       A literal <newline> cannot occur within an ERE; the escape
       sequence '\n' can be used to represent a <newline>.  A <newline>
       shall not be matched by a period operator.

                        Table: Escape Sequences in lex

       ┌─────────┬──────────────────────────┬──────────────────────────┐
       │ Escape  │                          │                          │
       │Sequence Description        Meaning          │
       ├─────────┼──────────────────────────┼──────────────────────────┤
       │\digits  │ A <backslash> character  │ The character whose      │
       │         │ followed by the longest  │ encoding is represented  │
       │         │ sequence of one, two, or │ by the one, two, or      │
       │         │ three octal-digit        │ three-digit octal        │
       │         │ characters (01234567).   │ integer. Multi-byte      │
       │         │ If all of the digits are │ characters require       │
       │         │ 0 (that is,              │ multiple, concatenated   │
       │         │ representation of the    │ escape sequences of this │
       │         │ NUL character), the      │ type, including the      │
       │         │ behavior is undefined.   │ leading <backslash> for  │
       │         │                          │ each byte.               │
       ├─────────┼──────────────────────────┼──────────────────────────┤
       │\xdigits │ A <backslash> character  │ The character whose      │
       │         │ followed by the longest  │ encoding is represented  │
       │         │ sequence of hexadecimal- │ by the hexadecimal       │
       │         │ digit characters         │ integer.                 │
       │         │ (01234567abcdefABCDEF).  │                          │
       │         │ If all of the digits are │                          │
       │         │ 0 (that is,              │                          │
       │         │ representation of the    │                          │
       │         │ NUL character), the      │                          │
       │         │ behavior is undefined.   │                          │
       ├─────────┼──────────────────────────┼──────────────────────────┤
       │\c       │ A <backslash> character  │ The character 'c',       │
       │         │ followed by any          │ unchanged.               │
       │         │ character not described  │                          │
       │         │ in this table or in the  │                          │
       │         │ table in the Base        │                          │
       │         │ Definitions volume of    │                          │
       │         │ POSIX.1‐2017, Chapter 5, │                          │
       │         │ File Format Notation     │                          │
       │         │ ('\\', '\a', '\b', '\f', │                          │
       │         │ '\n', '\r', '\t', '\v'). │                          │
       └─────────┴──────────────────────────┴──────────────────────────┘
       Note:  If a '\x' sequence needs to be immediately followed by a
              hexadecimal digit character, a sequence such as "\x1""1"
              can be used, which represents a character containing the
              value 1, followed by the character '1'.

       The order of precedence given to extended regular expressions for
       lex differs from that specified in the Base Definitions volume of
       POSIX.1‐2017, Section 9.4, Extended Regular Expressions.  The
       order of precedence for lex shall be as shown in the following
       table, from high to low.

       Note:  The escaped characters entry is not meant to imply that
              these are operators, but they are included in the table to
              show their relationships to the true operators. The start
              condition, trailing context, and anchoring notations have
              been omitted from the table because of the placement
              restrictions described in this section; they can only
              appear at the beginning or ending of an ERE.

                         Table: ERE Precedence in lex

          ┌──────────────────────────────────┬──────────────────────┐
          │   Extended Regular Expression    Precedence      │
          ├──────────────────────────────────┼──────────────────────┤
          │collation-related bracket symbols │ [= =]  [: :]  [. .]  │
          │escaped characters                │ \<special character> │
          │bracket expression                │ [ ]                  │
          │quoting                           │ "..."                │
          │grouping                          │ ( )                  │
          │definition                        │ {name}               │
          │single-character RE duplication   │ * + ?                │
          │concatenation                     │                      │
          │interval expression               │ {m,n}                │
          │alternation                       │ |                    │
          └──────────────────────────────────┴──────────────────────┘
       The ERE anchoring operators '^' and '$' do not appear in the
       table. With lex regular expressions, these operators are
       restricted in their use: the '^' operator can only be used at the
       beginning of an entire regular expression, and the '$' operator
       only at the end. The operators apply to the entire regular
       expression. Thus, for example, the pattern "(^abc)|(def$)" is
       undefined; it can instead be written as two separate rules, one
       with the regular expression "^abc" and one with "def$", which
       share a common action via the special '|' action (see below). If
       the pattern were written "^abc|def$", it would match either "abc"
       or "def" on a line by itself.

       Unlike the general ERE rules, embedded anchoring is not allowed
       by most historical lex implementations. An example of embedded
       anchoring would be for patterns such as "(^| )foo( |$)" to match
       "foo" when it exists as a complete word. This functionality can
       be obtained using existing lex features:

           ^foo/[ \n]      |
           " foo"/[ \n]    /* Found foo as a separate word. */

       Note also that '$' is a form of trailing context (it is
       equivalent to "/\n") and as such cannot be used with regular
       expressions containing another instance of the operator (see the
       preceding discussion of trailing context).

       The additional regular expressions trailing-context operator '/'
       can be used as an ordinary character if presented within double-
       quotes, "/"; preceded by a <backslash>, "\/"; or within a bracket
       expression, "[/]".  The start-condition '<' and '>' operators
       shall be special only in a start condition at the beginning of a
       regular expression; elsewhere in the regular expression they
       shall be treated as ordinary characters.

   Actions in lex
       The action to be taken when an ERE is matched can be a C program
       fragment or the special actions described below; the program
       fragment can contain one or more C statements, and can also
       include special actions. The empty C statement ';' shall be a
       valid action; any string in the lex.yy.c input that matches the
       pattern portion of such a rule is effectively ignored or skipped.
       However, the absence of an action shall not be valid, and the
       action lex takes in such a condition is undefined.

       The specification for an action, including C statements and
       special actions, can extend across several lines if enclosed in
       braces:

           ERE <one or more blanks> { program statement
                                      program statement }

       The program statements shall not contain unbalanced curly brace
       preprocessing tokens.

       The default action when a string in the input to a lex.yy.c
       program is not matched by any expression shall be to copy the
       string to the output. Because the default behavior of a program
       generated by lex is to read the input and copy it to the output,
       a minimal lex source program that has just "%%" shall generate a
       C program that simply copies the input to the output unchanged.

       Four special actions shall be available:

           |   ECHO;   REJECT;   BEGIN

       |         The action '|' means that the action for the next rule
                 is the action for this rule.  Unlike the other three
                 actions, '|' cannot be enclosed in braces or be
                 <semicolon>-terminated; the application shall ensure
                 that it is specified alone, with no other actions.

       ECHO;     Write the contents of the string yytext on the output.

       REJECT;   Usually only a single expression is matched by a given
                 string in the input.  REJECT means ``continue to the
                 next expression that matches the current input'', and
                 shall cause whatever rule was the second choice after
                 the current rule to be executed for the same input.
                 Thus, multiple rules can be matched and executed for
                 one input string or overlapping input strings. For
                 example, given the regular expressions "xyz" and "xy"
                 and the input "xyz", usually only the regular
                 expression "xyz" would match. The next attempted match
                 would start after z.  If the last action in the "xyz"
                 rule is REJECT, both this rule and the "xy" rule would
                 be executed. The REJECT action may be implemented in
                 such a fashion that flow of control does not continue
                 after it, as if it were equivalent to a goto to another
                 part of yylex().  The use of REJECT may result in
                 somewhat larger and slower scanners.

       BEGIN     The action:

                     BEGIN newstate;

                 switches the state (start condition) to newstate.  If
                 the string newstate has not been declared previously as
                 a start condition in the Definitions section, the
                 results are unspecified. The initial state is indicated
                 by the digit '0' or the token INITIAL.

       The functions or macros described below are accessible to user
       code included in the lex input. It is unspecified whether they
       appear in the C code output of lex, or are accessible only
       through the -l l operand to c99 (the lex library).

       int yylex(void)
             Performs lexical analysis on the input; this is the primary
             function generated by the lex utility. The function shall
             return zero when the end of input is reached; otherwise, it
             shall return non-zero values (tokens) determined by the
             actions that are selected.

       int yymore(void)
             When called, indicates that when the next input string is
             recognized, it is to be appended to the current value of
             yytext rather than replacing it; the value in yyleng shall
             be adjusted accordingly.

       int yyless(int n)
             Retains n initial characters in yytext, NUL-terminated, and
             treats the remaining characters as if they had not been
             read; the value in yyleng shall be adjusted accordingly.

       int input(void)
             Returns the next character from the input, or zero on end-
             of-file. It shall obtain input from the stream pointer
             yyin, although possibly via an intermediate buffer. Thus,
             once scanning has begun, the effect of altering the value
             of yyin is undefined. The character read shall be removed
             from the input stream of the scanner without any processing
             by the scanner.

       int unput(int c)
             Returns the character 'c' to the input; yytext and yyleng
             are undefined until the next expression is matched. The
             result of using unput() for more characters than have been
             input is unspecified.

       The following functions shall appear only in the lex library
       accessible through the -l l operand; they can therefore be
       redefined by a conforming application:

       int yywrap(void)
             Called by yylex() at end-of-file; the default yywrap()
             shall always return 1. If the application requires yylex()
             to continue processing with another source of input, then
             the application can include a function yywrap(), which
             associates another file with the external variable FILE *
             yyin and shall return a value of zero.

       int main(int argc, char *argv[])
             Calls yylex() to perform lexical analysis, then exits. The
             user code can contain main() to perform application-
             specific operations, calling yylex() as applicable.

       Except for input(), unput(), and main(), all external and static
       names generated by lex shall begin with the prefix yy or YY.

EXIT STATUS         top

       The following exit values shall be returned:

        0    Successful completion.

       >0    An error occurred.

CONSEQUENCES OF ERRORS         top

       Default.

       The following sections are informative.

APPLICATION USAGE         top

       Conforming applications are warned that in the Rules section, an
       ERE without an action is not acceptable, but need not be detected
       as erroneous by lex.  This may result in compilation or runtime
       errors.

       The purpose of input() is to take characters off the input stream
       and discard them as far as the lexical analysis is concerned. A
       common use is to discard the body of a comment once the beginning
       of a comment is recognized.

       The lex utility is not fully internationalized in its treatment
       of regular expressions in the lex source code or generated
       lexical analyzer. It would seem desirable to have the lexical
       analyzer interpret the regular expressions given in the lex
       source according to the environment specified when the lexical
       analyzer is executed, but this is not possible with the current
       lex technology. Furthermore, the very nature of the lexical
       analyzers produced by lex must be closely tied to the lexical
       requirements of the input language being described, which is
       frequently locale-specific anyway. (For example, writing an
       analyzer that is used for French text is not automatically useful
       for processing other languages.)

EXAMPLES         top

       The following is an example of a lex program that implements a
       rudimentary scanner for a Pascal-like syntax:

           %{
           /* Need this for the call to atof() below. */
           #include <math.h>
           /* Need this for printf(), fopen(), and stdin below. */
           #include <stdio.h>
           %}

           DIGIT    [0-9]
           ID       [a-z][a-z0-9]*

           %%

           {DIGIT}+ {
               printf("An integer: %s (%d)\n", yytext,
                   atoi(yytext));
               }

           {DIGIT}+"."{DIGIT}*        {
               printf("A float: %s (%g)\n", yytext,
                   atof(yytext));
               }

           if|then|begin|end|procedure|function        {
               printf("A keyword: %s\n", yytext);
               }

           {ID}    printf("An identifier: %s\n", yytext);

           "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);

           "{"[^}\n]*"}"    /* Eat up one-line comments. */

           [ \t\n]+        /* Eat up white space. */

           .  printf("Unrecognized character: %s\n", yytext);

           %%

           int main(int argc, char *argv[])
           {
               ++argv, --argc;  /* Skip over program name. */
               if (argc > 0)
                   yyin = fopen(argv[0], "r");
               else
                   yyin = stdin;

               yylex();
           }

RATIONALE         top

       Even though the -c option and references to the C language are
       retained in this description, lex may be generalized to other
       languages, as was done at one time for EFL, the Extended FORTRAN
       Language. Since the lex input specification is essentially
       language-independent, versions of this utility could be written
       to produce Ada, Modula-2, or Pascal code, and there are known
       historical implementations that do so.

       The current description of lex bypasses the issue of dealing with
       internationalized EREs in the lex source code or generated
       lexical analyzer. If it follows the model used by awk (the source
       code is assumed to be presented in the POSIX locale, but input
       and output are in the locale specified by the environment
       variables), then the tables in the lexical analyzer produced by
       lex would interpret EREs specified in the lex source in terms of
       the environment variables specified when lex was executed. The
       desired effect would be to have the lexical analyzer interpret
       the EREs given in the lex source according to the environment
       specified when the lexical analyzer is executed, but this is not
       possible with the current lex technology.

       The description of octal and hexadecimal-digit escape sequences
       agrees with the ISO C standard use of escape sequences.

       Earlier versions of this standard allowed for implementations
       with bytes other than eight bits, but this has been modified in
       this version.

       There is no detailed output format specification. The observed
       behavior of lex under four different historical implementations
       was that none of these implementations consistently reported the
       line numbers for error and warning messages. Furthermore, there
       was a desire that lex be allowed to output additional diagnostic
       messages. Leaving message formats unspecified avoids these
       formatting questions and problems with internationalization.

       Although the %x specifier for exclusive start conditions is not
       historical practice, it is believed to be a minor change to
       historical implementations and greatly enhances the usability of
       lex programs since it permits an application to obtain the
       expected functionality with fewer statements.

       The %array and %pointer declarations were added as a compromise
       between historical systems.  The System V-based lex copies the
       matched text to a yytext array. The flex program, supported in
       BSD and GNU systems, uses a pointer. In the latter case,
       significant performance improvements are available for some
       scanners. Most historical programs should require no change in
       porting from one system to another because the string being
       referenced is null-terminated in both cases. (The method used by
       flex in its case is to null-terminate the token in place by
       remembering the character that used to come right after the token
       and replacing it before continuing on to the next scan.) Multi-
       file programs with external references to yytext outside the
       scanner source file should continue to operate on their
       historical systems, but would require one of the new declarations
       to be considered strictly portable.

       The description of EREs avoids unnecessary duplication of ERE
       details because their meanings within a lex ERE are the same as
       that for the ERE in this volume of POSIX.1‐2017.

       The reason for the undefined condition associated with text
       beginning with a <blank> or within "%{" and "%}" delimiter lines
       appearing in the Rules section is historical practice. Both the
       BSD and System V lex copy the indented (or enclosed) input in the
       Rules section (except at the beginning) to unreachable areas of
       the yylex() function (the code is written directly after a break
       statement). In some cases, the System V lex generates an error
       message or a syntax error, depending on the form of indented
       input.

       The intention in breaking the list of functions into those that
       may appear in lex.yy.c versus those that only appear in libl.a is
       that only those functions in libl.a can be reliably redefined by
       a conforming application.

       The descriptions of standard output and standard error are
       somewhat complicated because historical lex implementations chose
       to issue diagnostic messages to standard output (unless -t was
       given). POSIX.1‐2008 allows this behavior, but leaves an opening
       for the more expected behavior of using standard error for
       diagnostics.  Also, the System V behavior of writing the
       statistics when any table sizes are given is allowed, while BSD-
       derived systems can avoid it. The programmer can always precisely
       obtain the desired results by using either the -t or -n options.

       The OPERANDS section does not mention the use of - as a synonym
       for standard input; not all historical implementations support
       such usage for any of the file operands.

       A description of the translation table was deleted from early
       proposals because of its relatively low usage in historical
       applications.

       The change to the definition of the input() function that allows
       buffering of input presents the opportunity for major performance
       gains in some applications.

       The following examples clarify the differences between lex
       regular expressions and regular expressions appearing elsewhere
       in this volume of POSIX.1‐2017. For regular expressions of the
       form "r/x", the string matching r is always returned; confusion
       may arise when the beginning of x matches the trailing portion of
       r.  For example, given the regular expression "a*b/cc" and the
       input "aaabcc", yytext would contain the string "aaab" on this
       match. But given the regular expression "x*/xy" and the input
       "xxxy", the token xxx, not xx, is returned by some
       implementations because xxx matches "x*".

       In the rule "ab*/bc", the "b*" at the end of r extends r's match
       into the beginning of the trailing context, so the result is
       unspecified. If this rule were "ab/bc", however, the rule matches
       the text "ab" when it is followed by the text "bc".  In this
       latter case, the matching of r cannot extend into the beginning
       of x, so the result is specified.

FUTURE DIRECTIONS         top

       None.

SEE ALSO         top

       c99(1p), ed(1p), yacc(1p)

       The Base Definitions volume of POSIX.1‐2017, Chapter 5, File
       Format Notation, Chapter 8, Environment Variables, Chapter 9,
       Regular Expressions, Section 12.2, Utility Syntax Guidelines

COPYRIGHT         top

       Portions of this text are reprinted and reproduced in electronic
       form from IEEE Std 1003.1-2017, Standard for Information
       Technology -- Portable Operating System Interface (POSIX), The
       Open Group Base Specifications Issue 7, 2018 Edition, Copyright
       (C) 2018 by the Institute of Electrical and Electronics
       Engineers, Inc and The Open Group.  In the event of any
       discrepancy between this version and the original IEEE and The
       Open Group Standard, the original IEEE and The Open Group
       Standard is the referee document. The original Standard can be
       obtained online at http://www.opengroup.org/unix/online.html .

       Any typographical or formatting errors that appear in this page
       are most likely to have been introduced during the conversion of
       the source files to man page format. To report such errors, see
       https://www.kernel.org/doc/man-pages/reporting_bugs.html .

IEEE/The Open Group               2017                           LEX(1P)

Pages that refer to this page: awk(1p)cflow(1p)make(1p)yacc(1p)