Python Lex-Yacc(PLY)输入结束时的错误恢复

Statement   : Expressions

Expressions : Expression Expressions
            | 

Expression  : A END

Expression : A END
           | A error

from __future__ import print_function

# Tokens
tokens = ('A','END')

t_A   = r'A'
t_END = r'END'
t_ignore = " "

def t_error(t):
    print("Illegal character '%s'" % t.value[0])
    t.lexer.skip(1)

# Build the lexer
import ply.lex as lex
lex.lex()

# Rules
def p_statement_expr(p):
    '''statement : expressions'''
    print("parsed:",p[1])

def p_expressions(p):
    '''expressions : expression expressions'''
    p[0] = [p[1]] + p[2]

def p_expressions_empty(p):
    '''expressions : '''
    p[0] = list()

def p_expression_pharse(p):
    '''expression : A END
                  | A error'''
    p[0] = 'A'

def p_error(p):
    if p:
        print("Syntax error at '%s'" % p.value)
    else:
        print("Syntax error at EOI")

import ply.yacc as yacc
yacc.yacc()

while 1:
    try:
        s = raw_input('query > ')   # use input() on Python 3
    except EOFError:
        break
    yacc.parse(s)

我把它添加为一个新的答案(并且知道它对于赏金来说为时已晚:-()因为它是一种非常不同的方法.如果我们使用flex,它会更容易,因为它具有<的概念. < EOF>>只在文件末尾匹配的令牌.在考虑完之后,我意识到通过在词法分析器周围使用代理,将该功能添加到PLY而不对原始模块进行任何更改是非常简单的.通过__getattr__特殊方法,可以轻松实现代理.

我只是补充一下

>将在文件末尾发送的新令牌EOF
>围绕词法分析器的令牌方法的代理,在文件末尾返回第一遍的特殊EOF令牌,然后是正常的无
> eof令牌结束语句规则

并且仍然反转规则表达式：表达式表达式而不是表达式：表达式表达式允许立即减少

代码变成：

from __future__ import print_function

# Tokens
tokens = ('A','END','EOF')

t_A   = r'A'
t_END = r'END'
t_ignore = " "

def t_error(t):
    print("Illegal character '%s'" % t.value[0])
    t.lexer.skip(1)

# Build the lexer
import ply.lex as lex

orig_lexer = lex.lex()

class ProxyLexer(object):
    def __init__(self,lexer,eoftoken):
        self.end = False
        self.lexer = lexer
        self.eof = eoftoken
    def token(self):
        tok = self.lexer.token()
        if tok is None:
            if self.end :
                self.end = False
            else:
                self.end = True
                tok = lex.LexToken()
                tok.type = self.eof
                tok.value = None
                tok.lexpos = self.lexer.lexpos
                tok.lineno = self.lexer.lineno
        # print ('custom',tok)
        return tok
    def __getattr__(self,name):
        return getattr(self.lexer,name)

lexer = ProxyLexer(orig_lexer,'EOF')

# Rules
def p_statement_expr(p):
    '''statement : expressions EOF'''
    print("parsed:",p[1])

def p_expressions(p):
    '''expressions : expressions expression'''
    p[0] = p[1] + [p[2]]

def p_expressions_empty(p):
    '''expressions : '''
    p[0] = list()

def p_expression_pharse(p):
    '''expression : A END
                  | A error'''
    p[0] = 'A'

def p_error(p):
    if p:
        print("Syntax error at '%s'" % p.value)
    else:
        print("Syntax error at EOI")

import ply.yacc as yacc
parser = yacc.yacc()

while 1:
    try:
        s = raw_input('query > ')   # use input() on Python 3
    except EOFError:
        break
    parser.parse(s,lexer = lexer)

那样 ：

>原始语法不变>错误恢复方法仍然很简单,并且不依赖于剩余的语法>它可以很容易地扩展到复杂的解析器