如何在嗖嗖声中使用n-gram

发布时间：2020-12-16 21:33:06 所属栏目：Python 来源：网络整理

导读：我正在尝试使用n-gram来使用Whoosh进行“自动完成式”搜索.不幸的是我有点困惑.我做了一个像这样的索引： if not os.path.exists("index"): os.mkdir("index")ix = create_in("index",schema)ix = open_dir("index")writer = ix.writer()q = MyTable.select(

我正在尝试使用n-gram来使用Whoosh进行“自动完成式”搜索.不幸的是我有点困惑.我做了一个像这样的索引：

if not os.path.exists("index"):
    os.mkdir("index")
ix = create_in("index",schema)

ix = open_dir("index")

writer = ix.writer()
q = MyTable.select()
for item in q:
    print 'adding %s' % item.Title
    writer.add_document(title=item.Title,content=item.content,url = item.URL)
writer.commit()

然后我搜索它的标题字段,如下所示：

querystring = 'my search string'

parser = QueryParser("title",ix.schema)
myquery = parser.parse(querystring)

with ix.searcher() as searcher:
    results = searcher.search(myquery)
    print len(results)

    for r in results:
        print r

这很有效.但是我想在自动完成中使用它并且它与部分单词不匹配(例如,搜索“ant”将返回“ant”,而不是“antelope”或“anteater”).这当然会大大妨碍将其用于自动完成. Whoosh page说使用这个：

analyzer = analysis.NgramWordAnalyzer()
title_field = fields.TEXT(analyzer=analyzer,phrase=False)
schema = fields.Schema(title=title_field)

但我对此感到困惑.它似乎只是过程的“中间”,当我构建索引时,我是否必须将title字段包含为NGRAM字段(而不是TEXT)？我该如何进行搜索？所以当我搜索“蚂蚁”时,我得到[“蚂蚁”,“食蚁兽”,“羚羊”]等？

解决方法

我通过创建两个单独的字段来解决这个问题.一个用于实际搜索,一个用于建议. NGRAM或NGRAMWORDS字段类型可用于“模糊搜索”功能.在你的情况下,它将是这样的：

# not sure how your schema looks like exactly
schema = Schema(
    title=NGRAMWORDS(minsize=2,maxsize=10,stored=True,field_boost=1.0,tokenizer=None,at='start',queryor=False,sortable=False)
    content=TEXT(stored=True),url=title=ID(stored=True),spelling=TEXT(stored=True,spelling=True)) # typeahead field

if not os.path.exists("index"):
os.mkdir("index")
ix = create_in("index",url = item.URL)
    writer.add_document(spelling=item.Title) # adding item title to typeahead field
    self.addContentToSpelling(writer,item.content) # some method that adds some content words to typeheadfield if needed. The same way as above.
writer.commit()

然后在搜索时：

origQueryString = 'my search string'
words = self.splitQuery(origQueryString) # use tokenizers / analyzers or self implemented
queryString = origQueryString # would be better to actually create a query
corrector = ix.searcher().corrector("spelling")
for word in words:
    suggestionList = corrector.suggest(word,limit=self.limit)
    for suggestion in suggestionList:
         queryString = queryString + " " + suggestion # would be better to actually create a query      

parser = QueryParser("title",ix.schema)
myquery = parser.parse(querystring)

with ix.searcher() as searcher:
     results = searcher.search(myquery)
     print len(results)

    for r in results:
        print r

希望你明白这个主意.

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!