Elasticsearch分词器

版本7.3.0

elasticsearch内置分词器:

  • standard analyzer:会转大小写,去符号等(默认的是standard)
  • simple analyzer:转大小写,同时完全切割单词,去除单词以外的符号及数字
  • whitespace analyzer:仅仅根据空格切割
  • language analyzer(特定的语言的分词器,如english,英语分词器):识别单词并分词,转大小写

分词器流程:

  • character filter:一段文本进行分词之前,先进行预处理,如:<span>hello<span> --> hello,& --> and。
  • tokenizer:分词,hello you and me --> hello, you, and, me
  • token filter:lowercase,stop word,synonymom,dogs --> dog,Tom --> tom,a/the/an --> 删掉,small --> little

默认分词器standard执行流程:

  • standard tokenizer:以单词边界进行切分
  • standard token filter:什么都不做
  • lowercase token filter:将所有字母转换为小写
  • stop token filer(默认被禁用):移除停用词,比如a the it等等

分词器设置:

启用english停用词token filter:

PUT /index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "es_std": {
          "type": "standard",
          "stopwords": "_english_"
        }
      }
    }
  }
}

定制化自己的分词器案例:

PUT /index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "&_to_and": {
          "type": "mapping",
          "mappings": ["&=> and"]
        }
      },
      "filter": {
        "my_stopwords": {
          "type": "stop",
          "stopwords": ["the", "a"]
        }
      },
      "analyzer": {
        "analyzer_name": {
          "type": "custom",
          "char_filter": ["html_strip", "&_to_and"],
          "tokenizer": "standard",
          "filter": ["lowercase", "my_stopwords"]
        }
      }
    }
  }
}

评论

暂无评论