ES查询之分词器

发布日期: 2021-11-09

更新日期: 2021-11-09

文章字数: 688

阅读时长: 3 分

阅读次数: loading

概念

切词
mormallization: (提升recall召回率：能搜索到的结果的比率)

分析器

character filter（mapping）：分词的预处理（过滤无用字符、标点、转换一些&=and 《Elasticsearch》 -> Elasticsearch）
- HTML Strip Character Filter: html strip
  - a. 参数：escaped_tags 需要保留的html标签
- Mapping Character Filter： type mapping
- Pattern Repace Character Filter: type pattern_replace

# HTML Strip Character Filter
PUT index
{
	"settings":
	{
		"analysis":
		{
			"char_filter":
			{
				"my_char_filter":
				{
					"type":"html_strip",
					"escaped_tags":['a']
				}
			},
			"analyzer":
			{
				"my_analyzer":
				{
					"tokenizer":"keyword", #standard
					"char_filter":"my_char_filter"
					
				}
			}
		}
	}

}


GET index/_analyze
{
	"analyzer":"my_analyzer",
	"text": "mashibing <a><b>deu</b></a>"
}


 #Mapping Character Filter
PUT index
{
	"settings":{
		"analysis":{
			"char_filter":{
				"my_char_filter":{
					"type":"mapping",
					"mappigns":[
						".=>0",
						")=>1",
						"(=>2"
					]
				}
			},
			"analyzer":{
				"my_analyzer":{
					"tokenizer":"keyword",
					"char_filter":{
						"my_char_filter"
					}
				}
			}
		}
	}
}

GET index/_annlyze
{
	"analyzer":"my_analyzer",
	"text": "sds.sd(sd) dss "
}


#Pattern Repace Character Filter
#213-456-789
PUT index
{
	"settings":
	{
		"analysis":
		{
			"char_filter":
			{
				"my_char_filter":
				{
					"type":"pattern_replace",
					"pattern":"(\\d)-(?=\\d)",
					"replacement":"$1_"
				}
			},
			"analyzer":
			{
				"my_analyzer":
				{
					"tokenizer":"keyword", #standard
					"char_filter":"my_char_filter"
					
				}
			}
		}
	}

}

GET index/_analyze
{
	"analyzer":"my_analyzer",
	"text": "213-456-789"
}

tokenizer(分词器)：分词

token filter：停用词、时态转换、大小写转换、同义词转换、语气词处理等。比如has=>have him=>he apples=>apple the/oh/a=>干掉
1
2
3
4
5
6
GET /_annlyze
{
"tokenizer":"standard",
"filter":["lowercase"],
"text":"MA SHI BING"
}

ES 内置分词器 7.6 15种默认自带分词器

standard anaylzer: 默认分词器，中文支持的不理想，会逐字拆分。
1. max_token_length:最大令牌长度。如果看到令牌超过此长度，则将其max_token_length间隔分割。默认255
Pattern Tokenizer: 以正则匹配分隔符，把文字拆分成若干词项。
Simple Pattern Tokenizer: 一正则匹配词项，速度比Pattern

#自定义 分析器

PUT index
{
	"settings":{
		"analysis":{
			"char_filter":{
				"my_char_filter":{
					"type":"mapping",
					"mappings":["& => and"]
				}
			},
			"filter":{
				"test_filter":{
					"type": "stop",
					"stopword":["is","a","at","the"]
				}
			},
			"analyzer":{
				"my_analyzer":{
					"type":"custom",
					"char_filter":[
						"my_char_filter",
						"html_strip"
					],
					"filter":["lowercase","test_filter"],
					"tokenizer":"standard"
				}
			}
		}
	}
}

GET index/_analyze
{
	"tokenizer":"my_analyzer",
	"text":"TEST and T a T at S the T"
}