TYPES OF DICTIONARIES FULL

SELECT ts_rank_cd (to_tsvector('english', 'in the list of stop words'), to_tsquery('list & stop')) Ranks calculated for documents with and without stop words are quite different: The missing positions 1,2,4 are because of stop words. SELECT to_tsvector('english', 'in the list of stop words')

However, stop words do affect the positions in tsvector, which in turn affect ranking: For example, every English text contains words like a and the, so it is useless to store them in an index.

TYPES OF DICTIONARIES FULL

Therefore, they can be ignored in the context of full text searching. Stop words are words that are very common, appear in almost every document, and have no discrimination value. For example, a filtering dictionary could be used to remove accents from accented letters, as is done by the unaccent module. Filtering dictionaries are useful to partially normalize words to simplify the task of later dictionaries. For example, for an astronomy-specific search ( astro_en configuration) one could bind token type asciiword (ASCII word) to a synonym dictionary of astronomical terms, a general English dictionary and a Snowball English stemmer:ĪDD MAPPING FOR asciiword WITH astrosyn, english_ispell, english_stem Ī filtering dictionary can be placed anywhere in the list, except at the end where it'd be useless. The general rule for configuring a list of dictionaries is to place first the most narrow, most specific dictionary, then the more general dictionaries, finishing with a very general dictionary, like a Snowball stemmer or simple, which recognizes everything. Normally, the first dictionary that returns a non- NULL output determines the result, and any remaining dictionaries are not consulted but a filtering dictionary can replace the given word with a modified word, which is then passed to subsequent dictionaries. If it is identified as a stop word, or if no dictionary recognizes the token, it will be discarded and not indexed or searched for. When a token of that type is found by the parser, each dictionary in the list is consulted in turn, until some dictionary recognizes it as a known word. For each token type that the parser can return, a separate list of dictionaries is specified by the configuration. If no existing template is suitable, it is possible to create new ones see the contrib/ area of the PostgreSQL distribution for examples.Ī text search configuration binds a parser together with a set of dictionaries to process the parser's output tokens. Each predefined dictionary template is described below. There are also several predefined templates that can be used to create new dictionaries with custom parameters. PostgreSQL provides predefined dictionaries for many languages. NULL if the dictionary does not recognize the input token An array of lexemes if the input token is known to the dictionary (notice that one token can produce more than one lexeme)Ī single lexeme with the TSL_FILTER flag set, to replace the original token with a new token to be passed to subsequent dictionaries (a dictionary that does this is called a filtering dictionary)Īn empty array if the dictionary knows the token, but it is a stop word

Types of dictionaries