head	2.4;
access;
symbols
	REL9_0_0:2.2.16.2
	REL9_1_ALPHA1:2.4
	REL9_0_RC1:2.2.16.2
	REL9_0_BETA4:2.2.16.1
	REL9_0_STABLE:2.2.0.16
	REL9_0_BETA3:2.2
	REL9_0_BETA2:2.2
	REL8_3_11:2.2
	REL8_4_4:2.2
	REL9_0_BETA1:2.2
	REL9_0_ALPHA5_BRANCH:2.2.0.14
	REL9_0_ALPHA5:2.2
	REL8_3_10:2.2
	REL8_4_3:2.2
	REL9_0_ALPHA4:2.2
	REL9_0_ALPHA4_BRANCH:2.2.0.12
	REL8_5_ALPHA3:2.2
	REL8_5_ALPHA3_BRANCH:2.2.0.10
	REL8_3_9:2.2
	REL8_4_2:2.2
	REL8_5_ALPHA2:2.2
	REL8_5_ALPHA2_BRANCH:2.2.0.8
	REL8_3_8:2.2
	REL8_4_1:2.2
	REL8_5_ALPHA1:2.2
	REL8_5_ALPHA1_BRANCH:2.2.0.6
	REL8_4_STABLE:2.2.0.4
	REL8_4_0:2.2
	REL8_4_RC2:2.2
	REL8_4_RC1:2.2
	REL8_4_BETA2:2.2
	REL8_4_BETA1:2.2
	REL8_3_7:2.2
	REL8_3_6:2.2
	REL8_3_5:2.2
	REL8_3_4:2.2
	REL8_3_3:2.2
	REL8_3_2:2.2
	REL8_3_1:2.2
	REL8_3_STABLE:2.2.0.2
	REL8_3_0:2.2
	REL8_3_RC2:2.2
	REL8_3_RC1:2.2
	REL8_3_BETA4:2.1;
locks; strict;
comment	@# @;


2.4
date	2010.08.17.04.37.21;	author petere;	state Exp;
branches;
next	2.3;

2.3
date	2010.07.29.19.34.40;	author petere;	state Exp;
branches;
next	2.2;

2.2
date	2007.12.10.05.32.51;	author tgl;	state Exp;
branches
	2.2.16.1;
next	2.1;

2.1
date	2007.12.02.22.33.20;	author tgl;	state Exp;
branches;
next	;

2.2.16.1
date	2010.07.29.19.34.37;	author petere;	state Exp;
branches;
next	2.2.16.2;

2.2.16.2
date	2010.08.17.04.37.17;	author petere;	state Exp;
branches;
next	;


desc
@@


2.4
log
@Spell and markup checking
@
text
@<!-- $PostgreSQL: pgsql/doc/src/sgml/pgtrgm.sgml,v 2.3 2010/07/29 19:34:40 petere Exp $ -->

<sect1 id="pgtrgm">
 <title>pg_trgm</title>

 <indexterm zone="pgtrgm">
  <primary>pg_trgm</primary>
 </indexterm>

 <para>
  The <filename>pg_trgm</filename> module provides functions and operators
  for determining the similarity of text based on trigram matching, as
  well as index operator classes that support fast searching for similar
  strings.
 </para>

 <sect2>
  <title>Trigram (or Trigraph) Concepts</title>

  <para>
   A trigram is a group of three consecutive characters taken
   from a string.  We can measure the similarity of two strings by
   counting the number of trigrams they share.  This simple idea
   turns out to be very effective for measuring the similarity of
   words in many natural languages.
  </para>

  <note>
   <para>
    A string is considered to have two spaces
    prefixed and one space suffixed when determining the set
    of trigrams contained in the string.
    For example, the set of trigrams in the string
    <quote><literal>cat</literal></quote> is
    <quote><literal>  c</literal></quote>,
    <quote><literal> ca</literal></quote>,
    <quote><literal>cat</literal></quote>, and
    <quote><literal>at </literal></quote>.
   </para>
  </note>
 </sect2>

 <sect2>
  <title>Functions and Operators</title>

  <table id="pgtrgm-func-table">
   <title><filename>pg_trgm</filename> functions</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Function</entry>
      <entry>Returns</entry>
      <entry>Description</entry>
     </row>
    </thead>

    <tbody>
     <row>
      <entry><function>similarity(text, text)</function></entry>
      <entry><type>real</type></entry>
      <entry>
       Returns a number that indicates how similar the two arguments are.
       The range of the result is zero (indicating that the two strings are
       completely dissimilar) to one (indicating that the two strings are
       identical).
      </entry>
     </row>
     <row>
      <entry><function>show_trgm(text)</function></entry>
      <entry><type>text[]</type></entry>
      <entry>
       Returns an array of all the trigrams in the given string.
       (In practice this is seldom useful except for debugging.)
      </entry>
     </row>
     <row>
      <entry><function>show_limit()</function></entry>
      <entry><type>real</type></entry>
      <entry>
       Returns the current similarity threshold used by the <literal>%</>
       operator.  This sets the minimum similarity between
       two words for them to be considered similar enough to
       be misspellings of each other, for example.
      </entry>
     </row>
     <row>
      <entry><function>set_limit(real)</function></entry>
      <entry><type>real</type></entry>
      <entry>
       Sets the current similarity threshold that is used by the <literal>%</>
       operator.  The threshold must be between 0 and 1 (default is 0.3).
       Returns the same value passed in.
      </entry>
     </row>
    </tbody>
   </tgroup>
  </table>

  <table id="pgtrgm-op-table">
   <title><filename>pg_trgm</filename> operators</title>
   <tgroup cols="3">
    <thead>
     <row>
      <entry>Operator</entry>
      <entry>Returns</entry>
      <entry>Description</entry>
     </row>
    </thead>

    <tbody>
     <row>
      <entry><type>text</> <literal>%</literal> <type>text</></entry>
      <entry><type>boolean</type></entry>
      <entry>
       Returns <literal>true</> if its arguments have a similarity that is
       greater than the current similarity threshold set by
       <function>set_limit</>.
      </entry>
     </row>
    </tbody>
   </tgroup>
  </table>
 </sect2>

 <sect2>
  <title>Index Support</title>

  <para>
   The <filename>pg_trgm</filename> module provides GiST and GIN index
   operator classes that allow you to create an index over a text column for
   the purpose of very fast similarity searches.  These index types support
   the <literal>%</> similarity operator (and no other operators, so you may
   want a regular B-tree index too).
  </para>

  <para>
   Example:

<programlisting>
CREATE TABLE test_trgm (t text);
CREATE INDEX trgm_idx ON test_trgm USING gist (t gist_trgm_ops);
</programlisting>
or
<programlisting>
CREATE INDEX trgm_idx ON test_trgm USING gin (t gin_trgm_ops);
</programlisting>
  </para>

  <para>
   At this point, you will have an index on the <structfield>t</> column that
   you can use for similarity searching.  A typical query is
<programlisting>
SELECT t, similarity(t, '<replaceable>word</>') AS sml
  FROM test_trgm
  WHERE t % '<replaceable>word</>'
  ORDER BY sml DESC, t;
</programlisting>
   This will return all values in the text column that are sufficiently
   similar to <replaceable>word</>, sorted from best match to worst.  The
   index will be used to make this a fast operation even over very large data
   sets.
  </para>

  <para>
   The choice between GiST and GIN indexing depends on the relative
   performance characteristics of GiST and GIN, which are discussed elsewhere.
   As a rule of thumb, a GIN index is faster to search than a GiST index, but
   slower to build or update; so GIN is better suited for static data and GiST
   for often-updated data.
  </para>
 </sect2>

 <sect2>
  <title>Text Search Integration</title>

  <para>
   Trigram matching is a very useful tool when used in conjunction
   with a full text index.  In particular it can help to recognize
   misspelled input words that will not be matched directly by the
   full text search mechanism.
  </para>

  <para>
   The first step is to generate an auxiliary table containing all
   the unique words in the documents:

<programlisting>
CREATE TABLE words AS SELECT word FROM
        ts_stat('SELECT to_tsvector(''simple'', bodytext) FROM documents');
</programlisting>

   where <structname>documents</> is a table that has a text field
   <structfield>bodytext</> that we wish to search.  The reason for using
   the <literal>simple</> configuration with the <function>to_tsvector</>
   function, instead of using a language-specific configuration,
   is that we want a list of the original (unstemmed) words.
  </para>

  <para>
   Next, create a trigram index on the word column:

<programlisting>
CREATE INDEX words_idx ON words USING gin(word gin_trgm_ops);
</programlisting>

   Now, a <command>SELECT</command> query similar to the previous example can
   be used to suggest spellings for misspelled words in user search terms.
   A useful extra test is to require that the selected words are also of
   similar length to the misspelled word.
  </para>

  <note>
   <para>
    Since the <structname>words</> table has been generated as a separate,
    static table, it will need to be periodically regenerated so that
    it remains reasonably up-to-date with the document collection.
    Keeping it exactly current is usually unnecessary.
   </para>
  </note>
 </sect2>

 <sect2>
  <title>References</title>

  <para>
   GiST Development Site
   <ulink url="http://www.sai.msu.su/~megera/postgres/gist/"></ulink>
  </para>
  <para>
   Tsearch2 Development Site
   <ulink url="http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/"></ulink>
  </para>
 </sect2>

 <sect2>
  <title>Authors</title>

  <para>
   Oleg Bartunov <email>oleg@@sai.msu.su</email>, Moscow, Moscow University, Russia
  </para>
  <para>
   Teodor Sigaev <email>teodor@@sigaev.ru</email>, Moscow, Delta-Soft Ltd.,Russia
  </para>
  <para>
   Documentation: Christopher Kings-Lynne
  </para>
  <para>
   This module is sponsored by Delta-Soft Ltd., Moscow, Russia.
  </para>
 </sect2>

</sect1>
@


2.3
log
@Fix indentation of verbatim block elements

Block elements with verbatim formatting (literallayout, programlisting,
screen, synopsis) should be aligned at column 0 independent of the surrounding
SGML, because whitespace is significant, and indenting them creates erratic
whitespace in the output.  The CSS stylesheets already take care of indenting
the output.

Assorted markup improvements to go along with it.
@
text
@d1 1
a1 1
<!-- $PostgreSQL: pgsql/doc/src/sgml/pgtrgm.sgml,v 2.2 2007/12/10 05:32:51 tgl Exp $ -->
d133 1
a133 1
   want a regular btree index too).
@


2.2
log
@Make an editorial pass over the newly SGML-ified contrib documentation.
Fix lots of bad markup, bad English, bad explanations.

Second round of commits.  pgcrypto and pgstandby still to go...
@
text
@d1 1
a1 1
<!-- $PostgreSQL$ -->
d139 1
a139 1
  <programlisting>
d142 1
a142 1
  </programlisting>
d144 1
a144 1
  <programlisting>
d146 1
a146 1
  </programlisting>
d152 1
a152 2
  </para>
  <programlisting>
d157 1
a157 2
  </programlisting>
  <para>
a185 1
  </para>
d187 1
a187 1
  <programlisting>
d190 1
a190 1
  </programlisting>
a191 1
  <para>
a200 1
  </para>
d202 1
a202 1
  <programlisting>
d204 1
a204 1
  </programlisting>
a205 1
  <para>
@


2.2.16.1
log
@Fix indentation of verbatim block elements

Block elements with verbatim formatting (literallayout, programlisting,
screen, synopsis) should be aligned at column 0 independent of the surrounding
SGML, because whitespace is significant, and indenting them creates erratic
whitespace in the output.  The CSS stylesheets already take care of indenting
the output.

Assorted markup improvements to go along with it.
@
text
@d1 1
a1 1
<!-- $PostgreSQL: pgsql/doc/src/sgml/pgtrgm.sgml,v 2.2 2007/12/10 05:32:51 tgl Exp $ -->
d139 1
a139 1
<programlisting>
d142 1
a142 1
</programlisting>
d144 1
a144 1
<programlisting>
d146 1
a146 1
</programlisting>
d152 2
a153 1
<programlisting>
d158 2
a159 1
</programlisting>
d188 1
d190 1
a190 1
<programlisting>
d193 1
a193 1
</programlisting>
d195 1
d205 1
d207 1
a207 1
<programlisting>
d209 1
a209 1
</programlisting>
d211 1
@


2.2.16.2
log
@Spell and markup checking
@
text
@d1 1
a1 1
<!-- $PostgreSQL: pgsql/doc/src/sgml/pgtrgm.sgml,v 2.2.16.1 2010/07/29 19:34:37 petere Exp $ -->
d133 1
a133 1
   want a regular B-tree index too).
@


2.1
log
@Ensure that all <sect1> and <refentry> tags have IDs.  This is needed
to ensure that the resulting webpages have predictable URLs, instead of
ever-changing numeric IDs.  The new contrib docs were the biggest
offender, but some old stuff had the problem too.  Also, rename a couple
of new contrib sgml files for consistency's sake.
@
text
@d1 2
d5 1
a5 1
 
d11 4
a14 2
  The <literal>pg_trgm</literal> module provides functions and index classes
  for determining the similarity of text based on trigram matching.
d18 2
a19 1
  <title>Trigram (or Trigraph)</title>
d21 5
a25 8
   A trigram is a set of three consecutive characters taken
   from a string.  A string is considered to have two spaces
   prefixed and one space suffixed when determining the set
   of trigrams that comprise the string.
  </para>
  <para>
   eg. The set of trigrams in the word "cat" is "  c", " ca", 
   "at " and "cat".
d27 14
d44 5
a48 4
  <title>Public Functions</title>
  <table>
   <title><literal>pg_trgm</literal> functions</title>
   <tgroup cols="2">
d52 1
d56 1
d59 2
a60 1
      <entry><literal>real similarity(text, text)</literal></entry>
d62 4
a65 6
       <para>
        Returns a number that indicates how closely matches the two
        arguments are.  A zero result indicates that the two words
        are completely dissimilar, and a result of one indicates that
        the two words are identical.
       </para>
d69 2
a70 1
      <entry><literal>real show_limit()</literal></entry>
d72 2
a73 6
       <para>
        Returns the current similarity threshold used by the '%'
        operator.  This in effect sets the minimum similarity between
        two words in order that they be considered similar enough to
        be misspellings of each other, for example.
       </para>
d77 2
a78 1
      <entry><literal>real set_limit(real)</literal></entry>
d80 4
a83 4
       <para>
        Sets the current similarity threshold that is used by the '%'
        operator, and is returned by the show_limit() function.
       </para>
d87 2
a88 1
      <entry><literal>text[] show_trgm(text)</literal></entry>
d90 3
a92 4
       <para>
        Returns an array of all the trigrams of the supplied text
        parameter.
       </para>
d95 8
d104 10
a113 1
      <entry>Operator: <literal>text % text (returns boolean)</literal></entry> 
d115 3
a117 6
       <para>
        The '%' operator returns TRUE if its two arguments have a similarity
        that is greater than the similarity threshold set by set_limit(). It
        will return FALSE if the similarity is less than the current
        threshold.
       </para>
d126 2
a127 1
  <title>Public Index Operator Class</title>
d129 5
a133 4
   The <literal>pg_trgm</literal> module comes with the 
   <literal>gist_trgm_ops</literal> index operator class that allows a
   developer to create an index over a text column for the purpose
   of very fast similarity searches.
d135 1
d137 2
a138 3
   To use this index, the '%' operator must be used and an appropriate
   similarity threshold for the application must be set. Example:
  </para>
d143 6
d150 2
a151 2
   At this point, you will have an index on the t text column that you
   can use for similarity searching. Example:
d154 4
a157 9
SELECT
        t,
        similarity(t, 'word') AS sml
FROM
        test_trgm
WHERE
        t % 'word'
ORDER BY
        sml DESC, t;
d161 11
a171 2
   similar to 'word', sorted from best match to worst.  The index will
   be used to make this a fast operation over very large data sets.
d177 1
d180 3
a182 1
   with a full text index.
d184 1
d189 1
d191 2
a192 2
CREATE TABLE words AS SELECT word FROM 
        stat('SELECT to_tsvector(''simple'', bodytext) FROM documents');
d194 1
d197 4
a200 6
   <structfield>bodytext</> that we wish to search.  The use of the
   <literal>simple</> configuration with the <function>to_tsvector</>
   function, instead of just using the already
   existing vector is to avoid creating a list of already stemmed
   words.  This way, only the original, unstemmed words are added
   to the word list.
d202 1
d206 1
d208 1
a208 1
CREATE INDEX words_idx ON words USING gist(word gist_trgm_ops);
d210 1
d212 4
a215 19
   or
  </para>
  <programlisting>
CREATE INDEX words_idx ON words USING gin(word gist_trgm_ops);
  </programlisting>
  <para>
   Now, a <literal>SELECT</literal> query similar to the example above can be 
   used to suggest spellings for misspelled words in user search terms. A
   useful extra clause is to ensure that the similar words are also
   of similar length to the misspelled word.
  </para>
  <para>
   <note>
    <para>
     Since the <structname>words</> table has been generated as a separate,
     static table, it will need to be periodically regenerated so that
     it remains up to date with the document collection.
    </para>
   </note>
d217 9
d230 1
d243 1
d251 1
a251 1
   Documentation: Christopher Kings-Lynne 
@

