Sync our Snowball stemmer dictionaries with current upstream

Enterprise / PostgreSQL - Tom Lane [sss.pgh.pa.us] - 24 September 2018 21:29 EDT

We haven't touched these since text search functionality landed in core in 2007 :-(. While the upstream project isn't a beehive of activity, they do make additions and bug fixes from time to time. Update our copies of these files.

Also update our documentation about how to keep things in sync, since they're not making distribution tarballs these days. Fortunately, their source code turns out to be a breeze to build.

Notable changes:

- The non-UTF8 version of the hungarian stemmer now works in LATIN2 not LATIN1.

- New stemmers have appeared for arabic, indonesian, irish, lithuanian, nepali, and tamil. These all work in UTF8, and the indonesian and irish ones also work in LATIN1.

(There are some new stemmers that I did not incorporate, mainly because their names don't match the underlying languages, suggesting that they're not to be considered mainstream.)

Worth noting: the upstream Nepali dictionary was contributed by Arthur Zakirov.

initdb forced because the contents of snowball_create.sql have changed.

Still TODO: see about updating the stopword lists.

Arthur Zakirov, minor mods and doc work by me

Discussion: https://postgr.es/m/20180626122025.GA12647@zakirov.localdomain Discussion: https://postgr.es/m/20180219140849.GA9050@zakirov.localdomain

fd582317e1 Sync our Snowball stemmer dictionaries with current upstream.
doc/src/sgml/textsearch.sgml | 10 +-
src/backend/snowball/Makefile | 16 +-
src/backend/snowball/README | 53 +-
src/backend/snowball/dict_snowball.c | 86 +-
.../snowball/libstemmer/stem_ISO_8859_1_danish.c | 130 +-
.../snowball/libstemmer/stem_ISO_8859_1_dutch.c | 298 ++-
.../snowball/libstemmer/stem_ISO_8859_1_english.c | 542 +++---
.../snowball/libstemmer/stem_ISO_8859_1_finnish.c | 513 +++---
.../snowball/libstemmer/stem_ISO_8859_1_french.c | 577 +++---
.../snowball/libstemmer/stem_ISO_8859_1_german.c | 418 ++---
.../libstemmer/stem_ISO_8859_1_indonesian.c | 414 +++++
.../snowball/libstemmer/stem_ISO_8859_1_irish.c | 490 +++++
.../snowball/libstemmer/stem_ISO_8859_1_italian.c | 372 ++--
.../libstemmer/stem_ISO_8859_1_norwegian.c | 120 +-
.../snowball/libstemmer/stem_ISO_8859_1_porter.c | 329 ++--
.../libstemmer/stem_ISO_8859_1_portuguese.c | 461 +++--
.../snowball/libstemmer/stem_ISO_8859_1_spanish.c | 375 ++--
.../snowball/libstemmer/stem_ISO_8859_1_swedish.c | 102 +-
...9_1_hungarian.c => stem_ISO_8859_2_hungarian.c} | 751 +++-----
.../snowball/libstemmer/stem_ISO_8859_2_romanian.c | 286 ++-
.../snowball/libstemmer/stem_KOI8_R_russian.c | 313 ++--
.../snowball/libstemmer/stem_UTF_8_arabic.c | 1683 +++++++++++++++++
.../snowball/libstemmer/stem_UTF_8_danish.c | 130 +-
src/backend/snowball/libstemmer/stem_UTF_8_dutch.c | 298 ++-
.../snowball/libstemmer/stem_UTF_8_english.c | 542 +++---
.../snowball/libstemmer/stem_UTF_8_finnish.c | 515 +++---
.../snowball/libstemmer/stem_UTF_8_french.c | 577 +++---
.../snowball/libstemmer/stem_UTF_8_german.c | 412 ++---
.../snowball/libstemmer/stem_UTF_8_hungarian.c | 763 +++-----
.../snowball/libstemmer/stem_UTF_8_indonesian.c | 414 +++++
src/backend/snowball/libstemmer/stem_UTF_8_irish.c | 490 +++++
.../snowball/libstemmer/stem_UTF_8_italian.c | 372 ++--
.../snowball/libstemmer/stem_UTF_8_lithuanian.c | 850 +++++++++
.../snowball/libstemmer/stem_UTF_8_nepali.c | 424 +++++
.../snowball/libstemmer/stem_UTF_8_norwegian.c | 120 +-
.../snowball/libstemmer/stem_UTF_8_porter.c | 329 ++--
.../snowball/libstemmer/stem_UTF_8_portuguese.c | 461 +++--
.../snowball/libstemmer/stem_UTF_8_romanian.c | 286 ++-
.../snowball/libstemmer/stem_UTF_8_russian.c | 359 ++--
.../snowball/libstemmer/stem_UTF_8_spanish.c | 375 ++--
.../snowball/libstemmer/stem_UTF_8_swedish.c | 102 +-
src/backend/snowball/libstemmer/stem_UTF_8_tamil.c | 1915 ++++++++++++++++++++
.../snowball/libstemmer/stem_UTF_8_turkish.c | 1680 ++++++++---------
src/backend/snowball/libstemmer/utilities.c | 43 +-
src/backend/snowball/stopwords/nepali.stop | 304 ++++
src/bin/initdb/initdb.c | 12 +
src/include/catalog/catversion.h | 2 +-
src/include/snowball/libstemmer/header.h | 2 +
.../snowball/libstemmer/stem_ISO_8859_1_danish.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_dutch.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_english.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_finnish.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_french.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_german.h | 4 +-
.../libstemmer/stem_ISO_8859_1_hungarian.h | 16 -
.../libstemmer/stem_ISO_8859_1_indonesian.h | 16 +
.../snowball/libstemmer/stem_ISO_8859_1_irish.h | 16 +
.../snowball/libstemmer/stem_ISO_8859_1_italian.h | 4 +-
.../libstemmer/stem_ISO_8859_1_norwegian.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_porter.h | 4 +-
.../libstemmer/stem_ISO_8859_1_portuguese.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_spanish.h | 4 +-
.../snowball/libstemmer/stem_ISO_8859_1_swedish.h | 4 +-
.../libstemmer/stem_ISO_8859_2_hungarian.h | 16 +
.../snowball/libstemmer/stem_ISO_8859_2_romanian.h | 4 +-
.../snowball/libstemmer/stem_KOI8_R_russian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_arabic.h | 16 +
.../snowball/libstemmer/stem_UTF_8_danish.h | 4 +-
src/include/snowball/libstemmer/stem_UTF_8_dutch.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_english.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_finnish.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_french.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_german.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_hungarian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_indonesian.h | 16 +
src/include/snowball/libstemmer/stem_UTF_8_irish.h | 16 +
.../snowball/libstemmer/stem_UTF_8_italian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_lithuanian.h | 16 +
.../snowball/libstemmer/stem_UTF_8_nepali.h | 16 +
.../snowball/libstemmer/stem_UTF_8_norwegian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_porter.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_portuguese.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_romanian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_russian.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_spanish.h | 4 +-
.../snowball/libstemmer/stem_UTF_8_swedish.h | 4 +-
src/include/snowball/libstemmer/stem_UTF_8_tamil.h | 16 +
.../snowball/libstemmer/stem_UTF_8_turkish.h | 4 +-
88 files changed, 13093 insertions(+), 7303 deletions(-)

Upstream: git.postgresql.org


  • Share