Are there character collections for all internatio

问题:

I am trying to parse utf-8 strings into "bite sized" segments. For example, I would like to break down a text into "sentences".

Is there a comprehensive collection of characters (or regex) that correspond to end of sentences in all languages? I'm looking for something that would capture the Latin period, exclamation and interrogation marks, the Chinese and Japanese full stop, etc.

Something like the above but for the equivalent of a comma would be great too.

回答1:

I haven’t encountered any compilations of such information, and I would expect it to be a major effort to collect it. For some widely used languages, you could get the information from The Chicago Manual of Style. There is some information about punctuation marks commonly used in different languages at http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters-other.html but is covers just a small set of languages and does not distinguish sentence-terminating characters.

Using just characters won’t be enough, since e.g. in English, the full stop “.” occurs in many contexts where it does not terminate a sentence, as in “e.g.” or in “1.5”.

回答2:

You need to look at code points with the \p{Sentence_Break=STerm} or \p{Sentence_Break=ATerm} properties that also have the \p{Terminal_Punctuation} property. Running the unichars script against Unicode v6.1, we learn that these code points meet all those criteria:

$ unichars -gas '[\p{Sentence_Break=STerm}\p{Sentence_Break=ATerm}]' '\p{Terminal_Punctuation}'
U+00021 ‭ !  GC=Po SC=Common       EXCLAMATION MARK
U+0002E ‭ .  GC=Po SC=Common       FULL STOP
U+0003F ‭ ?  GC=Po SC=Common       QUESTION MARK
U+00589 ‭ ։  GC=Po SC=Common       ARMENIAN FULL STOP
U+0061F ‭ ؟  GC=Po SC=Common       ARABIC QUESTION MARK
U+006D4 ‭ ۔  GC=Po SC=Arabic       ARABIC FULL STOP
U+00700 ‭ ܀  GC=Po SC=Syriac       SYRIAC END OF PARAGRAPH
U+00701 ‭ ܁  GC=Po SC=Syriac       SYRIAC SUPRALINEAR FULL STOP
U+00702 ‭ ܂  GC=Po SC=Syriac       SYRIAC SUBLINEAR FULL STOP
U+007F9 ‭ ߹  GC=Po SC=Nko          NKO EXCLAMATION MARK
U+00964 ‭ ।  GC=Po SC=Common       DEVANAGARI DANDA
U+00965 ‭ ॥  GC=Po SC=Common       DEVANAGARI DOUBLE DANDA
U+0104A ‭ ၊  GC=Po SC=Myanmar      MYANMAR SIGN LITTLE SECTION
U+0104B ‭ ။  GC=Po SC=Myanmar      MYANMAR SIGN SECTION
U+01362 ‭ ።  GC=Po SC=Ethiopic     ETHIOPIC FULL STOP
U+01367 ‭ ፧  GC=Po SC=Ethiopic     ETHIOPIC QUESTION MARK
U+01368 ‭ ፨  GC=Po SC=Ethiopic     ETHIOPIC PARAGRAPH SEPARATOR
U+0166E ‭ ᙮  GC=Po SC=Canadian_Aboriginal CANADIAN SYLLABICS FULL STOP
U+01803 ‭ ᠃  GC=Po SC=Common       MONGOLIAN FULL STOP
U+01809 ‭ ᠉  GC=Po SC=Mongolian    MONGOLIAN MANCHU FULL STOP
U+01944 ‭ ᥄  GC=Po SC=Limbu        LIMBU EXCLAMATION MARK
U+01945 ‭ ᥅  GC=Po SC=Limbu        LIMBU QUESTION MARK
U+01AA8 ‭ ᪨  GC=Po SC=Tai_Tham     TAI THAM SIGN KAAN
U+01AA9 ‭ ᪩  GC=Po SC=Tai_Tham     TAI THAM SIGN KAANKUU
U+01AAA ‭ ᪪  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAAN
U+01AAB ‭ ᪫  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAANKUU
U+01B5A ‭ ᭚  GC=Po SC=Balinese     BALINESE PANTI
U+01B5B ‭ ᭛  GC=Po SC=Balinese     BALINESE PAMADA
U+01B5E ‭ ᭞  GC=Po SC=Balinese     BALINESE CARIK SIKI
U+01B5F ‭ ᭟  GC=Po SC=Balinese     BALINESE CARIK PAREREN
U+01C3B ‭ ᰻  GC=Po SC=Lepcha       LEPCHA PUNCTUATION TA-ROL
U+01C3C ‭ ᰼  GC=Po SC=Lepcha       LEPCHA PUNCTUATION NYET THYOOM TA-ROL
U+01C7E ‭ ᱾  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION MUCAAD
U+01C7F ‭ ᱿  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION DOUBLE MUCAAD
U+0203C ‭ ‼  GC=Po SC=Common       DOUBLE EXCLAMATION MARK
U+0203D ‭ ‽  GC=Po SC=Common       INTERROBANG
U+02047 ‭ ⁇  GC=Po SC=Common       DOUBLE QUESTION MARK
U+02048 ‭ ⁈  GC=Po SC=Common       QUESTION EXCLAMATION MARK
U+02049 ‭ ⁉  GC=Po SC=Common       EXCLAMATION QUESTION MARK
U+02E2E ‭ ⸮  GC=Po SC=Common       REVERSED QUESTION MARK
U+03002 ‭ 。 GC=Po SC=Common       IDEOGRAPHIC FULL STOP
U+0A4FF ‭ ꓿  GC=Po SC=Lisu         LISU PUNCTUATION FULL STOP
U+0A60E ‭ ꘎  GC=Po SC=Vai          VAI FULL STOP
U+0A60F ‭ ꘏  GC=Po SC=Vai          VAI QUESTION MARK
U+0A6F3 ‭ ꛳  GC=Po SC=Bamum        BAMUM FULL STOP
U+0A6F7 ‭ ꛷  GC=Po SC=Bamum        BAMUM QUESTION MARK
U+0A876 ‭ ꡶  GC=Po SC=Phags_Pa     PHAGS-PA MARK SHAD
U+0A877 ‭ ꡷  GC=Po SC=Phags_Pa     PHAGS-PA MARK DOUBLE SHAD
U+0A8CE ‭ ꣎  GC=Po SC=Saurashtra   SAURASHTRA DANDA
U+0A8CF ‭ ꣏  GC=Po SC=Saurashtra   SAURASHTRA DOUBLE DANDA
U+0A92F ‭ ꤯  GC=Po SC=Kayah_Li     KAYAH LI SIGN SHYA
U+0A9C8 ‭ ꧈  GC=Po SC=Javanese     JAVANESE PADA LINGSA
U+0A9C9 ‭ ꧉  GC=Po SC=Javanese     JAVANESE PADA LUNGSI
U+0AA5D ‭ ꩝  GC=Po SC=Cham         CHAM PUNCTUATION DANDA
U+0AA5E ‭ ꩞  GC=Po SC=Cham         CHAM PUNCTUATION DOUBLE DANDA
U+0AA5F ‭ ꩟  GC=Po SC=Cham         CHAM PUNCTUATION TRIPLE DANDA
U+0AAF0 ‭ ꫰  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHAN
U+0AAF1 ‭ ꫱  GC=Po SC=Meetei_Mayek MEETEI MAYEK AHANG KHUDAM
U+0ABEB ‭ ꯫  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHEI
U+0FE52 ‭ ﹒ GC=Po SC=Common       SMALL FULL STOP
U+0FE56 ‭ ﹖ GC=Po SC=Common       SMALL QUESTION MARK
U+0FE57 ‭ ﹗ GC=Po SC=Common       SMALL EXCLAMATION MARK
U+0FF01 ‭ ！ GC=Po SC=Common       FULLWIDTH EXCLAMATION MARK
U+0FF0E ‭ ． GC=Po SC=Common       FULLWIDTH FULL STOP
U+0FF1F ‭ ？ GC=Po SC=Common       FULLWIDTH QUESTION MARK
U+0FF61 ‭ ｡  GC=Po SC=Common       HALFWIDTH IDEOGRAPHIC FULL STOP
U+11047 ‭