Complete Suffix Array

Question

A suffix array will index all the suffixes for a given list of strings, but what if you're trying to index all the possible unique substrings? I'm a bit new at this, so here's an example of what I mean:

Given the string

abcd

A suffix array indexes (at least to my understanding)

(abcd,bcd,cd,d)

I would like to index (all the substrings)

(abcd,bcd,cd,d,abc,bc,c,ab,b,a)

Is a suffix array what I'm looking for? If so, what do I do to get all the substrings indexed? If not, where should I be looking? Also what would I google for to contrast "all substrings" vs "suffix substrings"?

Answer 1

The suffix array does what you need already, because every substring is a prefix of one of the suffixes. Specifically, given your suffix array

abcd bcd cd d

and assume you are looking for substring "bc", then you can find that by looking for all suffixes that start with "bc" (there is only one in this case, "bcd"). Since a suffix array is lexicographically sorted, finding all suffixes that share a certain prefix corresponds to a binary search across the suffix array, and the result will be one continuous range of entries of the suffix array.

However, there are optimised search methods using the suffix array combined with auxiliary data structures, such as the LCP (longest-common prefix) array, or wavelet trees. See Navarro's 2007 survey for a description of such methods (DOI 10.1145/1216370.1216372).

To take into account the comments made below, I suggest combining each suffix with the number of substrings it represents. In a simple example like the above this would be

4 abcd
3 bcd
2 bc
1 d

because, for example, the first suffix "abcd" represents the 4 substrings "a", "ab", "abc", "abcd". However, in a more complex example, say for the string "abcabxdabe", the first two entries of the suffix array would be

10 abcabxdabe
1 abe

because the second entry represents substrings "a", "ab" and "abe", but "a" and "ab" are also represented by the first entry.

How to calculate the number of substrings an entry represents? --> The length of the suffix minus the length of the longest prefix it has in common with the previous suffix. E.g. in the "abe" example, that is 3 (its length) minus 2 (the length of "ab", the longest prefix it shares with the previous entry). So these numbers can be generated in one pass over the suffix array, and even faster if you have also generated the LCP (longest-common prefix) array.

The next step would be to generate accumulated counts:

10 abcabxdabe
11 abe
16 abxdabe
...

and then to find an efficient way to make use of the accumulated counts. E.g. if you want to get the 13th substring lexicographically, you'd have to find the first entry that has an accumulated count greater than or equal to 13. That would be "16 abxdabe" above. Then remove the prefix it shares with the previous entry (yields "xdabe"), and then jump to the position after the 2nd character (because the previous entry has accumulated count 11, and 13-11==2), so you get "abxd" as the 13th substring lexicographically.

Answer 2

As has been answered already, substrings are prefixes of suffixes. Sometimes you'd like perhaps to go the other way and get suffixes of prefixes.

Beyond that, it's unclear what you're looking for with "unique substrings." I'd suggest you look up the words: type, token, maximal, supermaximal. You should have no trouble finding these in the suffix array literature.

Answer 3

You should use a variation of 'Trie'. Essentially, if you have ABCD, create tree which is a merger of paths: root->A->B->C->D, root->B->C->D, root->C->D and root->D. Now, at every node keep a list of locations where string root->.->.->node was observed.