python regex: capture parts of multiple strings th

Question

I am trying to capture sub-strings from a string that looks similar to

'some string, another string, '

I want the result match group to be

('some string', 'another string')

my current solution

>>> from re import match
>>> match(2 * '(.*?), ', 'some string, another string, ').groups()
('some string', 'another string')

works, but is not practicable - what I am showing here of course is massively reduced in terms of complexity compared to what I'm doing in the real project; I want to use one 'straight' (non-computed) regex pattern only. Unfortunately, my attempts have failed so far:

This doesn't match (None as result), because {2} is applied to the space only, not to the whole string:

>>> match('.*?, {2}', 'some string, another string, ')

adding parentheses around the repeated string has the comma and space in the result

>>> match('(.*?, ){2}', 'some string, another string, ').groups()
('another string, ',)

adding another set of parantheses does fix that, but gets me too much:

>>> match('((.*?), ){2}', 'some string, another string, ').groups()
('another string, ', 'another string')

adding a non-capturing modifier improves the result, but still misses the first string

>>> match('(?:(.*?), ){2}', 'some string, another string, ').groups()
('another string',)

I feel like I'm close, but I can't really seem to find the proper way.

Can anyone help me ? Any other approaches I'm not seeing ?

---

Update after the first few responses:

First up, thank you very much everyone, your help is greatly appreciated! :-)

As I said in the original post, I have omitted a lot of complexity in my question for the sake of depicting the actual core problem. For starters, in the project I am working on, I am parsing large amounts of files (currently tens of thousands per day) in a number (currently 5, soon ~25, possibly in the hundreds later) of different line-based formats. There is also XML, JSON, binary and some other data file formats, but let's stay focussed.

In order to cope with the multitude of file formats and to exploit the fact that many of them are line-based, I have created a somewhat generic Python module that loads one file after the other, applies a regex to every line and returns a large data structure with the matches. This module is a prototype, the production version will require a C++ version for performance reason which will be connected over Boost::Python and will probably add the subject of regex dialects to the list of complexities.

Also, there are not 2 repetitions, but an amount varying between currently zero and 70 (or so), the comma is not always a comma and despite what I said originally, some parts of the regex pattern will have to be computed at runtime; let's just say I have reason to try and reduce the 'dynamic' amount and have as much 'fixed' pattern as possible.

So, in a word: I must use regular expressions.

---

Attempt to rephrase: I think the core of the problem boils down to: Is there a Python RegEx notation that e.g. involves curly braces repetitions and allows me to capture

'some string, another string, '

into

('some string', 'another string')

?

Hmmm, that probably narrows it down too far - but then, any way you do it is wrong :-D

---

Second attempt to rephrase: Why do I not see the first string ('some string') in the result ? Why does the regex produce a match (indicating there's gotta be 2 of something), but only returns 1 string (the second one) ?

The problem remains the same even if I use non-numeric repetition, i.e. using + instead of {2}:

>>> match('(?:(.*?), )+', 'some string, another string, ').groups()
('another string',)

Also, it's not the second string that's returned, it is the last one:

>>> match('(?:(.*?), )+', 'some string, another string, third string, ').groups()
('third string',)

Again, thanks for your help, never ceases to amaze me how helpful peer review is while trying to find out what I actually want to know...

Answer 1

No offense, but you obviously have a lot to learn about regexes, and what you're going to learn, ultimately, is that regexes can't handle this job. I'm sure this particular task is doable with regexes, but then what? You say you have potentially hundreds of different file formats to parse! You even mentioned JSON and XML, which are fundamentally incompatible with regexes.

Do yourself a favor: forget about regexes and learn pyparsing instead. Or skip Python entirely and use a standalone parser generator like ANTLR. In either case, you'll probably find that grammars for most of your file formats have already been written.

Answer 2

I think the core of the problem boils down to: Is there a Python RegEx notation that e.g. involves curly braces repetitions and allows me to capture 'some string, another string, ' ?

I don't think there is such a notation.

But regexes are not a matter of only NOTATION , that is to say the RE string used to define a regex. It is also a matter of TOOLS, that is to say functions.

Unfortunately, I can't use findall as the string from the initial question is only a part of the problem, the real string is a lot longer, so findall only works if I do multiple regex findalls / matches / searches.

You should give more information without delaying: we could understand more rapidly what are the constraints. Because in my opinion, to answer to your problem as it has been exposed, findall() is indeed OK:

import re

for line in ('string one, string two, ',
             'some string, another string, third string, ',
             # the following two lines are only one string
             'Topaz, Turquoise, Moss Agate, Obsidian, '
             'Tigers-Eye, Tourmaline, Lapis Lazuli, '):

    print re.findall('(.+?), *',line)

Result

['string one', 'string two']
['some string', 'another string', 'third string']
['Topaz', 'Turquoise', 'Moss Agate', 'Obsidian', 'Tigers-Eye', 'Tourmaline', 'Lapis Lazuli']

Now, since you "have omitted a lot of complexity" in your question, findall() could incidentally be unsufficient to hold this complexity. Then finditer() will be used because it allows more flexibility in the selection of groups of a match

import re

for line in ('string one, string two, ',
             'some string, another string, third string, ',
             # the following two lines are only one string
             'Topaz, Turquoise, Moss Agate, Obsidian, '
             'Tigers-Eye, Tourmaline, Lapis Lazuli, '):

    print [ mat.group(1) for mat in re.finditer('(.+?), *',line) ]

gives the same result and can be complexified by writing other expression in place of mat.group(1)

Answer 3

import re

regex = " *((?:[^, ]| +[^, ])+) *, *((?:[^, ]| +[^, ])+) *, *"

print re.match(regex, 'some string, another string, ').groups()
# ('some string', 'another string')
print re.match(regex, ' some string, another string, ').groups()
# ('some string', 'another string')
print re.match(regex, ' some string , another string, ').groups()
# ('some string', 'another string')

Answer 4

In order to sum this up, it seems I am already using the best solution by constructing the regex pattern in a 'dynamic' manner:

>>> from re import match
>>> match(2 * '(.*?), ', 'some string, another string, ').groups()
('some string', 'another string')

the

2 * '(.*?)

is what I mean by dynamic. The alternative approach

>>> match('(?:(.*?), ){2}', 'some string, another string, ').groups()
('another string',)

fails to return the desired result due to the fact that (as Glenn and Alan kindly explained)

with match, the captured content gets overwritten with each repetition of the capturing group

Thanks for your help everyone! :-)

Answer 5

As described, I think this regex works fine:

import re
thepattern = re.compile("(.+?)(?:,|$)") # lazy non-empty match 
thepattern.findall("a, b, asdf, d")     # until comma or end of line
# Result:
Out[19]: ['a', ' b', ' asdf', ' d']

The key here is to use findall rather than match. The phrasing of your question suggests you prefer match, but it isn't the right tool for the job here -- it is designed to return exactly one string for each corresponding group ( ) in the regex. Since your 'number of strings' is variable, the right approach is to use either findall or split.

If this isn't what you need, then please make the question more specific.

Edit: And if you must use tuples rather than lists:

tuple(Out[19])
# Result
Out[20]: ('a', ' b', ' asdf', ' d')

Answer 6

Unless there's much more to this problem than you've explained, I don't see the point in using regexes. This is very simple to deal with using basic string methods:

[s.strip() for s in mys.split(',') if s.strip()]

Or if it has to be a tuple:

tuple(s.strip() for s in mys.split(',') if s.strip())

The code is more readable too. Please tell me if this fails to apply.

---

EDIT: Ok, there is indeed more to this problem than it initially seemed. Leaving this for historical purposes though. (Guess I'm not 'disciplined' :) )