I process log files with python. Let´s say that I have a log file that contains a line which is START
and a line that is END
, like below:
START
one line
two line
...
n line
END
What I do want is to be able to store the content between the START
and END
lines for further processing.
I do the following in Python:
with open (file) as name_of_file:
for line in name_of_file:
if 'START' in line: # We found the start_delimiter
print(line)
found_start = True
for line in name_of_file: # We now read until the end delimiter
if 'END' in line: # We exit here as we have the info
found_end=True
break
else:
if not (line.isspace()): # We do not want to add to the data empty strings, so we ensure the line is not empty
data.append(line.replace(',','').strip().split()) # We store information in a list called data we do not want ','' or spaces
if(found_start and found_end):
relevant_data=data
And then I process the relevant_data
.
Looks to far complicated for the purity of Python, and hence my question: is there a more Pythonic way of doing this?
Thanks!
To perform that, you can use
iter(callable, sentinel)
discussed in this post , that will read until a sentinel value is reached, in your case 'END' (after applying.strip()
).You are right that there is something not OK with having a nested loop over the same iterator. File objects are already iterators, and you can use that to your advantage. For example, to find the first line with a
START
in it:This will raise a
StopIteration
if there is no such line. It also sets the file pointer to the beginning of the first line you care about.Getting the last line without anything that comes after it is a bit more complicated because it's difficult to set external state in a generator expression. Instead, you can make a simple generator:
The generator will yield nothing if you don't have a
START
, but it will yield all the lines until the end if there is noEND
, so it differs a little from your implementation. You would use the generator to replace your loop entirely:Wrapping the generator in
list
immediately processes it, so the lines persist even after you close the file. The iterator can be used repeatedly because at the end of your call, the file pointer will be just past theEND
line:The relatively lesser known form of
iter
converts any callable object that accepts no arguments into an iterator. The end is reached when the callable returns the sentinel value, in this case an empty list.This is a mission for regular expressions
re
, for example:The
re.S
is necessary for spanning multiple lines.