Go from RGB to scalars in matplotlib colormap

Question

I am using matplotlib to convert arrays of 400 floats (read from a text file) to images of 100x100 pixels, using a pre-defined colormap:

vec = [ float(i) for i in l.split()[1:] ]
v = normalise(np.array(vec).reshape((20,20)))
im = Image.fromarray(np.uint8(cm.coolwarm(v)*255))
im = im.resize((100, 100))

This works fine. But now I'd like to reverse this process. That is, given an image, I'd like to go from the RGB values of each pixel back to scalars (to retrieve my original array). How would I do that?

NB: my question is related to this one, except that I don't want to take any image and convert it back to an array -- only the ones that have been created by my colormap in the first place.