How to create pivot with totals (margins) in Panda

Question

For example, I have a very simple data frame:

values = pd.Series(i for i in range(5))
rows = pd.Series(['a', 'b', 'a', 'a', 'b'])
columns = pd.date_range('20130101',periods=5)

df = pd.DataFrame({'values': values, 'rows': rows, 'columns': columns})

and how it look:

              columns rows  values
0 2013-01-01 00:00:00    a       0
1 2013-01-02 00:00:00    b       1
2 2013-01-03 00:00:00    a       2
3 2013-01-04 00:00:00    a       3
4 2013-01-05 00:00:00    b       4

I have success when I try to make pivot without margins(totals):

pivot = pd.pivot_table(
    data=df,
    rows='rows',
    cols='columns',
    values='values',
    margins=False
)

how it look:

columns  2013-01-01  2013-01-02  2013-01-03  2013-01-04  2013-01-05
rows                                                               
a                 0         NaN           2           3         NaN
b               NaN           1         NaN         NaN           4

but if I want create pivot with margins:

pivot = pd.pivot_table(
    data=df,
    rows='rows',
    cols='columns',
    values='values',
    margins=True
)

I receive error:

Traceback (most recent call last):
  File "./test.py", line 17, in <module>
    margins=True
  File "/usr/local/lib/python2.6/dist-packages/pandas/tools/pivot.py", line 135, in pivot_table
    cols=cols, aggfunc=aggfunc)
  File "/usr/local/lib/python2.6/dist-packages/pandas/tools/pivot.py", line 174, in _add_margins
    piece[all_key] = margin[key]
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/frame.py", line 2119, in __setitem__
    self._set_item(key, value)
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/frame.py", line 2166, in _set_item
    NDFrame._set_item(self, key, value)
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/generic.py", line 679, in _set_item
    self._data.set(key, value)
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/internals.py", line 1781, in set
    self.insert(len(self.items), item, value)
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/internals.py", line 1801, in insert
    new_items = self.items.delete(loc)
  File "/usr/local/lib/python2.6/dist-packages/pandas/core/index.py", line 2610, in delete
    new_labels = [np.delete(lab, loc) for lab in self.labels]
  File "/usr/lib/pymodules/python2.6/numpy/lib/function_base.py", line 3339, in delete
    "invalid entry")
ValueError: invalid entry

• Python version: 2.6.8
• Pandas version: 0.12.0
• System: Debian Linux with 3.2.0 Kernel, 64bit.

Thanks.

Answer 1

I can reproduce your issue. It sounds like a bug. At least I found that reassigning the column names workaround the issue:

df.columns = ['rows', 'columns', 'values']

pd.pivot_table(
    ...:     data=df,
    ...:     rows='rows',
    ...:     cols='columns',
    ...:     values='values',
    ...:     margins=True)
Out[34]: 
columns                     a    b  All
rows                                   
2013-01-01 00:00:00  0.000000  NaN    0
2013-01-02 00:00:00       NaN  1.0    1
2013-01-03 00:00:00  2.000000  NaN    2
2013-01-04 00:00:00  3.000000  NaN    3
2013-01-05 00:00:00       NaN  4.0    4
All                  1.666667  2.5    2