How to calculate Dynamodb item size? Getting valid

Question

ValidationException: ValidationException: 400 Bad Request {u'message': u'Item size has exceeded the maximum allowed size', u'__type': u'com.amazon.coral.validate#ValidationException'}

The item object I have, has size of 92004 Bytes

>>> iii
<boto.dynamodb2.items.Item object at 0x7f7922c97190>
>>> iiip = iii.prepare_full() # it is now in dynamodb format e.g. "Item":{"time":{"N":"300"}, "user":{"S":"self"}} 

>>> len(json.dumps(iiip))
92004
>>>

The size I get 92004 is less than 400KB, Why do I see the above mentioned error when saving the item?

Any pointers?

EDIT:

I played around with different sizes of data,

>>> i00['Resources'] = "A" * 66848; len(json.dumps(i00))
68481
>>> i = Item(ct.table, data=i00); i.save()
True
>>> i.delete()
True
>>> i00['Resources'] = "A" * 66849; len(json.dumps(i00))
68482
>>> i = Item(ct.table, data=i00); i.save()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/dynamodb2/items.py", line 455, in save
    returned = self.table._put_item(final_data, expects=expects)
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/dynamodb2/table.py", line 835, in _put_item
    self.connection.put_item(self.table_name, item_data, **kwargs)
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 1510, in put_item
    body=json.dumps(params))
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 2842, in make_request
    retry_handler=self._retry_handler)
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/connection.py", line 954, in _mexe
    status = retry_handler(response, i, next_sleep)
  File "/var/www/virtualenv/ken/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 2882, in _retry_handler
    response.status, response.reason, data)
ValidationException: ValidationException: 400 Bad Request
{u'message': u'Item size has exceeded the maximum allowed size', u'__type': u'com.amazon.coral.validate#ValidationException'}

In other words, the size of cloudtrail data has to be less than 68482 bytes. I wonder why they claim it to be 400KB. Clearly, I am missing something.

Answer 1

Can you share your input data if no issues? Are you trying to insert bulk data using a flat file as input? Looks like dynamoDB is not able to interpret new line or is treating all records as single record!! I got a similar error, but for hash key field. I was trying bulk data load using hive scripts. I realized that the attributes should be tab separated, and by fixing the input format, error was fixed for me!!

Try inserting single record at a time. If you don't get the above error, then it is to do with the format of the data!!

Answer 2

Answering my own question since it might help someone with the same problem. I contacted aws technical support and here is the explanation:

I had 5 indexes on my dynamodb table, since the data is replicated for each index; the total data = 68481 * (5 + 1) = 410886 which is close to 400KB.

I feel this is missing from Dynamodb documentation and it'd be nice it amazon adds it.

So, to summarize, the total data (item size) that ends up being saved in dynamodb table is = Acutal data * (number of indexes + 1).