Raised Google Drive API Per-user limit, still gett

问题:

Similar to Raising Google Drive API per-user limit does not prevent rate limit exceptions

In Drive API Console, quotas looks like this:

Despite Per-user limit being set to an unnecessarily high requests/sec, I am still getting rate errors at the user-level.

What I'm doing:

I am using approx 8 threads uploading to Drive, and they are ALL implementing a robust exponential back-off of 1, 2, 4, 8, 16, 32, 64 sec back-off respectively (pretty excessive back-off, but necessary imho). The problem can still persists through all of this back-off in some of the threads.

Is there some other rate that is not being advertised / cannot be set?

I'm nowhere near the requests/sec, and still have 99.53% total quota. Why am I still getting userRateLimitExceeded errors?

回答1:

userRateLimitExceeded is flood protection basically. Its used to prevent people from sending to many requests to fast.

Indicates that the user rate limit has been exceeded. The maximum rate limit is 10 qps per IP address. The default value set in Google Developers Console is 1 qps per IP address. You can increase this limit in the Google Developers Console to a maximum of 10 qps.

You need to slow your code down, by implementing Exponential Backoff.

1. Make a request to the API
2. Receive an error response that has a retry-able error code
3. Wait 1s + random_number_milliseconds seconds
4. Retry request
5. Receive an error response that has a retry-able error code
6. Wait 2s + random_number_milliseconds seconds
7. Retry request
8. Receive an error response that has a retry-able error code
9. Wait 4s + random_number_milliseconds seconds
10. Retry request
11. Receive an error response that has a retry-able error code
12. Wait 8s + random_number_milliseconds seconds
13. Retry request
14. Receive an error response that has a retry-able error code
15. Wait 16s + random_number_milliseconds seconds
16. Retry request
17. If you still get an error, stop and log the error.

The idea is that every time you see that error you wait a few seconds then try and send it again. If you get the error again you wait a little longer.

Quota user:

Now I am not sure how your application works but, If all the quests are coming from the same IP this could cause your issue. As you can see by the Quota you get 10 requests per second / per user. How does Google know its a user? They look at the IP address. If all your reuqests are coming from the same IP then its one user and you are locked to 10 requests per second.

You can get around this by adding QuotaUser to your request.

quotaUser - Alternative to userIp. Link

1. Lets you enforce per-user quotas from a server-side application even in cases when the user's IP address is unknown. This can occur, for example, with applications that run cron jobs on App Engine on a user's behalf.
2. You can choose any arbitrary string that uniquely identifies a user, but it is limited to 40 characters.
3. Overrides userIp if both are provided.
4. Learn more about capping usage.

If you send a different quotauser on every reqest, say a random number, then Google thinks its a different user and will assume that its only one request in the 10 seconds. Its a little trick to get around the ip limitation when running server applications that request everything from the same IP.