MLlib MatrixFactorizationModel recommendProducts(u

I trained a MatrixFactorizationModel model using ALS.train() and now using model.recommendProducts(user, num) to get the top recommended products, but the code fails on some users with the following error:

  user_products = model.call("recommendProducts", user, prodNum)
  File "/usr/lib/spark/python/pyspark/mllib/common.py", line 136, in call
    return callJavaFunc(self._sc, getattr(self._java_model, name), *a)
  File "/usr/lib/spark/python/pyspark/mllib/common.py", line 113, in callJavaFunc
    return _java2py(sc, func(*args))
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o68.recommendProducts.
: java.util.NoSuchElementException: next on empty iterator
    at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
    at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
    at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
    at scala.collection.IterableLike$class.head(IterableLike.scala:91)
    at scala.collection.mutable.WrappedArray.scala$collection$IndexedSeqOptimized$$super$head(WrappedArray.scala:34)
    at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
    at scala.collection.mutable.WrappedArray.head(WrappedArray.scala:34)
    at org.apache.spark.mllib.recommendation.MatrixFactorizationModel.recommendProducts(MatrixFactorizationModel.scala:117)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)

As you can see in first line above, I am running

user_products = model.call("recommendProducts", user, prodNum)

instead of

user_products = model.recommendProducts(user, prodNum)

because the latter is not implemented in 1.3.0 pyspark which I am using. Anyhow, it correctly returns prediction for some users, but then it fails on others.

I understand that it probably does not have the exact number of predictions I am requesting, I would expect it would return fewer.

Short answer:

You have trained model on ratings where userID was in range [0;N]
You have asked for a recommendation for a userID=N+x, where x is a positive integer number. This caused the exception.

The rest of the answer:

You are allowed to ask for recommendations within the userID space used during the training phase
You are allowed to ask for as many recommendations as the product space used during the training phase

When you want to make recommendation for a new user you have to add information about his or her taste to a training dataset.

You can find examples covering Collaborative Filtering in Spark Summit 2014 Hands-on Exercises.

(If I missed anything please correct me in comment below)