how to get p value for logistic regression in spar

Question

How can I get p-value for logistic regression in Spark MLlib using Java. How to find the probability of the classified class. The following is the code i have tried with:

SparkConf sparkConf = new SparkConf().setAppName("GRP").setMaster("local[*]");
SparkContext ctx = new SparkContext(sparkConf);

LabeledPoint pos = new LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0));
String path = "dataSetnew.txt";

JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(ctx, path).toJavaRDD();
JavaRDD<LabeledPoint>[] splits = data.randomSplit(new double[] {0.6, 0.4}, 11L);
JavaRDD<LabeledPoint> training = splits[0].cache();
JavaRDD<LabeledPoint> test = splits[1];   

final org.apache.spark.mllib.classification.LogisticRegressionModel model = 
    new LogisticRegressionWithLBFGS()
        .setNumClasses(2)
        .setIntercept(true)
        .run(training.rdd());    

JavaRDD<Tuple2<Object, Object>> predictionAndLabels = test.map(
    new org.apache.spark.api.java.function.Function<LabeledPoint, Tuple2<Object, Object>>() {
        public Tuple2<Object, Object> call(LabeledPoint p) {
          Double prediction = model.predict(p.features());
         // System.out.println("prediction :"+prediction);
          return new Tuple2<Object, Object>(prediction, p.label());
        }
      }
    );   

Vector denseVecnew = Vectors.dense(112,110,110,0,0,0,0,0,0,0,0);
Double prediction = model.predict(denseVecnew);
Vector weightVector = model.weights();          
System.out.println("weights : "+weightVector);           
System.out.println("intercept : "+model.intercept());       
System.out.println("forecast”+ prediction);    
ctx.stop();

Answer 1

For binary classification you can use LogisticRegressionModel.clearThreshold method. After it is called predict will return raw scores

instead of labels. These are in range [0, 1] and can be interpreted as probabilities.

See clearThreshold docs.