Pyspark dataframe operator “IS NOT IN”

问题:

I would like to rewrite this from R to Pyspark, any nice looking suggestions?

array <- c(1,2,3)
dataset <- filter(!(column %in% array))

回答1:

In pyspark you can do it like this:

array = [1, 2, 3]
dataframe.filter(dataframe.column.isin(*array) == False)

Or using the binary NOT operator:

dataframe.filter(~dataframe.column.isin(*array))

回答2:

Take the operator ~ which means contrary :

df_filtered = df.filter(~df["column_name"].isin([1, 2, 3]))

回答3:

df_result = df[df.column_name.isin([1, 2, 3]) == False]

回答4:

slightly different syntax and a "date" data set:

toGetDates={'2017-11-09', '2017-11-11', '2017-11-12'}
df= df.filter(df['DATE'].isin(toGetDates) == False)

回答5:

You can also loop the array and filter:

array = [1, 2, 3]
for i in array:
    df = df.filter(df["column"] != i)