tm custom removePunctuation except hashtag

问题:

I have a Corpus of tweets from twitter. I clean this corpus (removeWords, tolower, delete URls) and finally also want to remove punctuation.

Here is my code:

tweetCorpus <- tm_map(tweetCorpus, removePunctuation, preserve_intra_word_dashes = TRUE)

The problem now is, that by doing so I also loose the hashtag (#). Is there a way to remove punctuation with tm_map but remain the hashtag?

回答1:

You could adapt the existing removePunctuation to suit your needs. For example

removeMostPunctuation<-
function (x, preserve_intra_word_dashes = FALSE) 
{
    rmpunct <- function(x) {
        x <- gsub("#", "\002", x)
        x <- gsub("[[:punct:]]+", "", x)
        gsub("\002", "#", x, fixed = TRUE)
    }
    if (preserve_intra_word_dashes) { 
        x <- gsub("(\\w)-(\\w)", "\\1\001\\2", x)
        x <- rmpunct(x)
        gsub("\001", "-", x, fixed = TRUE)
    } else {
        rmpunct(x)
    }
}

Which will give you

removeMostPunctuation("hello #hastag @money yeah!! o.k.")
# [1] "hello #hastag money yeah ok"

and when you use it with tm_map, but sure to wrap it in content_transformer()

tweetCorpus <- tm_map(tweetCorpus, content_transformer(removeMostPunctuation),
    preserve_intra_word_dashes = TRUE)

回答2:

The qdap package that I maintain has the strip function to handle this where you can specify characters not to strip:

library(qdap)

strip("hello #hastag @money yeah!! o.k.", char.keep="#")

Here it is applied to a Corpus:

library(tm)

tweetCorpus <- Corpus(VectorSource("hello #hastag @money yeah!! o.k."))
tm_map(tweetCorpus, content_transformer(strip), char.keep="#")

Also qdap has the sub_holder function that does essentially what Mr. Flick's removeMostPunctuation function does if that's useful

removeMostPunctuation <- function(text, keep = "#") {
    m <- sub_holder(keep, text)
    m$unhold(strip(m$output))
}

removeMostPunctuation("hello #hastag @money yeah!! o.k.")

## "hello #hastag money yeah ok"