Annotate x-axis with N in faceted plot, but preser

I asked a question yesterday about annotating the x-axis with N in a faceted plot using a minimal example that turns out to be too simple, relative to my real problem. The answer given there works in the case of complete data, but if you have missing facets you would like to preserve, the combination of facet_wrap options drop=FALSE and scales="free_x" triggers an error: "Error in if (zero_range(from) || zero_range(to)) { : missing value where TRUE/FALSE needed"

Here is a new, less-minimal example. The goal here is to produce a large graph with two panels using grid.arrange; the first showing absolute values over time by treatment group; the second showing the change from baseline over time by treatment group. In the second panel, we need a blank facet when vis=1.

# setup
library(ggplot2)
library(plyr)
library(gridExtra)

trt      <- factor(rep(LETTERS[1:2],150),ordered=TRUE)
vis      <- factor(c(rep(1,150),rep(2,100),rep(3,50)),ordered=TRUE)
id       <- c(c(1:150),c(1:100),c(1:50))
val      <- rnorm(300)
data     <- data.frame(id,trt,vis,val)
base     <- with(subset(data,vis==1),data.frame(id,trt,baseval=val))
data     <- merge(data,base,by="id")
data     <- transform(data,chg=ifelse(vis==1,NA,val-baseval))
data.sum <- ddply(data, .(vis, trt), summarise, N=length(na.omit(val)))
data     <- merge(data,data.sum)
data     <- transform(data, trtN=paste(trt,N,sep="\n"))
mytheme  <- theme_bw() + theme(panel.margin = unit(0, "lines"), strip.background = element_blank())

# no missing facets
plot.a   <- ggplot(data) + geom_boxplot(aes(x=trtN,y=val,group=trt,colour=trt), show.legend=FALSE) +
            facet_wrap(~ vis, drop=FALSE, switch="x", nrow=1, scales="free_x") +
            labs(x="Visit") + mytheme
# first facet should be blank
plot.b   <- ggplot(data) + geom_boxplot(aes(x=trtN,y=chg,group=trt,colour=trt), show.legend=FALSE) +
            facet_wrap(~ vis, drop=FALSE, switch="x", nrow=1, scales="free_x") +
            labs(x="Visit") + mytheme

grid.arrange(plot.a,plot.b,nrow=2)

You can add a blank layer to draw all the facets in your second plot. The key is that you need a variable that exists for every level of vis to use as your y variable. In your case you can simply use the variable you used in your first plot.

ggplot(data) + 
    geom_boxplot(aes(x = trtN, y = chg, group = trt, colour = trt), show.legend = FALSE) +
    geom_blank(aes(x = trtN, y = val)) +
    facet_wrap(~ vis, switch = "x", nrow = 1, scales = "free_x") +
    labs(x="Visit") + mytheme