ggplot: How to connect certain bars in a chart wit

At the moment I am replicating/updating certain graphics for a poster presentation. I managed to replicate colours, values, the bar-style and background of the graphic. But there's an arrow-label missing, highlighting a value-difference. I wonder whether there's an useful option via ggplot (lines or arrows), worth the effort - alternatively I have to draw some arrows with another graphic-software ...

Here's what I try to do:

Here's what I already have:

The data and code:

weight <- c(113.2158, 108.5404, 98.75564, 93.93759)
sex <- c("m","m", "f","f")
time <- c("t0", "t1", "t0", "t1")
data <- data.frame(weight, sex, time)
library(ggplot2)

ggplot(data, aes(x = factor(sex), y = weight, group=time, fill=factor(time))) + 
  geom_bar(position="dodge", stat = "identity") +
  theme(legend.position = c(0.8, 0.9),                                
        axis.ticks = element_blank(),                                 
        axis.title.x=element_blank(),                                 
        axis.title.y=element_blank(),                                 
        panel.background = element_blank(),                           
        panel.grid.major.x = element_blank() ,                        
        panel.grid.major.y = element_line( size=.1, color="grey" ),
        legend.key = element_rect(size = 2),
        legend.key.size = unit(1.5, 'lines')) +       
  guides(fill=guide_legend(title="time")) +                             
  scale_fill_manual(values=c("#b6181f", "#f6b8bb"), labels=c("t0", "t1")) +
  scale_x_discrete(labels = c("male, n = 57", "female, n = 133"))  +
  coord_cartesian(ylim = c(90,120)) +
  scale_y_continuous(breaks=c(90,95,100,105,110,115,120)) + 
  geom_text(aes(x=factor(sex), label=round(weight, digits=2)), position = position_dodge(width = 1), vjust = -0.25) +
  geom_line(aes(x=factor(sex), label=round(weight, digits=2)), position = position_dodge(width = 1), vjust = -0.25)

Thank you very much for ideas.

Here's a possible solution that tries to replicate the spirit of the original plot, except for the replacement of elbowed arrows with straight arrows. (If you want arrows with multiple elbows, it's probably easier to achieve the effect in PowerPoint...)

Define function for calculating difference in bar height within each group:

fun.data <- function(x){
  return(data.frame(y = max(x) + 1,
                    label = paste0(round(diff(x), 2), "cm")))
}

Define function for axis breaks (this is more flexible than hard coding the axis labels, especially if you need to change coord_cartesian's range):

ab = 5 # let axis break labels be multiples of 5
fun.breaks <- function(limits){seq(ceiling(limits[1] / ab) * ab,
                                   floor(limits[2] / ab) * ab,
                                   by = ab)}

Plot:

ggplot(data,
       aes(x = sex, y = weight, group = time, fill = time, label = round(weight, 2))) +

  # this segment creates the bar plot with labels just inside the top of each bar
  # note: geom_col() is equivalent to geom_bar(stat = "identity"), with less typing
  geom_col(position = "dodge") +
  geom_text(position = position_dodge(width = 0.9),
            vjust = 1.1) +

  # this segment creates the arrows & labels for the differences
  geom_line(aes(group = sex), position = position_nudge(0.1),
            arrow = arrow()) +
  stat_summary(aes(x = sex, y = weight), 
               geom = "label",
               fun.data = fun.data,
               fontface = "bold", fill = "lightgrey",
               inherit.aes = FALSE) +

  # set scales
  # note: use named vectors for fill & x to ensure that labels are mapped correctly
  scale_fill_manual(name = "time", 
                    values = c("t0" = "#b6181f", "t1" = "#f6b8bb"),
                    labels = c("beginning", "after 3 months")) +
  scale_x_discrete(name = "", 
                   labels = c("f" = "female, n = 133", "m" = "male, n = 57")) +
  scale_y_continuous(name = "",
                     breaks = fun.breaks,
                     minor_breaks = NULL) +

  # other cosmetic aspects
  coord_cartesian(ylim = c(90, 120)) +
  theme_classic() +
  theme(panel.grid.major.y = element_line(colour = "grey"))