Borrow checker doesn't realize that `clear` dr

Question

The following code reads space-delimited records from stdin, and writes comma-delimited records to stdout. Even with optimized builds it's rather slow (about twice as slow as using, say, awk).

use std::io::BufRead;

fn main() {
    let stdin = std::io::stdin();
    for line in stdin.lock().lines().map(|x| x.unwrap()) {
        let fields: Vec<_> = line.split(' ').collect();
        println!("{}", fields.join(","));
    }
}

One obvious improvement would be to use itertools to join without allocating a vector (the collect call causes an allocation). However, I tried a different approach:

fn main() {
    let stdin = std::io::stdin();
    let mut cache = Vec::<&str>::new();
    for line in stdin.lock().lines().map(|x| x.unwrap()) {
        cache.extend(line.split(' '));
        println!("{}", cache.join(","));
        cache.clear();
    }
}

This version tries to reuse the same vector over and over. Unfortunately, the compiler complains:

error: `line` does not live long enough
 --> src/main.rs:7:22
  |
7 |         cache.extend(line.split(' '));
  |                      ^^^^
  |
note: reference must be valid for the block suffix following statement 1 at 5:39...
 --> src/main.rs:5:40
  |
5 |     let mut cache = Vec::<&str>::new();
  |                                        ^
note: ...but borrowed value is only valid for the for at 6:4
 --> src/main.rs:6:5
  |
6 |     for line in stdin.lock().lines().map(|x| x.unwrap()) {
  |     ^

error: aborting due to previous error

Which of course makes sense: the line variable is only alive in the body of the for loop, whereas cache keeps a pointer into it across iterations. But that error still looks spurious to me: since the cache is cleared after each iteration, no reference to line can be kept, right?

How can I tell the borrow checker about this?

Answer 1

The only way to do this is to use transmute to change the Vec<&'a str> into a Vec<&'b str>. transmute is unsafe and Rust will not raise an error if you forget the call to clear here. You might want to extend the unsafe block up to after the call to clear to make it clear (no pun intended) where the code returns to "safe land".

use std::io::BufRead;
use std::mem;

fn main() {
    let stdin = std::io::stdin();
    let mut cache = Vec::<&str>::new();
    for line in stdin.lock().lines().map(|x| x.unwrap()) {
        let cache: &mut Vec<&str> = unsafe { mem::transmute(&mut cache) };
        cache.extend(line.split(' '));
        println!("{}", cache.join(","));
        cache.clear();
    }
}

Answer 2

In this case Rust doesn't know what you're trying to do. Unfortunately, .clear() does not affect how .extend() is checked.

The cache is a "vector of strings that live as long as the main function", but in extend() calls you're appending "strings that live only as long as one loop iteration", so that's a type mismatch. The call to .clear() doesn't change the types.

Usually such limited-time uses are expressed by making a long-lived opaque object that enables access to its memory by borrowing a temporary object with the right lifetime, like RefCell.borrow() gives a temporary Ref object. Implementation of that would be a bit involved and would require unsafe methods for recycling Vec's internal memory.

In this case an alternative solution could be to avoid any allocations at all (.join() allocates too) and stream the printing thanks to Peekable iterator wrapper:

for line in stdin.lock().lines().map(|x| x.unwrap()) {
    let mut fields = line.split(' ').peekable();
    while let Some(field) = fields.next() {
        print!("{}", field);
        if fields.peek().is_some() {
            print!(",");
        }
    }
    print!("\n");
}

BTW: Francis' answer with transmute is good too. You can use unsafe to say you know what you're doing and override the lifetime check.

Answer 3

Itertools has .format() for the purpose of lazy formatting, which skips allocating a string too.

use std::io::BufRead;
use itertools::Itertools;

fn main() {
    let stdin = std::io::stdin();
    for line in stdin.lock().lines().map(|x| x.unwrap()) {
        println!("{}", line.split(' ').format(","));
    }
}

---

(A digression, something like this is a “safe abstraction” in the littlest sense of the solution in another answer here:

fn repurpose<'a, T: ?Sized>(mut v: Vec<&T>) -> Vec<&'a T> {
    v.clear();
    unsafe {
        transmute(v)
    }
}

)