Why is Rails giving me “Can't verify CSRF toke

I am getting a "Can't verify CSRF token authenticity" in Rails production. My questions are:

Why is it doing this?
How can I fix it?

Here's my Heroku logs (some values anonymized):

2016-02-13T01:18:54.118956+00:00 heroku[router]: at=info method=POST path="/login" host=[MYURL] request_id=[ID STRING] fwd="FWDIP" dyno=web.1 connect=0ms service=6ms status=422 bytes=1783  
2016-02-13T01:18:54.116581+00:00 app[web.1]: Started POST "/login" for [IPADDRESS] at 2016-02-13 01:18:54 +0000  
2016-02-13T01:18:54.119372+00:00 app[web.1]: Completed 422 Unprocessable Entity in 1ms  
2016-02-13T01:18:54.118587+00:00 app[web.1]: Processing by SessionsController#create as HTML
2016-02-13T01:18:54.118637+00:00 app[web.1]:   Parameters: {"utf8"=>"✓", "authenticity_token"=>"[BIGLONGRANDOMTOKENSTRING]", "session"=>{"email"=>"[FRIENDSEMAILADDRESS]", "password"=>"[FILTERED]", "remember_me"=>"0"}, "commit"=>"Log in"}  
2016-02-13T01:18:54.119082+00:00 app[web.1]: Can't verify CSRF token authenticity  
2016-02-13T01:18:54.120565+00:00 app[web.1]:  
2016-02-13T01:18:54.120567+00:00 app[web.1]:  ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):  
2016-02-13T01:18:54.120569+00:00 app[web.1]:   vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.0/lib/action_controller/metal/request_forgery_protection.rb:181:in `handle_unverified_request'  
.  
.  
.etc

The only manifestation I'm aware of is when my friend tries to log in using Safari on his iPhone 5. His user account was created right about 6 months ago. I'm 99% sure that he accessed the site just fine with his phone at that time. Since then he hasn't logged in and I'm not aware of any changes I've made to the login/auth codes. Yesterday I had him hit my site for the first time in ~6 months and now he gets the CSRF error.

This issue doesn't happen to any other user account (that I'm aware of) or on any other device. In fact, logging in to his account from his older iPhone 4 works just fine.

I have a decent amount of dev experience but am totally new to web dev and everything Rails.

Here's what I have:

class ApplicationController < ActionController::Base
  # Prevent CSRF attacks by raising an exception.
  # For APIs, you may want to use :null_session instead.
  protect_from_forgery with: :exception
  include SessionsHelper
end

Application layout:

<!DOCTYPE html>
<html>
<head>
<title><%= full_title(yield(:title)) %></title>
<meta name="viewport" content="width=device-width,initial-scale=1">
<%= stylesheet_link_tag 'application', media: 'all' %>
<%= javascript_include_tag 'application' %>
<%= csrf_meta_tags %>
<%= render 'layouts/shim' %>
</head>
<body>
<%= render 'layouts/header' %>
<div class="container">
<% flash.each do |message_type, message| %>
<div class="alert alert-<%= message_type %>"><%= message %></div>
<% end %>
<%= yield %>
<%= render 'layouts/footer' %>
<%= debug(params) if Rails.env.development? %>
</div>
</body>
</html>

My secrets file looks like this:

# Do not keep production secrets in the repository,
# instead read values from the environment.
production:
secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>

And I have a production environment var on Heroku for the secret_key_base.

def log_in(user)
  session[:user_id] = user.id
end

def remember(user)
  user.remember
  cookies.permanent.signed[:user_id] = user.id
  cookies.permanent[:remember_token] = user.remember_token
end

Here's what I've done:
I began developing my app by following everything in Michael Hartl's Rails Tutorial to the letter up to/through Chapter 10. Most relevant is Chapter 8. Specifically, my app uses all the security / cookies / user auth stuff exactly as in the tut. I don't do anything fancy in my app...no AJAX or anything like that. I even yanked turbolinks.

My project has spanned the last 18 months so I'm not 100% positive which version I began on. I know it was 4.1.X and it was probably 4.1.6. I also am unsure the date I upgraded but at some point did to what I'm presently running; 4.2.0.

I've read just about every post I can find on the web regarding problems with CSRF + Rails. Seems like for almost everything I've read, the cause and solution have to do with AJAX or Devise, neither of which apply to me. iFrame issues are another common source on the web, which I'm neither using.

I've used my app's password reset feature to no avail. I tried changing protect_from_forgery to with: :reset_session. The only thing this changes is the Rails exception page is no longer displayed. But it won't let him go to any page requiring authentication. It just takes him back to root because I have this line in my routes:

get '*path' => redirect('/')

I don't want to clear his cookies/cache etc because I have dozens of other existing user accounts that I don't want to have to manually fix.

Frequently suggested solutions are some variant of turning off security, which I don't want to do for obvious reasons.

Some other things I have changed but haven't had a chance to test yet (because I don't have easy access to my friend's iPhone):

I changed the appstore name in session_store.rb:

Rails.application.config.session_store :cookie_store, key: '[NEWNAME]'

Ran the following commands:
heroku run rake assets:clean
heroku run rake assets:precompile

I am about to embark on a deep dive here, especially section 3.

Thanks for reading/considering. Any tips/ideas/suggestions/pointers would be greatly appreciated!

Turns out this gentleman was having the exact same issue as me and was able to create a repro case that worked for me. If I'm understanding right, Safari is caching the page but nuking the session. This causes the authenticity_token value to look legit'ish in my rails params but protect_from_forgery fails when verifying the token because the session was nuked.

The solution then is two fold: turn off caching and handle CSRF exceptions. You still need to handle exceptions even if you turn off caching because some browsers (e.g. Safari) don't respect no-cache settings. In this case a CSRF issue arises and hence the need to handle that also.

The workaround for me was to handle the CSRF exception by killing off all my cookies and session data, flash an "oops" message and redirect them to the login page. The redirect will pull down a fresh auth token that will verify when doing the login post. This idea came from here:

It is common to use persistent cookies to store user information, with cookies.permanent for example. In this case, the cookies will not be cleared and the out of the box CSRF protection will not be effective. If you are using a different cookie store than the session for this information, you must handle what to do with it yourself:

rescue_from ActionController::InvalidAuthenticityToken do |exception|
  sign_out_user # Example method that will destroy the user cookies
end

The above method can be placed in the ApplicationController and will be called when a CSRF token is not present or is incorrect on a non-GET request.

cookies.permanent is exactly what I was using. So I implemented the above tip like this:

class ApplicationController < ActionController::Base  
  include SessionsHelper  
  protect_from_forgery with: :exception  
  before_filter :set_cache_headers  
  rescue_from ActionController::InvalidAuthenticityToken do |exception|  
    cookies.delete(:user_id)
    cookies.delete(:remember_token)
    session.delete(:user_id)
    @current_user = nil  
    flash[:danger] = "Oops, you got logged out. If this keeps happening please contact us. Thank you!"  
    redirect_to login_path  
  end  

  def set_cache_headers  
    response.headers["Cache-Control"] = "no-cache, no-store, max-age=0, must-revalidate"  
    response.headers["Pragma"] = "no-cache"  
    response.headers["Expires"] = "Fri, 01 Jan 1990 00:00:00 GMT"  
  end  
end

Incidentally, after implementing the fix and verifying it worked in dev, my friend's phone was still unable to login, albeit with different behavior. Upon investigation I found he had "all cookies blocked" in his iPhone 5 Safari settings. This caused other weird behavior that made it hard to sort out which issue was causing what. The tipoff came when I realized I couldn't use his phone to log in to any online accounts (e.g. yahoo mail etc.). Going in to his Safari settings and allowing cookies solved things and now everything works great on his phone (and everywhere else that I'm aware of).