Having problems with Ruby file from Dashing

Question

I am having trouble with twitter_user.rb, which is supposed to get the number of tweets, followers, and following of a given Twitter username.

I assume that I am supposed to replace TWITTER_USERNAME in line 9 with the Twitter username that I am interested in. I did that and started dashing but I got:

scheduler caught exception:
undefined method '[]' for nil:NilClass
/.../jobs/twitter_user.rb:19:in 'block in <top (required)>'

It looks like the problem is with line 19 which is:

tweets = /profile["']>[\n\t\s]*<strong>([\d.,]+)/.match(response.body)[1].delete('.,').to_i

Can anybody tell me what is going on and how to fix it?

Answer 1

I have resolved by substituting this line:

followers = /<strong>([\d.]+)<\/strong> Follower/.match(response.body)[0].delete('.,').to_i

with these two:

followers_count_metadata = /followers_count":[\d]+/.match(response.body)
followers = /[\d]+/.match(followers_count_metadata.to_s).to_s

Answer 2

I managed to solve the same issue for myself by using the twitter API instead to pull out the relevant information. It seems the web page had changed too much for the scraping to work and it could also stop working again at no notice as various people have already said...

This is the solution I used.

#### Get your twitter keys & secrets:
#### https://dev.twitter.com/docs/auth/tokens-devtwittercom
Twitter.configure do |config|
  config.consumer_key = 'YOUR_CONSUMER_KEY'
  config.consumer_secret = 'YOUR_CONSUMER_SECRET'
  config.oauth_token = 'YOUR_OAUTH_TOKEN'
  config.oauth_token_secret = 'YOUR_OAUTH_SECRET'

end

twitter_username = 'foobugs'

MAX_USER_ATTEMPTS = 10
user_attempts = 0

SCHEDULER.every '10m', :first_in => 0 do |job|
  begin
    tw_user = Twitter.user("#{twitter_username}")
    if tw_user
        tweets = tw_user.statuses_count
        followers = tw_user.followers_count
        following = tw_user.friends_count

        send_event('twitter_user_tweets', current: tweets)
        send_event('twitter_user_followers', current: followers)
        send_event('twitter_user_following', current: following)

    end
  rescue Twitter::Error => e
    user_attempts = user_attempts +1
    puts "Twitter error #{e}"
    puts "\e[33mFor the twitter_user widget to work, you need to put in your twitter API keys in the jobs/twitter_user.rb file.\e[0m"
    sleep 5
    retry if(user_attempts < MAX_USER_ATTEMPTS)
  end
end

Answer 3

This is untested code, but it's a general idea how it should have been written. If you clone the source on the original page you can adjust it for your own purposes (i.e. fix it):

require 'nokogiri'

doc = Nokogiri::XML(content)

tweets = doc.at('profile strong').text.delete('.,').to_i
following = doc.at('following strong').text.delete('.,').to_i
followers = doc.at('followers strong').text.delete('.,').to_i

The above three lines can be reduced to something like:

tweets, following, followers = %w[profile following followers].map{ |tag|
  doc.at("#{ tag } strong").text.delete(',.').to_i
}

Again, without a usable sample of the XML/HTML I can't do much more, but as a practice we (programmers) shouldn't use regular expressions to try to parse XML or HTML. It's much to easy to break a pattern with either of those types of files.

Answer 4

Your assumption is incorrect. The program is looking for an environment variable called TWITTER_USERNAME that is set to the relevant user name. If that variable doesn't exist then the code uses foobugs instead.

If you would rather modify the code than set up an environment variable, then change

twitter_username = ENV['TWITTER_USERNAME'] || 'foobugs'

to

twitter_username = 'myusername'