Setting up thinking sphinx after server reboot (Ra

Question

Problem: I am trying to get sphinx running again after server reboot. There seems to be no sphinx.conf file when I try to start it running:

>searchd

Sphinx 2.0.4-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

FATAL: no readable config file (looked in /etc/sphinxsearch/sphinx.conf, ./sphinx.conf).

I have run:

rake thinking_sphinx:configure
rake thinking_sphinx:index
rake thinking_sphinx:start

The problem is for some reason no etc/sphinxsearch/sphinx.conf file is being created... I am new to thinking_sphinx and this might not be the only problem (with the site), but it doesn't seem to be set up fully. For out put and more information read below:

Background info: I am working on a project I didn't set up initially. We rebooted the server to see some of the changes we made in a constants file. But after the reboot the project no longer displays when you navigate to the site. When you put in the straight ip address it just says "Welcome to Nginx".

The port is open and working through our hosting server, so I was told I have to restart some services. One of the issues I came upon was with thinking_sphinx. This was the rake tasks for sphinx site I referenced. As well as common configuration issues for sphinx.

I set up the sphinx.yml development paths (we aren't using production). Then I ran

>rake thinking_sphinx:index

which seems to have worked even though it output some warnings:

Generating Configuration to /home/potato/streetpotato/config/development.sphinx.conf
   (0.2ms)  SELECT @@global.sql_mode, @@session.sql_mode;
Sphinx 2.0.4-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/home/potato/streetpotato/config/development.sphinx.conf'...
indexing index 'bar_core'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 14080 kb
collected 249 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 249 docs, 32394 bytes
total 0.254 sec, 127298 bytes/sec, 978.49 docs/sec
indexing index 'bar_delta'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 14080 kb
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.003 sec, 0 bytes/sec, 0.00 docs/sec
skipping non-plain index 'bar'...
indexing index 'synonym_core'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 13568 kb
collected 3 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 3 docs, 103 bytes
total 0.003 sec, 30356 bytes/sec, 884.17 docs/sec
indexing index 'synonym_delta'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 13568 kb
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
skipping non-plain index 'synonym'...
indexing index 'user_core'...
WARNING: collect_hits: mem_limit=0 kb too low, increasing to 13568 kb
collected 100 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 100 docs, 3146 bytes
total 0.013 sec, 239348 bytes/sec, 7608.03 docs/sec
skipping non-plain index 'user'...
total 11 reads, 0.000 sec, 3.8 kb/call avg, 0.0 msec/call avg
total 37 writes, 0.000 sec, 2.5 kb/call avg, 0.0 msec/call avg

Then I ran

>rake thinking_sphinx:configure

Generating Configuration to /home/potato/streetpotato/config/development.sphinx.conf
(0.2ms)  SELECT @@global.sql_mode, @@session.sql_mode;

Lastly running:

>rake thinking_sphinx:start

Started successfully (pid 29623).

Now even though my log says:

[Fri Nov 16 19:34:29.820 2012] [29623] accepting connections

There is still no sphinx.conf file being generated and when I try to use the searchd command it still gives me the error...

>searchd --stop

Sphinx 2.0.4-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

FATAL: no readable config file (looked in /etc/sphinxsearch/sphinx.conf, ./sphinx.conf).

I am at a loss, I know this is super long but only because I am so lost and trying to give as much information as possible. I got further then I did yesterday with this but it still doesn't seem to be fully working. I might have to do more set up with unicorn or thin as well. I'm just trying to figure out how to get the site back up and running again... If any one has run into similar issues with their site going down after reboot and got it back up (specifically a rails project on Nginx and unicorn or thin using sphinx) any insight would be appreciated.

Thanks,

Alan

Answer 1

So yeah, this is a hole in ThinkingSphinx (IMHO) -- you can start the sphinxd server using the various rake tasks (which generate the config as needed) ... but this doesn't work in production.

On a project I worked in last year (running on a Linux server) we created an /etc/init.d script to start sphinxd -- it takes options including a path to the configuration file. We did our deploys with capistrano, and put generated code in app/shared -- a directory outside of the source tree. I believe there are some predefined capistrano tasks that will rebuild the Rails-specific config files when models change or otherwise affect what Sphinx does (same as the rake task you mention).

This was one of those cases for us where we had been putting off site search for a long time, and one of our developers got it "all set up" in an afternoon. Getting it deployed took a lot more work.

(Just saw answer from @prakash-murthy -- he provides some details of how to specify config path when you initialized sphinxd). But the trick is to have it start when the system starts and pointing to the config that ThinkingSphinx generates.)

Answer 2

Calm down!! :-)

Firstly, you don't need a /etc/sphinxsearch/sphinx.conf file; that is just the default file that searchd tries to use when you don't specify any configuration file.

As your log output shows, your rails application is using /home/potato/streetpotato/config/development.sphinx.conf file when it starts the searchd process.

Run ps -fe | grep searchd on your dev machine; you should see something like this as the output:

501 14128     1   0   0:00.00 ttys004    0:00.00 searchd --pidfile --config /home/potato/streetpotato/config/development.sphinx.conf
501 14130 13546   0   0:00.00 ttys004    0:00.01 grep searchd

So rails app calls searchd with --config /home/potato/streetpotato/config/development.sphinx.conf argument, to specify a different conf file.

From your logs, it is clear that thinkingsphinx is running fine. You can confirm it further by logging into rails console and running a search method on one of the models which have thinking_sphinx indexes defined on them.

Eg: If your app has an Article model as shown in the above link, the following command will show all articles having National Parks in them:

$ rails console
> Article.search( "National Parks" )
=> [#<Article id: 15,... >, #<Article id: 22,...>,...]

The real problem is the application not showing after restarting the server. That has nothing to do with thinking sphinx which is running fine.

Try rolling back all the changes made in the constants file that you mention above, and make sure the application is working fine. Then start making the changes one by one and isolate the one change that breaks your application.

Answer 3

Ok so after a day n a half I finally set it all up and got it running (it was more then just sphinx). I also had to get nginx and unicorn up and running in the background, since we didn't have scripts set up to restart them when the server was rebooted...

When rebooting the server you have to restart some services before the app will be accessible:

1) thinking_sphinx
reference sites
http://pat.github.com/ts/en/rake_tasks.html
http://www.claytonlz.com/2010/09/thinkingsphinx-conf-problems/

a)create/modify  app/config/sphinx.yml

    development:
      morphology: stem_en
      port: 9312
      bin_path: "/usr/bin"             # set up the path to binary for searchd
      searchd_binary_name: searchd
      indexer_binary_name: indexer
      #mem_limit: 128M
    test:
      morphology: stem_en
      port: 9312
      mem_limit: 128M
    production:
      morphology: stem_en
      port: 9312
      mem_limit: 512M
    # the searchd ip, in case it's not on localhost
    # address: 10.10.0.0
    # this is by default included in db/sphinx
    # searchd_file_path: "/path/to/shared/folder/sphinx"

b)rake thinking_sphinx:index
c)rake thinking_sphinx:configure       # creates config/development.sphinx.conf which helps define sphinx's indexing

d)# then you have to start sphinx, there are 2 ways to do this

    rake thinking_sphinx:start
    rake thinking_sphinx:stop

    OR

    searchd
    searchd --stop

    # only the rake commands worked for me, when I tried to run searchd
    # I got an error FATAL: no readable config file (looked in /etc/sphinxsearch/sphinx.conf, ./sphinx.conf).
    # for some reason we dont have a sphinx.conf file, but the rake commands work without it

e)# once you start thinking_sphinx check log/searchd.log file for the line
    [Fri Nov 16 19:34:29.820 2012] [29623] accepting connections

2) nginx
reference site:
http://wiki.nginx.org/CommandLine

a) check that nginx is up and running
    i) start server 
        # to check where nginx resides type in this into server console
        which nginx
        # whatever path it gives you is how you start the server this is my path
        /usr/sbin/nginx 
    ii) stop server
        /usr/sbin/nginx -s stop # use the path given by which command

3) unicorn (starting app server)
reference site:
http://codelevy.com/2010/02/09/getting-started-with-unicorn.html

a) test if unicorn will run after previous changes
    unicorn_rails -p 3000
    # the site should now be up and running, check that it is
    # console should now log the different actions you do on the site 

b) create unicorn.rb in config folder (if none is there)
    # only start this step if the step above got the site running
    # close the console or exit the process you started above
    # contents of unicorn.rb 
    worker_processes 2      #(starts 2 child processes, not completely neccissary)
    preload_app true
    timeout 30
    listen 3000

    after_fork do |server, worker|
      ActiveRecord::Base.establish_connection
    end

c) run unicorn in the background
    # make sure you exited the process above before running this
    unicorn_rails -c config/unicorn.rb -D

    # this was giving me an error that it said was logged by stderr
    # I got the command to run by adding a command to the front
    http://stackoverflow.com/questions/2325152/check-for-stdout-or-stderr

    exec 2> /dev/null unicorn_rails -c config/unicorn.rb -D


d) (optional) check stats from starting unicorn

    i) pgrep -lf unicorn_rails
        #sample output
        5374 unicorn_rails master -c config/unicorn.rb -D             
        5388 unicorn_rails worker[0] -c config/unicorn.rb -D    # not needed currently  
        5391 unicorn_rails worker[1] -c config/unicorn.rb -D    # not needed currently

    ii) cat tmp/pids/unicorn.pid # from inside the streetpotato folder
        #sample output
        5374