Huge json object to excel?

问题:

I have a PHP application which has a table with more then 10000 rows and I am trying to export that into an excel sheet via my ROR application but my request is getting timed out on the server by PHP application.So I was wondering is there any elegant way to solve this problem. I come up with 2 solutions. First is doing a batch processing (need to read about as I am new with it) and the other is that my php application will send a large json object and my ruby app will read that json object write the data to excel sheet and sends back the excel sheet.So I wanted to ask whether is there any better way to deal with this problem ? And how can I convert json to excel I did google and did found excel to json but not vice versa. Any suggestions?

回答1:

I had some time so I constructed a json to excel csv converter in ruby1.9:

It reads from one file an writes it into an other file.

json2excel.rb

def json2excel(fromFile, toFile)
  pos = 0
  while true
    c = fromFile.read(1);pos += 1
    if c == ' ' or c == "\n" or c == "\r"
      # whitespace
    elsif c == '['
      # first bracket begins!
      attributes = []
      while true
        c = fromFile.read(1);pos += 1
        if c == '{'
          # now an object starts
          object = Hash.new
          while true
            puts "!!!"
            c = fromFile.read(1);pos += 1
            if c == '"'
              # new attribute starts
              name = ""
              while true
                c = fromFile.read(1);pos += 1
                if c == '"'
                  break
                else
                  name += c
                end
              end
              attributes << name unless attributes.include? name
              # scan for :
              while true
                c = fromFile.read(1);pos += 1
                if c == ':'
                  break
                elsif  c == ' ' or c == '\n' or c == '\r' # whitespace is ok
                else raise "4malformed json for excel conversion! char: #{c.inspect} position: #{pos}"
                end
              end
              # scan for staring value
              while true
                c = fromFile.read(1);pos += 1
                if c == '"'
                  # string follows
                  value = ""
                  value << c
                  while true
                    c = fromFile.read(1);pos += 1
                    value << c
                    if c == '"'
                      break
                    end
                  end
                  c = fromFile.read(1);pos += 1
                  break
                elsif  c == ' ' or c == '\n' or c == '\r' # whitespace is ok
                elsif "1234567890".include? c
                  # number follows
                  value = ""
                  value << c
                  while true
                    c = fromFile.read(1);pos += 1
                    if "1234567890".include? c
                      value << c
                    else break
                    end
                  end
                  break
                elsif c == "t"
                  # true follows
                  c = fromFile.read(3);pos += 3
                  if c != "rue"
                    raise "excpected true but found t#{c.inspect} position: #{pos}"
                  end
                  value = "true"
                  c = fromFile.read(1);pos += 1
                  break
                elsif c == "f"
                  # false follows
                  c = fromFile.read(4);pos += 4
                  if c != "alse"
                    raise "excpected false but found f#{c.inspect} position: #{pos}"
                  end
                  value = "false"
                  c = fromFile.read(1);pos += 1
                  break
                else raise "5malformed json for excel conversion! char: #{c.inspect} position: #{pos}"
                end
              end
              # value starts
              object[name] = value
              puts object
            end
            puts "c: #{c.inspect}"
            if c == "," 
              # comma is ok! just take many of them, does not hurt.
            elsif  c == ' ' or c == '\n' or c == '\r'
              # whitespace is ok
            elsif c == "}"
              # object ends!
              break
            else raise "3malformed json for excel conversion! char: #{c.inspect} position: #{pos}"
            end
          end
          attributes.each{|attr|
            value = object[attr]
            raise "expected object #{object} to have attribute #{attr} position: #{pos}" if value.nil?
            toFile.write(value)
            toFile.write(',')
          }
          toFile.write("\"\"\r\n") # this is the csv new line. a new object begins here
        elsif  c == ' ' or c == '\n' or c == '\r'
          # whitespace is ok
        elsif c == ']'
          attributes.each{ |attr|
            toFile.write(attr.inspect)
            toFile.write(",")
          }
          toFile.write("\"\"\r\n") # this is the csv new line. a new object begins here
          # the end of the file
          c = fromFile.read()
          if c != ''
            raise "end of listing was reached. skipping #{c.size} character after position #{pos}: #{c.inspect}"
          end
          break
        elsif c == ','
          # comma is ok! just take many of them, does not hurt.
        else
          raise "2malformed json for excel conversion! char: #{c.inspect} position: #{pos}"
        end
      end
      break
    else
      raise "1malformed json for excel conversion! char: #{c.inspect} position: #{pos}"
    end
  end
end

json2excel(File.open('json.txt'), File.open('excel.csv', 'wb'))

json.txt

[{"id": 1,"pro_id": 3,"pro_name": "asdf","cli_id": 113,"cli_name": "tyuryt"} , {"id": 1,"pro_id": 3,"pro_name": "asdf","cli_id": 113,"cli_name": "tyuryt"}]

excel.csv

1,3,"asdf",113,"tyuryt",""
1,3,"asdf",113,"tyuryt",""
"id","pro_id","pro_name","cli_id","cli_name",""

your column names are in the end of the file.

if new attributes are introduced after the first object not all columns will be of equal element count.

Note: it does not load everything into memory but writes it to the file as soon as possible.

What it does not do:

• negative numbers
• numbers with a . in them
• strings with " in it.