Please forgive me if I have provided more information than required for this question. :D
I am building an application that pulls large JSON data-sets from a remote machine. However, I am working within a secure environment that separates application servers with firewalls, etc. Because of this I have had to do a bit of fudging (using SSH) to get the data I need. I have requested additional ports to be opened so I could bypass using SSH but was denied.
Here is the physical path to get my data over SSH: My Application Server -> Secure Server with Curl -> Multiple unique application servers with JSON data.
To abide by the security rules; I wrote the CFML code below on my application server. (server is running Linux, Lucee 4.5, MySQL and Apache)
<cfset secure_server = "secure_server.my.example.com">
<cftry>
<cfexecute
name="/usr/bin/ssh"
arguments="#secure_server# ./RemoteDirectory/server_list.sh"
timeout="10"
variable="server_list">
</cfexecute>
<cfcatch type="any">
<cfoutput>
#cfcatch.message#<br>#cfcatch.detail#<br>
</cfoutput>
</cfcatch>
</cftry>
The above code simply SSHs to the secure server and runs a script called server_list.sh. The important bit of the script is below:
curl -s -k -b cookie.jar -c cookie.jar https://unique_application_server/Path/To/Server/list -H "Accept:application/json"
That curl script returns a JSON string that looks similar (but with far more data) to this:
{
"os" : "Linux ",
"fqdn" : "server1.my.example.com",
"disk_status" : "OK",
"system_model" : "HP",
"type" : "Server"
},
{
"os" : "Linux ",
"fqdn" : "server2.my.example.com",
"disk_status" : "OK",
"system_model" : "Dell",
"type" : "Server"
},
The amount of JSON data sent, at this time, is about 3MB. However, that data will grow exponentially over time.
Once I get the JSON back to my application server; I use CF's deserializeJSON function then grab just the field I need. In this case its the FQDN field. Since the data from the JSON is a full domain, I needed to chomp just the hostname. The "new_label" code takes care of that. There is probably a better REGEX or a better method than the one I am using, but this works.
<cfset arrayOfStructs = deserializeJson(server_list)>
<select multiple="multiple" class="searchable" name="searchable[]">
<cfloop array="#arrayOfStructs#" index="PARENT">
<cfset new_label = ReReplace(PARENT.fqdn, "(.[^.]*)$", "")>
<cfset new_label = ReReplace(new_label, "(.[^.]*)$", "")>
<cfset new_label = ReReplace(new_label, "(.[^.]*)$", "")>
<cfoutput><option value="#PARENT.fqdn#">#new_label#</option></cfoutput>
</cfloop>
</select>
The above code populates the field with the hostnames of servers from the JSON. The OPTION field would look something like this:
<option value="server1.my.example.com">server1</option>
A couple things of note that may (or may not) help. I am using jQuery and a multi-select.js that turns the SELECT into a side-by-side field. The multi-slect is from here: http://loudev.com/
Second, all servers in the path are fairly powerful and are all connected over 10gb. So besides the firewalls between, there should not be any bottle necks.
No matter the browser I use, the code above works, but is rather slow when it populates the hostname field. Because the JSON data I am pulling updates about every minute, I am not able to store it long term in the database on my application server.
Knowing all that, my question is: How could I speed up the form field so the user does not have to wait for the page to load 2500+ servernames?
Could I leverage jQuery or the MySQL database to speed things up without running into stale data? Could it simply be my method of getting the hostname from the FQDN? Should I scrap this method and try something else? Have I overlooked something here on Stack that has my answer? (I spent a lot of time searching Q/A here but found nothing. Perhaps I was not looking hard enough.)
Any other information you nice folks need? Thanks! :D
UPDATE --
@Alex. I do that very test now. Code is below. Good suggestion though!
<cfset tickBegin = GetTickCount()>
<!--- THE CODE THAT RUNS THE JSON QUERY ABOVE --->
<cfset tickEnd = GetTickCount()>
<cfset testTime = tickEnd - tickBegin>
<cfoutput>Page(#testTime#ms)</cfoutput>
As far as caching the data, that is part of my question. How to cache? What method to use? How can I cache the JSON without displaying stale data? Is this an instance where AJAX would be useful?
UPDATE 2 --
Could I be going about this the wrong way? I know that sometimes you will need to display large amounts of data back to your users. Would simply adding a BlockUI and a progress bar stating please wait be an OK solution? I would prefer to figure out a better solution honestly but I am open to suggestions for sure! Thoughts?
