How to cache in a Blackberry BrowserField

Question

I am creating a Blackberry application to display a full screen web view of a certain site. I have a working browserfield that displays properly but navigation from page to page is slower than that of the native browser. The browserfield does not seem to have a built in cache causing the load time to be slow. When I add the following code to manage the cache the site no longer displays properly.

BrowserFieldScreen.java:

import net.rim.device.api.browser.field2.*;
import net.rim.device.api.script.ScriptEngine;
import net.rim.device.api.system.*;
import net.rim.device.api.ui.*;
import net.rim.device.api.ui.component.*;
import net.rim.device.api.ui.container.*;
import org.w3c.dom.Document;

class BrowserFieldScreen extends MainScreen
{
    BrowserField browserField;
    LoadingScreen load = new LoadingScreen();;

    public BrowserFieldScreen()
    {   
        browserField = new BrowserField();
        browserField.getConfig().setProperty(
            BrowserFieldConfig.JAVASCRIPT_ENABLED, 
            Boolean.TRUE);
        browserField.getConfig().setProperty(
            BrowserFieldConfig.NAVIGATION_MODE, 
            BrowserFieldConfig.NAVIGATION_MODE_POINTER);
        browserField.getConfig().setProperty(
            BrowserFieldConfig.CONTROLLER, 
            new CacheProtocolController(browserField));

        browserField.requestContent("http://www.stackoverflow.com");
        add(browserField);
    }
}

CacheProtocolController.java:

import javax.microedition.io.HttpConnection;
import javax.microedition.io.InputConnection;

import net.rim.device.api.browser.field2.BrowserField;
import net.rim.device.api.browser.field2.BrowserFieldRequest;
import net.rim.device.api.browser.field2.ProtocolController;

public class CacheProtocolController extends ProtocolController{

    // The BrowserField instance
    private BrowserField browserField;

    // CacheManager will take care of cached resources 
    private CacheManager cacheManager;

    public CacheProtocolController(BrowserField browserField) {
        super(browserField);
        this.browserField = browserField;
    }

    private CacheManager getCacheManager() {
        if ( cacheManager == null ) {
            cacheManager = new CacheManagerImpl();
        }
        return cacheManager;
    }

    /**
     * Handle navigation requests (e.g., link clicks)
     */
    public void handleNavigationRequest(BrowserFieldRequest request) 
        throws Exception 
    {
        InputConnection ic = handleResourceRequest(request);
        browserField.displayContent(ic, request.getURL());
    }

    /**
     * Handle resource request 
     * (e.g., images, external css/javascript resources)
     */
    public InputConnection handleResourceRequest(BrowserFieldRequest request) 
        throws Exception 
    {
        // if requested resource is cacheable (e.g., an "http" resource), 
            // use the cache
        if (getCacheManager() != null 
            && getCacheManager().isRequestCacheable(request)) 
            {
                InputConnection ic = null;
                // if requested resource is cached, retrieve it from cache
                if (getCacheManager().hasCache(request.getURL()) 
                    && !getCacheManager().hasCacheExpired(request.getURL())) 
                {
                    ic = getCacheManager().getCache(request.getURL());
                }
                // if requested resource is not cached yet, cache it
                else 
                {
                ic = super.handleResourceRequest(request);
                    if (ic instanceof HttpConnection) 
                    {
                        HttpConnection response = (HttpConnection) ic;
                        if (getCacheManager().isResponseCacheable(response)) 
                        {
                        ic = getCacheManager().createCache(request.getURL(), 
                             response);
                        }
                }
            }
            return ic;
        }
        // if requested resource is not cacheable, load it as usual
        return super.handleResourceRequest(request);
    }

}

CacheManager.java:

import javax.microedition.io.HttpConnection;
import javax.microedition.io.InputConnection;

import net.rim.device.api.browser.field2.BrowserFieldRequest;

public interface CacheManager {
    public boolean isRequestCacheable(BrowserFieldRequest request);
    public boolean isResponseCacheable(HttpConnection response);
    public boolean hasCache(String url);
    public boolean hasCacheExpired(String url);
    public InputConnection getCache(String url);
    public InputConnection createCache(String url, HttpConnection response);
    public void clearCache(String url);
}

CacheManagerImpl.java:

import java.io.IOException;
import java.io.InputStream;
import java.util.Date;
import java.util.Hashtable;

import javax.microedition.io.HttpConnection;
import javax.microedition.io.InputConnection;

import net.rim.device.api.browser.field2.BrowserFieldRequest;
import net.rim.device.api.browser.field2.BrowserFieldResponse;
import net.rim.device.api.io.http.HttpHeaders;


public class CacheManagerImpl implements CacheManager {

    private static final int MAX_STANDARD_CACHE_AGE = 2592000;
    private Hashtable cacheTable;

    public CacheManagerImpl() {
        cacheTable = new Hashtable();
    }

    public boolean isRequestCacheable(BrowserFieldRequest request) {
        // Only HTTP requests are cacheable
        if (!request.getProtocol().equals("http")) {
            return false;
        }

        // Don't cache the request whose method is not "GET".
        if (request instanceof HttpConnection) {
            if (!((HttpConnection) request).getRequestMethod().equals("GET")) 
            {
                return false;
            }
        }

        // Don't cache the request with post data.
        if (request.getPostData() != null) {
                return false;
        }

        // Don't cache authentication request.
        if (request.getHeaders().getPropertyValue("Authorization") != null) {
            return false;
        }        

        return true;        
    }

    public boolean isResponseCacheable(HttpConnection response) {
        try {
            if (response.getResponseCode() != 200) {
                return false;
            }
        } catch (IOException ioe) {
            return false;
        }

        if (!response.getRequestMethod().equals("GET")) {
            return false;
        }

        if (containsPragmaNoCache(response)) {
            return false;
        }

        if (isExpired(response)) {
            return false;
        }

        if (containsCacheControlNoCache(response)) {
            return false;
        }

        if ( response.getLength() <= 0 ) {
            return false;
        }

        // additional checks can be implemented here to inspect
        // the HTTP cache-related headers of the response object

        return true;
    }

    private boolean isExpired(HttpConnection response) {
        try 
        {
            // getExpiration() returns 0 if not known
            long expires = response.getExpiration(); 
            if (expires > 0 && expires <= (new Date()).getTime()) {
                return true;
            }    
            return false;
        } catch (IOException ioe) {
            return true;
        }
    }

    private boolean containsPragmaNoCache(HttpConnection response) {
        try 
        {
            if (response.getHeaderField("pragma") != null 
                && response.getHeaderField("pragma")
                           .toLowerCase()
                           .indexOf("no-cache") >= 0) 
            {
                return true;
            } 

            return false;
        } catch (IOException ioe) {
            return true;
        }
    }

    private boolean containsCacheControlNoCache(HttpConnection response) {
        try {
            String cacheControl = response.getHeaderField("cache-control");
            if (cacheControl != null) {
                cacheControl = removeSpace(cacheControl.toLowerCase());
                if (cacheControl.indexOf("no-cache") >= 0 
                    || cacheControl.indexOf("no-store") >= 0 
                    || cacheControl.indexOf("private") >= 0 
                    || cacheControl.indexOf("max-age=0") >= 0) {
                    return true;        
                }

                long maxAge = parseMaxAge(cacheControl);
                if (maxAge > 0 && response.getDate() > 0) {
                    long date = response.getDate();
                    long now = (new Date()).getTime();                    
                    if (now > date + maxAge) {
                        // Already expired
                        return true;
                    }
                }
            } 

            return false;
        } catch (IOException ioe) {
            return true;
        }
    }    

    public InputConnection createCache(String url, HttpConnection response) {

        byte[] data = null;
        InputStream is = null;
        try {
            // Read data
            int len = (int) response.getLength();
            if (len > 0) {
                is = response.openInputStream();
                int actual = 0;
                int bytesread = 0 ;
                data = new byte[len];
                while ((bytesread != len) && (actual != -1)) {
                    actual = is.read(data, bytesread, len - bytesread);
                    bytesread += actual;
                }
            }       
        } catch (IOException ioe) {
            data = null;
        } finally {
            if (is != null) {
                try {
                    is.close();
                } catch (IOException ioe) {
                }
            }
            if (response != null) {
                try {
                    response.close();
                } catch (IOException ioe) {
                }
            } 
        }

        if (data == null) {
            return null;
        } 

        // Calculate expires
        long expires = calculateCacheExpires(response);

        // Copy headers
        HttpHeaders headers = copyResponseHeaders(response);

        // add item to cache
        cacheTable.put(url, new CacheItem(url, expires, data, headers));

        return new BrowserFieldResponse(url, data, headers);
    }

    private long calculateCacheExpires(HttpConnection response) {
        long date = 0;
        try {
            date = response.getDate();
        } catch (IOException ioe) {
        }

        if (date == 0) {
            date = (new Date()).getTime();
        }

        long expires = getResponseExpires(response);

        // If an expire date has not been specified assumes the maximum time
        if ( expires == 0 ) {
            return date + (MAX_STANDARD_CACHE_AGE * 1000L);
        }

        return expires;
    }

    private long getResponseExpires(HttpConnection response) {
        try {
            // Calculate expires from "expires"
            long expires = response.getExpiration();
            if (expires > 0) {
                return expires;
            }

            // Calculate expires from "max-age" and "date"
            if (response.getHeaderField("cache-control") != null) {
                String cacheControl = removeSpace(response
                                               .getHeaderField("cache-control")
                                               .toLowerCase());
                long maxAge = parseMaxAge(cacheControl);
                long date = response.getDate();

                if (maxAge > 0 && date > 0) {
                    return (date + maxAge);
                }
            }
        } catch (IOException ioe) {
        }

        return 0;
    }

    private long parseMaxAge(String cacheControl) {
        if (cacheControl == null) {
            return 0;
        }

        long maxAge = 0;
        if (cacheControl.indexOf("max-age=") >= 0) {
            int maxAgeStart = cacheControl.indexOf("max-age=") + 8;
            int maxAgeEnd = cacheControl.indexOf(',', maxAgeStart);
            if (maxAgeEnd < 0) {
                maxAgeEnd = cacheControl.length();
            }

            try {
                maxAge = Long.parseLong(cacheControl.substring(maxAgeStart,
                                                               maxAgeEnd));
            } catch (NumberFormatException nfe) {
            }
        }

                // Multiply maxAge by 1000 to convert seconds to milliseconds
                maxAge *= 1000L;
        return maxAge;
    }

    private static String removeSpace(String s) {
        StringBuffer result= new StringBuffer();
        int count = s.length();
        for (int i = 0; i < count; i++) {
            char c = s.charAt(i);
            if (c != ' ') {
                result.append(c);
            }
        }

        return result.toString();
    }

    private HttpHeaders copyResponseHeaders(HttpConnection response) {
        HttpHeaders headers = new HttpHeaders();
        try {
            int index = 0;
            while (response.getHeaderFieldKey(index) != null) {
                headers.addProperty(response.getHeaderFieldKey(index),
                                    response.getHeaderField(index));
                index++;
            }
        } catch (IOException ioe) {
        }

        return headers;
    }    

    public boolean hasCache(String url) {
        return cacheTable.containsKey(url);
    }

    public boolean hasCacheExpired(String url) {
        Object o = cacheTable.get(url);

        if (o instanceof CacheItem) {
            CacheItem ci = (CacheItem) o;
            long date = (new Date()).getTime();
            if (ci.getExpires() > date) {
                return false;
            } else {
                // Remove the expired cache item
                clearCache(url);
            }
        }

        return true;
    }

    public void clearCache(String url) {
        cacheTable.remove(url);
    }    

    public InputConnection getCache(String url) {
        Object o = cacheTable.get(url);        
        if (o instanceof CacheItem) {
            CacheItem ci = (CacheItem) o;
            return new BrowserFieldResponse(url, 
                                            ci.getData(), 
                                            ci.getHttpHeaders());
        }        
        return null;
    }
}

CacheItem.java:

import net.rim.device.api.io.http.HttpHeaders;

public class CacheItem {

    private String  url;    
    private long    expires;    
    private byte[] data;
    private HttpHeaders httpHeaders;

    public CacheItem(String url, 
                     long expires, 
                     byte[] data, 
                     HttpHeaders httpHeaders)
    {
        this.url = url;
        this.expires = expires;
        this.data = data;
        this.httpHeaders = httpHeaders;
    }

    public String getUrl() {
        return url;
    }

    public long getExpires() {
        return expires;
    }

    public byte[] getData() {
        return data;
    }

    public HttpHeaders getHttpHeaders() {
        return httpHeaders;
    }
}

Any help that can be giving towards this will be greatly appreciated. This really has me stumped. Thanks.

UPDATE: It looks like the caching only works at a certain level of the Blackberry libraries. I have added logic to check the current Software level and turn on the caching if it is supported by the device's current software level. This provides me with a good work around, but i would still like to know if there is a better way for the caching to work with all devices.

UPDATE 2 Based on comments: The site no longer displaying properly pertains to site not displaying the proper layout, images and text. It basically give a white background with links and text displaying as a bulleted list, all formatting removed.

Answer 1

I've been looking at your code, and the only thing I've found there's wrong with it, is you are completely ignoring the possibility of response.getLength(); returning less than zero (in CacheManagerImpl.createCache()). Although this didn't happen to me at the stackoverflow.com page, some pages use Transfer-Encoding: chunked, which means Content-Length is not present. This is, however, well handled, and should not cause the cache to fail (it would only be less effective).

I suggest testing your code on smaller problems, one step at a time. First, create cacheable page that only contains some text (like "hello") without any HTML tags. That should work pretty well, and in case it does not, it shouldn't be hard to determine where the data are getting lost. Or try to manually create cache item that does not expire and contains a webpage with no (external) stylesheet nor images, and see if it's even possible to pass it to BrowserField the way you do it. Then build on, add an image, add a style sheet so you can corner the problem.

The code is written very nicely, but at this point, it is not possible to help you because there are no evident flaws in the code and you are not explaining yourself very well, it is not clear how the error manifests itself, if it is every time or random, ... If I had a Blackberry device, I could probably try running the code for myself, but i don't.