XPages performance - 2 apps on same server, 1 runs

Question

We have been having a bit of a nightmare this last week with a business critical XPage application, all of a sudden it has started crawling really badly, to the point where I have to reboot the server daily and even then some pages can take 30 seconds to open.

The server has 12GB RAM, and 2 CPUs, I am waiting for another 2 to be added to see if this helps.

The database has around 100,000 documents in it, with no more than 50,000 displayed in any one view. The same database set up as a training application with far fewer documents, on the same server always responds even when the main copy if crawling.

There are a number of view panels in this application - I have read these are really slow. Should I get rid of them and replace with a Repeat control?

There is also Readers fields on the documents containing Roles, and authors fields as it's a workflow application.

I removed quite a few unnecessary views from the back end over the weekend to help speed it up but that has done very little.

Any ideas where I can check to see what's causing this massive performance hit? It's only really become unworkable in the last week but as far as I know nothing in the design has changed, apart from me deleting some old views.

Answer 1

Try to get more info about state of your server and application.

Hardware troubleshooting is summarized here: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Domino_Server_performance_troubleshooting_best_practices

According to your experience - only one of two applications is slowed down, it is rather code problem. The best thing is to profile your code: http://www.openntf.org/main.nsf/blog.xsp?permaLink=NHEF-84X8MU

To go deeper you can start to look for semaphore locks: http://www-01.ibm.com/support/docview.wss?uid=swg21094630, or to look at javadumps: http://lazynotesguy.net/blog/2013/10/04/peeking-inside-jvms-heap-part-2-usage/ and NSDs http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Using_NSD_A_Practical_Guide/$file/HND202%20-%20LAB.pdf and garbage collector Best setting for HTTPJVMMaxHeapSize in Domino 8.5.3 64 Bit.

This presentation gives a good overview of Domino troubleshooting (among many others on the web).

Answer 2

Ok so we resolved the performance issues by doing a number of things. I'll list the changes we did in order of the improvement gained, starting with the simple tweaks that weren't really noticeable.

1. Defrag Domino drive - it was showing as 32% fragmented and I thought I was on to a winner but it was really no better after the defrag. Even though IBM docs say even 1% fragmentation can cause performance issues.

2. Reviewed all the main code in the application and took a number of needless lookups out when they can be replaced with applicationScope variables. For instance on the search page, one of the drop down choices gets it's choices by doing an @Unique lookup on all documents in the database. Changed it to a keyword and put that in the application Scope.

3. Removed multiple checks on database.queryAccessRole and put the user's roles in a sessionScope.

4. DB had 103,000 documents - 70,000 of them were tiny little docs with about 5 fields on them. They don't need to be indexed by the FTIndex so we moved them in to a separate database and pointed the data source to that DB when these docs were needed. The FTIndex went from 500mb to 200mb = faster indexing and searches but the overall performance on the app was still rubbish.

5. The big one - I finally got around to checking the application properties, advanced tab. I set the following options : Optimize document table map (ran copystyle compact) Dont overwrite free space Dont support specialized response hierarchy Use LZ1 compression (ran copystyle compact with options to change existing attachments -ZU) Dont allow headline monitoring Limit entries in $UpdatedBy and $Revisions to 10 (as per domino documentation)

And also dont allow the use of stored forms.

Now I don't know which one of these options was the biggest gain, and not all of them will be applicable to your own apps, but after doing this the application flies! It's running like there are no documents in there at all, views load super fast, documents open like they should - quickly and everyone is happy.

Until the http threads get locked out - thats another question of mine that I am about to post so please take a look if you have any idea of what's going on :-)

Thanks to all who have suggested things to try.