5 minutes of effort = 1600% increase in performance

Posted by:

So, the other day I was humming along for one of our larger clients – putting the finishing touches on one of their applications. The app in question handles matching of vacancies against applications. The administrator needs to quickly have the ability to browse what vacancies have what status. 

One vacancy can have one of three different statuses:

  1. View – Show the selected application for this vacancy.
  2. List – Lists all matching applications for this particular vacancy.
  3. Missing – No current applications exists that match the criteria for this vacancy.

Don't worry too much about the business logic, I only provide a quick background to give you some idea of what I was trying to achieve.

 

Oh, I nearly forgot! Of course this is an IBM Domino solution as we are a "True Blue" IBM Premium Business partner. Unfortunately this client was stuck with a fairly old IBM Domino server: 8.5.2, so I wasn't too keen on using an XPages solution. Why? Because in my humble opinion XPages just hadn't matured enough yet. (Missing HTML5 features, no SSJS debugger, old DOJO release, loooong "first hit" boot times and so on – Yes, all of this can be mended, worked around, patched etc. But sometimes you can just say enough is enough and go with old school Domino development.)

Anywho, my first inclination was to create a WebQuerySave agent that did the matching ServerSide before presenting the result to the user, a very common approach. But the performance was really bad – The WQS agent took around 4000ms although the workload was light and number of documents fairly low (in the hundreds anyway). The page itself took around a second to load with an empty cache, so we we're looking at a total page load time here in the neighborhood of five seconds – not acceptable.

So I used an approach that's known as perceived performance. Let's take a clock as example – If you use a clock that doesn't show seconds, time seems to move slower. If there's a lot of stuff going on, the general experience is that things are happening quicker. So I extracted my LotusScript code and placed it in an agent that get's called thru AJAX. This way the page loads quickly, the user can start to interact with it immediately and the state of each vacancy trickles in as the server sees fit.

Below is a screenshot of the application, waiting for the AJAX request to comeback with the data. (The only vacancies that know its state are the ones that are booked, they're marked with "Visa" in the column to the right, the rest are pending).

 

Finally, here we have the first icarnation of the code:

Nothing too odd about the above, I would even dare to say a fairly common approach. I do what I can to speed up the process by using NotesViewEntryCollections and the ColumnValue-property. Using one view as the source, that has a column that combines the particular criteria for that vacancy. The key can look something like this:

2015-05-27DSurgeon (Date + Slot + Role)

We use this key to find any matching requests, if none is found it's added to the string that's returned to the AJAX request. We only need the ones that doesn't have a match (Missing) as the first state (View) is stored with the vacancy and the second state (List) are all that are not missing. Ideally "Missing" should always be very few documents so the data transfer over the wire should also be low, increasing performance further.

 

So, everything was "hunky-dory" then. The application loaded quickly and felt responsive to the user, but… The bad performance of the LotusScript code was nawing at me… 

After a murky night of coding I came up with the following:

Option Public
Option Declare
Sub Initialize
	Dim s As New NotesSession 
	Dim db As NotesDatabase 
	Dim vRequest As NotesView
	Dim vMatchVacancy As NotesView
	Dim veRequest As NotesViewEntry 
	Dim veMatchVacancy As NotesViewEntry 
	Dim rowIds As String
	Dim key As String
	Dim i As Integer
	Dim count As Long
	Dim hasEmptySlots As Boolean
	
	Dim navMatch As NotesViewNavigator 
	Dim navRequest As NotesViewNavigator 
	Dim ve As NotesViewEntry
	
	On Error GoTo handler
	
	Set db = s.Currentdatabase

	Set vMatchVacancy = db.getView("vMatchVacancy")
	Set vRequest = db.getView("vRequest-list-match")
	
	Call vRequest.refresh()
	Call vMatchVacancy.refresh()
	
	Set navMatch = vMatchVacancy.createViewNav()
	Set navRequest = vRequest.createViewNav()
	
	count = vMatchVacancy.Allentries.Count
	i = 0
	
	' do not do AutoUpdates
	vMatchVacancy.AutoUpdate = False
	vRequest.AutoUpdate = False
	
	' enable cache for max buffering
	navMatch.BufferMaxEntries = 100
	' if we are not interested in the number of children, we can go a little faster
	navMatch.EntryOptions = Vn_entryopt_nocountdata
	

	Set veMatchVacancy = navMatch.GetFirst
	
	Print "content-type: text;charset=utf-8;"
		
	While ( i <  count )  
		hasEmptySlots = False
		key = veMatchVacancy.Columnvalues(0)
		
		Set veRequest = vRequest.Getentrybykey(key, True)
		
		If(veRequest Is Nothing) Then
			Print  rowIds & veMatchVacancy.Columnvalues(1) & ","
		End if
		
		Set veMatchVacancy=navMatch.getNext(veMatchVacancy)
		i = i + 1
	Wend

	
exitSub:
	Exit Sub 
	
handler: 
	MsgBox db.filepath & "agentMatchVacancys - " & Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	Print Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	
	Resume exitSub 
	
End Sub

The big difference here is the use of the NotesViewNavigator. When using the NotesViewNavigator object you have the opportunity to use it's cache – "BufferMaxEntries". I set it to 100 in my case as the view will show no more then 100 rows at a time. I also set the EntryOptions to "VN_ENTRYOPT_NOCOUNTDATA" as I have no parent/child relationship in the view.

All in all the the running time of the agent went from 4000ms to 25ms! Pretty darn impressive if anyone were to ask me!

 

This technique is nothing new, but the performance gains are so huge I thought it's well wort repeating.

I was heavily influenced by this article: "Fast Retrieval of View Data Using the ViewNavigator Cache – V8.52" and I highly recommend you check that out for more details.

 

[EDIT]

After re-running the performance tests in my local test environment I've "only" managed a 200% increase in performance. Beware, YMMV….

5

AD110 – IBM Lotus Domino XPages Go Zoom!

Posted by:

This session talked briefly about suitable tools that you may use the debug and/or get a glimpse behind the sceen regarding frontend peformance:

Firebug
Chrome Developer Tools
Speed Tracer
Page Speed
Yslow
IE Developer Toolbar
Opera Dragonfly
Weinre
Fiddler
XPages Toolbox

Most of them should you be fairly familiar with. Firebug, at the very least. Personally I prefer Chrome Developer Tools, because I prefer Chrome and I find their Dev tools less prone to crashing.

For IE Development the IE Developer Toolbar is a must.

Fiddler is great across everything that uses HTTP/S. It sits in the middle as a proxy and record a lots of useful information as well as the ability to simulate modem speeds.

XPages a toolbox is also a welcome addition to be able to get a backstage view of what takes up most of the time in you XPages application.

The session went on and talked, in great detail, about the differences between the execution modes. To summarize:

Full refresh – Is the most expensive. Recalculates and sends the entire page.
Partial refresh – Recalculates the entire page, but just sends the part of the page you specify.
Partial execution + Partial refresh – Just computes the requested area and returns that.

For best performance, use the last combination specified above where ever possible.

Other suggestions are:
Use get requests instead of post. This skips a lot of steps in the life cycle and as such reduces the load on the server and calculates quicker.
Set all containers, that don’t require input, to read only. Again, saves computing power.
Use the dataCache-property on every Domino view source you can.
Use the viewsScope/Request scope to cache objects and variable. Caching reduces CPU utilization and using short lived scoped variables reduces the memory footprint.

Can’t remember if they mentioned this, but in reality modern web applications main bottleneck isn’t the server, it’s the number of http requests the page does. There for you will get the biggest performance boost if you enable Application Preferences / XPages-tab / “Use runtime optimized JavaScript and CSS resources”. (Domino 8.5.3 only) This merges most (not all) JavaScript and CSS external references you might have into one request per type.

Combine all of the above and your XPage applications will be screaming!

20120123-190448.jpg

1