5 minutes of effort = 1600% increase in performance

Posted by:

So, the other day I was humming along for one of our larger clients – putting the finishing touches on one of their applications. The app in question handles matching of vacancies against applications. The administrator needs to quickly have the ability to browse what vacancies have what status. 

One vacancy can have one of three different statuses:

  1. View – Show the selected application for this vacancy.
  2. List – Lists all matching applications for this particular vacancy.
  3. Missing – No current applications exists that match the criteria for this vacancy.

Don't worry too much about the business logic, I only provide a quick background to give you some idea of what I was trying to achieve.

 

Oh, I nearly forgot! Of course this is an IBM Domino solution as we are a "True Blue" IBM Premium Business partner. Unfortunately this client was stuck with a fairly old IBM Domino server: 8.5.2, so I wasn't too keen on using an XPages solution. Why? Because in my humble opinion XPages just hadn't matured enough yet. (Missing HTML5 features, no SSJS debugger, old DOJO release, loooong "first hit" boot times and so on – Yes, all of this can be mended, worked around, patched etc. But sometimes you can just say enough is enough and go with old school Domino development.)

Anywho, my first inclination was to create a WebQuerySave agent that did the matching ServerSide before presenting the result to the user, a very common approach. But the performance was really bad – The WQS agent took around 4000ms although the workload was light and number of documents fairly low (in the hundreds anyway). The page itself took around a second to load with an empty cache, so we we're looking at a total page load time here in the neighborhood of five seconds – not acceptable.

So I used an approach that's known as perceived performance. Let's take a clock as example – If you use a clock that doesn't show seconds, time seems to move slower. If there's a lot of stuff going on, the general experience is that things are happening quicker. So I extracted my LotusScript code and placed it in an agent that get's called thru AJAX. This way the page loads quickly, the user can start to interact with it immediately and the state of each vacancy trickles in as the server sees fit.

Below is a screenshot of the application, waiting for the AJAX request to comeback with the data. (The only vacancies that know its state are the ones that are booked, they're marked with "Visa" in the column to the right, the rest are pending).

Loading the state for each vacancy separately

(I had to pause the loading in the debugger because it’s just too quick to catch otherwise…)

 

Finally, here we have the first icarnation of the code:

Option Public
Option Declare
Use "helpers"
Sub Initialize
	Dim s As New NotesSession 
	Dim db As NotesDatabase 
	Dim vRequest As NotesView
	Dim vMatchVacancy As NotesView
	Dim vcRequest As NotesViewEntryCollection 
	Dim vcMatchVacancy As NotesViewEntryCollection 
	Dim veRequest As NotesViewEntry 
	Dim veMatchVacancy As NotesViewEntry 
	Dim rowIds As String
	Dim key As String
	Dim count As integer
	
	Print "content-type: text;charset=utf-8;"
	
	On Error GoTo handler 
	
	Set db = s.Currentdatabase
	Set vRequest = db.getView("vRequest-list")
	Set vMatchVacancy = db.getView("vMatchVacancy")
	
	Call vRequest.refresh()
	Call vMatchVacancy.refresh()
	
	Set vcMatchVacancy = vMatchVacancy.Allentries
	
	Set veMatchVacancy = vcMatchVacancy.Getfirstentry()
	
	While Not veMatchVacancy Is Nothing
		key = veMatchVacancy.Columnvalues(0)
		
		Set veRequest = vRequest.Getentrybykey(key, true)
		
		If(veRequest Is Nothing) Then
			rowIds = rowIds & veMatchVacancy.Columnvalues(1) & ","
		End If
		
		Set veMatchVacancy = vcMatchVacancy.Getnextentry(veMatchVacancy)
	Wend
	
	
	Print rowIds
	
exitSub:
	Exit Sub 
	
handler: 
	MsgBox db.filepath & "agentMatchVacancys - " & Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	Print Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	
	Resume exitSub 
End Sub

Nothing too odd about the above, I would even dare to say a fairly common approach. I do what I can to speed up the process by using NotesViewEntryCollections and the ColumnValue-property. Using one view as the source, that has a column that combines the particular criteria for that vacancy. The key can look something like this:

2015-05-27DSurgeon (Date + Slot + Role)

We use this key to find any matching requests, if none is found it's added to the string that's returned to the AJAX request. We only need the ones that doesn't have a match (Missing) as the first state (View) is stored with the vacancy and the second state (List) are all that are not missing. Ideally "Missing" should always be very few documents so the data transfer over the wire should also be low, increasing performance further.

 

So, everything was "hunky-dory" then. The application loaded quickly and felt responsive to the user, but… The bad performance of the LotusScript code was nawing at me… 

After a murky night of coding I came up with the following:

Option Public
Option Declare
Sub Initialize
	Dim s As New NotesSession 
	Dim db As NotesDatabase 
	Dim vRequest As NotesView
	Dim vMatchVacancy As NotesView
	Dim veRequest As NotesViewEntry 
	Dim veMatchVacancy As NotesViewEntry 
	Dim rowIds As String
	Dim key As String
	Dim i As Integer
	Dim count As Long
	Dim hasEmptySlots As Boolean
	
	Dim navMatch As NotesViewNavigator 
	Dim navRequest As NotesViewNavigator 
	Dim ve As NotesViewEntry
	
	On Error GoTo handler
	
	Set db = s.Currentdatabase

	Set vMatchVacancy = db.getView("vMatchVacancy")
	Set vRequest = db.getView("vRequest-list-match")
	
	Call vRequest.refresh()
	Call vMatchVacancy.refresh()
	
	Set navMatch = vMatchVacancy.createViewNav()
	Set navRequest = vRequest.createViewNav()
	
	count = vMatchVacancy.Allentries.Count
	i = 0
	
	' do not do AutoUpdates
	vMatchVacancy.AutoUpdate = False
	vRequest.AutoUpdate = False
	
	' enable cache for max buffering
	navMatch.BufferMaxEntries = 100
	' if we are not interested in the number of children, we can go a little faster
	navMatch.EntryOptions = Vn_entryopt_nocountdata
	

	Set veMatchVacancy = navMatch.GetFirst
	
	Print "content-type: text;charset=utf-8;"
		
	While ( i <  count )  
		hasEmptySlots = False
		key = veMatchVacancy.Columnvalues(0)
		
		Set veRequest = vRequest.Getentrybykey(key, True)
		
		If(veRequest Is Nothing) Then
			Print  rowIds & veMatchVacancy.Columnvalues(1) & ","
		End if
		
		Set veMatchVacancy=navMatch.getNext(veMatchVacancy)
		i = i + 1
	Wend

	
exitSub:
	Exit Sub 
	
handler: 
	MsgBox db.filepath & "agentMatchVacancys - " & Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	Print Error & Chr(13) + "Module: " & CStr( GetThreadInfo(1) ) & ", Line: " & CStr( Erl ) 
	
	Resume exitSub 
	
End Sub

The big difference here is the use of the NotesViewNavigator. When using the NotesViewNavigator object you have the opportunity to use it's cache – "BufferMaxEntries". I set it to 100 in my case as the view will show no more then 100 rows at a time. I also set the EntryOptions to "VN_ENTRYOPT_NOCOUNTDATA" as I have no parent/child relationship in the view.

All in all the the running time of the agent went from 4000ms to 25ms! Pretty darn impressive if anyone were to ask me!

 

This technique is nothing new, but the performance gains are so huge I thought it's well wort repeating.

I was heavily influenced by this article: "Fast Retrieval of View Data Using the ViewNavigator Cache – V8.52" and I highly recommend you check that out for more details.

 

[EDIT]

After re-running the performance tests in my local test environment I've "only" managed a 200% increase in performance. Beware, YMMV….

5