Performance Tuning in Grails 2.4.x: Part 2
In Part 1 I shared how I recently tuned an internal time tracking and invoicing app built on Grails 2.4.x app in order to make it perform more quickly. The app is hosted on AWS (Ubuntu) and served using Tomcat 7. Part 1 spoke about Java melody and 2nd level hibernate caching.
Soldiering on…
After enabling and tuning the hibernate caches, my app was looking pretty good on my local laptop. Java melody had pointed out some slow to respond controller calls, and I threw some old-fashioned log statements into the corresponding service methods so I could benchmark the improvements before and after caching.
e.g.
[code language=”groovy”]
def t1= System.currentTimeMillis()
def totals = timesheet.getDailyTotals()
def t2= System.currentTimeMillis()
log.debug(“timesheet.getDailyTotals() Exec Time=${t2-t1} millis")
[/code]
I was able to see a huge performance improvement (~3-15x) in my service methods. Also, melody showed a marked improvement when it came to template generation performance as a result of sped up underlying domain obj manipulations. Overall hibernate caching made a huge impact on my local tests.
So, I was excited to put the app into QA and get ultimately get it out to production. Unfortunately, as often happens, performance on the remote (AWS hosted) QA environment was nothing like performance on my local laptop. That is, some of the pages were performing well, but a few of them were still dog slow.
For the slow pages, Chrome Developer Tools helped me realize that the problem was not actually anything happening on the server, but the fact that I was passing very large html pages back as text from the server as the result of some overzealous looping constructs in the gsps. There was no question that this code should be rearchitected – in fact there is a story in the project backlog to convert from gsp to Angular. However, for the moment I needed to get something out the door and move on to other things.
The solution was to enable gzip compression. Enabling compression for text content is a near-mandatory optimization when configuring web servers like ngnix or Apache, but the fact that we were serving a war from Tomcat without a web server front (as many grails apps seem to) caused me to forget to do that. I had never enabled compression on Tomcat directly. Initially I looked for a Grails plugin and found the following aging plugins:
https://grails.org/Compress+Plugin
http://grails.org/plugin/ui-performance
However, my experience with the diagnostic and ehcache plugins (see Part 1) made me realize that perhaps it’s wiser to just use the plugins you really need. More plugins mean more potential for conflicts and potential difficulties down the road when it comes to upgrading your libraries. This is definitely a tradeoff – there is a certain elegance to keeping as much configuration as possible in Config.groovy.
In this case, I modified the tomcat server.xml file directly by adding the following xml attributes to the Connector tag as per this link.
http://viralpatel.net/blogs/enable-gzip-compression-in-tomcat/
compression="on" compressionMinSize="2048" noCompressionUserAgents="gozilla, traviata" compressableMimeType="text/html,text/xml"
Hot mama, massive page load improvements! This worked like a charm and somewhat eliminated my chagrin at having built the app using server side templating. We still plan to move over to an Angular frontend when we implement our next round of UX improvements, but at least the app will perform ok until then and there is no immediate urgency.
Now that the app was performing nicely in QA, it was time to move to production.
To Production!
In software, when you first think you are done you are probably not. After deploying to production, a couple of new problems emerged.
1) The /monitoring path that melody exposes was public and not secured by Shiro (authorization library). In fact, it was publicly available so anyone on the web could look at the guts of my app and wreak havoc, or at least stick out their tongues at my object counts.
2) I realized that I needed to externalize the ehcache.xml file. One of the first commits we made involved externalization of Config.groovy and DataSource.groovy so that the app would not have to be redeployed with every minor config change. In the same way, I wanted the ability to tweak the caches via ehcache.xml (add memory, change time to live, etc) without rebuilding and redeploying war. Building a war in grails, incidentally, is not a fast process unless you have some serious server horsepower.
Securing melody
This was more of a roller coaster than I expected.
There are a ton of links out there suggesting that while Shiro used to have problems securing melody (because of url filter load order issue), that problem no longer exists as of melody build 1.14. Perhaps somebody has been able to get Shiro to pick up the /melody path that maps to the melody servlet. For yours truly, this didn’t work.. Instead, I ended up going the more traditional realm authentication route suggested by the melody docs. It was kind of ugly and it meant mussing with more Tomcat configs, but it worked.
In order to secure javamelody, we need to get The solution involved adding some configuration to the web.xml file as described here. https://code.google.com/p/javamelody/wiki/UserGuideAdvanced#5._Security_with_a_collect_server
There is a gotcha. While we are happy to include web.xml authentication for production and QA, we don’t actually want this in development mode (running locally). Locally, we want to be able to tweak javamelody to our heart’s content without authenticating, and we don’t want to run within container or to deal with the java realm. Ideally I could have just used the -Djavamelody.plugin-authentication-disabled=true flag as advertised in the javamelody docs while running locally (by adding this jvm param to GRAILS_OPTS). Unfortunately that didn’t help to disable authentication as advertised, and when running locally I was presented with the following nastiness http://drt.io/foavq. Also, I didn’t want to run within a tomcat container locally, which would have allowed me to configure the realm.
Fortunately, I already had a hack in place for using a separate version of web.xml depending on the environment.
This hack is as follows. From BuildConfig.groovy:
[code language=”groovy”]
//from http://stackoverflow.com/questions/5888980/environment-specific-web-xml-in-grails
//differentiating because in prod we want secure (http auth) java melody. in dev we don’t care, leave it open
switch (grails.util.Environment.getCurrent().getName()) {
case "development":
if (new File("/${basedir}/src/templates/war/web_dev.xml").exists()) {
grails.config.base.webXml = "file:${basedir}/src/templates/war/web_dev.xml"
}
break;
default:
if (new File("/${basedir}/src/templates/war/web_prod.xml").exists()) {
grails.config.base.webXml = "file:${basedir}/src/templates/war/web_prod.xml"
}
break;
}
[/code]
In this way I can use a separate web.xml depending on the environment I’m running in.
Externalizing ehcache.xml
I wanted to keep my cache settings on the filesystem in production in QA, so that I could tweak them at will without having to rebuild the war, which is time-consuming.
Unfortunately, the ehcache.xml file which is needed for development was being written to WEB-INF/classes/ehcache.xml by default, which means it was being pulled into the war file when it was created. As a result, QA and Prod would use the ehcache.xml inside the war file instead of the version of ehcache.xml on the filesystem.
Basically I needed to prevent ehcache.xml from being bundled with the war files meant for my QA and Prod environments.
The way to do this was to create an _Events.groovy script that included a eventCreateWarStart directive. When the start create war event occurs, my script is run, and the ehcache.xml file is removed before it can be included.
[code language=”groovy”]
// Remove the ehcache jar before the war is bundled – we want to use external ehcache dir
eventCreateWarStart = { warName, stagingDir ->
println "${stagingDir} Excluding ehcache.xml so that changes can more easily be made – please make sure ehcache.xml is in tomcat lib dir so it gets included in app classpath"
def location = "${stagingDir.getAbsolutePath()}/WEB-INF/classes/ehcache.xml"
def file = new File(location)
if(file.exists()) {
println "found ehcache.xml at ${location}"
ant.delete(file:location)
println "${location} deleted (excluded from war)"
} else {
println "could not find ehcache.xml at ${location}…ignoring exclusion attempt"
}
}
[/code]
I then created a sym link from tomcat’s lib dir to the place where I keep all my application configs. Of course, this makes the config globally accessible to tomcat, which is ok in my case since I have only one grails app running on the server.
/usr/share/tomcat7/lib/ehcache.xml -> /home/ubuntu/.grails/config/moment/ehcache.xml
I also externalized Config.groovy and DataSource.groovy to that same location for the same reason. No one wants to rebuild a war file when there is a time-sensitive issue that requires a config change.
Conclusion
After following the steps detailed above, my app was nicely configurable and performing much better.
I hope this can help someone out there, or at least might serve as yet another testament to the puzzle-chasing masochism that is software development.
