Performance Tuning in Grails 2.4.x: Part 1

Over the weekend I made performance optimizations to an internal web app that we use for time tracking and invoicing. The app runs on tomcat and is built using grails 2.4.2. Grails 2.4 included some major changes including asset pipelining. This is the first time I’ve tuned a grails app, and I had to do quite a bit of web crawling in order to find my way. I thought I would share some of the helpful tips I encountered, parts of the configuration I used, as well as some dead ends that could waste your time.

In this post I’ll mostly talk about simple diagnostic setup with melody and hibernate caching. In the next post I’ll talk about compression, externalizing cache configs and locking down your diagnostics dashboard in production (and not in dev).

To start off with I looked into monitoring tools. This helpful link suggested a number of profiling tools. I was familiar with melody, and it was great to see it so tightly integrated with grails. In fact the grails melody plugin has the same versioning as the underlying melody core – something you don’t usually see in plugins that integrate third party libraries, but a nice convention I think.

Melody was a cinch to get up and running in my local environment, and was extremely useful in diagnosing bottlenecks. Later I’ll describe how I locked down melody in prod while at the same time keeping it enabled in dev – that was a little tedious.

To enable melody I added the following line to the plugins section of my BuildConfig.groovy.

compile ":grails-melody:1.52.0"

I also tried to install the app-info and perf4j (logging) plugins, but was less successful. Neither of these appear to be working with grails 2.4.x at the present time.

After adding the melody plugin, running the app now gave me access to the melody dashboard at localhost:8080/[appName]/monitoring. While navigating the app, I could quickly see that some queries were taxing the database. In the app I was tuning, domain objects needed to be accessed frequently in generating statistics. Going to the database was obviously slowing things down drastically.

Fortunately, grails is built on hibernate, which has built-in support for second level caching as well as query caching. While first level caching is enabled by default in hibernate, second level caching is not. Second level caching is more configurable than first level caching. While first level caching works within a single hibernate session, second level caching works across sessions and is useful for more extensive optimizations. My app’s domain objects had very frequent read access, but infrequent write access.

According to this helpful SO post, a responder gives reasons why you might not want to enable the 2nd level cache. I thought it was insightful – there is a lot of misinformation out there about 2nd level caching.

1. there are so many entities and they are so changing that the number of cache hits would be very low, and that the second-level cache handling would in fact consume more time and memory than a solution without cache
2. the application is clustered, and the cost and complexity of a distributed second-level cache is too high, for a low gain
3. other non-hibernate applications write to the same database, and the cache has thus a big risk of returning stale data, which is not acceptable
4. everything goes very well without a second-level cache, and there is no reason to make the application more complex than it is.

None of these fit my situation, so I got to work on configuring the cache.

Hibernate supports Terracotta’s ehcache. I enabled the cache in my Datasource.groovy file as follows:

[code language=”groovy”]
hibernate {
    generate_statistics=true
    cache.use_second_level_cache = true
    cache.use_query_cache = true
    cache.region.factory_class = ‘net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory’
}
[/code]

Notice that I enabled statistics, which allows melody to tell you what is happening.

You might notice that I opted to use a factory class SingletonEhCacheRegionFactory as opposed to what is recommended in the grails docs:

cache.provider_class=‘org.hibernate.cache.EhCacheProvider

This is because certain grails commands (the db migration command “dbm-update”, for example) make attempts to load the cache manager more than once, which then causes problems because ehcache needs the cache managers to be unique. Naming the ehcache in ehcache.xml as some people recommend will not help, because it’s actually the exact same cache that gets loaded. Using the Singleton version saves you from running into this issue, and I didn’t observe any negatives from using that approach. Here is the error we avoid:

net.sf.ehcache.CacheException: Another CacheManager with same name ‘moment-cache’ already exists in the same VM.

I then modified BuildConfig, because I wanted to get the latest version of ehcache. If you’re using hibernate 3 like I am, you will get a pretty old version of ehcache included by default.

Exclude the ehcache transitive dependency:

[code language=”groovy”]
runtime (‘:hibernate:3.6.10.16’) {
     excludes ‘ehcache-core’
}
[/code]

in dependencies section of BuildConfig, include the version of ehcache that you’re interested in:

[code language=”groovy”]
compile ‘net.sf.ehcache:ehcache-core:2.6.8’
[/code]

I also had the grails cache plugin in my BuildConfig. This cache plugin is needed for 2nd level cache wiring, somehow magically.

[code language=”groovy”]
compile ‘:cache:1.1.7’
[/code]

After DataSource.groovy, I then included an ehcache.xml file in grails-app/conf and proceeded to define some caches. I did try the grails cache-ehcache plugin, but unfortunately it wasn’t picking up the conf/CacheConfig.groovy configuration file as advertised. I decided to go with less plumbing and use ehcache.xml directly. I also figured that I would externalize the cache configuration so that I wouldn’t have to rebuild the war in production in the event that changes were needed. I’ll describe externalizing the optimizations for production later.

In my case, I created a simple default cache with eternal=“true”. I also set statistics=“true” so that melody will show the hit/miss ratios are for each of your caches. melody will also tell you how each cache is configured, which is useful for debugging a wonky cache. I used the default cache for the domain objects that were few in number and I would always want in the cache.

<defaultCache
     eternal="true"
     overflowToDisk="false"
     statistics="true">

In my app, the Project domain is something that we would always want in the cache and never want to expire – even if some projects become inactive, I would still want to derive calculations from them. So for Project, I don’t do anything explicit in ehcache.xml – i just let the default cache apply. TimesheetEntry, on the other hand, is something pretty low level that is time-sensitive. I created a custom cache for this type of object. Note that I needed to use the full package name in ehcache.xml – you’ll be hosed if you don’t do that.

<cache name="com.grio.moment.TimesheetEntry"
    eternal="false"
    timeToIdleSeconds="86400"
    statistics="true"/>

In terms of resource allocation for the caches, I just kept it simple and added maxBytesLocalHeap=“512M” to my ehcache tag. This means that all caches will pull from that one big chunk of memory. In my case I didn’t have any special reason to throttle resources for a particular cache.

For all the domain objects that you want to cache, include the following:

[code language=”groovy”]
static mapping = {
   cache true
}
[/code]

This tells the grails cache plugin that the domain is eligible to be cached. If ehcache is enabled in hibernate (we did that above), then ehcache will be used.

I also had some domain objects that were hierarchical. For example my Client domain contains a list of projects of type Project. In these cases we want to enable read-write caching on the collection itself.

To handle this,

[code language=”groovy”]
static mapping = {
cache true
   projects cache: true
}
[/code]

Ok that’s it for now. In my case the caching worked beautifully and the app sped up substantially (~4x improvement for many of my pages). In a perfect world I would have liked to have patched the broken 3rd party plugins, but unfortunately there wasn’t enough time for that – internal apps get short shrift next to client work and daily consulting company juggling! :)

In Part 2, I’ll talk about another additional optimizations, as well as getting melody and ehcache configured for production.