Jahia fine-tuning tips you need to know

Please find below a series of key tips you need to know regarding Jahia production server fine-tuning.

During the development stage:

1)      NEVER use your development database in production

While developing or customizing your new set of templates or portlets, you will most likely generate some Java exceptions. This will cause the application server crashes or require some Java processes to be terminated quite abruptly. All these actions may corrupt unfinished Jahia transactions and indirectly also corrupt your underlying database. Therefore never use a development database in production. Always migrate your set of templates, once fully tested, on a fresh or on a stable production server. If you want to test your templates with some real content, always copy the database and files from the production server to your development workstation. Never do the opposite.

2)      Use the Jahia API with care

If you plan to develop advanced Jahia features in addition to basic templates (for example if you plan to automatically create a default set of pages for each new virtual site or if you plan to automatically import external content into the Jahia content repository), please take care to fully understand the Jahia model and to fully and extensively test your custom developments before putting them in production and in the hand of the end-users. Certain Jahia operations are quite complex and tricky (e.g. the page creation operation). Having full access to the source code and to all the underlying Jahia classes, doesn’t imply that you can use them as is without fully understanding the Jahia object model. You then risk corrupting the integrity of the database by forgetting to integrate certain mandatory checks or certain tests in your code.

Moreover the Jahia team does not guarantee that the Jahia API will not change from a point release to another. So your workarounds or your advanced features may not work anymore in the next releases. Consequently, think twice before implementing such advanced features. Always think if it isn’t better to contribute your changes to the community or to sponsor a longer term development rather than implementing a short term custom workaround which risks becoming rapidly unmaintainable and may corrupt your whole content model if badly implemented.

3)      Use OSCache to cache fragments

OSCache can perfectly be used for fragments of the page, which take much time to render and where the result is always the same, especially if it is the same on every page (header, footer, menu, last 5 news,...). However you might want to use it only in LIVE mode, if you want to show the edit menu items in EDIT mode.

You could do it like this:

<%

ContainerBean boxCont = (ContainerBean) pageContext.getAttribute("boxContainerBean");

int boxContainerId = boxCont.getId();

boolean isCacheActive = jData.gui().isNormalMode();

%>

<cache:cache time='<%= isCacheActive ? 120 : 0 %>'

    key='<%="box" + (new Integer(boxContainerId)).toString()%>'>

    [... the display of the box...]

</cache:cache>

In this example the box will be cached in LIVE mode for 120 seconds. You can decide whether you want to cache a fragment for minutes or hours. It depends how often it changes. For instance the header and footer will not change often. For the navigation, it might happen more frequent that new pages are coming or old pages are being deleted. In order to prevent 404 errors, if a fragment is still showing a page, which has been deleted, you could also call

<cache:flush key='<%="box" + (new Integer(boxContainerId)).toString()%>' />

in EDIT mode on the JSP, where the container gets changed. If you are using a cluster, then you should also consider the configuration, described here: http://www.opensymphony.com/oscache/wiki/Clustering.html  .

For the key, it also depends whether you are using access rights on the container list, so each user might have a different view. For this case you might want to add the user-id to the key, then the fragment will be cached per user. You should surely limit the time on these fragments.

To use the OSCache you need to deploy the JAR to the lib directory of the Jahia webapp, you need to add the oscache.tld to the web.xml and then you can use the taglib in your templates.

4)      Avoid loading the webapps/portlets if they are not necessary

Each time you create a new virtual site, Jahia re-deploys a new copy of all the available webapps/portlets in the following directory:

X:\jahia\tomcat\webapps\jahia\WEB-INF\var\shared_components

By default and for demo purposes, each web application is an independent servlet which launches its own embedded Hypersonic database. Each web application is then loaded in memory with its own embedded database at startup -even if you do not use it on your new virtual site. This behavior can rapidly cause “Out of Memory” problems and/or some performance issues.

Therefore we recommend that all the unnecessary webapps are removed from this directory, especially if you only use the Jahia CMS features. If necessary, you can re-deploy any of them at a later point by simply copying the original war/ear file in the virtual site hot-deployment directory available here:

E:\jahia\tomcat\webapps\jahia\WEB-INF\var\new_webapps\Your_Virtual_Site

Please note that if another virtual site has already deployed the webapp you want to use, you can share the same instance of this webapp and reuse it in your current virtual site. Just go in the “Manage Portlet” menu in the Admin Center and click on “Share Components with other sites”. This will avoid launching a new copy of this webapp and its database in memory.

Finally, certain web applications can be configured to use another data source than the embedded one (e.g. the database used for your Jahia installation). So instead of launching a separate database for each web application, it is recommend to centralize all your web application data into one centralized storage system. This will also simplify database backups.

Warning: Not all the web applications provided by default with Jahia have been developed to be able to use another data source. Modifications to the web application may be required.

Once in production:

5)      Reduce logs verbosity

Do not forget to reduce log verbosity on the production server. Jahia makes use of the Log4J  library for all debugging info. Log4J defines some logging levels as follows (from the more to the less verbose):

ALL < DEBUG < INFO < WARN < ERROR < FATAL < OFF

ß   More Verbose                                         Less Verbose à

By default, point releases of Jahia (as opposed to CVS builds) have the log levels set to INFO . If you want to increase the log level to trace a problem, you will need to modify the log4j.xml file located in the following directory:

%TOMCAT_HOME%/webapps/jahia/WEB-INF/etc/config/

(e.g. C:\jahia405 \tomcat\webapps\jahia\WEB-INF\etc\config )

At the bottom of the file, you have the <root>... </root> part. Change the <priority value="info"/> to <priority value="debug"/> for example to have more debugging information in the console.

Don't forget to change back the values to INFO , as DEBUG log level has a pretty important impact on performance. If you encounter a problem afterwards, you can perfectly temporarily switch the production server to DEBUG mode (for example to have the time to get the full stack trace exceptions). This can be done without rebooting since Jahia automatically detects all the changes in log4j.xml every 30 seconds.

6)      Modify your default JVM settings

We recommend to use the following JVM settings (for Tomcat set in catalina.bat or catalina.sh) when using Sun JVMs:

set CATALINA_OPTS=%CATALINA_OPTS% -server -Xms128m -Xmx1536m -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:NewRatio=4 -Dsun.io.useCanonCaches=false

-server

Use Server Hotspot VM. Must be the first option.

Starting with Sun JDK 5.0 this option is set automatically for Solaris or Linux, whenever the machine has at least 2 CPUs and at least 2GB of physical memory. On Microsoft Windows platforms the parameter is never set automatically. So we recommend to set it manually.

-Xms128m -Xmx1536m

Jahia is quite memory consuming, so you may typically want to increase the -Xmx maximum memory value to the highest possible one supported by your system or configure the JVM settings for your specific usage. More information about the SUN JVM configurations can is available here:

  http://java.sun.com/docs/hotspot/VMOptions.html 

As the maximum heap size is limited on 32-bit systems, we are recommending to use a 64-bit operating system.

-XX:MaxPermSize=128m

There are reported issues with Jahia, when the JVM ran out of space in the permanent generation heap. Therefore we recommend to increase the value to 128 MB.

-XX:+UseParNewGC -XX:NewRatio=4

If you are using a Sun JVM 1.4.1_x or 1.4.2_x on a machine with 2 or more CPUs, then we strongly recommend using the parallel copying collector and to use a ratio of young generation to old generation of 4 (or even 3), what means that the space for young objects is one fifth of the heap size (ratio 1:4). The default is in most set ups 8, thus the young generation is too small and the garbage collection will slow down your system. Starting with Sun JDK 5.0 the default settings regarding garbage collection are better, but especially on Windows systems with more than 2 CPUs we are also recommending you to try those two settings and see whether you get a better performance.

-Dsun.io.useCanonCaches=false

This parameter is required to support Windows file names.

You may also install and use other JVMs (IBM, SUN, etc…). Be warned though that although these JVMs may run faster, they haven’t necessarily been tested with Jahia. Small differences may then occur and lead to unexpected behaviors especially under heavy loads. If you change the default SUN JVM, we suggest carefully monitoring your site over a period of time in order to verify that everything is running smoothly. Some of the settings we described are not available in JVMs of the other vendors

7)      Do not use HSQLDB in production.

Jahia is by default prepackaged with Hypersonic SQL. This is a small Java database embedded in Jahia used for demo or development purposes. We do not suggest using this database in production. If you want to use a free and open source database, please consider using MySQL or PostgreSQL.

Warning : Large commercial databases such as Oracle are by default configured in order to optimize large and complex transactional SQL requests. This is not the default behavior of Jahia which, due to its low granularity and its object-oriented model, launches thousands of small SQL requests per page. So except if you are familiar with complex database fine-tuning, we suggest you use a midrange database product such as MySQL which is by default better adapted to Jahia.

8)      Fine-tune your connection pool and index your database

This point is one of the most important. Jahia relies heavily on a fast and efficient database. So it is critical not to forget:

a)      to allocate enough connections to the database

b)      to correctly index your database

Jahia relies on the Apache Database Connection Pool. You can consult a detailed description of all the available configuration options here: http://jakarta.apache.org/commons/dbcp/configuration.html  . By default Jahia is already installed with the maxActive value increased to 100 (instead of 8 ).

Regarding database performance, Jahia is by default delivered without indexes as they are strongly related to your choice of database. However you may find a list of Jahia indexes you can use here: http://www.jahia.net/jahia/Jahia/site/jahia_net/pid/589#6 

Database indexing is critical to get the best performance out of Jahia. This option should not be considered optional, but as a mandatory one.

If your database supports query caching, please enable it. For example MySQL 4.0.1+ integrates an internal cache. By default this cache is not turned on. Using it will increase a lot the overall Jahia performance (more information available on the MySQL cache here: http://dev.mysql.com/doc/mysql/en/query-cache.html  )

Jahia 4.2 also can use the client side prepared statement pool, while in the versions before we were using a custom implementation. Now you can set the settings in %TOMCAT_HOME%/webapps/jahia/WEB-INF/etc/config/ dbpool.properties

boolean.poolPreparedStatements = true

int.maxOpenPreparedStatements = 0 (indefinite)

9)      Install your Tomcat as a service

Once in production, you may want to run your Tomcat application server as a service. Please refer to the Jahia technical FAQ to get more information about how to run Tomcat as a service: http://www.jahia.net/jahia/Jahia/site/jahia_net/pid/589#5  or to the Tomcat documentation.

10)  Fine-tune Tomcat configuration

For your production system, please ensure that the following Tomcat parameters are set in %TOMCAT_HOME%/conf/web.xml :

11)  Periodically backup your Jahia environment

Do not forget to customize automatic backup procedures of your Jahia environment. You may want to read the following Jahia FAQ to get more information about how to backup Jahia: http://www.jahia.net/jahia/Jahia/site/jahia_net/pid/589#14  or read the Jahia Administration Guide. Basically to be sure not to forget anything, backup everything in your Tomcat directory and your entire database.

12)  Monitor your Jahia Server

Once in production and running, you may also want to monitor your Jahia server in order to react promptly in case of downtime. Please use monitoring tools such as IPSentry ( http://www.ipsentry.com/  ) or others to monitor your Jahia server.

13)  Do not forget to upgrade to the latest Jahia release

Before sending email in the Jahia mailing lists or calling a Jahia expert, please upgrade to the latest available Jahia release. Dozens of bugs are corrected from a service pack to another. It is possible that the bug or the performance issue you are currently experiencing is already corrected in the latest release. In any case, it is highly recommended, for security and stability reasons, to be up to date with your Jahia release.

14)  Do not forget to remove or modify the password of the default Jahia users

Last but not least, do not forget to remove in production the default Jahia users. Usually, Jahia is installed with the default administrator “ siteadmin ”. So please do not forget to modify the default password of this user or to delete him.

15)  Fine-tune some Jahia properties

In Jahia 4.2 there are some new parameters for fine tuning your Jahia server. You should edit:

%TOMCAT_HOME%/webapps/jahia/WEB-INF/etc/config/jahia .properties

maxParallelProcessings = 10

With this parameter you can control how many threads in Jahia should be used to do page render processing. Requests, which are served from the HTML cache are not affected. This way you can prevent for instance OutOfMemory exceptions, when the load is high, and also the performance will be more stable. The default it is set to 10 threads, but you can raise this value depending of the power of your production server. By simulating a heavy load with parallel requests from different users or for different pages, you could determine, how many processing threads are still improving your performance.

PageGenerationWaitTime = 1800000

If all processing threads are working, then this setting is used for the waiting threads, which need to wait until the first processing thread finishes its work. In order to not wait endlessly, there is a timeout in milliseconds, which is set to 1800000 on default (30 minutes).

PageCounterIncrementActivated = false

In Jahia there is an internal counter, which counts how often a page has been accessed as such that it had to be created and not served from cache. If this information is not very interesting for you, because you want to count ALL page accesses and not just the ones, which were not served from cache, you should deactivate this counter and use a tool, which generates statistics from your WebServer log file.

PreloadHomepageFieldsActivated = true

If there are many containers from your homepage, which are accessed or displayed on every page of your system (for instance: site-settings, quick-links, menu, header-links, footer-links or other general fields), then you should try to set this parameter to true. This way the field and container IDs of the homepage will be preloaded also on other pages only once, otherwise it is done for each access on a container list, which is kept on the homepage.

Finally you may still do some other advanced fine-tuning to get better performance:

16)  Preload your site in memory

You may want to preload pages in the Jahia HTML cache proxy after a reboot or during low activity period (e.g. during the night). You can use any kind of offline site browser utility such as wget or httrack ( http://www.httrack.com  ) to preload Jahia pages in memory.

Please do not forget to limit the maximum number of connections per second if you want such a preloading to be as transparent for the end-users as possible. If you have lots of pages in your site(s), you may also want to limit the maximum number of possible entries in the front-end Jahia HTML cache proxy in order to avoid any possible JVM out-of-memory exceptions (e.g. for 3’000 or 4’000 entries set the following value in the jahia.properties file to “ max_cached_HTMLCache = 4000 ”).

Also note that the sitemap engine uses a “?matrix=1111072580718” parameter in order to force page refresh for certain browsers. You should then customize your utility in order to avoid browsing the sitemap or to avoid to include parameters beginning with “?”.

17)  Precompile your set of templates

If you do not plan to change your set of templates often, you may precompile your templates with the jspc command or another similar tool. By default, the Jahia engines (i.e. the edition pop-ups) are already pre-compiled.

For Jahia 4.2 we also introduce a new servlet, which you can trigger after deployment of new templates or template changes. Simply call simply call http://  <yourservername:port>/jahia/precompileServlet and then select an option by clicking on the link.

18)  Install a partial or full load-balancing Jahia environment

To ensure higher availability or to try to get faster performance, you may want to clusterize part or all of your Jahia environment.

As mentioned before, Jahia is rather SQL-intensive. You may therefore decide to clusterize your underlying database. Free tools such as C-JDBC
( http://c-jdbc.objectweb.org/  ) will allow you to easily create a transparent cluster of free or commercial databases.

The Jahia search engine (Apache Lucene: http://jakarta.apache.org/lucene  ) and especially the indexing of PDF files (through the PDFbox library: http://www.pdfbox.org/  ) takes up a lot of CPU time. If you plan to manage and store thousands of PDF within Jahia, you may then want to run a separate Jahia server for indexing purposes only. Please refer to the Jahia Clustering Guide for more information about how to install such a server ( http://www.jahia.org/download/documentation/Jahia_Clustering_HOW-TO_v2.pdf  ).

Finally you may also want to install two (or more) load-balanced Jahia servers. Such an installation will ensure a better availability and better performances. Here again, please refer to the Jahia Clustering Guide for more information.

19)  Identify possible errors and performance bottlenecks (if any)

If all the options mentioned above fail and you still encounter some important performance bottlenecks, either your system configuration, environment settings, database or your Jahia templates may be badly set, corrupt or introduces a conflict..

We then suggest using various tools in order to better diagnose the issue:

·         Jahia Doctor ( http://cvspub.jahia.org/cgi-bin/cvsweb.cgi/jahiadoctor/  ): The Jahia team developed a convenient utility to carry out integrity tests on your Jahia database. This utility attempts to automatically fix certain problems.
Warning: This tool is a low-level development tool. Please use only against a fully backed-up copy of your database and, if automatic-fixes are enabled, please make in-depth tests on the resulting database before migrating it back into production.

·         SQL Profiler ( http://sqlprofiler.jahia.org/  ) : the Jahia team has also developed a small utility to automatically regroup and identify the most frequently used SQL queries and, if necessary, to automatically generate new database indexes. This tool may be useful if you have developed new SQL-intensive Jahia features.
 

·         Borland OptimizeIt ( www.borland.com  ): finally, if you still experience latency or other performance bottlenecks, you may want to install a Java monitoring tools such as Borland OptimizeIt in order to identify which Jahia class or which library is consuming all your resources.

Finally if you still cannot determine the source of your problem, please contact a Jahia expert. You could first submit your problem to the free Jahia mailing lists. Please do not forget to give as much information as possible (OS used, Jahia build used, database used; stack trace of the exception (if any), etc…) so that the community can help you identify and solve the issue. If this is not possible by email, you can buy some commercial support tickets ( www.jahia.com/support  ). Jahia experts will then spend some time re-installing your full Jahia environment and carry out extensive performance tests and diagnostics.