Atlassian Suite - SSL with Apache

Why Another Tutorial on the Same Subject?

It is true that the subject is already covered in the official documentation, but in my opinion not in the best way. My reasons for creating this post (tutorial) are:

• In the official documentation every Atlassian product has its own documentation about SSL and Apache reverse proxy, and these tutorials are written either by different people, or in different time, so they are different in some parts, although covering the same thing. This can be confusing (and it was in my case). As you'll see here, the procedure is almost exactly the same no matter which of the products you're dealing with.
• Official documentation does not explain Java keystore issues.
• I'll do the things in slightly different way. Actually I'll use the way described in JIRA documentation for all other products.

JIRA

Here I'll mostly repeat everything already explained at Integrating JIRA with Apache using SSL page from the official documentation. The reason for repeating these steps is that I'll use them later for all other products.

I assume that you've already installed JIRA, and that it is available at the moment at http://jiraserver:8080.

Before starting you need to decide what you actually want to do, of course. (As a wise man said: "Knowing where you want to go will significantly improve your chances to actually get there.") Here you need to decide about the domain. Here are the options:

• JIRA will be published under domain's root (i.e. https://mydomain.com, https://jira.mydomain.com, https://anything.mydomain.com, etc.)
• JIRA will be published with context path (i.e. https://mydomain.com/jira, https://anything.mydomain.com/acontextpath, etc.)

In my case, I'll go with context path because this way I can use the same SSL certificate for all products (i.e. https://itenlight.com/jira, https://itenlight.com/confluence, etc.)

Preparing the Certificate

I won't go into details about how you can create or acquire a certificate, but I can recommend www.sslshopper.com as a good resource that can solve many SSL certificate related issues. Here I'll just go through few things needed for completing this tutorial.

You have few options for getting the certificate:

• Create so-called self signed certificate (by using OpenSSL or similar tool);
• If you have certificate authority (CA) server in your environment, you can create certificate signed by this CA;
• Buy the certificate from a trusted provider, thus ensuring famous green https prefix for your website. Since these days you can buy such certificate very cheap (even for $6 per year), I suggest this option.

If you decide to go with self-signed certificate, you'll have one private key file, and one certificate file. But if you select any of other two options, you'll have one private key file, one certificate file, and one or more chain certificates. In my case I've bought the certificate from Comodo, so i have the following:

• Root CA Certificate - AddTrustExternalCARoot.crt
• Intermediate CA Certificate - COMODORSAAddTrustCA.crt
• Intermediate CA Certificate - COMODORSADomainValidationSecureServerCA.crt
• Certificate file - itenlight_com.crt
• Private key file - itenlight_com.key

In case of self-signed certificate you'll have only the last two, and you don't have to do any preparation, so you can skip the rest of this section.

If you have a certificate signed by some authority (either private CA server or trusted provider), you'll have several certificates in the chain (actual number depends on the authority), and you'll have to concatenate them to the same file. Before doing this you should know about the hierarchy of the certificates from the chain. In my case the first from the list above is the root CA, which has issued a certificate for the second from the list (the first intermediate), which further has issued a certificate for the third from the list (the second intermediate), which finally issued the certificate for me. It is root-to-cert (top-to-bottom) order. While concatenating them to single file, you'll go in the opposite direction:

$ # Create new file:
$ touch itenlight_com_with_chain.crt
$ 
$ # Concatenate content of all certificate files from the chain, cert to root (bottom to top):
$ cat itenlight_com.crt >> itenlight_com_with_chain.crt
$ cat COMODORSADomainValidationSecureServerCA.crt >> itenlight_com_with_chain.crt
$ cat COMODORSAAddTrustCA.crt >> itenlight_com_with_chain.crt
$ cat AddTrustExternalCARoot.crt >> itenlight_com_with_chain.crt

Note: *.crt files are nothing more than text files, so you can accomplish the same by using any text editor (i.e. Notepad++), except Windows' native editor Notepad and some other Microsoft's editors (i.e. Visual Studio Code) which are suffering from Windows' carriage-return-new-line problem. To avoid any confusion about this issue I prefer using command line for this.

Now we can place private key file and certificate with chain to appropriate locations. On Ubuntu these are:

$ mv itenlight_com.key /etc/ssl/private/
$ mv itenlight_com_with_chain.crt /etc/ssl/certs/

The next thing to do is to set minimal necessary permissions. On Ubuntu this can be done by:

$ chown root:ssl-cert /etc/ssl/private/itenlight_com.key
$ chmod 0440 /etc/ssl/private/itenlight_com.key
$ chown root:root /etc/ssl/certs/itenlight_com_with_chain.crt
$ chmod 0644 /etc/ssl/certs/itenlight_com_with_chain.crt

The certificate is now ready for Apache.

Configuring Tomcat

1. Stop JIRA.
2. Locate server.xml file in conf subdirectory of JIRA installation directory. In my case installation directory is /opt/atlassian/jira, so the file is /opt/atlassian/jira/conf/server.xml.

3. If you decided to go with context path (i.e. https://mydomain.com/jira), you need to change path attribute of Context element so that it contains your context path. For example, if you want to publish JIRA on https://mydomain.com/jira, path attribute should have value "/jira", as in the following snippet from server.xml file:

   ...
   <Engine name="Catalina" defaultHost="localhost">
       <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
           <Context path="/jira"
                    docBase="${catalina.home}/atlassian-jira"
                    reloadable="false"
                    useHttpOnly="true">
   ...

   If you decided to publish JIRA under domain's root, without context path (i.e. https://mydomain.com or https://anything.mydomain.com), the same path attribute should contain only slash character ("/").

4. Within the same server.xml file locate Connector element within Service element. It'll look like (but not so nicely indented):

   ...
   <Service name="Catalina">
    
           <Connector port="8080"
                      maxThreads="150"
                      minSpareThreads="25"
                      connectionTimeout="20000"
                      enableLookups="false"
                      maxHttpHeaderSize="8192"
                      protocol="HTTP/1.1"
                      useBodyEncodingForURI="true"
                      redirectPort="8443"
                      acceptCount="100"
                      disableUploadTimeout="true"/>
   ...

5. Here you'll do the following:

   1. Remove redirectPort="8443" attribute. It is not necessary, but I like to do so to prevent Tomcat to even think about any redirection.
   2. Create another copy of the Connector element with all its attributes immediately below the existing one (to have two identical Connector elements).

   3. In the second Connector element change port value to 8081, and add three more attributes, so that finally you get:

      ...
      <service name="Catalina">
       
              <Connector port="8080"
                         maxThreads="150"
                         minSpareThreads="25"
                         connectionTimeout="20000"
                         enableLookups="false"
                         maxHttpHeaderSize="8192"
                         protocol="HTTP/1.1"
                         useBodyEncodingForURI="true"
                         acceptCount="100"
                         disableUploadTimeout="true"/>
       
              <Connector port="8081"
                         maxThreads="150"
                         minSpareThreads="25"
                         connectionTimeout="20000"
                         enableLookups="false"
                         maxHttpHeaderSize="8192"
                         protocol="HTTP/1.1"
                         useBodyEncodingForURI="true"
                         acceptCount="100"
                         disableUploadTimeout="true"
        
                         scheme="https"
                         proxyName="itenlight.com"
                         proxyPort="443"
                         secure="true"/>
      ...

      I need to bring your attention to proxyName attribute. It should contain domain name without context path (i.e. mydomain.com, anything.mydomain.com, etc.), even if context path is used. For example, although I want to publish JIRA on https://itenlight.com/jira, I've set proxyName="itenlight.com".

   4. Start JIRA.

Configuring Apache

The first thing we'll do with Apache is to ensure that necessary modules are installed. On Ubuntu we can do this by executing:

$ a2enmod proxy_http ssl

When the modules are installed restart Apache. On Ubuntu:

$ service apache2 restart

The next thing to do is to create / modify virtual host that will be used. Depending on your decision about using context path or not, there are two configurations. Virtual host configuration without context path:

<VirtualHost jira.mydomain.com:443>
 
    ServerName jira.mydomain.com
 
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
 
    SSLEngine               On
         
    SSLCertificateFile      /etc/ssl/certs/jira_mydomain_com_with_chain.crt
    SSLCertificateKeyFile   /etc/ssl/private/jira_mydomain_com.key
         
    ProxyPass               /       http://jiraserver:8081/
    ProxyPassReverse        /       http://jiraserver:8081/
 
</VirtualHost>

Few notes:

• Certificate file names are changed here comparing to my example from above (jira_mydomain_com_with_chain.crt is used instead of itenlight_com_with_chain.crt, and jira_mydomain_com.key is used instead of itenlight_com.key).
• We are proxying to port 8081 (the second Connector element we've created above).

As an example of virtual host settings in case when context path is used (my case), I'll provide my own configuration, with my domain name. And since I'll publish other Atlassian products in the same way, I'll include them also:

<VirtualHost itenlight.com:443>
 
    ServerName itenlight.com
 
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
 
    SSLEngine               On
         
    SSLCertificateFile      /etc/ssl/certs/itenlight_com_with_chain.crt
    SSLCertificateKeyFile   /etc/ssl/private/itenlight_com.key
         
    ProxyRequests           Off
    ProxyPreserveHost       On
         
    ProxyPass               /crowd      http://crowdserver:8096/crowd
    ProxyPassReverse        /crowd      http://crowdserver:8096/crowd
 
    ProxyPass               /openidserver   http://crowdserver:8096/openidserver
    ProxyPassReverse        /openidserver   http://crowdserver:8096/openidserver
 
    ProxyPass               /jira       http://jiraserver:8081/jira
    ProxyPassReverse        /jira       http://jiraserver:8081/jira
 
    ProxyPass               /confluence http://confluenceserver:8091/confluence
    ProxyPassReverse        /confluence http://confluenceserver:8091/confluence
 
    ProxyPass               /bitbucket  http://bitbucketserver:7991/bitbucket connectiontimeout=5 timeout=300
    ProxyPassReverse        /bitbucket  http://bitbucketserver:7991/bitbucket
         
    ProxyPass               /fisheye    http://fisheyeserver:8060/fisheye
    ProxyPassReverse        /fisheye    http://fisheyeserver:8060/fisheye
 
    Redirect permanent      /   https://www.itenlight.com
 
</VirtualHost>

In the last line I've set permanent redirection to www.itenlight.com if path does not correspond to any of Atlassian products.

After changing virtual hosts you'll need to restart Apache again.

Configuring Base URL

At this moment you should be able to access JIRA by using the new https URL, so please do. As soon as you login, JIRA will probably warn you that something is wrong with URL. It's because JIRA doesn't expect you to reach it with this address. No matter if you've got the warning or not, you should configure JIRA with the new URL. To do this, you need to go to JIRA Administration, System tab, and change "Base URL" field to contain your new base URL (i.e. https://anything.mydomain.com, https://mydomain.com/anything, etc.). In this field you need to enter the full new URL that includes schema part (https://), domain part (with subdomain if used), and context path (if used). Save the changes.

This is it - you've finished configuring JIRA SSL.

Confluence, Bitbucket and Crowd

As I've already mentioned, configuration of almost all other products will be the same. To be precise, configuration for all other products which are using Tomcat container is the same, and the only product in my list that doesn't use it is FishEye. We'll deal with FishEye later. I'll repeat the procedure for Tomcat based products in short:

• Prepare the certificate in the same exact way as you've done for JIRA. If you'll use context paths under the same domain - you'll use the same certificate, so there's nothing to do in this step.
• Stop the product.
• Locate server.xml file. The actual location depends on the installation directory you've selected while installing the product. In table below I've show the locations in my case.
• Set context path appropriately.
• Remove redirectPort attribute from currently used Connector element.
• Create another copy of Connector element, change port attribute (in my case I've simply incremented existing port value for one), and add four new attributes (the same ones as in JIRA).
• Start the product.
• Configure Apache in the same exact way we've done with JIRA, but with appropriate server names, domain names and context paths.
• Restart Apache.
• Configure base URL.

FishEye

As already mentioned, configuring FishEye SSL is different since it doesn't use Tomcat (it uses Jetty), but other parts of the procedure are still the same:

• You'll prepare SSL certificate in the same way;
• You'll configure Apache virtual host in the same way.

Configuration of FishEye itself can be done either through FishEye web interface, or by editing config.xml file. I'll do this through interface. Go to FishEye Administration and select "Server" option (under "Global Settings"). There you'll want to set the following:

That's all that needs to be done at FishEye's side. Once you finish with configuring Apache, you'll have FishEye up and running with SSL.

Java Keystore

Java and its dealing with SSL certificates is one of the most famous causes for developer's and administrator's headaches. But once you understand what's actually happening there - everything becomes easier. I won't go into details on the subject here, but I will provide an overview so that you can understand what is all about. In the subject we're dealing with here, the problem arises when one Atlassian product tries to communicate with another. When you try to establish Application Links, for example. It's always problem if you are using self-signed certificates, but it also may happen even if you are using a certificate bought from trusted provider. Here I'll provide simplified description of the problem:

• For example, Confluence tries to communicate with JIRA (you are trying to establish application link between them).
• JIRA gets URL address of Confluence instance (i.e. https://itenlight.com/confluence), but when it tries to communicate Confluence Java figures out that SSL certificate used is unknown, and refuses to establish the connection. As consequence you'll get some "connection error" or "host unreachable" message. Even worse, often such message does not precisely describes the problem, but introduces more confusion.
• It will happen even if JIRA itself uses the same exact certificate as Confluence does (i.e. https://itenlight.com/jira and https://itenlight.com/confluence - both using the same SSL certificate).

The only way to resolve the issue is to tell Java that everything is OK with the certificate. In order to do so we need to understand how Java checks particular certificate's validity. What Java actually does is that it searches for this certificate, or for the issuer's certificate, in a list of trusted certificates. This list of trusted certificates resides in Java keystore. (Note: If you are using self-signed certificate Java will search for the certificate itself, but if you are using a certificate issued by some authority, Java will be satisfied if the issuer's certificate is found in the list, even if the actually used certificate itself is absent.) Obviously we need to add our certificate to keystore, but we have to know where the keystore is. Things are getting a more complicated here since there can be more than one keystore. Java will search for the certificate in so-called user's keystore (~/.keystore), in Java JRE instance's keystore (i.e. /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts), and some Java applications are allowing specifying custom store. Again, I won't go into details about every option. Instead I'll add certificates to JRE's keystore.

To recap, we'll add our SSL certificates to /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts keystore. Another thing we need to know in order to add certificates to the store is keystore password. Default password (when JRE is installed) is "changeit", and chances are that it is still password in your case. Very few people know how the password can be changed, and even they avoiding changing it in fear that it may break some installed Java applications (this is my case - I've never changed this password). So once we know which keystore we'll use, and the password of the keystore, we can start adding certificates. We'll do this by using keytool command (see www.sslshopper.com for more details). If keytool command isn't in your path, you can reach it by its full path (i.e. /usr/lib/jvm/java-8-oracle/jre/bin/keytool).

If you're using self-signed certificate, you'll have to add only one certificate to the keystore - the certificate used:

$ # Navigate to the directory where keytool resides:
$ cd /usr/lib/jvm/java-8-oracle/jre/bin
$ # Importing the certificate:
$ keytool -import -trustcacerts -alias itenlight -file ~/itenlight_com.crt -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts -storepass changeit

Notes:

• -alias argument: (itenlight in my case) isn't important - you can choose the alias it freely;
• -file argument: It should point to the actual certificate file. Here I've assumed that actual certificate file resides in home directory (~/itenlight_com.crt);
• -keystore argument: It should point to the actual keystore file;
• -storepass argument: Already explained above. If not specified the command will prompt for password.

If you're using using certificate signed by some CA, you'll need to add all certificates from the chain (in top-to-bottom order), except for the certificate used. In my case:

$ # Navigate to the directory where keytool resides:
$ cd /usr/lib/jvm/java-8-oracle/jre/bin
$ # Importing the certificates:
$ keytool -import -trustcacerts -alias itenlight -file ~/AddTrustExternalCARoot.crt -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts -storepass changeit
$ keytool -import -trustcacerts -alias itenlight -file ~/COMODORSAAddTrustCA.crt -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts -storepass changeit
$ keytool -import -trustcacerts -alias itenlight -file ~/COMODORSADomainValidationSecureServerCA.crt -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts -storepass changeit

Have we finished? Well, no. As already mentioned we've updated keystore of one JRE instance, and it'll be enough for all Java applications that are running by using this JRE instance. But if you have more than one JRE instance installed (i.e. multiple JRE versions running side-by-side), you'll have to repeat the same process for every [JRE_DIRECTORY]/lib/security/cacerts keystore file. Since JIRA and Confluence are installed with their own JRE instance, there are at least two more keystore files you'll need to update this way:

• [JIRA_INSTALL_DIRECTORY]/jre/lib/security/cacerts
• [CONFLUENCE_INSTALL_DIRECTORY]/jre/lib/security/cacerts

Now we've finished. The final confirmation that Java keystores are updated appropriately will be successful establishing application links.

Application Links

If you've done everything correctly, you should be able to create application links between Atlassian products. For example, in Confluence you need to go to Confluence Administration, and then select "Application Links" (under "Administration" section). Here you should be able to create the links between the applications by using new https URLs. Finally, you should end up with something like this:

Image