Skip to content

Troubleshooting

Networking#

TLS version support#

Tower Enterprises relies on Java 11 (Amazon Corretto), which does not default to TLS v1.2, which may lead to some issues with 3rd party integrations which enforce TLS v1.2 (such as Azure Active Directory OIDC). Such problems can be addressed using the environment variables for the JDK, by explicitly setting TLS v1.2 as the default version.

1
_JAVA_OPTIONS="-Dmail.smtp.ssl.protocols=TLSv1.2

"SocketTimeoutException: connect timed out" error#

This problem is often observed while trying to launch workflows from a self-hosted Git server, for e.g. Bitbucket, Gitlab etc.

This error signals that the backend/cron container cannot connect to the Git remote host. This can be caused by a missing proxy configuration.

Click to expand error log!
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
ERROR i.s.t.c.GlobalErrorController - Oops... Unable to process request - Error ID: 6h3HBUkaPe03vgzoDPc5HO
java.net.SocketTimeoutException: connect timed out
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:289)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
        at nextflow.scm.RepositoryProvider.checkResponse(RepositoryProvider.groovy:167)
        at nextflow.scm.RepositoryProvider.invoke(RepositoryProvider.groovy:136)
        at nextflow.scm.RepositoryProvider.memoizedMethodPriv$invokeAndParseResponseString(RepositoryProvider.groovy:218)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1259)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
        at nextflow.scm.RepositoryProvider$_closure2.doCall(RepositoryProvider.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at groovy.lang.Closure.call(Closure.java:412)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at groovy.lang.Closure.call(Closure.java:428)
        at nextflow.scm.RepositoryProvider.invokeAndParseResponse(RepositoryProvider.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:193)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:61)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
        at nextflow.scm.BitbucketRepositoryProvider.getCloneUrl(BitbucketRepositoryProvider.groovy:114)
        at nextflow.scm.AssetManager.memoizedMethodPriv$getGitRepositoryUrl(AssetManager.groovy:394)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1259)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
        at nextflow.scm.AssetManager$_closure1.doCall(AssetManager.groovy)
        at nextflow.scm.AssetManager$_closure1.doCall(AssetManager.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at groovy.lang.Closure.call(Closure.java:412)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at groovy.lang.Closure.call(Closure.java:406)
        at nextflow.scm.AssetManager.getGitRepositoryUrl(AssetManager.groovy)

SOLUTION

Update the HTTP proxy configuration in the backend and cron environment. eg

1
2
export http_proxy="http://PROXY_SERVER:PORT"
export https_proxy="https://PROXY_SERVER:PORT"

"SSLHandshakeException: PKIX path building failed" error#

PKIX error example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        ...

This error occurs when a target server's SSL certificate cannot be verified due to an incomplete Certificate Chain of Trust on the calling system. You may encounter this error due to:

  1. A private Certificate Authority's (CA) public certificate not being available in your Nextflow / Nextflow Tower instance.
  2. Missing intermediate certificates in your Nextflow / Nextflow Tower instance.

SOLUTION 1: Adding a private CA certificate to a Nextflow container#

To add a copy of your private CA's public certificate to a container running the Nextflow process, do the following:

  1. Acquire a copy of your private CA's public certificate (i.e. ca.crt).
  2. Copy ca.crt to a storage medium that is programmatically available to your container.
  3. Define a pre-run script that does the following:
    1. Downloads the file to the container from the storage medium.
    2. Imports the certificate into your Java truststore.
      1
      2
      3
      # Example
      aws s3 cp s3://your-bucket/path/to/ca.crt /path/to/local/file
      keytool -import -trustcacerts -keystore path/to/cacerts -storepass changeit  -alias aliasName -file path/to/certificate.cer
      

SOLUTION 2: Adding intermediate certificates to your instance#

To allow Java to automatically download missing intermediate certificates, activate the enableAIAcaIssuers system property via an environment variable:

1
export JAVA_OPTS="-Dcom.sun.security.enableAIAcaIssuers=true"

  1. For Tower-based connectivity problems, set the environment variable within your Tower implementation.

  2. For Nextflow container connectivity problems, set the environment variable using a pre-run script.

Please note that this fix is dependent on the JVM version. See here for further details.

Email server#

How to configure SMTP gateway, which does not require authentication?#

SOLUTION

Since the SMTP gateway allows sending email without the need to specify a user name and passwords, the user and password should be set to null.

Please replace the mail section in your tower.yml with the following.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
mail:
  from: "${TOWER_CONTACT_EMAIL}"
  smtp:
    host: ${TOWER_SMTP_HOST}
    port: ${TOWER_SMTP_PORT}
    user: null
    password: null
    auth: false
    starttls:
      enable: false
      required: false

Unable to receive emails for TOWER_CONTACT_EMAIL#

This error occurs due to the organizational security policy on for emails.

In case you've setup the SMTP server correctly and the emails are sent correctly via Tower, but they are rejected by your organizational email ID

SOLUTION

You need to setup/configure the spf, dkim and dmarc for your domain.

For further assistance, please contact your IT staff.

Database#

Sign in fails with java.sql.SQLException in the backend log#

While trying to log-in, after the authentication, the Oops... Unable to process request error message is observed.

Click to expand error log!
1
2
3
4
5
6
io.micronaut.transaction.exceptions.CannotCreateTransactionException: Could not open Hibernate Session for transaction
…
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
…
java.sql.SQLException: The server time zone value ‘CEST’ is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the ‘serverTimezone’ configuration property) to use a more specific time zone value if you want to utilize time zone support.
…

SOLUTION

Generally, this means that the webapp is not able to connect to the database and JDBC client needs to specify the time zone value via serverTimezone.

To resolve this issue for Europe/Amsterdam time zone, please update the value of TOWER_DB_URL as shown below

1
export TOWER_DB_URL": "jdbc:mysql://<YOUR_DATABASE_IP>:3306/tower?serverTimezone=Europe/Amsterdam"

Authentication#

Sign in fails with a 500 error code frontend logs while using OpenID connect provider.#

Click to expand error log!
1
*8317 upstream sent too big header while reading response header from upstream, client: 10.170.157.186, server: localhost, request: "GET /oauth/callback

SOLUTION

This happens when using a OpenID connect, the callback request could send too big HTTP headers causing the Tower web server to report that error message.

The solution consists of rebuilding the frontend container adding the following proxy directives in the /etc/nginx/nginx.conf file.

1
2
3
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
Back to top