A while ago I posted about getting wildfly working. That is now working. The issue was with wildfly itself. At this point, I'm running in to issues getting ejbca deployed and installed.
Running the "ant runinstall" command I get an error saying it can't communicate with the app server or ejbca isn't installed properly:
[java] Error: CLI could not contact EJBCA instance. Either your application server is not up and running, EJBCA has not been deployed successfully, or some firewall rule is blocking the CLI from the application server.
Upon looking at the server.log file for wildfly (see below), I see a lot of services that are failing that could prevent ejbca from deploying cleanly. I followed the documentation here: https://doc.primekey.com/ejbca743/ejbca-installation/application-servers/wildfly-18-jboss-eap-7-3 pretty closely and really just cut and paste for the most part (the exception is that the mail config for me uses port 25, and I changed all the passwords for key and trust stores). The only difference is that I'm running a more current version of wildfly (version 25 final). I'm wondering if the reason for these isssues is the version of wildfly, or maybe I did something wrong. What am I missing here?
Thanks,
Scott
2021-11-18 17:54:21,716 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "mariadb-java-client.jar" (runtime-name: "mariadb-java-client.jar")
2021-11-18 17:54:21,906 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service org.wildfly.security.trust-manager.httpsTM: org.jboss.msc.service.StartException in service org.wildfly.security.trust-manager.httpsTM: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: ELY04031: TrustManagerFactory algorithm [PKIX] does not support certificate revocation
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:122)
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:64)
at org.wildfly.security.ssl.X509RevocationTrustManager$Builder.build(X509RevocationTrustManager.java:343)
at org.wildfly.extension.elytron.SSLDefinitions$2.lambda$createX509RevocationTrustManager$1(SSLDefinitions.java:857)
at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:61)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
... 6 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:74)
... 12 more
<snip>
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "datasources"),
("data-source" => "ejbcads") ) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => ["jboss.jdbc-driver.mariadb-java-client_jar"],
"WFLYCTL0180: Services with missing/unavailable dependencies" =>
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.driver-demander.java:/EjbcaDS is missing [jboss.jdbc-driver.mariadb-java-client_jar]"
]
}
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "elytron"),
("trust-manager" => "httpsTM") ) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.trust-manager.httpsTM" => "Failed to start service
Caused by: java.lang.IllegalStateException: ELY04031: TrustManagerFactory algorithm [PKIX] does not support certificate revocation
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty"}}
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "datasources"),
("data-source" => "ejbcads") ) - failure description: {
"WFLYCTL0412: Required services that are not installed:" =>
"jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.jdbc-driver.mariadb-java-client_jar" ,
"WFLYCTL0180: Services with missing/unavailable dependencies" =>
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.driver-demander.java:/EjbcaDS is missing [jboss.jdbc-driver.mariadb-java-client_jar]",
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar]"
]
}
2021-11-18 17:54:24,159 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "mariadb-java-client.jar" (runtime-name : "mariadb-java-client.jar")
2021-11-18 17:54:24,159 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "ejbca.ear" (runtime-name : "ejbca.ear")
2021-11-18 17:54:24,163 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0184: New missing/unsatisfied dependencies:
service jboss.jdbc-driver.mariadb-java-client_jar (missing) dependents: [service jboss.driver-demander.java:/EjbcaDS, service org.wildfly.data-source.ejbcads]
WFLYCTL0186: Services which failed to start: service org.wildfly.security.trust-manager.httpsTM: Failed to start service
WFLYCTL0448: 6 additional services are down due to their dependencies being missing or failed
2021-11-18 17:54:24,222 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server</snip></init></init></init></init></init>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I see at least two errors in your log.
1. Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
This looks to have something todo with your TLS truststore configuration. As you say that you modified that, there may be something wrong with your modifications.
WFLYCTL0412: Required services that are not installed:" =>
"jboss.jdbc-driver.mariadb-java-client_jar",
This tells us that the MariaDB jdbc driver (jar) was not installed correctly, it is either not picked up and deployed by WildFly, or it is deployed under the wrong name.
Do one step at a time with the installation, checking the server.log as you go. Start with JDBC driver installation and check server.log that it is actually deployed by WildFly.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Tomas, no luck. I'm still doing one step at a time. I'm not sure where the string "jboss.jdbc-driver.mariadb-java-client_jar" is defined but it isn't anywhere in the documented step:
Yup. I know... I blew away the wildfly installation and am starting again. The attached log is from where I just stopped. Right after the reload when adding the datastore.
That looks good. You should continue with the next step. Be sure to monitor server.log when ejbca.ear is deployed, if a database connection fails (the most common error here) it will fail with nice errors in the log.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, I'll keep an eye on the logs. Worst case... Lets say I get the same issue once I run through the step... If I send you the logs (and maybe my database.conf from the ejbca directory tree, Is that all all you'll need to troubleshoot?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks like that step, generates the httpsTM error messages below. Did I miss a step or do something wrong? The missing keystor and truststore warnings started at the "Configure TLS" step in that document. I'm assuming that, since I don't have certificates yet, that those errors go way once I do the ant runinstall step.
I know this isn't the issue with the datastores, but we'll get there.
Yeah, I tried using the quick install script and it didn't work either. I've been following those installation instructions to the letter. The part where it starts failing is in the enable ocsp revocation step. That post included the server.log from that step. I'm not sure why there are missing services or if that is indeed normal. But its the first time missing services are coming up.
If you are thinking that those errors are normal, there is another error I'm getting when running ant clean. and maybe that's causing bigger issues. I don't know. I figured one issue at a time. I do want to leverage my pkcs11 smartcard so it'll be important at some point.
You have to post more of the logs. Just the small error snippet is hard to debug without the other information that is printed all the way back from when you run the command.
I don't know why SunP11 would be called when you run an ant command. Did you configure something with P11 in your Java?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Let me know how you want me to do send more logs. I can tail the server.log while running commands so you can see what's going on if that helps. The send you the entire output. I could be wrong with where things go sideways.
As for the pkcs11 part. I have no idea. I just installed headless and the opensc and pkcs11 packages.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I created a full log using typescript. Basically you see everything that is on my screen. I tailed server.log so that as I ran command any output is on my screen as well. Sorry for the hideous metacharacters and backspaces.
A while ago I posted about getting wildfly working. That is now working. The issue was with wildfly itself. At this point, I'm running in to issues getting ejbca deployed and installed.
Running the "ant runinstall" command I get an error saying it can't communicate with the app server or ejbca isn't installed properly:
[java] Error: CLI could not contact EJBCA instance. Either your application server is not up and running, EJBCA has not been deployed successfully, or some firewall rule is blocking the CLI from the application server.
Upon looking at the server.log file for wildfly (see below), I see a lot of services that are failing that could prevent ejbca from deploying cleanly. I followed the documentation here: https://doc.primekey.com/ejbca743/ejbca-installation/application-servers/wildfly-18-jboss-eap-7-3 pretty closely and really just cut and paste for the most part (the exception is that the mail config for me uses port 25, and I changed all the passwords for key and trust stores). The only difference is that I'm running a more current version of wildfly (version 25 final). I'm wondering if the reason for these isssues is the version of wildfly, or maybe I did something wrong. What am I missing here?
Thanks,
Scott
2021-11-18 17:54:21,716 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "mariadb-java-client.jar" (runtime-name: "mariadb-java-client.jar")
2021-11-18 17:54:21,906 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service org.wildfly.security.trust-manager.httpsTM: org.jboss.msc.service.StartException in service org.wildfly.security.trust-manager.httpsTM: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: ELY04031: TrustManagerFactory algorithm [PKIX] does not support certificate revocation
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:122)
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:64)
at org.wildfly.security.ssl.X509RevocationTrustManager$Builder.build(X509RevocationTrustManager.java:343)
at org.wildfly.extension.elytron.SSLDefinitions$2.lambda$createX509RevocationTrustManager$1(SSLDefinitions.java:857)
at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:61)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
... 6 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
at org.wildfly.security.ssl.X509RevocationTrustManager.<init>(X509RevocationTrustManager.java:74)
... 12 more
<snip>
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "datasources"),
("data-source" => "ejbcads")
) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => ["jboss.jdbc-driver.mariadb-java-client_jar"],
"WFLYCTL0180: Services with missing/unavailable dependencies" =>
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.driver-demander.java:/EjbcaDS is missing [jboss.jdbc-driver.mariadb-java-client_jar]"
]
}
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "elytron"),
("trust-manager" => "httpsTM")
) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.trust-manager.httpsTM" => "Failed to start service
Caused by: java.lang.IllegalStateException: ELY04031: TrustManagerFactory algorithm [PKIX] does not support certificate revocation
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty"}}
2021-11-18 17:54:24,155 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: (
("subsystem" => "datasources"),
("data-source" => "ejbcads")
) - failure description: {
"WFLYCTL0412: Required services that are not installed:" =>
"jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.jdbc-driver.mariadb-java-client_jar"
,
"WFLYCTL0180: Services with missing/unavailable dependencies" =>
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar",
"jboss.driver-demander.java:/EjbcaDS is missing [jboss.jdbc-driver.mariadb-java-client_jar]",
"org.wildfly.data-source.ejbcads is missing [jboss.jdbc-driver.mariadb-java-client_jar]"
]
}
2021-11-18 17:54:24,159 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "mariadb-java-client.jar" (runtime-name : "mariadb-java-client.jar")
2021-11-18 17:54:24,159 INFO [org.jboss.as.server] (ServerService Thread Pool -- 39) WFLYSRV0010: Deployed "ejbca.ear" (runtime-name : "ejbca.ear")
2021-11-18 17:54:24,163 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0184: New missing/unsatisfied dependencies:
service jboss.jdbc-driver.mariadb-java-client_jar (missing) dependents: [service jboss.driver-demander.java:/EjbcaDS, service org.wildfly.data-source.ejbcads]
WFLYCTL0186: Services which failed to start: service org.wildfly.security.trust-manager.httpsTM: Failed to start service
WFLYCTL0448: 6 additional services are down due to their dependencies being missing or failed
2021-11-18 17:54:24,222 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server</snip></init></init></init></init></init>
Looking at the data store issue a bit more... I looked to see if I created a datastore... I did:
No idea why it is undefined... So I removed and re-added it. Still says undefined though. Not sure what's going on there.
I see at least two errors in your log.
1. Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
This looks to have something todo with your TLS truststore configuration. As you say that you modified that, there may be something wrong with your modifications.
"jboss.jdbc-driver.mariadb-java-client_jar",
This tells us that the MariaDB jdbc driver (jar) was not installed correctly, it is either not picked up and deployed by WildFly, or it is deployed under the wrong name.
Do one step at a time with the installation, checking the server.log as you go. Start with JDBC driver installation and check server.log that it is actually deployed by WildFly.
Thanks Tomas... I'm doing a new installation. This time using wildfly 18 since it is the exact version that the documentation is following.
Following the docs to the letter, I still get:
Is that undefined expected?
Tomas, no luck. I'm still doing one step at a time. I'm not sure where the string "jboss.jdbc-driver.mariadb-java-client_jar" is defined but it isn't anywhere in the documented step:
That mariadb module is clearly being deployed:
Why would EJBCA not see that jar file?
@anatom, I don't mean to be a pest. But, do you have any insight that will help figure out the problem?
You have to check the server.log for logs when you add the data-source.
There are no errors from the command... The only thing in the logs is this line (for the datasource addition, the reload generates alot):
That doesn't tell me anything. What does reload say?
(no ejbca.ear deployed, just plain WildFly with adding the JDBC driver and the data-source).
Yup. I know... I blew away the wildfly installation and am starting again. The attached log is from where I just stopped. Right after the reload when adding the datastore.
That looks good. You should continue with the next step. Be sure to monitor server.log when ejbca.ear is deployed, if a database connection fails (the most common error here) it will fail with nice errors in the log.
OK, I'll keep an eye on the logs. Worst case... Lets say I get the same issue once I run through the step... If I send you the logs (and maybe my database.conf from the ejbca directory tree, Is that all all you'll need to troubleshoot?
@anatom, I'm a little closer to a cause for at least one of the errors. Following the docs here: https://doc.primekey.com/ejbca743/ejbca-installation/application-servers/wildfly-18-jboss-eap-7-3, there is a step:
It looks like that step, generates the httpsTM error messages below. Did I miss a step or do something wrong? The missing keystor and truststore warnings started at the "Configure TLS" step in that document. I'm assuming that, since I don't have certificates yet, that those errors go way once I do the ant runinstall step.
I know this isn't the issue with the datastores, but we'll get there.
Sounds reasonable.
Cool. I posted some information around the failure and when it happens in the process. Any ideas?
I don't see where that is posted?
This post here https://sourceforge.net/p/ejbca/discussion/123123/thread/aabe5fe10c/?limit=50#6639/a73c/37b9 from yesterday
Hey @anatom. I'm assuming that the post here: https://sourceforge.net/p/ejbca/discussion/123123/thread/aabe5fe10c/?limit=50#6639/a73c/37b9 wasn't able to help you figure out what I'm doing wrong. Maybe we can try this a different way... What documentation/steps do you follow when building ejbca? Do you just use the create script that comes with ejbca?
Right now, I have a bunch of certs that are expired so I'm kinda screwed at the moment.
Scott
I answered that below. I could not figure out what you mean in the post, it looks good? I don't understand where or what doesn't work for you.
For a from-scratch installation I follow this:
https://doc.primekey.com/ejbca/ejbca-installation
I've done that hundreds of times...
or for a test spin this:
https://hub.docker.com/r/primekey/ejbca-ce
or:
https://aws.amazon.com/marketplace/pp/prodview-u2xdo5mkuilke?sr=0-1&ref_=beagle&applicationId=AWSMPContessa
I've run the quick-install script as well, but it's ageing and will be removed in the future.
Yeah, I tried using the quick install script and it didn't work either. I've been following those installation instructions to the letter. The part where it starts failing is in the enable ocsp revocation step. That post included the server.log from that step. I'm not sure why there are missing services or if that is indeed normal. But its the first time missing services are coming up.
If you are thinking that those errors are normal, there is another error I'm getting when running ant clean. and maybe that's causing bigger issues. I don't know. I figured one issue at a time. I do want to leverage my pkcs11 smartcard so it'll be important at some point.
You have to post more of the logs. Just the small error snippet is hard to debug without the other information that is printed all the way back from when you run the command.
I don't know why SunP11 would be called when you run an ant command. Did you configure something with P11 in your Java?
Let me know how you want me to do send more logs. I can tail the server.log while running commands so you can see what's going on if that helps. The send you the entire output. I could be wrong with where things go sideways.
As for the pkcs11 part. I have no idea. I just installed headless and the opensc and pkcs11 packages.
The full console log of the error you just pasted above would be a good start.
I created a full log using typescript. Basically you see everything that is on my screen. I tailed server.log so that as I ran command any output is on my screen as well. Sorry for the hideous metacharacters and backspaces.
Skip everything from the "Optional Configuration".
This is things that you can do afterwards if needed.
The errors seems to happen after some of the optional configuration performed.
Regards,
Tomas