JobScheduler / Bugs / #181 Failed to start "JOC Cockpit"

Andreas - 2020-10-05

Hi Silvas,

the installation was completed and it was completed successfully. In your joc_install.xml you use the setting <entry key="launchJetty" value="yes"/> which causes JOC to be started immediately after installation which results in a problem. The log out output tells you:

Job for joc.service failed because a timeout was exceeded. See "systemctl status joc.service" and "journalctl -xe" for details.

You could follow the above hint and execute the journalctl -xe command accordingly. The problem occurs when starting JOC Cockpit by use of systemd. I therefore suggest to proceed in two steps:

Start JOC Cockpit manually with /opt/sos-berlin..com/joc/jetty/bin/jetty.sh start. Then check ./jetty_base/logs/yyyy-mm-dd.stderrout.log for errors. If there are no errors then JOC Cockpit should be available for your browser from http://<host>:4446. If this works then the problem is with the systemd configuration.

Check the systemd service file that has been created during installation. You should find it in /usr/lib/systemd/system/joc.service. The above error message tells you that a timeout occurred, the journalctl command should provide more detail about the root cause. Possible errors include that the PID file is not found/not accessible/not available in good time for systemd which is why systemd kills the JOC Cockpit process.

Best regards
Andreas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Hi Andreas,

Thanks for the update.

Using the script (jetty.sh) I managed to upload the service, but it was not possible to access the interface../jetty.sh start

2020-10-05 20:58:28.152:INFO::main: Console stderr/stdout captured to /home/my_user/sos-berlin.com/joc/jetty_base/logs/2020_10_05.stderrout.log

After starting from jetty.sh, I tried again ./setup joc_install.xml, but the return was time_out:

.....

+-----------------------------------------------------------
|   Configuration of ./resources/joc/jobscheduler.hibernate.cfg.xml
+-----------------------------------------------------------
... jobscheduler.hibernate.cfg.xml of a previous installation found
... nothing to do
... done

+-----------------------------------------------------------
|   Install/Update Jetty daemon for JOC Cockpit
+-----------------------------------------------------------
stop daemon "joc"
Stopping Jetty: OK
The daemon "joc" is already configured for auto start.
Job for joc.service failed because a timeout was exceeded.
See "systemctl status joc.service" and "journalctl -xe" for details.
Failed to start "JOC Cockpit". Refer to log in /home/my_user/sos-berlin.com/joc/jetty_base/logs.
[ Processing finished ]
[ Writing the uninstaller data ... ]
[ Automated installation done ]

systemctl status joc.service

   Loaded: loaded (/usr/lib/systemd/system/joc.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Mon 2020-10-05 21:06:14 -03; 24s ago
  Process: 3024970 ExecStart=/opt/sos-berlin.com/joc/jetty/bin/jetty.sh start (code=exited, status=0/SUCCESS)

I really don't know what I could be doing wrong. Have you been through this?

Thank you very much in advance!

Captura de tela de 2020-10-05 21-08-42.png

Andreas - 2020-10-06

Hi Silvas,

let's be precise about instructions: I was not asking to use systemctl status joc.service, I was asking to use journalctl -xeafter installation and to report the output.

You have two alternative options to start JOC Cockpit:

run ./jetty.sh and forward the log file ./jetty_base/logs/yyyy-mm-dd.stderrout log after you tried to login. If you cannot login (as verified by your screenshot) then this log file should report an error message. Your latest post reports the output of the command, not the output written to the log file.

run systemctl start joc.service to force use of systemd and then report the output of the command journalctl -xe

Before using one of the above options check that JOC Cockpit is stopped, e.g. with ps -ef | grep joc

BTW: why install release 1.13.3? You might consider to use a current release 1.13.6 . One more point: you seem to install on top of a previous release (which one?). As your screenshot shows that an account scheduler01 is used I suggest to check if this account is configured with ./jetty_base/resource/joc/shiro.ini.active - either as a local account or by availability of an LDAP configuration.

Best regards
Andreas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andreas - 2020-10-06

status: open --> pending

assigned_to: Andreas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvas - 2020-10-06

Hi Andreas, good moornig.

Thanks for the updates.

jetty.sh start successfully executed, but we are still experiencing login problems.

Attached I am sending the log and the shiro.

I have not yet been able to capture the return of some evidence about the installation in journalctl -xe. Any updates I communicate.

Regarding the joc version, I installed jobscheduler 1.13.3 because we used a much earlier version. We need to migrate our work structure to another environment and we are installing new versions, but jobscheduler 1.13.3 does not open the interface, it asks for an updated version of the joc.

Once we are able to install joc 1.13.3, we can think about migrating to a newer version. The important thing now is to be able to complete the installation :)

Thank you very much for your willingness and help.

2020_10_06.stderrout.log

shiro.ini.active

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andreas - 2020-10-06

Hi Silvas,

as explained before: your installation is fine. The problem is about the configuration of login and authentication.

The JOC Cockpit log file reports: "There is no user with username/password combination of scheduler01: Could not login with user: scheduler01 password: ........"

Consider the shiro.ini.active file: it tells you that only the default account "root" is configured. No account "scheduler01" and no LDAP integration is configured. Should you have had a shiro.ini (or shiro.ini.active) file in place with a previous release then copy the working file under the name "shiro.ini" to the current JOC Cockpit installation. Then login from the GUI (no restart of JOC Cockpit required

Best regards
Andreas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvas - 2020-10-06

Andreas,

Thank you very much, you are amazing.

Before we finish the topic, could you clarify one more question? My version of the job is quite old, 1.10 ..., accessible by the host: port / jobscheduler / operations_gui /, we don't use shiro.ini

I do not have a shiro.ini configured on any virtual machine, but I believe it is possible to configure it.

I can change the value root = $shiro1 $SHA-512 ........... by schduler01 = $shiro1 $SHA-512 ..... and save shiro.ini.active or will I have problems with that?

Again, thank you very much for clarifying the questions posed in the topic, it helped me a lot.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andreas - 2020-10-06

Hi Silvas,

thank you, the problem is narrowed down to the fact that you might have missed the information that the default account for JOC Cockpit is "root" with the password "root".

The account "scheduler01" most probably has been used in your 1.10 release with the "Classic" JOC running directly in the Master and being configured to use Apache as a reverse proxy for PAM authentication.

That being said you can simply rename "root" to "scheduler01" in shiro.ini.active. Please consider to copy the modified file shiro.ini.active to shiro.ini. With your next login to JOC Cockpit it will check for existence of the file shiro.ini and will apply the changes. As a result the file shiro.ini is renamed to shiro.ini.backup and the file shiro.ini.active is created to prove the settings that have been applied.

Best regards
Andreas

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvas - 2020-10-07

Hi Andreas,

Login successful, thanks.

Now I need to integrate my yade 1.13.3 with the joc interface. I have the profiles configured on the machine but it did not appear in the joc :).

I believe I can find something about it in the documentation, I'll look it up.

Thank you very much.

Captura de tela de 2020-10-07 18-19-56.png

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andreas - 2020-10-07

Hi Silvas,

I appreciate your believe in the power of documentation, but please consider there is a 5 years gap between releases 1.10 and 1.13. In between the YADE was moved to XML configuration. Find details from the YADE Documentation.

There are two approaches:

If you operate a larger number of YADE configurations then you might want to automatically convert them. To this purpose you could download the XML Editor and convert your *.ini profiles to *.xml profiles. Such *.xml files you can import in the JOC Cockpit Configurations view with the YADE tab.

For a smaller number of YADE configurations you might prefer to simply type existing configuration settings into the YADE tab of the Configuration view.

The final step is to parameterize the YADE job with just two parameters:

settings: the location of the *.xml settings file. This would be ./config/live/sos/.configuration/yade/yade.xml (or a different XML file in this directory)

profile: this is the name of the profile ("profile id") that you added with the YADE tab of the Configuration view.

Best regards
Andreas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvas - 2020-10-15

Hi Andreas,

Thanks for the help, I'm already setting up jobscheduler and jade, using joc.

I have questions about my .ini files. Is it possible to continue using the .ini files or is migration to xml necessary?

Another question would be about the job parameters placed on this link: https://kb.sos-berlin.com/display/PKB/Job+JobSchedulerFTPReceive
I tried to configure ftp_parallel in my profile in the .ini file but at the time of execution I didn't see the difference. Are these parameters valid for XML only?

Thank you.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andreas - 2020-10-28

Hi Silvas,

in fact you can use .ini files. The YADE provides an implicit compatibility mode to use .ini files. Precisely this means that new features, e.g. two factor authentication, are available for file transfer configurations with .xml files only, however, the reduced feature set that you can use from .ini files still is supported.

Therefore, it is not a bad idea to switch to .xml configuration, e.g. for use with the upcoming branch 2 (JS7) of JobScheduler, however, as long as you use releases 1.x the .ini configuration files will be supported.

Concerning parallel transfer availability admittedly the docs are not correct which translates to the fact that you hit a problem zone: what is stated with the parallel transfer article does not match the current capabilities: the feature YADE-402 did not make it into a current release - for whatever reasons. Instead, the feature for concurrent transfer is on the roadmap.

The perfect excuse for the missing YADE-402 feature is the fact that most users prefer to parallelize file transfer at job level, i.e. to run a number of YADE jobs in parallel.

You are strongly invited to give YADE-402 a vote or to comment accordingly. Chances are not too bad that this will force this feature to the roadmap.

Best regards
Andreas

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Silvas - 2020-10-30

Hi Andreas,

Thank you for your attention and additional information about what is being developed. I will follow the updates closely.

I would like a suggestion about transferring many files recursively.

I use recursive = true and operation = copy to fetch approximately 200k of files from an external host. This transfer takes a long time and I have a lot of problems with that.

After studying the yade documentation, I found some possibilities that could solve my problem. I used the FileList parameter and put all the directories and files I need to transfer into a file. However, I don't know what happened, but mistakes do happen:

*

YADE - Managed File Transfer *

-----www.sos-berlin.com----- *

*

+------------Source------------
| Protocol = sftp
| Host = xx
| IP = xxx.90
| User = my_user
| AuthMethod = my_pub_key
| AuthFile = ***
| FileList = /dir/dir/jade-configs/file_list.txt
| ErrorWhenNoFilesFound = false
| Remove = true

+------------Target------------
| Protocol = sftp
| Host = xxx.81
| IP = xxx.81
| User = my_user_internal
| AuthMethod = publickey
| AuthFile = ***
| Directory = /dir_target/
| OverwriteFiles = true
| AtomicSuffix = .tmp

11:07:42.076[ INFO]main - SOSVfs-D-0101: Try to connect to host 'x' at Port 'x'.
11:07:43.909[ INFO]main - SOSVfs_D_133: user 'my_user' logged in.
11:07:43.910[ INFO]main - SOSVfs-D-0101: Try to connect to host 'x' at Port 'x'.
11:07:44.248[ INFO]main - SOSVfs_D_133: user 'x' logged in.
11:17:45.449[ INFO]main - 67388 files found.
11:17:45.457[ERROR]main - [/source_dir/source_dir_recursive/file_to_transfer_20201027.txt]java.lang.Exception: SftpATTRS is null
java.lang.Exception: SftpATTRS is null
at com.sos.VirtualFileSystem.SFTP.SOSVfsSFtpFileJCraft.getModificationDateTime(SOSVfsSFtpFileJCraft.java:128) [com.sos-berlin.jobscheduler.virtual-file-system-1.13.3.jar:1.13.3]
at com.sos.VirtualFileSystem.DataElements.SOSFileListEntry.setSourceFileTransfer(SOSFileListEntry.java:1221) [com.sos-berlin.jobscheduler.virtual-file-system-1.13.3.jar:1.13.3]
at com.sos.DataExchange.SOSDataExchangeEngine.sendFiles(SOSDataExchangeEngine.java:1027) [com.sos-berlin.jade.jade-engine-1.13.3.jar:1.13.3]
at com.sos.DataExchange.SOSDataExchangeEngine.transfer(SOSDataExchangeEngine.java:1331) [com.sos-berlin.jade.jade-engine-1.13.3.jar:1.13.3]
at com.sos.DataExchange.SOSDataExchangeEngine.execute(SOSDataExchangeEngine.java:440) [com.sos-berlin.jade.jade-engine-1.13.3.jar:1.13.3]
at com.sos.DataExchange.SOSDataExchangeEngineMain.Execute(SOSDataExchangeEngineMain.java:76) [com.sos-berlin.jade.jade-engine-1.13.3.jar:1.13.3]
at com.sos.DataExchange.SOSDataExchangeEngineMain.main(SOSDataExchangeEngineMain.java:58) [com.sos-berlin.jade.jade-engine-1.13.3.jar:1.13.3]
11:17:45.473[ERROR]main - SOSVfs_E_229: error. unable to transfer data, reason: com.sos.JSHelper.Exceptions.JobSchedulerException: SOSVfs_E_226: .. file '/source_dir/source_dir_recursive/file_to_transfer' does not exist.
11:17:45.474[ERROR]main - SOSDataExchangeEngine.TRANSFER_ABORTED
11:17:45.475[ INFO]main - Rollback aborted files.
11:17:45.508[ INFO]main -

execution status = failure. Errors reported.
successful transfers = 0
skipped transfers = 0
failed transfers = 67388
last error = SOSVfs_E_226: .. file '/source_dir/source_dir_recursive/file_to_transfer' does not exist.

11:17:45.508[ERROR]main - Execute: Error occurred ...: SOSVfs_E_226: .. file '/source_dir/source_dir_recursive/file_to_transfer' does not exist., exit-code 99 raised

The file exists in the directory, but due to the reported error, it appears that yade cannot find the file.

I would like to understand what are the best possibilities for transferring many files, recursively.

Grateful for the attention.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Failed to start "JOC Cockpit"

JobScheduler workload automation to execute jobs and workflows

Group

Searches

Help

#181 Failed to start "JOC Cockpit"

Discussion