Hi all,
We've installed GTM V63010 on a brand new server with Ubuntu 2022.
We encounter some recurrent errors than we can't reproduce on developper machine (see logs extracts below).
As you can read, we've got many GTM-F-NOPRINCIO even if when no user operation are done on this database.
Does anyone has any idea ?
Mar 22 18:17:00 GTM-MUMPS[27853]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FC2CA4675E4.
Mar 22 18:17:58 GTM-MUMPS[27860]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F0C0E6675E4.
Mar 22 18:17:59 GTM-MUMPS[27861]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F72F44675E4.
Mar 22 18:18:00 GTM-MUMPS[27862]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F38A78675E4.
Mar 22 18:18:01 GTM-MUMPS[27863]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FA7C0C675E4.
Mar 22 18:18:58 GTM-MUMPS[27868]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F18640675E4.
Mar 22 18:18:59 GTM-MUMPS[27869]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FEC568675E4.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If I look at the times - it looks like a background job that is called by cron or something like that. Maybe the stdout is not set right for this job?
If this is not the case, please post the source code of the suspect routine...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We're using GT.M as a database server, responding to frontend requests through xinetd.
No background job is scheduled for that. And those requests seem to be local.
Our problem is that we can't percisely define which routine is causing this trouble.
Is there a way to have more details on this kind of error ?
Can it be solved by upgrading the database version (moving to V63011) ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
NOPRINCIO errors indicate the loss of the PRINCIPAL device, e.g., start a GT.M program that produces output then close the terminal session.
NOPRINCIO errors are fatal, so when GT.M detects one due to a second unsuccessful attempt to READ or WRITE to the device, it terminates the process immediately, possibly leaving an application state that is out of design.
From the Messages & Recovery Procedures Manual:
NOPRINCIO, Unable to dddd principal device: DDDD at LLLL due to: SSSS
Run Time Fatal: This indicates that GT.M attempted to, but could not, READ from, or WRITE to (direction indicated by dddd), the PRINCIPAL device, and therefore attempted to issue an appropriate error, for example, an IOEOF, TERMHANGUP, or TERMWRITE at location LLLL, with a status of SSSS. However, if the error handling does not prevent any and all subsequent READs and WRITEs to the no longer available PRINCIPAL device, the next subsequent I/O error shuts down the process immediately to prevent mysteriously lost output, or, worse, an indefinite loop. The NOPRINCIO message appears in the operator log
Action: The NOPRINCIO error message is FATAL which does not drive device or trap handlers and terminates the process. This termination does not allow any application level orderly shutdown and, depending on the application, may lead to out-of-design application state. Therefore FIS recommends appropriate application level error handling that recognizes the preceding error and performs an orderly shutdown without issuing any additional READ or WRITE to the principal device. The most common causes for the principal device to cease to exist involve terminal sessions or socket connections (including those from processes started by inetd/xinetd). When the remote client terminates the connection, the underlying PRINCIPAL device becomes inaccessible making any subsequent attempt to READ from, or WRITE to, it hopeless. In the case of terminals, a user closing the window of a session without cleanly exiting from the GT.M process sets up the case that can drive this error. GT.M does not issue NOPRINCIO errors from Direct Mode, because it is a developer tool, or at the completion of a HEREDOC in a shell script. However, this means a HEREDOC must use ZHALT to return a specific status to the shell, and that a $ETRAP that bounces a process into Direct Mode terminates without evidence.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Also:
V6.3-011 introduced gtm_hupenable and VIEW(“HUPENABLE”) to produce an error on principal device disconnect. This may be related to NOPRINCIO issues, but also to whether your environment expects/relies on processes continuing to run after a principal device disconnect.
From the release note:
USE for a terminal device recognizes NOHUPENABLE and HUPENABLE deviceparameters
to determine whether to recognize a terminal disconnect on $PRINCIPAL as an error. The
gtm_hupenable environment variable, if defined to a one (1) T(rue) or Y(es), determines the initial setting for the process as HUPENABLE; if undefined or defined to zero (0) F(alse) or N(o), it specifies the initial setting as NOHUPENABLE. By default, GT.M ignores terminal disconnects which can be handy for breaking and resuming connections with long-running processes. Previously, there was no way to change the default behavior. (GTM-9036)
...
TERMHANGUP, Terminal has disconnected
Run Time Error: This indicates that the terminal serving as the PRINCIPAL device has disconnected. By default, GT.M ignores terminal "hang-ups," which can allow the terminal to reconnect at a later time to a process that does not need the terminal to continue work. You can enable recognition of Principal device disconnects with USE $PRINCIPAL:HUPENABLE or by starting the process with the gtm_hupenable set to 1, TRUE or YES, or disable them with USE $PRINCIPAL:NOHUPENABLE.
Action: When a process receives this error it must avoid any further READs from, or WRITEs to,
$PRINCIPAL, typically by shutting down in a wholesome fashion. Failure to do so causes GT.M to terminate the process with a NOPRINCIO message to the operator log.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
An additional note about considering the move to GT.M V6.3-011:
Version V6.3-011 and after give more context for NOPRINCIO errors .if you are on V6 there’s no change to the database content format. In fact, if you are on V5, the database upgrade to V6 is automatic. However, once you use the expanded V6 capabilities of bigger keys and records, there’s no going back.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi all,
We've installed GTM V63010 on a brand new server with Ubuntu 2022.
We encounter some recurrent errors than we can't reproduce on developper machine (see logs extracts below).
As you can read, we've got many GTM-F-NOPRINCIO even if when no user operation are done on this database.
Does anyone has any idea ?
Mar 22 18:17:00 GTM-MUMPS[27853]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FC2CA4675E4.
Mar 22 18:17:58 GTM-MUMPS[27860]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F0C0E6675E4.
Mar 22 18:17:59 GTM-MUMPS[27861]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F72F44675E4.
Mar 22 18:18:00 GTM-MUMPS[27862]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F38A78675E4.
Mar 22 18:18:01 GTM-MUMPS[27863]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FA7C0C675E4.
Mar 22 18:18:58 GTM-MUMPS[27868]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007F18640675E4.
Mar 22 18:18:59 GTM-MUMPS[27869]: %GTM-F-NOPRINCIO, Unable to write to principal device -- generated from 0x00007FEC568675E4.
If I look at the times - it looks like a background job that is called by cron or something like that. Maybe the stdout is not set right for this job?
If this is not the case, please post the source code of the suspect routine...
We're using GT.M as a database server, responding to frontend requests through xinetd.
No background job is scheduled for that. And those requests seem to be local.
Our problem is that we can't percisely define which routine is causing this trouble.
Is there a way to have more details on this kind of error ?
Can it be solved by upgrading the database version (moving to V63011) ?
NOPRINCIO errors indicate the loss of the PRINCIPAL device, e.g., start a GT.M program that produces output then close the terminal session.
NOPRINCIO errors are fatal, so when GT.M detects one due to a second unsuccessful attempt to READ or WRITE to the device, it terminates the process immediately, possibly leaving an application state that is out of design.
From the Messages & Recovery Procedures Manual:
NOPRINCIO, Unable to dddd principal device: DDDD at LLLL due to: SSSS
Run Time Fatal: This indicates that GT.M attempted to, but could not, READ from, or WRITE to (direction indicated by dddd), the PRINCIPAL device, and therefore attempted to issue an appropriate error, for example, an IOEOF, TERMHANGUP, or TERMWRITE at location LLLL, with a status of SSSS. However, if the error handling does not prevent any and all subsequent READs and WRITEs to the no longer available PRINCIPAL device, the next subsequent I/O error shuts down the process immediately to prevent mysteriously lost output, or, worse, an indefinite loop. The NOPRINCIO message appears in the operator log
Action: The NOPRINCIO error message is FATAL which does not drive device or trap handlers and terminates the process. This termination does not allow any application level orderly shutdown and, depending on the application, may lead to out-of-design application state. Therefore FIS recommends appropriate application level error handling that recognizes the preceding error and performs an orderly shutdown without issuing any additional READ or WRITE to the principal device. The most common causes for the principal device to cease to exist involve terminal sessions or socket connections (including those from processes started by inetd/xinetd). When the remote client terminates the connection, the underlying PRINCIPAL device becomes inaccessible making any subsequent attempt to READ from, or WRITE to, it hopeless. In the case of terminals, a user closing the window of a session without cleanly exiting from the GT.M process sets up the case that can drive this error. GT.M does not issue NOPRINCIO errors from Direct Mode, because it is a developer tool, or at the completion of a HEREDOC in a shell script. However, this means a HEREDOC must use ZHALT to return a specific status to the shell, and that a $ETRAP that bounces a process into Direct Mode terminates without evidence.
From a prior post:
Can it be solved by upgrading the database version (moving to V63011) ?
From the GT.M V6.3-011 Release Note:
V6.3-011 provides ... additional context for NOPRINCIO syslog messages.
Also:
V6.3-011 introduced gtm_hupenable and VIEW(“HUPENABLE”) to produce an error on principal device disconnect. This may be related to NOPRINCIO issues, but also to whether your environment expects/relies on processes continuing to run after a principal device disconnect.
From the release note:
USE for a terminal device recognizes NOHUPENABLE and HUPENABLE deviceparameters
to determine whether to recognize a terminal disconnect on $PRINCIPAL as an error. The
gtm_hupenable environment variable, if defined to a one (1) T(rue) or Y(es), determines the initial setting for the process as HUPENABLE; if undefined or defined to zero (0) F(alse) or N(o), it specifies the initial setting as NOHUPENABLE. By default, GT.M ignores terminal disconnects which can be handy for breaking and resuming connections with long-running processes. Previously, there was no way to change the default behavior. (GTM-9036)
...
TERMHANGUP, Terminal has disconnected
Run Time Error: This indicates that the terminal serving as the PRINCIPAL device has disconnected. By default, GT.M ignores terminal "hang-ups," which can allow the terminal to reconnect at a later time to a process that does not need the terminal to continue work. You can enable recognition of Principal device disconnects with USE $PRINCIPAL:HUPENABLE or by starting the process with the gtm_hupenable set to 1, TRUE or YES, or disable them with USE $PRINCIPAL:NOHUPENABLE.
Action: When a process receives this error it must avoid any further READs from, or WRITEs to,
$PRINCIPAL, typically by shutting down in a wholesome fashion. Failure to do so causes GT.M to terminate the process with a NOPRINCIO message to the operator log.
An additional note about considering the move to GT.M V6.3-011:
Version V6.3-011 and after give more context for NOPRINCIO errors .if you are on V6 there’s no change to the database content format. In fact, if you are on V5, the database upgrade to V6 is automatic. However, once you use the expanded V6 capabilities of bigger keys and records, there’s no going back.