#429 setRecordFormat is failing with InternalErrorException

[Wim Jongman]

We are using sequentialfile and keyedfile since a long time. As of recently, the setRecordFormat() method of these classes is failing. We are getting an internalerrorexception (see attached file). However we do not get it all the time.

We call the java classes from an RPGLE program.

I have spent all week debugging this and now I am at my wits end.

I have checked the following cases:

1. Is adopted authority fooling me (we relied on this in the past)
   Ran it with qsecofr
   Ran it with user profile swap
   Ran it while changing the userprofile to userclass SECOFR spcaut USRCLS
2. Are the IFS jar files not accessible
   All files are accessible
3. Is it the PTF level of the JT400
   Just ran it with JT400 10.2
4. Is it the java runtime level
   It is failing on 1.8.0_155 but also on 1.8.0_122

When debugging I found the following error:

Jan 28, 2020 2:06:40 PM - INFO - Setting Record Format START - /qsys.lib/OMHD_PRDS.lib/QRPGLESRC.file/OMX128.mbr
Jan 28, 2020 2:06:40 PM - ERROR - SetRecordFormat
Jan 28, 2020 2:06:40 PM - ERROR - com.ibm.as400.access.InternalErrorException
Jan 28, 2020 2:06:40 PM - ERROR - Return code: 6 (unknown)

The file to be processed does exist (it is a source member) and is accessible by the user running the job.

The code I run is:

    file = new SequentialFile(system, filename);
    file.setRecordFormat();

I can't debug any further without setting up the JT400 dev environment. Also the InteralErrorException with return code 6 suggests it might be something deeper.

We have hundreds of installations (at customer sites) running this software, most work fine but others see the same thing I see.

Thanks for taking a look at this.

Cheers,

Wim

PS,
I have changed my code in some areas to use jdbc instead, but this is kind of painful since I have to access members. This involves creating qtemp aliases when using jdbc

[Zhang Ze]

hi Wim Jongman,
The error that you see means there is no output in the output file after ruan a command.
Can you try a command, replace the library and file with yours,
DSPFFD FILE(" + library_ + "/" + file_ + ") OUTPUT(*OUTFILE) OUTFILE(QTEMP/JT4FFD)
Then check what output in QTEMP/JT4FFD

[Wim Jongman]

Hi Zhang Ze

The file that I am processing is an existing source member. Attached you will find the output of the DSPFFD and DSPF in various formats.

The only thing I do it is read all records. However, before I reach anywhere, setrecordformat is throwing the exception.

Let me know if you want to see the full source of a failing process.

Cheers, Wim

[Wim Jongman]

result of previous comment

[Wim Jongman]

Any news Zhang Ze?

[Zhang Ze]

Sorry, something were wrong that I said . the issue becasue there is no message in reply of command, Do you recreate this issue with a specific file? I am trying to recreate it and collect more trace. Can you enable the toolbox trace for a sample program, the trace may very large?

[Wim Jongman]

I google only to find jdbc trace. However, I am using record level access. Will this still work? Can you point me to instructions on how to activate the trace?

[Zhang Ze]

for application, set below in jvm.options or applicaiton jvm confguration file.
com.ibm.as400.access.Trace.category=ALL
com.ibm.as400.access.Trace.file=/tmp/out.txt

for sample java program,
java -Dcom.ibm.as400.access.Trace.category=ALL -Dcom.ibm.as400.access.Trace.category.file=/tmp/out.txt <your class="">
notes: if the out.txt can be created, you also can write the output of program to a file with >></your>

the trace may very large, use less step or small program, If for an applicaiton, remember disable the trace after collect trace.

[Wim Jongman]

Got it Zhange Ze. Thanks. Please find the trace attached. I have looked at it and it seems to bomb out in a DSPFFD statement. So I have ran that same statement from the command line:

DSPFFD FILE(OMHD_PRDS/QRPGLESRC) OUTPUT(OUTFILE) OUTFILE(QTEMP/JT4FFD)
Output file JT4FFD created in library QTEMP.
Member JT4FFD added to output file JT4FFD in library QTEMP.
3 records added to member JT4FFD in file JT4FFD in QTEMP.
DSPFFD FILE(OMHD_PRDS/QRPGLESRC) OUTPUT(OUTFILE) OUTFILE(QTEMP/JT4FFD)
3 records added to member JT4FFD in file JT4FFD in QTEMP.

[Wim Jongman]

Here is a trace of the same file but this time it all worked. I call them differently, the failing one is called in batch and this one is called interactively. However, if I submit the same process differently (with a less deep call stack than the failing job) it also works.

Could it be failing because of an activation group issue?

[Wim Jongman]

Hello Zhang Ze, did you find anything in the provided logs?

I have now also confirmed that it is not an activation group issue. The same error occurs in NEW but also in CALLLER

[Zhang Ze]

I notice a sigon error when setrecordformat before run the command. Still investigating.
Thread[main,5,main] Wed Feb 5 12:36:49:864 CET 2020 GSSToken is not available: org.ietf.jgss.GSSException, major code: 11, minor code: 0
major string: General failure, unspecified at GSSAPI level
minor string: Error creating name: com.ibm.security.krb5.KrbException, status code: 0
message: None
Thread[main,5,main] Wed Feb 5 12:36:49:865 CET 2020 Tolerated exception retrieving GSSToken:
org.ietf.jgss.GSSException, major code: 11, minor code: 0
major string: General failure, unspecified at GSSAPI level
minor string: Error creating name: com.ibm.security.krb5.KrbException, status code: 0
message: None
at com.ibm.security.jgss.i18n.I18NException.throwGSSException(Unknown Source)
at com.ibm.security.jgss.mech.krb5.w.b(Unknown Source)
at com.ibm.security.jgss.mech.krb5.w.a(Unknown Source)
at com.ibm.security.jgss.mech.krb5.w.<init>(Unknown Source)
at com.ibm.security.jgss.mech.krb5.Krb5MechFactory.getNameElement(Unknown Source)
at com.ibm.security.jgss.GSSManagerImpl.a(Unknown Source)
at com.ibm.security.jgss.GSSNameImpl.a(Unknown Source)
at com.ibm.security.jgss.GSSNameImpl.canonicalize(Unknown Source)
at com.ibm.security.jgss.GSSNameImpl.<init>(Unknown Source)
at com.ibm.security.jgss.GSSManagerImpl.createName(Unknown Source)
at com.ibm.as400.access.TokenManager.getGSSToken(TokenManager.java:43)
at com.ibm.as400.access.AS400.signon(AS400.java:4351)
at com.ibm.as400.access.AS400File.chooseImpl(AS400File.java:427)
at com.ibm.as400.access.AS400File.setRecordFormat(AS400File.java:2313)
at com.ibm.as400.access.AS400File.setRecordFormat(AS400File.java:2284)
at nl.remain.util.OMGetSourceFileData.setFile(OMGetSourceFileData.java:172)
at nl.remain.util.OMGetSourceFileData.setSource(OMGetSourceFileData.java:141)
at nl.remain.util.OMGetSourceFileData.<init>(OMGetSourceFileData.java:70)
at nl.remain.td.OMiSeriesSourceRelations.loadSourceFromIseries(OMiSeriesSourceRelations.java:661)
at nl.remain.td.OMiSeriesSourceRelations.loadSource(OMiSeriesSourceRelations.java:617)
at nl.remain.td.OMiSeriesSourceRelations.getSourceRelations(OMiSeriesSourceRelations.java:430)</init></init></init>

[Wim Jongman]

Any updates on this Zhang Ze? Sorry for being on your tail but our customers are waiting on a solution. This is breaking our product and we do not have a workaround.

[Zhang Ze]

Wim Jongman, What message shows when you run DSPFFD FILE(<liberary file="">) OUTPUT(*OUTFILE) OUTFILE(QTEMP/JT4FFD) in green screen(pcom)?
example, I see below
Additional Message Information </liberary>

Message ID . . . . . . : CPF9861 Severity . . . . . . . : 00
Message type . . . . . : Information
Date sent . . . . . . : 02/12/20 Time sent . . . . . . : 20:03:08

Message . . . . : Output file JT4FFD created in library QTEMP.

[Wim Jongman]

Hi Zhang Ze, thank you for your response.

Can you tell me what you are looking for? Maybe I can be of more use.

Anyway:

I ran DSPFFD two times.
On time on the command line and the second time just before I call the program that calls Java.

I also see a third call in the joblog that probably comes from jt400. I see a call from QYJSPART in the joblog.

I have attached the joblog output of all three events.

[Zhang Ze]

I mean what message returned when you ran the command DSPFFD FILE(<liberary file="">) OUTPUT(*OUTFILE) OUTFILE(QTEMP/JT4FFD) on the command line. the message returns on your screen directly, not out put file. it should return something like below:</liberary>

Message ID . . . . . . : CPF9861 Severity . . . . . . . : 00
Message type . . . . . : Information
Date sent . . . . . . : 02/12/20 Time sent . . . . . . : 20:03:08

Message . . . . : Output file JT4FFD created in library QTEMP.

I want to know what message that you see on your system.

[Wim Jongman]

Yes, I put that information in the attached file but here it is again:

                     Additional Message Information

Message ID . . . . . . : CPF9861 Severity . . . . . . . : 00
Message type . . . . . : Information
Date sent . . . . . . : 14-02-20 Time sent . . . . . . : 16:58:40

Message . . . . : Output file JT4FFD created in library QTEMP.

[Wim Jongman]

Any updates on this?

[Wim Jongman]

Hello Zhang Ze, any updates on this?

[Wim Jongman]

Ok, I give up on this ticket. I will ditch the record level access approach and use SQL.

[Zhang Ze]

Sorry, little busy recently, Still Investigate.
Do you use kerberso (single sign on), when setrecordformat there are another error, Thread[main,5,main] Wed Feb 5 12:36:49:864 CET 2020 GSSToken is not available: org.ietf.jgss.GSSException, major code: 11, minor code: 0
minor string: Error creating name: com.ibm.security.krb5.KrbException, status code: 0
message: None

Is there any kerberos error on the system?

[Wim Jongman]

I don't use kerberos or any other form of single signon