#212 FileOrderSource: Order doesn't get removed

[Martin Betcher]

I'm using JS7 2.5.1 in a Windows Environment and use several FileOrderSources. The Files from the orders that are created are getting moved to an archive folder, as instructed by the documentation. (https://kb.sos-berlin.com/display/JS7/JS7+-+File+Watching#JS7FileWatching-MovingFilesandRemovingFilesonWorkflowCompletion)
I do this mostly with YADE Jobs and sometimes with a shell script jobs, not with JITL jobs. However, the orders don't get removed like expected, instead enter a "COMPLETED" state as if the file still exists in the watched folder. I cannot remove the order manually, if I try I get this error (from the JOC log):

2023-03-03T09:07:26,519 ERROR JControllerProxy-34  c.s.j.c.ProblemHelper                        - BadRequestError: CannotDeleteWatchingOrder: Cannot delete a file watching order: #2023-03-02#F75304085900-FooBar:FooBar20230302-11.xml

The Problem: If there is a file created with the same name as one of a completed old order, there will be no new order for that new file. There is also no notification about a new file. The file has to be moved out of the watched folder manually for the old Order to leave the Workflow, and back in for it to be recognized as new file that needs a new order.

[Andreas]

Hi Martin,
thank you, your post asks the question: is it a bug or is it a feature? The answer is "it's a feature", it works as designed.

• The implementation is like this:
  • On completion of a workflow a file order checks if the incoming file is still available.
    • If the answer is "yes" then it is assumed that the workflow missed to move the file. In this situation the order remains in a COMPLETED state but cannot be removed as the underlying file still exists. This is the intended behavior.
    • If the answer is "no" then the order will leave the workflow.
• Your use case includes that
  • the file and the corresponding file order are initially created. Then a job in the workflow moves the file during processing of the workflow.
  • However, a new file with the same name as the original file is created before the current file order is completed. The JS7 considers this the same file as the original one as there is no way to determine if the newly created file in fact is new or maybe just resurrected after failure of connection to a file share.
• My recommendation is to avoid this situation:
  • You never know what time it takes to process a workflow (orders can be in a failed or suspended state etc.)
  • Therefore, adding files with the same name while an order is running for the previous version of the file is bad practice as the situation cannot be reliably resolved.
    • What mess is created if job1 in a workflow works on the older version of the file while job2 in the workflow finds a newer version?
    • The straightforward solution is to use unique file names that you add to the incoming directory.
    • The more complex solution is to add a "traffic light" mechanism by which a newer version of the incoming file is not created before the current file order is completed.
• A possible workaround is to
  • accept file orders in workflow 1 that implements a) a first job that will immediately move the incoming file to a different folder and assign a unique file name and b) an AddOrder Instruction to start workflow 2 that is parameterized from the moved file holding a unique file name.
    • the job can create a workflow variable that is used for assignment with the AddOrder instruction like this (assuming that for the shell job you map the built-in workflow variable $file to the environment variable $FILE):

TARGET_FILE="${FILE}-${RANDOM}"
mv "${FILE}" "${TARGET_FILE}"
echo "process_file=${TARGET_FILE}" >> $JS7_RETURN_VALUES

• The AddOrder Instruction receives an argument with an arbitrary name and the value: $process_file.
• you preferably can use shell commands based on randomization and date milliseconds to create unique file names
• JITL jobs for moving files are limited to seconds when it comes to uniqueness in rename operations.

Best regards
Andreas

[Martin Betcher]

Hi Andreas,
thanks for your response and your help. However there is one misconception: The files get moved out of the observed directory before the workflow reaches it's end and before a new file with the same name gets placed. The directory is empty for days or hours, yet the File Order doesn't get removed or can be removed via JOC. This contradicts your statement and the Wiki that the File Order Source should leave the workflow, if the file isn't present at the end of the workflow. This, to me, is a bug, or even two:
1. File Order not leaving the workflow, if the file isn't present anymore at the end of the workflow
2. Not being able to let the File Order leave the workflow even when the specified file isn't present anymore.

There are other processes at work that prevent the file from being overwritten, overwriting a file while it's being processed by the workflow is not possible in this specific situation.

I tried letting the workflow sleep for 10 seconds via Powershell, thinking it might be Windows not updating the directory quick enough, but that didn't help either.