#2065 Impossible to redirect the help console output into a file

[elgonzo]

VICE: GTK3VICE-3.8-win64-r45315
OS: Windows 11 x64 english

I found it impossible to redirect the console CLI help output of x64sc.exe (or x128.exe) into a file.
Attempting to do

x64sc.exe --help >clihelp.txt

still does output the CLI help in the console. The file clihelp.txt where the output should have been redirected into only contains a single line with the text "lib_init()".

This inability pretty much impacts anything that relies on redirecting stdout. For example, screen-wise output of the help by means of x64sc.exe --help | more is not working either anymore.

I am not certain but i suspect this got broken by the fix for https://sourceforge.net/p/vice-emu/bugs/1733/.

[Uffe Jakobsen]

Have you tried this:

x64sc.exe --help 2>clihelp.txt

AND/OR

x64sc.exe --help >clihelp.txt 2>&1

AND/OR

x64sc.exe --help 2>&1 | more

[elgonzo]

The help output is not on stderr, so trying to capture stderr won't help with the help ;-)

[gpz]

you probably need -no-redirect-streams on windows, like compyx said in ticket #1733

[elgonzo]

Without being dispectful, but this is terrible. Because a user won't ever know about this unless they got lucky spotting compyx' comment in the vast abyss that of the internet. It's also astronomically unlikely that a user would even think about trying to look up/find such an option, because that's not how stdout/stderr redirection is supposed to work. It's supposed to just work. Lets not talk about the "wat?" moment that is the file used as redirection target being actually created but with some random "lib_init()" text in it...

However, i understand why you guys implemented this option, because Windows console is frankly a bitch. Anyways, in a post below (in a few minutes), i will outline a solution that should make the Windows console behave without needing crutches like a some -no-redirect-streams options.

As an aside, i find it quite hilarious that to enable redirection i have to use an option that basically says "no redirect". ROFL...

[gpz]

As yoiu said, it should just work. but it doesn't. VICE contains a silly amount of code just to work around windows nonsense and to output the log into the console. Which then in turn breaks the redirect. Since we need the log more than the redirect, we choose that.

The core problem here seems to be that there is no way to determine whether stdout is currently attached to a terminal (where we need this silly extra code) or whether its being redirected to a file.

I am totally willing to merge a patch that removes this -no-redirect-streams options and just makes it work for that matter, but i doubt it can be done

[compyx]

Indeed a patch to fix this properly would be much appreciated, but I also doubt it can be done, seeing how we build with -mwindows which basically tells Windows we're running a GUI program and thus no std streams are needed/available.

There are some comments in the code about when WinAPI (or MSDN) lies about what functions return: https://sourceforge.net/p/vice-emu/code/HEAD/tree/trunk/vice/src/arch/shared/archdep_fix_streams.c#l47 (specifically the comments with "XXX:").

So have at it =)

[elgonzo]

Here i am going to outline a solution to have working console output while also having working stdout/stderr redirects without needing "magic" CLI incantations such as -no-redirect-streams.

Unfortunately, i am currently ill-equipped to get the tool-chain running for building VICE from source, hence why i also currently unable to submit my code suggestion as a diff or PR. And i don't know when i would be able to find enough time to get it all set up properly, so for the time being it's just me here throwing around code snippets.

For the sake of brevity, my code suggestions only deal with stdout, but stderr would be dealt with in the same manner.

There are two problems to address here. The first is detecting whether there is a redirection or not. The second is Windows' console not waiting for WinApps to end, thus happily displaying the prompt before the console output happens.

1. Making console output and redirection just work

A WinApp has no stdout/stderr assigned (its standard handles being of type FILE_TYPE_UNKNOWN), unless they are redirected. Only if there is no redirection, the respective standard stream needs to be rewired. The check for redirection has to happen before the console is attached.

Therefore:

FILE* fp_stdout;

HANDLE stdhandle = GetStdHandle(STD_OUTPUT_HANDLE);
DWORD stdFileType = GetFileType(stdhandle);
BOOL isConsoleStdOut = (stdFileType == FILE_TYPE_UNKNOWN || stdFileType == FILE_TYPE_CHAR);

if (AttachConsole(ATTACH_PARENT_PROCESS)) {
    if (isConsoleStdOut) {
        freopen_s(&fp_stdout, "CONOUT$", "wt", stdout);
        ...
    }

    ... same approach applies for stderr ...

    /*
        This does not deal with redirection, but with the console prompt;
        see section two below
     */
    if (isConsoleStdOut || isConsoleErrOut) {
        hasConsolePromptBackup = BackupAndClearConsolePrompt(&consolePromptBackup);
    }        
}

The logic here is somewhat different from what the function archdep_fix_streams() currently does. Note how here the standard handle is obtained (and its type checked) before the call to AttachConsole. archdep_fix_streams only obtains the standard handles after AttachConsole, those causing the trouble we are speaking about. Also note that the standard handle check includes the FILE_TYPE_UNKNOWN type (the type of which an unredirected standard handle of a WinApp would be).

That's it. This should be executed (for the Windows build only, obviously) independently of the -no-redirect-streams option. (If there is no other purpose for -no-redirect-streams it wouldn't be necessary anymore.) As far as i can tell, the only condition that should gate this code is the check for MSYSTEM (like archdep_fix_streams() already does.)

2. Tidying up the placement of the console prompt

Now, in the code snippet above there is one code line:

hasConsolePromptBackup = BackupAndRemoveConsolePrompt(&consolePromptBackup);

That's not about making redirection work, but addresses the second issue i mentioned: When starting WinApps, the Windows console won't wait for the WinApp to end but immediately prints the prompt. This leads to the prompt appearing before the program's console output. We can't make the console wait, but we can put lipstick on the pig.

For dealing with the prompt placement, two functions are implemented:

• BOOL BackupAndRemoveConsolePrompt(PCONSOLE_PROMPT_BACKUP)
  has to be called when attaching to console with stdout or stderr being not redirected.
  
  Parameter: A pointer to a CONSOLE_PROMPT_BACKUP struct that will be populated by the function with the backed up console prompt
  
  Returns: TRUE when prompt has been backed up; FALSE when prompt could not be backed up for some reason.
  
  
• void RestoreConsolePrompt(PCONSOLE_PROMPT_BACKUP)
  Call at the end of the entire console output (or the end of the program) to restore the prompt backed up by BackupAndRemoveConsolePrompt
  
  

Since we are dealing here with the Windows console, you'll see a lot of WINAPI stuff in their implementation (i hope i didn't unwittingly slip in any C++-only syntax by accident).

The entire code shown here below is only necessary for adjusting the placement of the console prompt. It is not needed for the console output and redirection to work. If you don't feel like taking on too much baggage and you are happy with just the console output and redirection working regardless of where the command prompt will appear in the console, you would only need the code in section 1 above (without the call to BackupAndRemoveConsolePrompt).

typedef struct _CONSOLE_PROMPT_BACKUP {
    PCHAR_INFO Buffer;
    SHORT Length;
} CONSOLE_PROMPT_BACKUP, *PCONSOLE_PROMPT_BACKUP;


BOOL BackupAndRemoveConsolePrompt(PCONSOLE_PROMPT_BACKUP pConsolePromptBackup)
{
    if (pConsolePromptBackup == NULL) {
        return FALSE;
    }

    pConsolePromptBackup->Buffer = NULL;
    pConsolePromptBackup->Length = 0;

    CONSOLE_SCREEN_BUFFER_INFO csbi;
    HANDLE hndStdOut = GetStdHandle(STD_OUTPUT_HANDLE);

    if (!GetConsoleScreenBufferInfo(hndStdOut, &csbi)) {
        return FALSE;
    }

    SHORT promptLength = csbi.dwCursorPosition.X;
    SHORT promptLineNumber = csbi.dwCursorPosition.Y;

    PCHAR_INFO pPromptBackupBuffer = (PCHAR_INFO)calloc(promptLength, sizeof(CHAR_INFO));
    if (pPromptBackupBuffer == NULL) {
        return FALSE;
    }

    SMALL_RECT promptRegion = { 0, promptLineNumber, promptLength, promptLineNumber };
    COORD bufferSize = { promptLength, 1 };
    COORD bufferCoord = { 0, 0 };

    BOOL isSuccess = ReadConsoleOutputW(
        hndStdOut,
        pPromptBackupBuffer,
        bufferSize,
        bufferCoord,
        &promptRegion
    );
    if (isSuccess) {
        DWORD charsWritten;
        COORD writeCoord = { 0, promptLineNumber };
        isSuccess = FillConsoleOutputCharacterW(
            hndStdOut,
            ' ',
            promptLength,
            writeCoord,
            &charsWritten
        );
        if (isSuccess) {
            COORD cursorPos = { 0, promptLineNumber };
            SetConsoleCursorPosition(hndStdOut, cursorPos);
        }
    }

    if (isSuccess) {
        pConsolePromptBackup->Buffer = pPromptBackupBuffer;
        pConsolePromptBackup->Length = promptLength;
    }
    else {
        free(pPromptBackupBuffer);
    }
    return isSuccess;
}


void FreeConsolePromptBackup(PCONSOLE_PROMPT_BACKUP pPromptBackupBuffer)
{
    if (pPromptBackupBuffer != NULL && pPromptBackupBuffer->Buffer != NULL) {
        free(pPromptBackupBuffer->Buffer);
        pPromptBackupBuffer->Buffer = NULL;
    }
}


void RestoreConsolePrompt(PCONSOLE_PROMPT_BACKUP pPromptBackupBuffer)
{
    if (pPromptBackupBuffer == NULL || pPromptBackupBuffer->Buffer == NULL) {
        return;
    }
    if (pPromptBackupBuffer->Length == 0)
    {
        FreeConsolePromptBackup(pPromptBackupBuffer);
        return;
    }

    CONSOLE_SCREEN_BUFFER_INFO csbi;
    HANDLE hndStdOut = GetStdHandle(STD_OUTPUT_HANDLE);

    if (!GetConsoleScreenBufferInfo(hndStdOut, &csbi))
    {
        FreeConsolePromptBackup(pPromptBackupBuffer);
        return;
    }

    if (csbi.dwCursorPosition.X != 0)
    {
        puts("");
        if (!GetConsoleScreenBufferInfo(hndStdOut, &csbi))
        {
            FreeConsolePromptBackup(pPromptBackupBuffer);
            return;
        }
    }

    SMALL_RECT promptRegion = { 0, csbi.dwCursorPosition.Y, pPromptBackupBuffer->Length, csbi.dwCursorPosition.Y };
    COORD bufferSize = { pPromptBackupBuffer->Length, 1 };
    COORD bufferCoord = { 0, 0 };
    BOOL isSuccess = WriteConsoleOutputW(
        hndStdOut,
        pPromptBackupBuffer->Buffer,
        bufferSize,
        bufferCoord,
        &promptRegion
    );
    if (isSuccess) {
        COORD cursorPos = { pPromptBackupBuffer->Length, csbi.dwCursorPosition.Y };
        SetConsoleCursorPosition(hndStdOut, cursorPos);
    }

    FreeConsolePromptBackup(pPromptBackupBuffer);
}

3. Demo app

A gist of a simple WinApp application demonstrating the code can be found here:
https://gist.github.com/elgonzo/6da34ae3a5f91be09bb0551657f421d1

Create a Windows desktop application project with the IDE/build tools of your choice, and use the code from the gist. Depending on your build tools, you might need to adopt annotations or the name the main method, but hopefully that shouldn't be much of a problem.

It prints about 4000 lines alternating between "Hello!" and "World!" using the puts function. You should be able to run the program to see the output on the console and also be able to redirect or pipe its console output just like a console app...

4. Be quick with the console output

If GTK Vice is doing console output to the console, it needs to do the entire console output it wants relatively quickly. Due to it being a WinApp (and not a console app), the console won't wait for Vice to end, but goes back into interactive mode right after starting the executable.

I saw that archdep_fix_streams() attempts to wire up stdin. But trying to wire it up to the console won't ever work because of the Windows console behavior regarding executing WinApps. If you really need un-redirected console input, or do un-redirected console output over longer time period (think console logging during the entire runtime of the emulator process), the approach with trying to wire up the WinApp executable with the console's stdout/stderr/stdin won't really work reliably, because the WinApp has no exclusive control over the console (unlike a console app).

[compyx]

I didn't see this when I posted my reply above. That's a pretty comprehensive explanation of what's happening and how to properly fix it, so thanks =)

I'll try to fiddle with archdep_fix_streams() this weekend or the coming week with the information provided above and see if I can get it working properly, hopefully indeed getting rid of the entire --no-redirect-streams option if I succeed.

Thanks again for the detailed information, it should be of great help.

[elgonzo]

I just noticed a mistake i made in the implementation of RestoreConsolePrompt. I used puts there to output a newline (can't manipulate the cursor position directly here, as the last output might be at the end/bottom of the console buffer, hence have to ensure a possibly needed shift of the lines in the console buffer). Which is fine if stdout is not redirected. But if stdout is redirected and stderr is not, RestoreConsolePrompt can't use puts, obviously. Need to replace it with a call to the WinApi WriteConsole. That's what i get for trying to be "cheap", lol...

[elgonzo]

One more update re BackupAndRemoveConsolePrompt and RestoreConsolePrompt functions.

I just noticed some strange issue triggered by the cls command in Powershell. Issuing a cls and then running my demo program (as shown in the gist), outputs correctly, with RestoreConsolePrompt placing the prompt and cursor at the right position correctly, as it is supposed to be. However doing cls before leads to PS now believing the input position to be the old cursor position before the cursor position has been changed by RestoreConsolePrompt. Which means, that now typed characters (or recalling a command from the history) appear at the old unadjusted cursor position. I wouldn't be surprised if there are other PS commands triggering the same weird behavior. Need to also check with PS.Core to see whether it also does funky things...

I therefore would suggest to not use BackupAndRemoveConsolePrompt and RestoreConsolePromptas currently implemented until i find a way to make Powershell behave.

[gpz]

until i find a way to make Powershell behave.

Ahem. please lets stick to cmd windows and fix that. powershell is of even less interest :)

[elgonzo]

cmd.exe works fine as far as i can tell. That's where i did all my testing before i did some checks in PS.

PS also works fine unless you do a cls beforehand. I didn't notice it earlier, because i didn't do a cls while testing in PS. Only now when i was dealing with the puts replacement in the RestoreConsolePrompt function that i mentioned earlier i stumbled over the weird PS thing triggered by cls...

UPDATE: "PS also works fine unless [...]" my arse!!!
Piping works fine PS, like .\foo.exe | more, the StdHandle being of type FILE_TYPE_PIPE. Hunky dory....
Redirecting like .\foo.exe >.\some.txtin PS is not working fine. The StdHandle is for some reason still of type FILE_TYPE_PIPE and not FILE_TYPE_DISK. Worse, while it creates the file, it doesn't write to it. There seems to be some PS-internal output buffer, but for some reason PS is not flushing this output buffer into the file. And if that buffer is filled and the program still keeps trying to write to stdout, the whole shebang freezes, waiting forever for the output buffer being flushed. How the flying eff can PS be so flustered by the exe merely being a Windows desktop app that it turns into stinkin' soup? Good lord...

[gpz]

It would be nice if you could massage what you have into an actual patch (and i guess "acts up a bit when using powershell" is still a lot better than "doesn't work at all"). Just don't break the log :)

(also CAUTION: there is still some log printing dangling around in the code that uses printf instead of the proper log functions - in particular if you have configured with --debug you will see this. that stuff should all be changed to log_printf in the long run. So if you see "some" log lines that behave different than the rest, this is the reason for it. You may enable logging to the monitor and check there - the printfs wont end up there in this case)