#1974 Any msys utility called from Apache->cgi->python freezes

MSYS
assigned
Cesar Strauss
Support
none
Unknown
False
2013-05-18
2013-05-17
Miroslav Stibor
No
#!/Python27/python -u
# File name: test.py

import subprocess
import datetime

def log(strIn):
  t = str(datetime.datetime.now())
  print(t + ": " + strIn)

def myTest():
  print "Content-type: text/html"
  print
  print "<html><body><pre>"
  log("start")
  process = subprocess.Popen(["c:/Program Files/msys/1.0/bin/date.exe"], stdout = subprocess.PIPE)
  # process = subprocess.Popen(["c:/temp/wbash/date.exe"], stdout = subprocess.PIPE)
  log("#1")
  print process.communicate()[0]
  log("end")
  print "</pre></body></html>"

myTest()

I tried to use MSYS unix utilities called from apache within python cgi script. The script is shown above. It doesn't matter i call bash, sed, date using Popen in that example. Called utility "freezes" for 14 s, in process explorer i always see it has 4 threads and no CPU consumption.

When i call python script above manually outside of apache script, everything works in expected way.

When I use other commands then MSYS commands, everything goes well too (within apache cgi/python). I migrated from MSYS to Windows bash (cygwin) to achieve solution.

Called from python (no cgi script solution), everything lasts 1/10 of second:
ActivePython 2.7.2.5 (ActiveState Software Inc.) based on
Python 2.7.2 (default, Jun 24 2011, 12:21:10) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import test
Content-type: text/html

<html><body><pre>
2013-05-17 13:20:23.203000: start
2013-05-17 13:20:23.234000: #1
Fri May 17 13:20:23 SEAST 2013
2013-05-17 13:20:23.312000: end
</pre></body></html>

Python cgi script called from apache lasts 14 seconds, doesn't take CPU:
2013-05-17 13:20:58.656000: start
2013-05-17 13:20:58.671000: #1
Fri May 17 13:21:22 SEAST 2013
2013-05-17 13:21:22.734000: end

My configuration:
Windows 7 Ultimate/Oracle VirtualBox/Windows 7 Ultimate
ActivePython 2.7.2.5
Apache httpd-2.0.64-win32-x86-no_ssl.msi
MSYS-1.0.11.exe

Discussion

1 2 > >> (Page 1 of 2)
  • Earnie Boyd
    Earnie Boyd
    2013-05-17

    MSYS-1.0.11.exe

    Upgrade and then report back. Issue reports against an old version do not fly well with any development community.

    • Download mingw-get from here.
    • Extract the archive to c:\MinGW
    • Open a cmd.exe window
    • Change directory to c:\MinGW
    • mingw-get update
    • mingw-get install mingw-get; # So you can upgrade later
    • mingw-get install msys
     
  • Earnie Boyd
    Earnie Boyd
    2013-05-17

    • status: unread --> open
    • Resolution: none --> old-version
     
  • Hello, I can't use mingw-get since my connection is limited and there's a lot of files being downloaded. However I started from a scratch in c:/MinGW, downloaded manually msys 1.0.18 and other fresh files needed (you can see below), run pi.bat, edited PATH, restarted Windows. Result is very same like described before.

    $ ls -l
    total 2018
    -rw-r--r-- 1 M Administrators 237267 May 17 19:49 bash-3.1.17-4-msys-1.0.16-bin.tar.lzma
    -rw-r--r-- 1 M Administrators 295789 May 17 19:53 coreutils-5.97-3-msys-1.0.13-bin.tar.lzma
    -rw-r--r-- 1 M Administrators 266676 May 17 19:53 coreutils-5.97-3-msys-1.0.13-ext.tar.lzma
    -rw-r--r-- 1 M Administrators   4719 May 17 22:14 libcharset-1.14-1-msys-1.0.17-dll-1.tar (1).lzma
    -rw-r--r-- 1 M Administrators  33055 May 17 22:15 libiconv-1.14-1-msys-1.0.17-bin.tar (1).lzma
    -rw-r--r-- 1 M Administrators 531196 May 17 22:14 libiconv-1.14-1-msys-1.0.17-dll-2.tar (1).lzma
    -rw-r--r-- 1 M Administrators  22647 May 17 22:12 libintl-0.18.1.1-1-msys-1.0.17-dll-8.tar (1).lzma
    -rw-r--r-- 1 M Administrators  36169 May 17 22:02 libregex-1.20090805-2-msys-1.0.13-dll-1.tar (1).lzma
    -rw-r--r-- 1 M Administrators  14687 May 17 22:06 libtermcap-0.20050421_1-2-msys-1.0.13-dll-0.tar.lzma
    -rw-r--r-- 1 M Administrators 610806 May 17 19:47 msysCORE-1.0.18-1-msys-1.0.18-bin.tar.lzma
    -rw-r--r-- 1 M Administrators   7872 May 17 19:48 msysCORE-1.0.18-1-msys-1.0.18-ext.tar.lzma
     
  • Earnie Boyd
    Earnie Boyd
    2013-05-17

    • labels: --> python, apache
    • Resolution: old-version --> none
     
  • Earnie Boyd
    Earnie Boyd
    2013-05-17

    Your use case isn't something we support much, it is well beyond the requirements for the Minimal SYStem mission statement. But it is worth declaring an issue; however, Cesar has little time to look. Alexey is working on MSYS2 based on more current release of Cygwin so eventually this may work but not anytime soon.

    One thing that might help is to make sure that the pipes are flushed in your Python script. I'm guessing that perhaps Apache hasn't seen the end of communication to process the data.

     
    • I've no idea. Yesterday i stumbled upon this link: 'Running subprocess.Popen under apache+mod_wsgi is always returning an error'. Perhaps there's a chance it is related bug.
      But then i tried other binaries, eg. from windows/system32 or from cygwin and it is being processed well. Therefore i tend to believe something in MinGW is broken.
      Apache server I use is not seen from Internet, Windows is restarted, output of script and script itself is minimal. And the result is always stable.

       
      • Earnie Boyd
        Earnie Boyd
        2013-05-17

        I doubt that the link that you provided has any bearing on this issue. There may indeed be a problem in MSYS but your use case scenario is under supported.

        I suggest flushing buffers because of a known issue which causes pty issues (which are implemented using pipes) when a non MSYS executable is involved. I do not know if that will resolve the issue but you'll need to try it to determine if that will help.

         
        Last edit: Earnie Boyd 2013-05-17
        • Thank you for response. I was playing with flushing. I was playing again with using files instead of writing to stdout/stderr/PIPEs. The result's always the same: MSYS app freezes when called by means of Apache.

          Could it be something that MSYS app is looking for stdin and can't handle it? Because I can play with sys.stdin file descriptor while i'm in python. When python script's called from apache, it says stdin: bad file descriptor.

          I don't press to resolve the bug, I just regret there's such bug and even bug which is very difficult to figure out, therefore someone else can get stuck too.

           
          Last edit: Miroslav Stibor 2013-05-18
  • Earnie Boyd
    Earnie Boyd
    2013-05-17

    BTW, Keith is working on reducing the download of the catalogue files so that ``mingw-get update'' will only download those that have a new timestamp. That should help eliminate the bulk downloads.

     
    • Keith Marshall
      Keith Marshall
      2013-05-17

      That's already working, in the latest snapshot on my project page. However, it doesn't help for the first time 'mingw-get update', because the aggregate catalogue furnished by mingw-get-inst.exe lacks the necessary issue time stamp keys in the package list entries.

      I'm currently working on mingw-get-setup.exe, which will supplant mingw-get-inst.exe; (I don't plan to update mingw-get-inst.exe ever again). mingw-get-setup.exe will furnish an aggregate catalogue as a free standing tarball, (bundled with an additional temporary DLL, for post-install use by mingw-get-setup.exe itself). Would it help if I were to post an advance copy of that?

       
1 2 > >> (Page 1 of 2)