Webware for Python

Geoffrey Talvola wrote:
> I think the original reason for loginid was: suppose someone logs in, then
> leaves their browser open for a while.  Their session expires.  Now suppose
> someone else comes up to the browser, uses the Back button to go back to the
> login screen, and then presses Forward to re-post the username and password.
> The loginid is supposed to prevent this from working.

Ok. There are other scenarios as well where you want things to happen 
only one time (for instance, database transactions). I wonder whether 
Webware could provide some methods to do this transparently so you don't 
have to invent and code this kind of things again and again.

> As long as your newly modified code deletes the loginid as soon as it is
> used for a login, then it's fine.

Yes. The code still immediately clears the whole session after reading 
the login id. You can use it only one time.

-- Christoph

Christoph Zwerschke wrote:
> Geoffrey Talvola wrote:
>> I think the original reason for loginid was: suppose someone logs
>> in, then leaves their browser open for a while.  Their session
>> expires.  Now suppose someone else comes up to the browser, uses the
>> Back button to go back to the login screen, and then presses Forward
>> to re-post the username and password. The loginid is supposed to
>> prevent this from working. 

A common scenario where things can get posted more than once is if the user
uses the Back button to go back to a page that was a result of a post, and
then presses the Refresh button.  The browser will put up a popup offering
to re-post the variables.  If you don't want that to happen, you can have
the servlet that handles the post do a redirect.  This seems to prevent the
browser from ever re-posting the form -- instead, pressing the Refresh
button will re-get the page that was redirected to.

> Ok. There are other scenarios as well where you want things to happen
> only one time (for instance, database transactions). I wonder whether
> Webware could provide some methods to do this transparently so you
> don't have to invent and code this kind of things again and again.

Seems like a good idea.

- Geoff

Ben Parker wrote:
> This is not a strict Webware question, more of a Python and Linux
> question, but I'm hoping someone may have some good insight into a fix.
>
> I have a web app that relies on data from a 3rd party, fetched via XML
> over HTTPS. I'm using urllib2 to connect to the remote server.
Are you keeping the old data between connections? or do you fetch the
data every time?  More importantly how often are actually connecting to
this 3rd party server to get data?  i have a similar application but I
only update my cache of the data on daily. 
>
> Unfortunately, the 3rd party is unstable, and when they have a problem,
> sometimes the appserver crashes with this error:
>
> Traceback (most recent call last):
>   File "WebKit/ThreadedAppServer.py", line 630, in run
>   File "WebKit/ThreadedAppServer.py", line 156, in mainloop
>   File "/usr/local/lib/python2.4/socket.py", line 169, in accept
> error: (24, 'Too many open files')
>
If you're keeping the old data then you can simply wrap your 3rd party
call into a try -except statement and use the old data if your update fails.
> I think this is a symptom of too many open connections to the 3rd party,
> which makes sense since these server crashes (only happened twice so
> far) are the same time as a reported "outage" by the 3rd party.
>
> I already know about increasing the number of file descriptors for
> Linux, which I did after the first occurance of this, but as I
> expected that only delayed the problem. I need a way to recover from
> this somehow without having the server crash.
>
> Thanks for any advice!
> Ben
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>
>

Jose

Jose Galvez wrote on 01/18/2006 12:16 PM:

>Ben Parker wrote:
>  
>
>>This is not a strict Webware question, more of a Python and Linux
>>question, but I'm hoping someone may have some good insight into a fix.
>>
>>I have a web app that relies on data from a 3rd party, fetched via XML
>>over HTTPS. I'm using urllib2 to connect to the remote server.
>>    
>>
>Are you keeping the old data between connections? or do you fetch the
>data every time?  More importantly how often are actually connecting to
>this 3rd party server to get data?  i have a similar application but I
>only update my cache of the data on daily. 
>  
>
I'm doing lots of caching wherever possible, but most of the data must 
be fetched in real-time. It is for hotel room availability, which 
changes constantly.

>>Unfortunately, the 3rd party is unstable, and when they have a problem,
>>sometimes the appserver crashes with this error:
>>
>>Traceback (most recent call last):
>>  File "WebKit/ThreadedAppServer.py", line 630, in run
>>  File "WebKit/ThreadedAppServer.py", line 156, in mainloop
>>  File "/usr/local/lib/python2.4/socket.py", line 169, in accept
>>error: (24, 'Too many open files')
>>
>>    
>>
>If you're keeping the old data then you can simply wrap your 3rd party
>call into a try -except statement and use the old data if your update fails.
>  
>
I have that already. I think the problem is that the sockets are not 
timing out or throwing any error, so the except block never triggers. 
Otherwise I'm already handling a few differnt kinds of socket errors. 
I've been adding except clauses whenever I discover a new type of error:

These are trapped for urllib2.urlopen calls:
socket.sslerror
IOError
AttributeError (thrown from somewhere in urllib2.urlopen when it gets a 
malformed response)
httplib.BadStatusLine

These are trapped during the read of the url object returned by 
urllib2.urlopen:
socket.error
AssertionError (again thrown somewhere inside urllib2


Regards - Ben

Ben Parker wrote on 01/18/2006 12:45 PM:

>>> Unfortunately, the 3rd party is unstable, and when they have a problem,
>>> sometimes the appserver crashes with this error:
>>>
>>> Traceback (most recent call last):
>>>  File "WebKit/ThreadedAppServer.py", line 630, in run
>>>  File "WebKit/ThreadedAppServer.py", line 156, in mainloop
>>>  File "/usr/local/lib/python2.4/socket.py", line 169, in accept
>>> error: (24, 'Too many open files')
>>>
>>>   
>>
>> If you're keeping the old data then you can simply wrap your 3rd party
>> call into a try -except statement and use the old data if your update 
>> fails.
>>  
>>
> I have that already. I think the problem is that the sockets are not 
> timing out or throwing any error, so the except block never triggers.
>
I'm going to partially answer my own question here:

Doh! There is no timeout in the Python 2.4 socket module by default. For 
some reason I had thought it was 2 mins. I am now calling 
socket.setdefaulttimeout in the __init__ of my context to set a default 
timeout of 2.5 mins, which causes these connections to time out instead 
of consuming all available files.

However, I need to control the timeout value on a per-socket basis. I 
don't want to use socket.setdefaulttimeout because of the variety of 
requests this application needs to make. There are some that can 
tolerate relatively short timeouts (20 secs or so), but there are 
critical transactions which could take a couple minutes to return from 
the remote server.

A quick check of the socket module reveals that since Python 2.3 one can 
set a timeout on a particular socket object. So now I need to find a way 
of doing that through urllib2 or httplib or work around it by rolling my 
own httplib. I'll post some more info if I get anything going.

In the meantime, if anyone has an application which is setting 
per-socket timeouts for https transactions I would be very interested to 
hear how you are doing it. :)

- Ben

Marcelo Ametller wrote:
> where encounter made administrative applications of management with
> webware, webkit, python ?

Did you have a look at the examples provided with Webware, particularly 
the Admin section? From there you can do mostly everything. You should 
also make yourself aquainted with the possibilities of the standard lib. 
You can use all of this in Webware.

-- Christoph

Marcelo Ametller wrote:
> I have a servlet with form and a button label “import”
> the action for the 'button' is a function that takes a file .txt and
> enters it, into a postgresql/table.
> 
> I need that this function goes off from the button, at the browser, but
> as it consumes many resources (memory,threads), that the fall of
> appserver does not cause.

Span a completely separate process. You can store the pid of the process 
in your session so servlets can check whether the process is still 
running. The process could also signal "done" by creating a certain file 
or writing something to the database.

Process management is described in section 6.1.5 of the Python standard 
lib docs.

-- Christoph

marcelo ametller wrote:
> I have a servlet, with a form, a button with label "generate".
> When the user press the button, de "action" run a function, it generate
> a pdf (reportlab).
> When the pdf have 200 items, that's ok. But when the pdf have 6000
> items, the appserver crash. The dmesg inform that there isn't memory...

Does this crash immediately after you start the appserver and generate 
the very first pdf? Or does it only happen after some time when more 
users have been on the server?

What do you get in the Webware logfile?

Do you create the pdf in memory only or do you write it to a file?

-- Christoph

On 1/21/06, Christoph Zwerschke <cito@...> wrote:
> marcelo ametller wrote:
> > I have a servlet, with a form, a button with label "generate".
> > When the user press the button, de "action" run a function, it generate
> > a pdf (reportlab).
> > When the pdf have 200 items, that's ok. But when the pdf have 6000
> > items, the appserver crash. The dmesg inform that there isn't memory...
>
> Does this crash immediately after you start the appserver and generate
> the very first pdf? Or does it only happen after some time when more
> users have been on the server?
>
> What do you get in the Webware logfile?
>
> Do you create the pdf in memory only or do you write it to a file?
>
> -- Christoph

Also, can do you this generation outside of Webware, say as a
standalone Python program? In the past, most, if not all, crashes have
been associated with compiled Python extensions.

-Chuck

Christoph Zwerschke wrote:
> marcelo ametller wrote:
> 
>> I have a servlet, with a form, a button with label "generate".
>> When the user press the button, de "action" run a function, it generate
>> a pdf (reportlab).
>> When the pdf have 200 items, that's ok. But when the pdf have 6000
>> items, the appserver crash. The dmesg inform that there isn't memory...
> 
> 
> Does this crash immediately after you start the appserver and generate
> the very first pdf? Or does it only happen after some time when more
> users have been on the server?

it only crash after some time... the time for generate de pdf is 20min.

> 
> What do you get in the Webware logfile?
nothing...


> 
> Do you create the pdf in memory only or do you write it to a file?
i not know that...
i paste de code here:

action for the botton:

	def generar(self):		
		fecha1=self.form.field('fecha1').value()
		fecha2=self.form.field('fecha2').value()
		
		self.session().setValue('fecha1', fecha1)
		self.session().setValue('fecha2', fecha2) 		
		url='/menu/fac/inf/v/libFactR1'
		self.response().sendRedirect(url)
		self.writeBody()	

part of the servlet:

def writeHTML(self):
		req=self.request()		
		self.fecha1=self.session().value('fecha1')
		self.fecha2=self.session().value('fecha2')
		self.empresa = empresa.nombre	
		self.titulo = "Resumen Libro Facturacion - Facturas generadas entre %s
- %s" % (self.fecha1,self.fecha2)

		fact= Fact1()
		#consulta sql (return 6000 regs for 10 columns)
		self.listado= fact.lisLibFactR1(self.fecha1,self.fecha2)
		
		buffer=StringIO()
		doc = SimpleDocTemplate(buffer,pagesize=A4,leftMargin=3*cm)
		doc.setTitle=self.titulo
		doc.setAuthor='operador'
		
		Story = []
		data=self.listado	
		colwidths=(10*cm,2*cm)
		t=Table( data, colwidths)
		t.setStyle(TableStyle([('VALIGN',(0,0),(-1,-1),'MIDDLE'),					
('GRID',(0,0),(-1,-1),  0,colors.grey),						('BOX', (0,0), (-1,-1),
1, colors.black),						]))

		Story.append(t)
		Story.append(Spacer(1,0.2*cm))
			
		doc.build(Story, onFirstPage=self.myFirstPage,
onLaterPages=self.myLaterPages)
		pdf=buffer.getvalue()
		buffer.close()
		
		self.response().setHeader('Content-Type', 'application/pdf')
		self.response().setHeader('Content-Length', str(len(pdf)))
		self.response().setHeader('Content-Disposition', 'inline;
filename="ReslibFact.pdf"')
		self.write(pdf)


thanks for you time...

Marcelo

marcelo ametller wrote:
> it only crash after some time... the time for generate de pdf is 20min.

That's way too long. If it really must take so long, you should spawn a 
separate process to create the pdf. Your webserver will probably quit 
the connection with timeout anyway.

>> What do you get in the Webware logfile?
> nothing...

You can insert print statements, and use unbuffered output for the 
AppServer, to see where the servlet is spinning. But after a look on 
your servlet I am almost sure it will be the doc.build() call.

>> Do you create the pdf in memory only or do you write it to a file?
> i not know that...
> i paste de code here:

That would be in memory. Can you check whether the StringIO is imported 
from cStringIO (not StringIO)?

But the real problem is probably here:

> 		t=Table( data, colwidths)

You should definitely use "LongTable" instead of "Table" if your table 
has so many rows. The time used by Table goes quadratic with the number 
of rows...

The rest looks fine.

Let us know whether that helps.

-- Christoph

Christoph Zwerschke wrote:
> marcelo ametller wrote:
> 
>> it only crash after some time... the time for generate de pdf is 20min.
> 
> 
> That's way too long. If it really must take so long, you should spawn a
> separate process to create the pdf. Your webserver will probably quit
> the connection with timeout anyway.
> 
>>> What do you get in the Webware logfile?
>>
>> nothing...
> 
> 
> You can insert print statements, and use unbuffered output for the
> AppServer, to see where the servlet is spinning. But after a look on
> your servlet I am almost sure it will be the doc.build() call.
> 
>>> Do you create the pdf in memory only or do you write it to a file?
>>
>> i not know that...
>> i paste de code here:
> 
> 
> That would be in memory. Can you check whether the StringIO is imported
> from cStringIO (not StringIO)?
yes: from cStringIO import StringIO
I suppose that use the memory of the servidor?



> 
> But the real problem is probably here:
> 
>>         t=Table( data, colwidths)

now use:

from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer,
LongTable, TableStyle

the time of that form is smaller...

> 
> 
> You should definitely use "LongTable" instead of "Table" if your table
> has so many rows. The time used by Table goes quadratic with the number
> of rows...
> 
> The rest looks fine.
> 
> Let us know whether that helps.


it's the appserver inf:

empezo doc.build
   13  2006-01-24 09:51:39  /menu/fac/inf/v/auxIva1

termino doc.build
   13  234.81 secs          /menu/fac/inf/v/auxIva

the pdf contain 180 pages.

the appserver not crash...

thanks...




-- 
Marcelo

Marcelo Ametller wrote:
> yes: from cStringIO import StringIO
> I suppose that use the memory of the servidor?

Yes, it provides a kind of memory file for Reportlab to write to instead 
a real file on the harddisk. In this case, it would not make much 
difference, but if you serve many small PDFs it is better. You also 
don't have to care about a unique temporary file name. Sometimes it may 
also make sense to cache the PDFs you created. I.e. you don't create a 
temporary filename, but a filename that is made from the input 
parameters (e.g. weekly_report_03_2006). If somebody else requests the 
same document, you can serve it directly from disk.

-- Christoph

On 1/18/06, Hank Freeman <hfreeman@...> wrote:
> I am working on learning MiddleKit, and having a great time.  Things have
> gone well until I try to actually build a Store from my model.  After
> creating a simple model, generating it, creating the database and it's
> objects, and loading my sample data all successfully, I've stumbled.  Whe=
n I
> try to do store.readModelFileNamed(_filename) I get "ModelError: (<path t=
o
> my model.mkmodel>', "Could not import module for class '<First class in m=
y
> model>' due to 'No module named Middle'.  If you added this class recentl=
y,
> you need to re-generate your model.')
>
> I have a Settings.config file that contains:
> {
>     'Package': 'Middle',
>     'SQLLOG': {'File': 'mk-sql.log' },
>     'Database': 'MyDbNameHere'
> }
> and my "Middle" directory does have (an empty) __init__.py file to identi=
fy
> it as a package.
>
> I suspect I'm making a common noob oversite here, but (admitting to being=
 a
> newbie) I cannot see it.  Any advice on common pitfalls I might be hittin=
g
> here?  Thanks in advance!

The directory containing your Middle directory needs to be in the
python path which is stored in sys.path at runtime, and augmented by
the environment variable PYTHONPATH when Python starts.

I like to make my code impervious to its environment when possible, so
my personal pref is to enhance sys.path in my Python code. Also:

- I also like to make my code impervious to what directory it's run
in, so I don't use absolute paths.
- sys.path[0] is often a blank string or the path of the main python
program. Whatever the case, I never pre-empt it.
- My program is usually "next door" to the Middle directory in another
directory like "bin" or "WebApp".

So given all that, a command line tool I write that uses Middle might say:

import os, sys
progPath =3D os.path.join(os.getcwd(), sys.argv[0])
progDir =3D os.path.dirname(progPath)
projectDir =3D os.path.dirname(projectDir)
assert os.path.exists(projectDir), projectDir
sys.path.insert(1, projectDir)
# now test it:
import Middle


For a WebKit app, you'd do this fixup during context initialization.

Or you could just set the PYTHONPATH environment variable and not
write any Python code.


Btw any time you have problems importing a module, you can "print
sys.path" to help you troubleshoot it, although I find that output
hard to read and do this instead:

for p in sys.path:
    print repr(p)


HTH,
Chuck

Thank you!  As long as I manually set projectDir to some valid path all that
works for me!  However, you specificaly said that absolute paths should be
avoided, and I agree.  So, how should projectDir be getting set without
doing it overtly?  For the moment this works, so I'm off high-center, but I
need to understand the more general coase for the longer haul.

--Hank
  -----Original Message-----
  From: webware-discuss-admin@...
[mailto:webware-discuss-admin@... Behalf Of Chuck
Esterbrook
  Sent: Thursday, January 19, 2006 11:23 AM
  To: webware-discuss@...
  Subject: Re: [Webware-discuss] "No Module named Middle"


               Sender ALLOWED   [ Remove ]  [ Block ]    details

              Vanquish Anti-Spam Control Panel




  On 1/18/06, Hank Freeman <hfreeman@...> wrote:
  > I am working on learning MiddleKit, and having a great time.  Things
have
  > gone well until I try to actually build a Store from my model.  After
  > creating a simple model, generating it, creating the database and it's
  > objects, and loading my sample data all successfully, I've stumbled.
When I
  > try to do store.readModelFileNamed(_filename) I get "ModelError: (<path
to
  > my model.mkmodel>', "Could not import module for class '<First class in
my
  > model>' due to 'No module named Middle'.  If you added this class
recently,
  > you need to re-generate your model.')
  >
  > I have a Settings.config file that contains:
  > {
  >     'Package': 'Middle',
  >     'SQLLOG': {'File': 'mk-sql.log' },
  >     'Database': 'MyDbNameHere'
  > }
  > and my "Middle" directory does have (an empty) __init__.py file to
identify
  > it as a package.
  >
  > I suspect I'm making a common noob oversite here, but (admitting to
being a
  > newbie) I cannot see it.  Any advice on common pitfalls I might be
hitting
  > here?  Thanks in advance!

  The directory containing your Middle directory needs to be in the
  python path which is stored in sys.path at runtime, and augmented by
  the environment variable PYTHONPATH when Python starts.

  I like to make my code impervious to its environment when possible, so
  my personal pref is to enhance sys.path in my Python code. Also:

  - I also like to make my code impervious to what directory it's run
  in, so I don't use absolute paths.
  - sys.path[0] is often a blank string or the path of the main python
  program. Whatever the case, I never pre-empt it.
  - My program is usually "next door" to the Middle directory in another
  directory like "bin" or "WebApp".

  So given all that, a command line tool I write that uses Middle might say:

  import os, sys
  progPath = os.path.join(os.getcwd(), sys.argv[0])
  progDir = os.path.dirname(progPath)
  projectDir = os.path.dirname(projectDir)
  assert os.path.exists(projectDir), projectDir
  sys.path.insert(1, projectDir)
  # now test it:
  import Middle


  For a WebKit app, you'd do this fixup during context initialization.

  Or you could just set the PYTHONPATH environment variable and not
  write any Python code.


  Btw any time you have problems importing a module, you can "print
  sys.path" to help you troubleshoot it, although I find that output
  hard to read and do this instead:

  for p in sys.path:
      print repr(p)


  HTH,
  Chuck


  -------------------------------------------------------
  This SF.net email is sponsored by: Splunk Inc. Do you grep through log
files
  for problems?  Stop!  Download the new AJAX search engine that makes
  searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
  http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
  _______________________________________________
  Webware-discuss mailing list
  Webware-discuss@...
  https://lists.sourceforge.net/lists/listinfo/webware-discuss

Chuck Esterbrook wrote:
> Thanks for the update. Do you know if that plug-in leaves it's
> signature in either an HTTP header or in the user agent string? It
> would be nice to know when troubleshooting in the future.

You can recognize prefetching from Firefox by the following header:
X-Moz: prefetch

Firefox can do this even without Fasterfox if a web page provides 
certain prefetch link tags or headers. The Fasterfox extension has a 
setting where it does prefetching more aggressively.

-- Christoph

Yes, my Fasterfox version is significantly older: 0.7.9 is what I am at now.

--Hank
  -----Original Message-----
  From: webware-discuss-admin@...
[mailto:webware-discuss-admin@... Behalf Of Christoph
Zwerschke
  Sent: Friday, January 13, 2006 3:28 AM
  To: webware-discuss@...
  Subject: Re: [Webware-discuss] Admin password rejected


               Sender ALLOWED   [ Remove ]  [ Block ]    details

              Vanquish Anti-Spam Control Panel




  Hank Freeman wrote:
  > Mystery solved.  I am using the "Faster Fox" plug-in.  It does
pre-fetching
  > of links on a page.  With the setting at "Turbo" (most aggressive) this
  > authentication fails.  For me, it fails at all settings above
"Courteous".
  > (YMMV)  Dialing back the aggressiveness of the plug-in resolved this
issue.
  > Thanks again for all the help.

  Thank you for the feedback. Now the reason for this problem is clear:
  Fasterfox loaded all the other Admin links in the background which
  resulted in opening a lot of login pages. Each call of the login page
  changed the login id in the session, so that the original login id did
  not fit any more.

  If I understand correctly, the login id is used to ensure nobody can
  circumvent the login page (e.h. by providing user and password directly
  as parameters in the URL). So I left the login id mechanism in the code,
  but changed it so that no new login id is created if there is already
  one in the current session. I have checked that in already. The Example
  and Admin pages are not really important, but intended to give people an
  idea how things should be done; so they should do it correctly.

  But there is still one thing that I do not understand: I checked this
  with Fasterfox, and I noticed these two bugs/features:
  1) Fasterfox makes the links lowercase before pref etching them. It does
  not work at all if links contain uppercase letters and are case
  sensitive (which is the case for the Webware Example/Admin pages!)
  2) Fasterfox checks whether links have an extension that looks like a
  static page (e.g. ".html"). If they have no extension (which is the case
  for the Webware Example/Admin pages), then they are not prefetched.
  So only after I changed these two things in Fasterfox, I could reproduce
  the described behavior. Are you using a different version of Fasterfox
  (I checked with 1.0.2 which seems to be current).

  -- Christoph


  -------------------------------------------------------
  This SF.net email is sponsored by: Splunk Inc. Do you grep through log
files
  for problems?  Stop!  Download the new AJAX search engine that makes
  searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
  http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
  _______________________________________________
  Webware-discuss mailing list
  Webware-discuss@...
  https://lists.sourceforge.net/lists/listinfo/webware-discuss

Hank Freeman wrote:
> Yes, my Fasterfox version is significantly older: 0.7.9 is what I am at now.

Ok - then all mysteries are solved ;-)

-- Christoph