Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

export all, git repo

2009-09-04
2013-05-14
  • I may be very broken :)

    I am trying to export ALL of the recipes I have spent 2 years importing from various software going back to the C=PET days.

    I can export selected one page at a time but I can not export all.

    Alternatively a way to clean up the database instead of exporting all and reimporting all would work if there is such a thing.

    Next I tried to get into the GIT repository read_only using the URL onm the SF project page with no luck, the CVS seems to me to be antique.

    I would love to take a look under the hood and see if I can fix my "problem" myself LOL.

    I have output of "gourmet -vvv" while trying to "export all" zipped up and can post it if that is a better option than trying to get fresh code to poke around in :)

    I posted long long ago about my import issues, and it seems that especially the DOS mealmaster files which do not use the IBM850 codepage or come from BBS's etc. where lots of ascii pictures are added need to be edited by hand before importing.

     
    • I just noticed the 8/23/2009 version and downloaded the deb and tarball.

      The CVS repository now seems even more antique and the GIT wont let me clone still.

      Export all still behaves the same way, nothing happens on the desktop and the almost identical output from gourmet -vvv (havent compared but they look the same to me).

      Probably should have said previosly:
      Ubuntu 9.04 Jaunty Jackalope AMD-64

       
      • tom
        tom
        2009-09-07

        I don't know what's going on with git -- I definitely have other users pulling the git repo without incident.

        With respect to your export error, it's particularly odd to get no error message and have it fail. I assume you're using the "export all" menu item. Do you see a traceback at the terminal when you run export all?

         
        • I am going to assume for the moment that my git software is ornery with something on SF since it does work on a couple of gnome projects and not any SF projects.

          I have recorded a video if that helps any but it is really not worth watching IMHO, it is me typing "gourmet" and the gourmet window coming up over the terminal window... then the text pasted below flies by and then ~20 minutes of nothing until I quit and the command prompt returns.

          Here is the whole of my terminal window from "gourmet" to closing gourmet

          After I click "export all" in the menu there is a few screens full of stuff and then nothing for a very long time. 

          I have edited out many repeats of the numbers etc.

          will@bigwill:~$ clear                                                                                                                                                                            
          will@bigwill:~$ gourmet
          /var/lib/python-support/python2.6/sqlalchemy/util.py:7: DeprecationWarning: the sets module is deprecated
            import inspect, itertools, new, operator, sets, sys, warnings, weakref                                
          Traceback (most recent call last):                                                                      
            File "/usr/share/gourmet/gourmet/GourmetRecipeManager.py", line 1053, in <lambda>                     
              None,_('Export all recipes to file'),lambda *args: self.do_export(export_all=True)),                
            File "/usr/share/gourmet/gourmet/GourmetRecipeManager.py", line 766, in do_export                     
              parent=self.app.get_toplevel())                                                                     
            File "/usr/share/gourmet/gourmet/exporters/exportManager.py", line 86, in offer_multiple_export       
              self.app.rd.include_linked_recipes(recs)                                                            
            File "/usr/share/gourmet/gourmet/backends/db.py", line 1365, in include_linked_recipes                
              self.ingredients_table.c.recipe_id.in_(ids)                                                         
            File "/var/lib/python-support/python2.6/sqlalchemy/sql/expression.py", line 1087, in execute          
              return e.execute_clauseelement(self, multiparams, params)                                           
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 1219, in execute_clauseelement
              return connection.execute_clauseelement(elem, multiparams, params)                                  
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 895, in execute_clauseelement
              return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 907, in _execute_compiled                                   
              self.__execute_raw(context)                                                                                                        
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 916, in __execute_raw                                       
              self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)                                    
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 960, in _cursor_execute                                     
              self._handle_dbapi_exception(e, statement, parameters, cursor)                                                                     
            File "/var/lib/python-support/python2.6/sqlalchemy/engine/base.py", line 942, in _handle_dbapi_exception                             
              raise exceptions.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)                               
          sqlalchemy.exceptions.OperationalError: (OperationalError) too many SQL variables u'SELECT ingredients.id, ingredients.recipe_id, ingredients.refid, ingredients.unit, ingredients.amount,
          ingredients.rangeamount, ingredients.item, ingredients.ingkey, ingredients.optional, ingredients.shopoptional, ingredients.inggroup, ingredients.position, ingredients.deleted \nFROM ingredients
            \nWHERE ingredients.refid AND ingredients.recipe_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
          ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
          ... lots of the same ...
          ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' [16045, 16046, 16047, 16048, 16049, 16050, 21922, 16051, 1, 2, 22420, 16052,
          3, 22421, 16053, 16054, 22422, 16055, 16056, 16057, 16058, 16059, 16060, 28842, 18, 22423, 16061, 22424, 28843, 4, 28845, 5, 28846, 22425, 6, 22426, 22427, 22428, 21924, 16062, 7, 21925, 8, 22429,
          16063, 20, 21, 22430, 22, 28847, 22431, 34574, 22432, 9, 10, 11, 28849, 34575, 12, 16064, 16065, 22433, 28851, 22434, 22435, 16066, 23, 13, 22436, 24, 35633, 35634, 16067, 16068, 28852, 16069,
          ... you get the idea ...
          21739, 26434, 21740,21741, 28838, 21742, 13095, 26435, 13096, 13097, 13098, 13099, 26436, 13100, 13101, 13103, 13104, 13105, 21743, 13106, 13107, 13108, 13109, 26437, 28839, 35631, 26438, 27432,
            35632, 21744, 27433]
          will@bigwill:~$

           
    • tom
      tom
      2009-09-07

      Oh -- a final question -- what format are you trying to export as?

       
    • It doesnt even ask me what format I want to export, it literally does NOTHING after the list of numbers above.

      It acts like there is something unplugged, I can go back to editing etc. no crash just nothing else that looks like an export.

      When I select a page of recipes and export it all works perfectly.

       
  • I am not sure if it was I missed posting or SF loosing a post but in any case after considerable will-foolery if the total number of recipes in a database is 999 or less export all works perfectly every time.

    1000 or greater and it fails as previously posted.

    My python is getting better SLOWLY ;) but coming along all the same!

     
  • tom
    tom
    2009-10-09

    This makes sense. It turns out there is a limit to how big a list you can handle SQL's &quot;in&quot; operator.

    The line your export is breaking on is a place where the program is exporting a linked recipe (e.g. a pie referencing a pie crust recipe). If the e.g. pie crust recipe is already in the export list, there's no extra work; if it's not, we add it to the export.

    This is handy, but apparently breaks at more than 999 recipes, so a new approach is needed. If you want to put your python/SQL skills to the test, go ahead and see if you can submit a patch.

    If you want to get really ambitious, you might look for other places where the &quot;in&quot; keyword is used in SQL and see if we might have similar bugs there as well.

     
  • already playing with fixing the export as a means of polishing python back to shiny ;)

    I will add everything else to the list and see what happens.

    What format do you prefer your patches?

    When I have done projects like this I prefer someone sending me an archive (I can open them all as far as I know) with the actual code  compiled/used when  done and then I merge them myself.

     
  • tom
    tom
    2009-10-19

    Patches against git head are fine. Since I haven't had many collaborators, I don't have a ton of experience with this, but if I recall correctly, emacs has modes that make it pretty easy to look at a patch and see what it's doing and accept/reject on a file-by-file basis if need be.