roundup-users — Go here to ask questions about how to use Roundup

Re: [Roundup-users] Issue tracker not working

[John P. R. <ro...@cs...>]

Hello Gianluca:

In message <d09...@tv...>,
Gianluca Ferrazzano writes:
>I took a look and we are using the classic template and I am working on 
>a copy at the moment

Good. I won't be available tomorrow, and wanted to make sure that this
wasn't a production issue.

>We did the Python 2-3 changes first to install the 2.2.0 roundup 
>version, but I went from 1.3.3 to 2.2.0 as I thought the reason it was 
>not working was because it was not fully updated.

Once you have done the 2-3 changes, only 2.2.0 or newer will work.
Anything earlier will fail to run.

How are you running Roundup?

  * stand alone server?
  * under another web server? which one: nginx, apache, lighttpd, other?
  * are you using a wsgi gateway like uwsgi or gnuicorn?
  * mod_python? (not suggested)
  * some other way?

There may be something in the upgrade path that we missed for a
particular deployment method. It's unlikely that the problem you have
is dependent on how you run Roundup, but this is an odd error inside
of Roundup so....

For testing purposes, you can use the stand alone server
(roundup-server).

Also while we are at it, what OS are you using?

It looks like you are using python 3.7 which is a bit old.

Is /usr/local/lib/python3.7 a Python virtual environment used for
installing Roundup? If not please create a virtual environment to
isolate your system Python install from the Roundup install.

>I am trying to access the home page, searching for specific issue 
>numbers works fine

Good.

>/tracker/status and /tracker/index?@columns=title both dont work

Hmm, you get the same traceback? Neither of those should be calling
reldate.

>i dont get an error when running, roundup-admin -i 
>tracker/home/directory display issue1

Good.

>I see so the problem could be with reldate, ok I will take a look and 
>see if theres anything I can do

Well the problem isn't with reldate. Reldate is getting an invalid
argument and complaining. The problem is that invalid value
(self._value).

I hate to suggest it, but at this point I would reset your
test copy to production. Then try the upgrade in stages:

   1.3.3 to 1.5.0 (using Python 2)
   1.5.0 to 1.6.1 (using Python 2)
   1.6.1 to 2.0.0 (and convert to Python 3)
   2.0.0 to 2.3.0 (Python 3 (2.3.0 was release a couple of weeks ago))

and test after every stage. It's possible the upgrade directions are
missing something that is needed in your environment. They could also
be unclear and lead you down the wrong path.

Have a great day.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Issue tracker not working

[Gianluca F. <gia...@tv...>]

Hello

I took a look and we are using the classic template and I am working on  
a copy at the moment

We did the Python 2-3 changes first to install the 2.2.0 roundup 
version, but I went from 1.3.3 to 2.2.0 as I thought the reason it was 
not working was because it was not fully updated.

I am trying to access the home page, searching for specific issue 
numbers works fine

/tracker/status and /tracker/index?@columns=title both dont work

i dont get an error when running, roundup-admin -i 
tracker/home/directory display issue1

I see so the problem could be with reldate, ok I will take a look and 
see if theres anything I can do

Thanks
Gianluca

On 26/07/2023 18:50, John P. Rouillard wrote:
> Hi Gianluca:
>
> In message <597...@tv...>,
> Gianluca Ferrazzano writes:
>> Yes we have a backup of the original database and tracker.
> Good.
>
>> Yes we have a backup of the tracker now.
> Good.
>
>> I am a beginner when it comes to python.
> Ok.
>
>> The database is postgresql.
>>
>> I'm not sure which template the tracker is based on, how would I find
>> this out?
> Look for a TEMPLATE-INFO.txt file in the tracker home directory.
>
> Are you working on your production tracker or are you testing the
> upgrade on a copy?
>
> I have included the rest of the original email below with more
> requests below. I think you stopped at the first quoted section that I
> was using for context for the rest of my questions.
>
> In message <37f...@tv...>,
> Gianluca Ferrazzano writes:
>
>> I have recently updates the Roundup issue tracker from 1.3.3 to 2.2.0,
>   That's a big jump. Did you do all the upgrades in one step, or did you
>   do it in multiple steps? For example:
>
>      * upgrade from 1.3.3 to 1.6.1 and verify things were working
>      * then do the 1.6.1 to 2.0.0 upgrade and convert to Python3
>      * then do the 2.0.0 to 2.2.0 upgrade
>
>   There is no reason I know of that doing it all in one step shouldn't
>   work. However, if you used mutiple steps, it would help localize it to
>   a particular release/releases.
>
>> I have done all the required steps but I am getting this error when I try
>> to access the issue tracker.
>   For all url references, '.../tracker/' means everything up to and
>   including the tracker designator (if present). For example Roundup
>   uses https://issues.roundup-tracker.org/ with no 'tracker' designator,
>   so '.../tracker/' would just be 'https://issues.roundup-tracker.org/'.
>   If your URL was 'http://tracker.mycompany.com/product/', that would
>   replace '.../tracker/'.
>
>   What page are you trying to load?
>
>      * an index page (e.g. .../tracker/issue)
>      * an item page (e.g. .../tracker/issue1)
>      * a search page (e.g. .../tracker/issue?@template=search)
>      * the home page  (e.g. .../tracker/)
>
>   Can you try loading the status index page:
>
>       .../tracker/status
>
>   or load an index page with only the issue titles:
>
>       .../tracker/index?@columns=title
>
>   by editing the URL in your browser. Neither of these pages should have
>   date calculations (IIRC). If there is no traceback for these, it will
>   tell us that the basic machinery is working.
>
>   Do you get an error if you run:
>
>      roundup-admin -i tracker/home/directory display issue1
>
>   where tracker/home/directory is the home directory for your tracker. (I
>   assume issue1 exists in the tracker. If you know an issue number you
>   can use it instead.)
>
>> Here is the full traceback, what should I do?
>   It looks like there is some date calculation going on and the _value
>   is not a Date but a string. I have never seen this before.
>
>    From the traceback this looks like the result of calling reldate. In
>   the classic template this only happens for the activity or creation
>   date. Check your templates (probably just the issue templates) using
>   grep or other tool and see if reldate shows up anywhere else than in:
>
>          tal:content="i/creation/reldate">&nbsp;</td>
>          tal:content="i/activity/reldate">&nbsp;</td>
>
>   You may have tripped across some weird edge case here.  Some
>   comparisions/operations that were allowed in Python 2 are not allowed
>   in Python 3. If your tracker has some string field that it's trying to
>   interpret as a date, Python 2 may have ignored the failed subtraction
>   but Python 3 won't. I doubt that this is the cause but it's the first
>   thought.
>
>   Have a good day.
>
> --
> 			-- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.
>

Re: [Roundup-users] Issue tracker not working

[John P. R. <ro...@cs...>]

Hi Gianluca:

In message <597...@tv...>,
Gianluca Ferrazzano writes:
>Yes we have a backup of the original database and tracker.

Good.

>Yes we have a backup of the tracker now.

Good.

>I am a beginner when it comes to python.

Ok.

>The database is postgresql.
>
>I'm not sure which template the tracker is based on, how would I find 
>this out?

Look for a TEMPLATE-INFO.txt file in the tracker home directory.

Are you working on your production tracker or are you testing the
upgrade on a copy?

I have included the rest of the original email below with more
requests below. I think you stopped at the first quoted section that I
was using for context for the rest of my questions.

In message <37f...@tv...>,
Gianluca Ferrazzano writes:

> I have recently updates the Roundup issue tracker from 1.3.3 to 2.2.0,

 That's a big jump. Did you do all the upgrades in one step, or did you
 do it in multiple steps? For example:

    * upgrade from 1.3.3 to 1.6.1 and verify things were working
    * then do the 1.6.1 to 2.0.0 upgrade and convert to Python3
    * then do the 2.0.0 to 2.2.0 upgrade

 There is no reason I know of that doing it all in one step shouldn't
 work. However, if you used mutiple steps, it would help localize it to
 a particular release/releases.

> I have done all the required steps but I am getting this error when I try
> to access the issue tracker.

 For all url references, '.../tracker/' means everything up to and
 including the tracker designator (if present). For example Roundup
 uses https://issues.roundup-tracker.org/ with no 'tracker' designator,
 so '.../tracker/' would just be 'https://issues.roundup-tracker.org/'.
 If your URL was 'http://tracker.mycompany.com/product/', that would
 replace '.../tracker/'.

 What page are you trying to load?

    * an index page (e.g. .../tracker/issue)
    * an item page (e.g. .../tracker/issue1)
    * a search page (e.g. .../tracker/issue?@template=search)
    * the home page  (e.g. .../tracker/)

 Can you try loading the status index page:

     .../tracker/status

 or load an index page with only the issue titles:

     .../tracker/index?@columns=title

 by editing the URL in your browser. Neither of these pages should have
 date calculations (IIRC). If there is no traceback for these, it will
 tell us that the basic machinery is working.

 Do you get an error if you run:

    roundup-admin -i tracker/home/directory display issue1

 where tracker/home/directory is the home directory for your tracker. (I
 assume issue1 exists in the tracker. If you know an issue number you
 can use it instead.)

> Here is the full traceback, what should I do?

 It looks like there is some date calculation going on and the _value
 is not a Date but a string. I have never seen this before.

  From the traceback this looks like the result of calling reldate. In
 the classic template this only happens for the activity or creation
 date. Check your templates (probably just the issue templates) using
 grep or other tool and see if reldate shows up anywhere else than in:

        tal:content="i/creation/reldate">&nbsp;</td>
        tal:content="i/activity/reldate">&nbsp;</td>

 You may have tripped across some weird edge case here.  Some
 comparisions/operations that were allowed in Python 2 are not allowed
 in Python 3. If your tracker has some string field that it's trying to
 interpret as a date, Python 2 may have ignored the failed subtraction
 but Python 3 won't. I doubt that this is the cause but it's the first
 thought.

 Have a good day.

--
			-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Issue tracker not working

[Gianluca F. <gia...@tv...>]

Hi John

Yes we have a backup of the original database and tracker.

Yes we have a backup of the tracker now.

I am a beginner when it comes to python.

The database is postgresql.

I'm not sure which template the tracker is based on, how would I find 
this out?

Thanks
Gianluca

On 26/07/2023 16:38, John P. Rouillard wrote:
> Hello Gianluca:
>
> I'm sorry you have run into issues. Let's see what we can do to fix it.
>
> Before we get started, a few questions.
>
> Do you have a backup of your original database and your original
> tracker before you started the upgrade?
>
> Do you have a backup of your database and your tracker as they are
> right now?
>
> How experienced are you with Python? Are you comfortable using
> the debugger for example.
>
> Which database (anydbm, sqlite, mysql, postgresql) are you using?
>
> Do you know which tracker template your tracker is based on?  I think
> 'classic' and 'minimal' were the only choices for 1.3.3 and previous.
>
> Also Ralf, Thomas, Bern, jerrykan any thoughts here? Ideas on how to
> set up an handler to get more info (e.g. local variable dump to see
> info about self/self._value would be nice)?  The TAL code is probably
> the area where I have the least experience.
>
> In message <37f...@tv...>,
> Gianluca Ferrazzano writes:
>> I have recently updates the Roundup issue tracker from 1.3.3 to 2.2.0,
> That's a big jump. Did you do all the upgrades in one step, or did you
> do it in multiple steps? For example:
>
>    * upgrade from 1.3.3 to 1.6.1 and verify things were working
>    * then do the 1.6.1 to 2.0.0 upgrade and convert to Python3
>    * then do the 2.0.0 to 2.2.0 upgrade
>
> There is no reason I know of that doing it all in one step shouldn't
> work. However, if you used mutiple steps, it would help localize it to
> a particular release/releases.
>
>> I have done all the required steps but I am getting this error when I try
>> to access the issue tracker.
> For all url references, '.../tracker/' means everything up to and
> including the tracker designator (if present). For example Roundup
> uses https://issues.roundup-tracker.org/ with no 'tracker' designator,
> so '.../tracker/' would just be 'https://issues.roundup-tracker.org/'.
> If your URL was 'http://tracker.mycompany.com/product/', that would
> replace '.../tracker/'.
>
> What page are you trying to load?
>
>    * an index page (e.g. .../tracker/issue)
>    * an item page (e.g. .../tracker/issue1)
>    * a search page (e.g. .../tracker/issue?@template=search)
>    * the home page  (e.g. .../tracker/)
>
> Can you try loading the status index page:
>
>     .../tracker/status
>
> or load an index page with only the issue titles:
>
>     .../tracker/index?@columns=title
>
> by editing the URL in your browser. Neither of these pages should have
> date calculations (IIRC). If there is no traceback for these, it will
> tell us that the basic machinery is working.
>
> Do you get an error if you run:
>
>    roundup-admin -i tracker/home/directory display issue1
>
> where tracker/home/directory is the home directory for your tracker. (I
> assume issue1 exists in the tracker. If you know an issue number you
> can use it instead.)
>
>> Here is the full traceback, what should I do?
> It looks like there is some date calculation going on and the _value
> is not a Date but a string. I have never seen this before.
>
>  From the traceback this looks like the result of calling reldate. In
> the classic template this only happens for the activity or creation
> date. Check your templates (probably just the issue templates) using
> grep or other tool and see if reldate shows up anywhere else than in:
>
>        tal:content="i/creation/reldate">&nbsp;</td>
>        tal:content="i/activity/reldate">&nbsp;</td>
>
> You may have tripped across some weird edge case here.  Some
> comparisions/operations that were allowed in Python 2 are not allowed
> in Python 3. If your tracker has some string field that it's trying to
> interpret as a date, Python 2 may have ignored the failed subtraction
> but Python 3 won't. I doubt that this is the cause but it's the first
> thought.
>
>> Full traceback:
>>
>> Traceback (most recent call last):
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/client.py", line 2021, in renderContext
>>      result = pt.render(self, None, None, **args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
>>      tal=1, strictinsert=0)()
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
>>      self.interpret(self.program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 571, in do_insertStructure_tal
>>      structure = self.engine.evaluateStructure(expr)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
>>      return expression(self)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/PythonExpr.py", line 91, in __call__
>>      return f()
>>    File "<string>", line 2, in f
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 978, in renderWith
>>      return pt.render(self._client, self.classname, req, **args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
>>      tal=1, strictinsert=0)()
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
>>      self.interpret(self.program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 680, in do_useMacro
>>      self.interpret(macro)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>>      self.do_optTag(stuff)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>>      return self.no_tag(start, program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>>      self.interpret(program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 705, in do_defineSlot
>>      self.interpret(slot)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>>      self.interpret(block)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>>      self.do_optTag(stuff)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>>      return self.no_tag(start, program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>>      self.interpret(program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>>      self.interpret(block)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 617, in do_loop_tal
>>      self.interpret(block)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>>      self.do_optTag(stuff)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>>      return self.no_tag(start, program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>>      self.interpret(program)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>>      self.interpret(block)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>>      handlers[opcode](self, args)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 492, in do_insertText_tal
>>      text = self.engine.evaluateText(stuff[0])
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 230, in evaluateText
>>      text = self.evaluate(expr)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
>>      return expression(self)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 198, in __call__
>>      return self._eval(econtext)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 193, in _eval
>>      return render(ob, econtext.vars)
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 99, in render
>>      ob = ob()
>>    File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 2241, in reldate
>>      interval = self._value - date.Date('.', translator=self._client)
>> TypeError: unsupported operand type(s) for -: 'str' and 'Date'
> Have a good day.
>
> --
> 				-- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Issue tracker not working

[John P. R. <ro...@cs...>]

Hello Gianluca:

I'm sorry you have run into issues. Let's see what we can do to fix it.

Before we get started, a few questions.

Do you have a backup of your original database and your original
tracker before you started the upgrade?

Do you have a backup of your database and your tracker as they are
right now?

How experienced are you with Python? Are you comfortable using
the debugger for example.

Which database (anydbm, sqlite, mysql, postgresql) are you using?

Do you know which tracker template your tracker is based on?  I think
'classic' and 'minimal' were the only choices for 1.3.3 and previous.

Also Ralf, Thomas, Bern, jerrykan any thoughts here? Ideas on how to
set up an handler to get more info (e.g. local variable dump to see
info about self/self._value would be nice)?  The TAL code is probably
the area where I have the least experience.

In message <37f...@tv...>,
Gianluca Ferrazzano writes:
>I have recently updates the Roundup issue tracker from 1.3.3 to 2.2.0,

That's a big jump. Did you do all the upgrades in one step, or did you
do it in multiple steps? For example:

  * upgrade from 1.3.3 to 1.6.1 and verify things were working
  * then do the 1.6.1 to 2.0.0 upgrade and convert to Python3
  * then do the 2.0.0 to 2.2.0 upgrade

There is no reason I know of that doing it all in one step shouldn't
work. However, if you used mutiple steps, it would help localize it to
a particular release/releases.

>I have done all the required steps but I am getting this error when I try 
>to access the issue tracker.

For all url references, '.../tracker/' means everything up to and
including the tracker designator (if present). For example Roundup
uses https://issues.roundup-tracker.org/ with no 'tracker' designator,
so '.../tracker/' would just be 'https://issues.roundup-tracker.org/'.
If your URL was 'http://tracker.mycompany.com/product/', that would
replace '.../tracker/'.

What page are you trying to load?

  * an index page (e.g. .../tracker/issue)
  * an item page (e.g. .../tracker/issue1)
  * a search page (e.g. .../tracker/issue?@template=search)
  * the home page  (e.g. .../tracker/)

Can you try loading the status index page:

   .../tracker/status

or load an index page with only the issue titles:

   .../tracker/index?@columns=title

by editing the URL in your browser. Neither of these pages should have
date calculations (IIRC). If there is no traceback for these, it will
tell us that the basic machinery is working.

Do you get an error if you run:

  roundup-admin -i tracker/home/directory display issue1

where tracker/home/directory is the home directory for your tracker. (I
assume issue1 exists in the tracker. If you know an issue number you
can use it instead.)

>Here is the full traceback, what should I do?

It looks like there is some date calculation going on and the _value
is not a Date but a string. I have never seen this before.

From the traceback this looks like the result of calling reldate. In
the classic template this only happens for the activity or creation
date. Check your templates (probably just the issue templates) using
grep or other tool and see if reldate shows up anywhere else than in:

      tal:content="i/creation/reldate">&nbsp;</td>
      tal:content="i/activity/reldate">&nbsp;</td>

You may have tripped across some weird edge case here.  Some
comparisions/operations that were allowed in Python 2 are not allowed
in Python 3. If your tracker has some string field that it's trying to
interpret as a date, Python 2 may have ignored the failed subtraction
but Python 3 won't. I doubt that this is the cause but it's the first
thought.

>Full traceback:
>
>Traceback (most recent call last):
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/client.py", line 2021, in renderContext
>     result = pt.render(self, None, None, **args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
>     tal=1, strictinsert=0)()
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
>     self.interpret(self.program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 571, in do_insertStructure_tal
>     structure = self.engine.evaluateStructure(expr)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
>     return expression(self)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/PythonExpr.py", line 91, in __call__
>     return f()
>   File "<string>", line 2, in f
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 978, in renderWith
>     return pt.render(self._client, self.classname, req, **args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
>     tal=1, strictinsert=0)()
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
>     self.interpret(self.program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 680, in do_useMacro
>     self.interpret(macro)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>     self.do_optTag(stuff)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>     return self.no_tag(start, program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>     self.interpret(program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 705, in do_defineSlot
>     self.interpret(slot)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>     self.interpret(block)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>     self.do_optTag(stuff)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>     return self.no_tag(start, program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>     self.interpret(program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>     self.interpret(block)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 617, in do_loop_tal
>     self.interpret(block)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
>     self.do_optTag(stuff)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
>     return self.no_tag(start, program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
>     self.interpret(program)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
>     self.interpret(block)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
>     handlers[opcode](self, args)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 492, in do_insertText_tal
>     text = self.engine.evaluateText(stuff[0])
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 230, in evaluateText
>     text = self.evaluate(expr)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
>     return expression(self)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 198, in __call__
>     return self._eval(econtext)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 193, in _eval
>     return render(ob, econtext.vars)
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 99, in render
>     ob = ob()
>   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 2241, in reldate
>     interval = self._value - date.Date('.', translator=self._client)
>TypeError: unsupported operand type(s) for -: 'str' and 'Date'

Have a good day.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Roundup-users] Issue tracker not working

[Gianluca F. <gia...@tv...>]

  Hello

I have recently updates the Roundup issue tracker from 1.3.3 to 2.2.0, I 
have done all the required steps but I am getting this error when I try 
to access the issue tracker.

Here is the full traceback, what should I do?

Thanks
Gianluca

Full traceback:

Traceback (most recent call last):
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/client.py", line 2021, in renderContext
     result = pt.render(self, None, None, **args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
     tal=1, strictinsert=0)()
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
     self.interpret(self.program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 571, in do_insertStructure_tal
     structure = self.engine.evaluateStructure(expr)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
     return expression(self)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/PythonExpr.py", line 91, in __call__
     return f()
   File "<string>", line 2, in f
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 978, in renderWith
     return pt.render(self._client, self.classname, req, **args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/engine_zopetal.py", line 94, in render
     tal=1, strictinsert=0)()
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 192, in __call__
     self.interpret(self.program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 680, in do_useMacro
     self.interpret(macro)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
     self.do_optTag(stuff)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
     return self.no_tag(start, program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
     self.interpret(program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 705, in do_defineSlot
     self.interpret(slot)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
     self.interpret(block)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
     self.do_optTag(stuff)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
     return self.no_tag(start, program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
     self.interpret(program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
     self.interpret(block)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 617, in do_loop_tal
     self.interpret(block)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 413, in do_optTag_tal
     self.do_optTag(stuff)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 398, in do_optTag
     return self.no_tag(start, program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 392, in no_tag
     self.interpret(program)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 644, in do_condition
     self.interpret(block)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 236, in interpret
     handlers[opcode](self, args)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/TAL/TALInterpreter.py", line 492, in do_insertText_tal
     text = self.engine.evaluateText(stuff[0])
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 230, in evaluateText
     text = self.evaluate(expr)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/TALES.py", line 224, in evaluate
     return expression(self)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 198, in __call__
     return self._eval(econtext)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 193, in _eval
     return render(ob, econtext.vars)
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/PageTemplates/Expressions.py", line 99, in render
     ob = ob()
   File "/usr/local/lib/python3.7/dist-packages/roundup-2.2.0-py3.7.egg/roundup/cgi/templating.py", line 2241, in reldate
     interval = self._value - date.Date('.', translator=self._client)
TypeError: unsupported operand type(s) for -: 'str' and 'Date'

-- 
Gianluca Ferrazzano
IT Executive
TVF
375 City Road
London EC1V 1NB

T:      +44 20 7837 3000
Email:gia...@tv...
Web:www.tvf.co.uk


This email, its contents and any files transmitted with it are confidential and may be legally privileged. It is intended solely for the addressee(s). If you are not the intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this email in error, please notify us immediately via telephone or fax and delete the material from your computer system.

TVF screens electronic mail for viruses but does not warrant that this electronic mail is free of any viruses. TVF accepts no liability for any damage caused by any virus transmitted by this electronic mail and recommends that you subject these to your virus checking procedures prior to use.

Re: [Roundup-users] Roundup Issue Tracker version 2.3.0 released

[Ralf S. <rs...@ru...>]

Thanks John!

[Ralf, back from vacation]
On Thu, Jul 13, 2023 at 12:17:12AM -0400, John P. Rouillard wrote:
> Hello All:
> 
> I'm proud to release version 2.3.0 of the Roundup issue
> tracker.  This release is a bugfix and feature
> release, so make sure to read `docs/upgrading.txt
> <https://www.roundup-tracker.org/docs/upgrading.html>`_ to
> bring your tracker up to date.
> 
> The changes, as usual, include some new features and many
> bug fixes.
> 
> Note that you should run ``roundup-admin ... migrate`` to
> update the database schema version. Do this before you use
> the web, command-line or mail interface and before any users
> access the tracker.
> 
> You can download it with::
> 
>    pip download roundup
> 
> then unpack and test/install the tarball. Also::
> 
>    pip install roundup
> 
> (preferably in a virtual environment) can be used.
> 
> Among the notable improvements from the 2.2.0 release are:
> 
> * Dockerfile demo mode implemented. This allows quick evaluation as
>   well as the ability to spin up a configured tracker to customise.
> 
> * SQLite backends can use WAL mode to reduce blocking between readers
>   and writers improving concurrent use.
> 
> * Redis can be used for session database with SQLite and dbm
>   backends. Provides a major performance improvement.
> 
> * roundup-mailgw can use OAUTH authentication to SMTP
>   server. (roundup-mailgw command line options changed as a result.)
> 
> * Postgres full text index can now be enabled.
> 
> * Modifications to in-reply-to threading when there are multiple
>   matches resulting in more predictable handling of messages.
> 
> * Many updates to documentation to make it scannable, useful and
>   work on mobile.
> 
> * Admin documentation includes a section on setting up Content
>   Security Policy (CSP) to better secure your Roundup trackers.
> 
> * REST now allows rate limiting headers to be accessed by client
>   JavaScript.
> 
> * Default number of rounds for PBKDF2 updated to 2M to account for
>   improvements in password crackers and CPU power.
> 
> * Support PBKDF2 with SHA512 for password storage to improve
>   resistance to password crackers.
> 
> * Deprecate SSHA password hash function.
> 
> * roundup-admin reindex can be done in batches to manage load
>   incurred by reindexing.
> 
> * roundup-admin can list available templates and their installed
>   locations. This is useful when installing via pip or in a docker
>   container as supporting files are not stored in the usual locations
>   like /usr/share/roundup.
> 
> * Crash fixes in detector handling
> 
> The file CHANGES.txt has a detailed list of feature additions and
> bug fixes (53) for each release. The most recent changes from
> there are at the end of this announcement. Also see the
> information in doc/upgrading.txt.
> 
> If you find bugs, please report them to issues AT roundup-tracker.org
> or create an account at https://issues.roundup-tracker.org and open a
> new ticket. If you have patches to fix the issues they can be attached
> to the email or uploaded to the tracker.
> 
> Upgrading
> =========
> 
> If you're upgrading from an older version of Roundup you *must* follow
> all the "Software Upgrade" guidelines given in the doc/upgrading.txt
> documentation.
> 
> Note that you should run ``roundup-admin ... migrate`` for
> all your trackers to update the database schema version. Do
> this before you use the web, command-line or mail interface
> and before any users access the tracker.
> 
> Roundup requires Python 2 newer than version 2.7.12 or Python 3 newer
> than or equal to version 3.6 for correct operation. (Python
> 3.4 or 3.5 may work, but are not tested.) Note that Python 2 support
> is being removed from the CI platforms, so you should deploy new
> trackers with Python 3 and plan on upgrading older trackers from Python
> 2 to Python 3. See the upgrade guide.
> 
> To give Roundup a try, just download (directions above), unpack and run::
> 
>     python demo.py
> 
> then open the url printed by the demo app.
> 
> Release info and download page:
>      https://pypi.org/project/roundup
> Source and documentation is available at the website:
>      https://www.roundup-tracker.org/
> Mailing lists - the place to ask questions:
>      https://sourceforge.net/p/roundup/mailman/
> 
> 
> About Roundup
> =============
> 
> Roundup is a simple-to-use and install issue-tracking system with
> command-line, web and e-mail interfaces. It is based on the winning design
> from Ka-Ping Yee in the Software Carpentry "Track" design competition.
> 
> Note: Ping is not responsible for this project. The contact for this
> project is rouilj at users.sourceforge.net. Use this address for
> security or other sensitive issues. Development discussions occur on
> the roundup-devel at lists.sourceforge.net mailing list. Tickets can
> be opened at https://issues.roundup-tracker.org.
> 
> Roundup manages a number of issues (with flexible properties such as
> "description", "priority", and so on) and provides the ability to:
> 
> (a) submit new issues,
> (b) find and edit existing issues, and
> (c) discuss issues with other participants.
> 
> The system facilitates communication among the participants by managing
> discussions and notifying interested parties when issues are edited. One of
> the major design goals for Roundup that it be simple to get going. Roundup
> is therefore usable "out of the box" with any Python 2.7.2+ (or 3.6+)
> installation. It doesn't even need to be "installed" to be operational,
> though an install script is provided.
> 
> It comes with five basic issue tracker templates
> 
> * a classic bug/feature tracker
> * a more extensive devel tracker for bug/features etc.
> * a responsive version of the devel tracker
> * a jinja2 version of the devel template (work in progress)
> * a minimal skeleton
> 
> and supports four database back-ends (anydbm, sqlite, mysql and postgresql).
> 
> Recent Changes
> ==============
> 
> >From 2.2.0 to 2.3.0
> 
> Fixed:
> ------
> 
> - Updated directions for verifying Roundup distribution using pgp.
> - Dockerfile healthcheck fixed so it works when trackers are
>   specified on command line. Also cleanup of unneeded
>   packages. (John Rouillard)
> - issue2551224 - Replace dbm db for sessions and otks when using
>   sqlite. New databases are created for session data (db-session)
>   and one time key data (db-otk). The data is ephemeral so no
>   need to migrate. (John Rouillard)
> - issue2551223 - Timestamps are truncated in mysql and postgresql
>   for session and otk database tables. Modify db schema to use a
>   numeric type that preserves more significant figures. See
>   upgrading.txt for required steps. (John Rouillard)
> - added more testing of BasicDatabase to support use of SQLite
>   for that purpose. Had to fix memory, rdbms and dbm edge cases
>   due to new tests. (John Rouillard)
> - issue2551138 - roundup-server with ssl under python2 throws
>   traceback on socket close. Not sure how this got fixed,
>   but after fixing issue2551137 it was not an issue anymore.
> - issue2551137 - roundup-server won't run with ssl under python3
>   Fixed by using SocketIO and manually adding buffering io and
>   catching SSL.ZeroReturnError indicating SSL has been shut down.
> - add caching header for text/javascript in addition to depricated
>   application/javascript. (John Rouillard)
> - Enable postgres-fts: fix indexer-common::get_indexer so it returns a
>   postgresql-fts Test code paths in get_indexer. (John Rouillard)
> - Fix Postgres native-fts, implement a two phase initialization of the
>   indexer. The native-fts one gets assigned after the database
>   connection is open. (John Rouillard)
> - fix crash if postgresql native-fts backend is asked to index content
>   with null bytes. (John Rouillard)
> - issue2551232 - modify in-reply-to threading when multiple matches
>   Change how in-reply-to threading works in the mailgw. If there is
>   more than one issue with a matching parent message, fall back to
>   subject matching. See upgrading.txt for details. (John Rouillard)
> - issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
> - issue2551246 - mitigation, document how -u doesn't work for
>   roundup-admin. (John Rouillard)
> - Document better that files in the template or static_files
>   directories accessed via @@file are available to any user with the
>   url. (John Rouillard)
> - Fix final exception handler in roundup-server to send proper
>   Content-Length header to the client. (John Rouillard)
> - Fix traceback if Origin header is missing. (John Rouillard)
> - issue2551250: Fix sorting of detectors even if there are two with the
>   same name and priority (can happen if they are created in two
>   different files).
> - Fix Traceback when a numeric order attribute is empty (Ralf
>   Schlatterbeck)
> - Update some template schema files to assign Register permissions for the
>   Anonymous user. Replaces the old Create permission. (John Rouillard)
> - Allow '*' and explicit origins in allowed_api_origins. Only return 
>   'Access-Control-Allow-Credentials' when not matching '*'. Fixes
>   security issue with rest when using '*'.  (John Rouillard)
> - issue2551263: In REST response expose rate limiting, sunset, allow
>   HTTP headers to calling JavaScript.  (John Rouillard)
> - issue2551257: When downloading an attached (user supplied file),
>   make sure that an 'X-Content-Type-Options: nosniff' header is sent.
>   (John Rouillard)
> - issue2551252 - default number of rounds for PKDF2 password increased
>   to 2,000,000.  (John Rouillard)
> - issue2551251 - migrate/re-encrypt PBKDF2 password if stored
>   password used a smaller number of rounds than set in
>   password_pbkdf2_default_rounds.  (John Rouillard)
> - upgrade from jquery-3.5.1 to jquery-3.6.3. Update user.help.html
>   to new version.  (John Rouillard)
> - Dockerfile scanned with hadolint. Fixed multiple issues. (John Rouillard)
> - fix crash due to invalid initialization/reset of configuration.py
>   option_validators. Crashed roundup-admin on second command if an
>   option_validator was added by a detector or extension. (John Rouillard)
> - Dockerfile uses dumb-init to properly wait for child/zombie
>   processes. Defense against child process starting from detector
>   and becoming a zombie when its roundup-server instance exits.
>   (John Rouillard)
> - Move installed frontend/Zope back to frontend/ZRoundup
>   directory. This better identifies the directory when copied into
>   the Zope framework. It also matches existing
>   documentation. (John Rouilard)
> - Multiple fixes/updates for installation documentation.
>   Including docker shell/admin/demo mdoes. (John Rouillard)
> - Invalid item identifiers passed to REST endpoint return a 404
>   rather than a 400 error. E.G. /rest/data/issue/issue4 (rather
>   than .../issue/4). (John Rouillard)
> - issue2551280 - sorted() method of MultilinkHTMLProperty is broken?
>   (Gabor Nagy report and fix; commit John Rouillard)
> 
> Features:
> ---------
> 
> - Add warning about limited Python 2 support lifetime to install and
>   upgrading docs. (John Rouillard)
> - Dockerfile supports demo mode for instant gratification
>   8-). Also supports shell and admin mode (John Rouillard)
> - Dockerfile build allows adding additional python packages via
>   pip, setting UID tracker is run under. (John Rouillard)
> - issue2551140 - Added redis as a session and otk database for use
>   with anydbm and sqlite primary databases. (John Rouillard)
> - issue2550559 - Pretty printing / formatting for Number types.
>   Added pretty(format='%0.3f') method to NumberHTMLProperty to
>   print numeric values. If value is None, return empty string
>   otherwise str() of value. (John Rouillard)
> - sqlite native-fts backend now uses the stopwords list in config.ini
>   to filter words from queries. (Stopwords are still indexed so that
>   phrase/proximity searches still work.) (John Rouillard)
> - sqlite databases use WAL mode when *created* to improve read
>   concurrency. Existing sqlite database still use rollback journal
>   mode. See upgrading.txt for details. (John Rouillard)
> - issue2551233 - create new roundup-admin command "templates" list all
>   template names, location and descriptions. Should help find where
>   /usr/share/roundup/templates is buried during some install
>   mechanisms. Does not need a tracker home to run. (John Rouillard)
> - Add OAuth authentication to the mailgw script. Now IMAPS can be used
>   with OAuth as required by several large cloud providers. Move command
>   line processing of the mailgw script to ``argparse``. Note that the
>   command line options of the mailgw have changed, see upgrading.txt for
>   details. (Ralf Schlatterbeck)
> - issue2551243: schema-dump.py enhanced with anti-CSRF headers. Flake8
>   cleanup and python2 support. (John Rouillard)
> - issue2551253 - new password hash PBDKF2-SHA512 added. Not
>   available by default. Follow directions in upgrading document
>   to use.  (John Rouillard)
> - roundup-admin migrate command reports the schema version.
> - issue2551262 - the mail gateway subject prefix now allows spaces
>   before/after prefix. Also allow spaces between classname and id
>   number in prefix designator. So "[ issue 23   ] subject" is parsed
>   like "[issue23] subject". (John Rouillard)
> - [doc]: add section on implementing CSP for Roundup to admin
>   doc. (John Rouillard)
> - issue2551265 - deprecate SSHA password hash method. Users using SSHA
>   passwords will have their passwords transprently upgraded to PBKDF2
>   derived hash on next login. (John Rouillard)
> - issue2551253 - Modify password PBKDF2 method to use SHA512. New
>   hash function using PBKDF2-SHA512 available. Will be default in
>   future. Directions for upgrading security by using it now is
>   included in upgrading.txt. (John Rouillard)
> - issue2551275 - Allow configuring max_children in roundup-server.
>   When using roundup-server in fork mode, allow raising number of
>   forked children above the default of 40. (Original patch by Joseph
>   Myers, config settings and docs by John Rouillard.)
> - roundup-admin genconfig does not need a tracker home to run. (John
>   Rouillard)
> - issue2551190 - Allow roundup-admin reindex to work in
>   batches. Running roundup-admin -i ... reindex issue:1-1000 will
>   reindex the first 1000 issues while reporting any missing issues
>   in the range. Also completion progress is reported when indexing a
>   specific class.
> - doc updates: add explanation for SQL code in 1.3.3->1.4.0 upgrade.
>   document schema table in rdbms backends and how to dump/extract
>   version from them. (John Rouillard)
> 
> --
> 				-- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.
> 
> 
> _______________________________________________
> Roundup-users mailing list
> Rou...@li...
> https://lists.sourceforge.net/lists/listinfo/roundup-users
> 

-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   www.runtux.com
Reichergasse 131, A-3411 Weidling       email: of...@ru...

[Roundup-users] Removing support for StructuredText (not reStructuredText)

[John P. R. <ro...@cs...>]

Hello all:

I was working in the templating code today and in testing I noticed
that I don't have StructuredText installed. So I wasn't testing with it.

I would like to remove the use of StructuredText (but not
reStructuredText) from the code base.

If there are no objections, it's gone in about a year in 2.4.0.

Here's the ticket: https://issues.roundup-tracker.org/issue2551285

Have a great weekend all.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[SCHLEMMER N. <Nor...@pd...>]

Hi John

Changed from #delete to #patch with { "@op":"action", "@action_name":"retire" } and it works just as using # delete

Thanks again
Norbert


-----Ursprüngliche Nachricht-----
Von: ro...@cs... <ro...@cs...> 
Gesendet: Donnerstag, 13. Juli 2023 21:57
An: SCHLEMMER Norbert <Nor...@pd...>
Cc: rou...@li...
Betreff: Re: AW: AW: [Roundup-users] Roundup rest API

*EXTERNAL source*


Hi Norbert:

In message <18f...@pd...>,
SCHLEMMER Norbert writes:
>>>The #patch command with "@op=remove" will be executed with return 
>>>code
>>>"200 OK"

I think this is correct. Without seeting the entire transaction (including payload) it is difficult to tell. I think you told it to remove the items in the payload without supplying a payload. So it removed nothing and reported an empty data section because nothing had changed.

>>Just a note, using the DELETE verb will do the same [...]
>>
>Using the DELETE command and having ADMIN role assigned makes it working well.
>The "deleted" issue will be set to retire and is excluded from search

I mispoke (mistyped??). To do the equivalent of DELETE using PATCH "@op=action" and "@action_name=retire" is used to retire an item. Compare this to @op=remove is used to modify the data according to the data payload.

If you use:

    curl -X DELETE -H 'If-match: "etag here"' -H 'x-requested-with: rest' .... tracker/rest/data/status/1

it works and a subsequent GET on tracker/rest/data/status

will not show status10.

Using:

   curl -u admin:adminpw -X PATCH  \
      -H "x-requested-with: rest" \
      -H "Content-Type: application/json" \
      -H "Origin: https://hostname" \
      -H "Referer: https://hostname/tracker" \
      -H 'If-match: "0ad78b0fd17712641b65ca1f8a60f7a0"' \
      --data-binary '{ "@op": "action", "@action_name": "retire" }' \
     https://hostname/tracker/rest/data/status/10

returns:

  {
    "data": {
        "id": "10",
        "type": "status",
        "link": "https://hostname/tracker/rest/data/status/10",
        "result": null
    }
  }

should also work. Note that using this with a user missing the restire/restore permission in a role, I get a 403 error.

  {
    "error": {
        "status": 403,
        "msg": "You do not have permission to retire or restore the status class."
    }
  }

Does this describe what you are seeing?

--
                                -- rouilj John Rouillard ===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[John P. R. <ro...@cs...>]

Hi Norbert:

In message <18f...@pd...>,
SCHLEMMER Norbert writes:
>>>The #patch command with "@op=remove" will be executed with return code 
>>>"200 OK"

I think this is correct. Without seeting the entire transaction
(including payload) it is difficult to tell. I think you told it to
remove the items in the payload without supplying a payload. So it
removed nothing and reported an empty data section because nothing had
changed.

>>Just a note, using the DELETE verb will do the same [...]
>>
>Using the DELETE command and having ADMIN role assigned makes it working well.
>The "deleted" issue will be set to retire and is excluded from search

I mispoke (mistyped??). To do the equivalent of DELETE using PATCH
"@op=action" and "@action_name=retire" is used to retire an
item. Compare this to @op=remove is used to modify the data according
to the data payload.

If you use:

    curl -X DELETE -H 'If-match: "etag here"' -H 'x-requested-with: rest' .... tracker/rest/data/status/1

it works and a subsequent GET on tracker/rest/data/status

will not show status10.

Using:
 
   curl -u admin:adminpw -X PATCH  \
      -H "x-requested-with: rest" \
      -H "Content-Type: application/json" \
      -H "Origin: https://hostname" \
      -H "Referer: https://hostname/tracker" \
      -H 'If-match: "0ad78b0fd17712641b65ca1f8a60f7a0"' \
      --data-binary '{ "@op": "action", "@action_name": "retire" }' \
     https://hostname/tracker/rest/data/status/10

returns:

  {
    "data": {
        "id": "10",
        "type": "status",
        "link": "https://hostname/tracker/rest/data/status/10",
        "result": null
    }
  }

should also work. Note that using this with a user missing the
restire/restore permission in a role, I get a 403 error.

  {
    "error": {
        "status": 403,
        "msg": "You do not have permission to retire or restore the status class."
    }
  }

Does this describe what you are seeing?

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[SCHLEMMER N. <Nor...@pd...>]

Using the DELETE command and having ADMIN role assigned makes it working well.
The "deleted" issue will be set to retire and is excluded from search

Thanks
Norbert

-----Ursprüngliche Nachricht-----
Von: ro...@cs... <ro...@cs...> 
Gesendet: Donnerstag, 13. Juli 2023 15:23
An: SCHLEMMER Norbert <Nor...@pd...>
Cc: rou...@li...
Betreff: Re: AW: [Roundup-users] Roundup rest API

*EXTERNAL source*


In message <6ad...@pd...>,
SCHLEMMER Norbert writes:
>Hello John
>
>Want to add the "delete / remove" issue function, but how ?
>
>The #patch command with "@op=remove" will be executed with return code 
>"200 OK"

Just a note, using the DELETE verb will do the same and is arguably easier to use.  @op=remove was added to match @op=restore since there is no HTTP verb that matches the restore function.

>but the issue is still visible and part of the search result

The issue will be retreivable if you use its full rest path
.../rest/data/issue/23 for example. There have been discussions about returning a 404 or hiding a retired item. But we haven't had any concrete ideas on impementing it. This can be a problem for users and GDPR. https://wiki.roundup-tracker.org/RemovingUsers may be helpful.

It should not be listed in an index view. E.G. .../rest/data/issue should not show the retired issue.

What type of search are you doing, what URL? Also what roles does the searching user have?

Have a great day.
--
                                -- rouilj John Rouillard ===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[John P. R. <ro...@cs...>]

In message <6ad...@pd...>,
SCHLEMMER Norbert writes:
>Hello John
>
>Want to add the "delete / remove" issue function, but how ?
>
>The #patch command with "@op=remove" will be executed with return
>code "200 OK"

Just a note, using the DELETE verb will do the same and is arguably
easier to use.  @op=remove was added to match @op=restore since there
is no HTTP verb that matches the restore function.

>but the issue is still visible and part of the search result

The issue will be retreivable if you use its full rest path
.../rest/data/issue/23 for example. There have been discussions about
returning a 404 or hiding a retired item. But we haven't had any
concrete ideas on impementing it. This can be a problem for users and
GDPR. https://wiki.roundup-tracker.org/RemovingUsers may be helpful.

It should not be listed in an index view. E.G. .../rest/data/issue
should not show the retired issue.

What type of search are you doing, what URL? Also what roles does the
searching user have?

Have a great day.
--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[SCHLEMMER N. <Nor...@pd...>]

Hello John

Want to add the "delete / remove" issue function, but how ?

The #patch command with "@op=remove" will be executed with return code "200 OK" but the issue is still visible and part of the search result

Any idea ?

Thanks
Norbert
 

-----Ursprüngliche Nachricht-----
Von: ro...@cs... <ro...@cs...> 
Gesendet: Mittwoch, 12. Juli 2023 22:44
An: SCHLEMMER Norbert <Nor...@pd...>
Cc: rou...@li...
Betreff: Re: [Roundup-users] Roundup rest API

*EXTERNAL source*


*This email contains one or more catchphrase(s), like password/passwort, voicemail, etc. which may be used by attackers.*

Hello Norbert:

In message <c05...@pd...>,
SCHLEMMER Norbert writes:
>Where can I find an "How-to" add a keyword / list of keywords to an issue ?

There isn't a How-To for that, but the info needed is in the rest guide.  You are looking for the example that has the string:
rest/data/issue/@poe/viz in it.


To expand a little. To create a new item (status, issue, user etc.) you POST a json data structure to the item endpoint:
tracker/rest/data/status for example.

To find out the json structure, perform a GET with ?@verbose=0 (described in the section "Using the POST method") on a similar item. For example on my customized tracker:

a GET to: tracker/rest/data/status/8?@verbose=0 produces.

{
    "data": {
        "id": "8",
        "type": "status",
        "link": "tracker/rest/data/status/8",
        "attributes": {
            "abbreviation": "D",
            "help": "Ticket has been closed. Will not be worked on.",
            "name": "dead",
            "order": 8,
            "requiredpermissions": null,
            "transitions": [
                "1",
                "2",
                "4"
            ]
        },
        "@etag": "\"a1974134dfc8410fc46163032f063c0d\""
    }
}

The attributes json blob is what you want to replicate to create a new status.  (Note this is from my tracker. It has attributes like "abbreviation" and "transitions" yours will be different.)

To create a new status, I created the file /tmp/new_status.json with this content:

 {
            "abbreviation": "Z",
            "help": "example status created via rest. Do not use.",
            "name": "zoid",
            "order": 11,
            "requiredpermissions": null,
            "transitions": []
        }


and submitted it using:

    curl -u admin:adminpassword -X POST \
    --data-binary @/tmp/new_status.json \
    -H "x-requested-with: rest" \
    -H "Content-Type: application/json" \
    -H "Origin: https://hostname" \
    -H "Referer: https://hostname/tracker" \
     https://hostname/tracker/rest/data/status

and got the result:

{
    "data": {
        "id": "10",
        "link": "https://hostname/tracker/rest/data/status/10"
    }
}

The x-requested-with and Content-Type headers are required. You may not need all the rest of the header fields. I have all the cross site request forgery protections engaged, so I need them.

The user can be any user who has create privs for the status item. Usually this is anybody with the Admin role and usually only the admin account has that role. You can see if another role has that ability by running:

  roundup-admin  -i demo security | sed -ne '/^Role/p' -e '/create status/p'
  Role "admin":
  Role "agent":
  Role "anonymous":
  Role "provisional user":
  Role "seeabout":
  Role "seeaboutprivdetails":
  Role "setanystatus":
  Role "user":

So we can see that no other role has the ability to create a status. (Note that the Admin role has the right to make CRUD operations on all items. Security doesn't list every item for Admin. It prints "User may create everything" instead.)

Does this help?

Have a great rest of the week.

--
                                -- rouilj John Rouillard ===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[SCHLEMMER N. <Nor...@pd...>]

Hello John

The problem wasn't creating the issue object, but adding a list of keywords to it.
I got it running by debugging and reverse engineering...

Thanks and all the best
Norbert

-----Ursprüngliche Nachricht-----
Von: ro...@cs... <ro...@cs...> 
Gesendet: Mittwoch, 12. Juli 2023 22:44
An: SCHLEMMER Norbert <Nor...@pd...>
Cc: rou...@li...
Betreff: Re: [Roundup-users] Roundup rest API

*EXTERNAL source*


*This email contains one or more catchphrase(s), like password/passwort, voicemail, etc. which may be used by attackers.*

Hello Norbert:

In message <c05...@pd...>,
SCHLEMMER Norbert writes:
>Where can I find an "How-to" add a keyword / list of keywords to an issue ?

There isn't a How-To for that, but the info needed is in the rest guide.  You are looking for the example that has the string:
rest/data/issue/@poe/viz in it.


To expand a little. To create a new item (status, issue, user etc.) you POST a json data structure to the item endpoint:
tracker/rest/data/status for example.

To find out the json structure, perform a GET with ?@verbose=0 (described in the section "Using the POST method") on a similar item. For example on my customized tracker:

a GET to: tracker/rest/data/status/8?@verbose=0 produces.

{
    "data": {
        "id": "8",
        "type": "status",
        "link": "tracker/rest/data/status/8",
        "attributes": {
            "abbreviation": "D",
            "help": "Ticket has been closed. Will not be worked on.",
            "name": "dead",
            "order": 8,
            "requiredpermissions": null,
            "transitions": [
                "1",
                "2",
                "4"
            ]
        },
        "@etag": "\"a1974134dfc8410fc46163032f063c0d\""
    }
}

The attributes json blob is what you want to replicate to create a new status.  (Note this is from my tracker. It has attributes like "abbreviation" and "transitions" yours will be different.)

To create a new status, I created the file /tmp/new_status.json with this content:

 {
            "abbreviation": "Z",
            "help": "example status created via rest. Do not use.",
            "name": "zoid",
            "order": 11,
            "requiredpermissions": null,
            "transitions": []
        }


and submitted it using:

    curl -u admin:adminpassword -X POST \
    --data-binary @/tmp/new_status.json \
    -H "x-requested-with: rest" \
    -H "Content-Type: application/json" \
    -H "Origin: https://hostname" \
    -H "Referer: https://hostname/tracker" \
     https://hostname/tracker/rest/data/status

and got the result:

{
    "data": {
        "id": "10",
        "link": "https://hostname/tracker/rest/data/status/10"
    }
}

The x-requested-with and Content-Type headers are required. You may not need all the rest of the header fields. I have all the cross site request forgery protections engaged, so I need them.

The user can be any user who has create privs for the status item. Usually this is anybody with the Admin role and usually only the admin account has that role. You can see if another role has that ability by running:

  roundup-admin  -i demo security | sed -ne '/^Role/p' -e '/create status/p'
  Role "admin":
  Role "agent":
  Role "anonymous":
  Role "provisional user":
  Role "seeabout":
  Role "seeaboutprivdetails":
  Role "setanystatus":
  Role "user":

So we can see that no other role has the ability to create a status. (Note that the Admin role has the right to make CRUD operations on all items. Security doesn't list every item for Admin. It prints "User may create everything" instead.)

Does this help?

Have a great rest of the week.

--
                                -- rouilj John Rouillard ===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Roundup-users] Roundup Issue Tracker version 2.3.0 released

[John P. R. <ro...@cs...>]

Hello All:

I'm proud to release version 2.3.0 of the Roundup issue
tracker.  This release is a bugfix and feature
release, so make sure to read `docs/upgrading.txt
<https://www.roundup-tracker.org/docs/upgrading.html>`_ to
bring your tracker up to date.

The changes, as usual, include some new features and many
bug fixes.

Note that you should run ``roundup-admin ... migrate`` to
update the database schema version. Do this before you use
the web, command-line or mail interface and before any users
access the tracker.

You can download it with::

   pip download roundup

then unpack and test/install the tarball. Also::

   pip install roundup

(preferably in a virtual environment) can be used.

Among the notable improvements from the 2.2.0 release are:

* Dockerfile demo mode implemented. This allows quick evaluation as
  well as the ability to spin up a configured tracker to customise.

* SQLite backends can use WAL mode to reduce blocking between readers
  and writers improving concurrent use.

* Redis can be used for session database with SQLite and dbm
  backends. Provides a major performance improvement.

* roundup-mailgw can use OAUTH authentication to SMTP
  server. (roundup-mailgw command line options changed as a result.)

* Postgres full text index can now be enabled.

* Modifications to in-reply-to threading when there are multiple
  matches resulting in more predictable handling of messages.

* Many updates to documentation to make it scannable, useful and
  work on mobile.

* Admin documentation includes a section on setting up Content
  Security Policy (CSP) to better secure your Roundup trackers.

* REST now allows rate limiting headers to be accessed by client
  JavaScript.

* Default number of rounds for PBKDF2 updated to 2M to account for
  improvements in password crackers and CPU power.

* Support PBKDF2 with SHA512 for password storage to improve
  resistance to password crackers.

* Deprecate SSHA password hash function.

* roundup-admin reindex can be done in batches to manage load
  incurred by reindexing.

* roundup-admin can list available templates and their installed
  locations. This is useful when installing via pip or in a docker
  container as supporting files are not stored in the usual locations
  like /usr/share/roundup.

* Crash fixes in detector handling

The file CHANGES.txt has a detailed list of feature additions and
bug fixes (53) for each release. The most recent changes from
there are at the end of this announcement. Also see the
information in doc/upgrading.txt.

If you find bugs, please report them to issues AT roundup-tracker.org
or create an account at https://issues.roundup-tracker.org and open a
new ticket. If you have patches to fix the issues they can be attached
to the email or uploaded to the tracker.

Upgrading
=========

If you're upgrading from an older version of Roundup you *must* follow
all the "Software Upgrade" guidelines given in the doc/upgrading.txt
documentation.

Note that you should run ``roundup-admin ... migrate`` for
all your trackers to update the database schema version. Do
this before you use the web, command-line or mail interface
and before any users access the tracker.

Roundup requires Python 2 newer than version 2.7.12 or Python 3 newer
than or equal to version 3.6 for correct operation. (Python
3.4 or 3.5 may work, but are not tested.) Note that Python 2 support
is being removed from the CI platforms, so you should deploy new
trackers with Python 3 and plan on upgrading older trackers from Python
2 to Python 3. See the upgrade guide.

To give Roundup a try, just download (directions above), unpack and run::

    python demo.py

then open the url printed by the demo app.

Release info and download page:
     https://pypi.org/project/roundup
Source and documentation is available at the website:
     https://www.roundup-tracker.org/
Mailing lists - the place to ask questions:
     https://sourceforge.net/p/roundup/mailman/


About Roundup
=============

Roundup is a simple-to-use and install issue-tracking system with
command-line, web and e-mail interfaces. It is based on the winning design
from Ka-Ping Yee in the Software Carpentry "Track" design competition.

Note: Ping is not responsible for this project. The contact for this
project is rouilj at users.sourceforge.net. Use this address for
security or other sensitive issues. Development discussions occur on
the roundup-devel at lists.sourceforge.net mailing list. Tickets can
be opened at https://issues.roundup-tracker.org.

Roundup manages a number of issues (with flexible properties such as
"description", "priority", and so on) and provides the ability to:

(a) submit new issues,
(b) find and edit existing issues, and
(c) discuss issues with other participants.

The system facilitates communication among the participants by managing
discussions and notifying interested parties when issues are edited. One of
the major design goals for Roundup that it be simple to get going. Roundup
is therefore usable "out of the box" with any Python 2.7.2+ (or 3.6+)
installation. It doesn't even need to be "installed" to be operational,
though an install script is provided.

It comes with five basic issue tracker templates

* a classic bug/feature tracker
* a more extensive devel tracker for bug/features etc.
* a responsive version of the devel tracker
* a jinja2 version of the devel template (work in progress)
* a minimal skeleton

and supports four database back-ends (anydbm, sqlite, mysql and postgresql).

Recent Changes
==============

>From 2.2.0 to 2.3.0

Fixed:
------

- Updated directions for verifying Roundup distribution using pgp.
- Dockerfile healthcheck fixed so it works when trackers are
  specified on command line. Also cleanup of unneeded
  packages. (John Rouillard)
- issue2551224 - Replace dbm db for sessions and otks when using
  sqlite. New databases are created for session data (db-session)
  and one time key data (db-otk). The data is ephemeral so no
  need to migrate. (John Rouillard)
- issue2551223 - Timestamps are truncated in mysql and postgresql
  for session and otk database tables. Modify db schema to use a
  numeric type that preserves more significant figures. See
  upgrading.txt for required steps. (John Rouillard)
- added more testing of BasicDatabase to support use of SQLite
  for that purpose. Had to fix memory, rdbms and dbm edge cases
  due to new tests. (John Rouillard)
- issue2551138 - roundup-server with ssl under python2 throws
  traceback on socket close. Not sure how this got fixed,
  but after fixing issue2551137 it was not an issue anymore.
- issue2551137 - roundup-server won't run with ssl under python3
  Fixed by using SocketIO and manually adding buffering io and
  catching SSL.ZeroReturnError indicating SSL has been shut down.
- add caching header for text/javascript in addition to depricated
  application/javascript. (John Rouillard)
- Enable postgres-fts: fix indexer-common::get_indexer so it returns a
  postgresql-fts Test code paths in get_indexer. (John Rouillard)
- Fix Postgres native-fts, implement a two phase initialization of the
  indexer. The native-fts one gets assigned after the database
  connection is open. (John Rouillard)
- fix crash if postgresql native-fts backend is asked to index content
  with null bytes. (John Rouillard)
- issue2551232 - modify in-reply-to threading when multiple matches
  Change how in-reply-to threading works in the mailgw. If there is
  more than one issue with a matching parent message, fall back to
  subject matching. See upgrading.txt for details. (John Rouillard)
- issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
- issue2551246 - mitigation, document how -u doesn't work for
  roundup-admin. (John Rouillard)
- Document better that files in the template or static_files
  directories accessed via @@file are available to any user with the
  url. (John Rouillard)
- Fix final exception handler in roundup-server to send proper
  Content-Length header to the client. (John Rouillard)
- Fix traceback if Origin header is missing. (John Rouillard)
- issue2551250: Fix sorting of detectors even if there are two with the
  same name and priority (can happen if they are created in two
  different files).
- Fix Traceback when a numeric order attribute is empty (Ralf
  Schlatterbeck)
- Update some template schema files to assign Register permissions for the
  Anonymous user. Replaces the old Create permission. (John Rouillard)
- Allow '*' and explicit origins in allowed_api_origins. Only return 
  'Access-Control-Allow-Credentials' when not matching '*'. Fixes
  security issue with rest when using '*'.  (John Rouillard)
- issue2551263: In REST response expose rate limiting, sunset, allow
  HTTP headers to calling JavaScript.  (John Rouillard)
- issue2551257: When downloading an attached (user supplied file),
  make sure that an 'X-Content-Type-Options: nosniff' header is sent.
  (John Rouillard)
- issue2551252 - default number of rounds for PKDF2 password increased
  to 2,000,000.  (John Rouillard)
- issue2551251 - migrate/re-encrypt PBKDF2 password if stored
  password used a smaller number of rounds than set in
  password_pbkdf2_default_rounds.  (John Rouillard)
- upgrade from jquery-3.5.1 to jquery-3.6.3. Update user.help.html
  to new version.  (John Rouillard)
- Dockerfile scanned with hadolint. Fixed multiple issues. (John Rouillard)
- fix crash due to invalid initialization/reset of configuration.py
  option_validators. Crashed roundup-admin on second command if an
  option_validator was added by a detector or extension. (John Rouillard)
- Dockerfile uses dumb-init to properly wait for child/zombie
  processes. Defense against child process starting from detector
  and becoming a zombie when its roundup-server instance exits.
  (John Rouillard)
- Move installed frontend/Zope back to frontend/ZRoundup
  directory. This better identifies the directory when copied into
  the Zope framework. It also matches existing
  documentation. (John Rouilard)
- Multiple fixes/updates for installation documentation.
  Including docker shell/admin/demo mdoes. (John Rouillard)
- Invalid item identifiers passed to REST endpoint return a 404
  rather than a 400 error. E.G. /rest/data/issue/issue4 (rather
  than .../issue/4). (John Rouillard)
- issue2551280 - sorted() method of MultilinkHTMLProperty is broken?
  (Gabor Nagy report and fix; commit John Rouillard)

Features:
---------

- Add warning about limited Python 2 support lifetime to install and
  upgrading docs. (John Rouillard)
- Dockerfile supports demo mode for instant gratification
  8-). Also supports shell and admin mode (John Rouillard)
- Dockerfile build allows adding additional python packages via
  pip, setting UID tracker is run under. (John Rouillard)
- issue2551140 - Added redis as a session and otk database for use
  with anydbm and sqlite primary databases. (John Rouillard)
- issue2550559 - Pretty printing / formatting for Number types.
  Added pretty(format='%0.3f') method to NumberHTMLProperty to
  print numeric values. If value is None, return empty string
  otherwise str() of value. (John Rouillard)
- sqlite native-fts backend now uses the stopwords list in config.ini
  to filter words from queries. (Stopwords are still indexed so that
  phrase/proximity searches still work.) (John Rouillard)
- sqlite databases use WAL mode when *created* to improve read
  concurrency. Existing sqlite database still use rollback journal
  mode. See upgrading.txt for details. (John Rouillard)
- issue2551233 - create new roundup-admin command "templates" list all
  template names, location and descriptions. Should help find where
  /usr/share/roundup/templates is buried during some install
  mechanisms. Does not need a tracker home to run. (John Rouillard)
- Add OAuth authentication to the mailgw script. Now IMAPS can be used
  with OAuth as required by several large cloud providers. Move command
  line processing of the mailgw script to ``argparse``. Note that the
  command line options of the mailgw have changed, see upgrading.txt for
  details. (Ralf Schlatterbeck)
- issue2551243: schema-dump.py enhanced with anti-CSRF headers. Flake8
  cleanup and python2 support. (John Rouillard)
- issue2551253 - new password hash PBDKF2-SHA512 added. Not
  available by default. Follow directions in upgrading document
  to use.  (John Rouillard)
- roundup-admin migrate command reports the schema version.
- issue2551262 - the mail gateway subject prefix now allows spaces
  before/after prefix. Also allow spaces between classname and id
  number in prefix designator. So "[ issue 23   ] subject" is parsed
  like "[issue23] subject". (John Rouillard)
- [doc]: add section on implementing CSP for Roundup to admin
  doc. (John Rouillard)
- issue2551265 - deprecate SSHA password hash method. Users using SSHA
  passwords will have their passwords transprently upgraded to PBKDF2
  derived hash on next login. (John Rouillard)
- issue2551253 - Modify password PBKDF2 method to use SHA512. New
  hash function using PBKDF2-SHA512 available. Will be default in
  future. Directions for upgrading security by using it now is
  included in upgrading.txt. (John Rouillard)
- issue2551275 - Allow configuring max_children in roundup-server.
  When using roundup-server in fork mode, allow raising number of
  forked children above the default of 40. (Original patch by Joseph
  Myers, config settings and docs by John Rouillard.)
- roundup-admin genconfig does not need a tracker home to run. (John
  Rouillard)
- issue2551190 - Allow roundup-admin reindex to work in
  batches. Running roundup-admin -i ... reindex issue:1-1000 will
  reindex the first 1000 issues while reporting any missing issues
  in the range. Also completion progress is reported when indexing a
  specific class.
- doc updates: add explanation for SQL code in 1.3.3->1.4.0 upgrade.
  document schema table in rdbms backends and how to dump/extract
  version from them. (John Rouillard)

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] Roundup rest API

[John P. R. <ro...@cs...>]

Hello Norbert:

In message <c05...@pd...>,
SCHLEMMER Norbert writes:
>Where can I find an "How-to" add a keyword / list of keywords to an issue ?

There isn't a How-To for that, but the info needed is in the rest
guide.  You are looking for the example that has the string:
rest/data/issue/@poe/viz in it.


To expand a little. To create a new item (status, issue, user etc.)
you POST a json data structure to the item endpoint:
tracker/rest/data/status for example.

To find out the json structure, perform a GET with ?@verbose=0
(described in the section "Using the POST method") on a similar
item. For example on my customized tracker:

a GET to: tracker/rest/data/status/8?@verbose=0 produces.

{
    "data": {
        "id": "8",
        "type": "status",
        "link": "tracker/rest/data/status/8",
        "attributes": {
            "abbreviation": "D",
            "help": "Ticket has been closed. Will not be worked on.",
            "name": "dead",
            "order": 8,
            "requiredpermissions": null,
            "transitions": [
                "1",
                "2",
                "4"
            ]
        },
        "@etag": "\"a1974134dfc8410fc46163032f063c0d\""
    }
}

The attributes json blob is what you want to replicate to create a new
status.  (Note this is from my tracker. It has attributes like
"abbreviation" and "transitions" yours will be different.)

To create a new status, I created the file /tmp/new_status.json with
this content:

 {
            "abbreviation": "Z",
            "help": "example status created via rest. Do not use.",
            "name": "zoid",
            "order": 11,
            "requiredpermissions": null,
            "transitions": []
        }


and submitted it using:

    curl -u admin:adminpassword -X POST \
    --data-binary @/tmp/new_status.json \
    -H "x-requested-with: rest" \
    -H "Content-Type: application/json" \
    -H "Origin: https://hostname" \
    -H "Referer: https://hostname/tracker" \
     https://hostname/tracker/rest/data/status

and got the result:

{
    "data": {
        "id": "10",
        "link": "https://hostname/tracker/rest/data/status/10"
    }
}

The x-requested-with and Content-Type headers are required. You may
not need all the rest of the header fields. I have all the cross site
request forgery protections engaged, so I need them.

The user can be any user who has create privs for the status
item. Usually this is anybody with the Admin role and usually only the
admin account has that role. You can see if another role has that
ability by running:

  roundup-admin  -i demo security | sed -ne '/^Role/p' -e '/create status/p'
  Role "admin":
  Role "agent":
  Role "anonymous":
  Role "provisional user":
  Role "seeabout":
  Role "seeaboutprivdetails":
  Role "setanystatus":
  Role "user":

So we can see that no other role has the ability to create a
status. (Note that the Admin role has the right to make CRUD operations
on all items. Security doesn't list every item for Admin. It prints
"User may create everything" instead.)

Does this help?

Have a great rest of the week.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Roundup-users] Roundup rest API

[SCHLEMMER N. <Nor...@pd...>]

Hello
Where can I find an "How-to" add a keyword / list of keywords to an issue ?

Thanks
Norbert

Re: [Roundup-users] Updating roundup

[John P. R. <ro...@cs...>]

Hello Gianluca:

In message <ef9...@tv...>,
Gianluca Ferrazzano writes:
>I am trying to update roundup from 1.4.x to 1.4.17 by following the 
>documentation and it says to run the script on the tracker but im not 
>sure what that means are you able to help

You should cut and paste the script into a file (for example
searchable.py). Then put the file in your tracker's home directory.

To run it, cd to the tracker home directory and run:

   python ./searchable.py

using the same python executable used to run roundup.

Just out of curiosity, how much experience do you have with Python?

Hope this helps.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] OperationalError: (1253, "COLLATION 'utf8mb3_bin' is not valid for CHARACTER SET 'utf8mb4'")

[John P. R. <ro...@cs...>]

Hi Gabor:

In message <20230616213123.150b935f@Dell>,
Nagy Gabor writes:
>Thanks for detailed response.
You're welcome.

>[...] I was very happy - and
>surprised - when I found Roundup, a very flexible bug tracker: That is
>exactly what I looked for, I did not believe such an open source
>project exists. It took a while until I understood how Roundup works,
>but now I find it very handy.)

I'm glad you are finding it useful. Please tell your friends 8-).

>First of all, I have no clue how I managed to create utf8mb4 encoded
>tables in my Roundup's sql database. Now I cannot create utf8mb4 table
>with Roundup 2.1.0 or 2.2.0 (no sql patches applied) even if I set
>mysql_charset = utf8mb4 in config.ini. This is mystery how my Roundup
>created such tables
>(I did not upgrade or configure mariadb after my utf8mb4 tables were
>created.)

Sorry, I don't have an answer for you.

>Now I think mysql_charset = utf8mb4 has no real effect in Roundup
>2.1.0. The only effect I see in the mysqldump of the database is:
>
>/*!40101 SET NAMES utf8mb4 */,
>
>but the databases still looks like this:
>
>CREATE TABLE `_status` (
>...
>) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci;

Hmm, the COLLATE looks wierd. Maybe that's how the utf8_general_ci
collation on the db gets displayed at the table level? (IIRC you said
utf8 is really utf8mb3 under the hood.)

>Based on your previous email, the following patch (of back_mysql.py of
>v2.1.0) solves the problems, plus the mysql_charset = utf8mb4 setting
>in config.ini:
>
>diff --git a/back_mysql.py.bak b/back_mysql.py
>index b0ec1c6..7b128a4 100644
>--- a/back_mysql.py.bak
>+++ b/back_mysql.py
>@@ -92,9 +92,9 @@ def db_create(config):
>     kwargs = connection_dict(config)
>     conn = MySQLdb.connect(**kwargs)
>     cursor = conn.cursor()
>-    command = "CREATE DATABASE %s"%config.RDBMS_NAME
>+    command = "CREATE DATABASE %s COLLATE utf8mb4_unicode_ci"%config.RDBMS_NAME

I may need to add a new collate setting or something.

>     if sys.version_info[0] > 2:
>-        command += ' CHARACTER SET utf8'
>+        command += ' CHARACTER SET utf8mb4'

This should be command += ' CHARACTER SET %s' %
config.RDBMS_MYSQL_CHARSET.  Using %s here I will claim is ok as the
source of the value is the administrator. Usually I use placeholders
tht are substituted at runtime to prevent SQL injection.

>     logging.info(command)
>     cursor.execute(command)
>     conn.commit()
>@@ -625,7 +625,7 @@ class Database(rdbms_common.Database):
>                 raise
> 
> class MysqlClass:
>-    case_sensitive_equal = 'COLLATE utf8_bin ='
>+    case_sensitive_equal = 'COLLATE utf8mb4_bin ='

Hmm, yeah..... I wonder if there is a standard for these. For a given
charset does:

   CHARSET_unicode_ci  (or CHARSET_generic_ci)
   CHARSET_bin

always exist?

>Discussion:
>1) The change to case_sensitive_equal = 'COLLATE utf8mb4_bin ='
>eliminates the Error in the subject of this email on utf8mb4 tables.

Good.

>(Unfortunately, utf8mb4_bin does not seem to be backward compatible,
>because if I do exact_match_spec search with this patch applied in an
>utf8mb3 table, I get the same error with the numbers 3 and 4
>interchanged. Great...)

Yeah, that makes sense. The mb3 and mb4 are different lengths and it
kind of makes sense they are incompatible. I assume you can convert
your mb3 tables to mb4 wuing SQL.

>2) The patches on db_create() has the effect that 'roundup-admin
>initalise' finally creates utf8mb4 encoded tables. I also tested the
>implicit table creation (when you add a new class to schema.py), those
>tables are also created with utf8mb4.

Great.

>(The "COLLATE utf8_general_ci" change of Roundup v2.2.0 is not enough to
>achieve this on my system.)

Understood. Looking back, I don't see why it would have worked.


>3) MySQL 8.0 documentation says this in
>https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html :
>"utf8: An alias for utf8mb3. In MySQL 8.0, this alias is deprecated;
>use utf8mb4 instead. utf8 is expected in a future release to become an
>alias for utf8mb4."
>So if we wait long enough, this problem might disappear automatically.
>:)

Well, for newly created tables yes. For people upgrading they will
start reporting the erros you ran into with mb4 queries running
against mb3 tables.

>+1) Of course, these hardcoded "utf8mb4 commands" in the source code are
>ugly (those are proof of concept codes), this should be somehow
>configurable, or the code should figure out (from mysql_charset) these
>codeparts.

If the CHARSET_* forms I mention above work, then I should be able to
handle the conversion. Apparently the unicode_ci forms perform worse
than the general_ci, but are more correct. There are also newer
varients unicode__unicode_520_ci. I hate to do it but I think I need
all three settings:

   mysql_charset
   mysql_collation_case
   mysql_collation_ignore_case

or some such.

At least one report I saw was that japanese collation was broken using
the unicode variants.

These changes won't make it into 2.3.0 I'm afraid. Unlike your other
change, this is playing with the data storage layer. So it needs some
more testing and planning. I guess I have to learn more mysql to
create tables and see if I can do some sort of correction/migration
via roundup-admin.

>P.S. I also thank you for your quick response to my bug report for
>the sorted() method of the class MultilinkHTMLProperty.

My pleasure. It was simple enough and just "looked right" (tm) (R). I
was able to walk though it by hand in the debugger and verify a few
cases. So I put it in untested. Not sure if it will come back to bite
me.

Have a great rest of your week.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Roundup-users] Updating roundup

[Gianluca F. <gia...@tv...>]

Hello

I am trying to update roundup from 1.4.x to 1.4.17 by following the 
documentation and it says to run the script on the tracker but im not 
sure what that means are you able to help

Thanks
Gianluca

-- 
Gianluca Ferrazzano
IT Executive
TVF
375 City Road
London EC1V 1NB

T:      +44 20 7837 3000
Email:  gia...@tv...
Web:    www.tvf.co.uk


This email, its contents and any files transmitted with it are confidential and may be legally privileged. It is intended solely for the addressee(s). If you are not the intended recipient, any disclosure, copying, distribution, or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this email in error, please notify us immediately via telephone or fax and delete the material from your computer system.

TVF screens electronic mail for viruses but does not warrant that this electronic mail is free of any viruses. TVF accepts no liability for any damage caused by any virus transmitted by this electronic mail and recommends that you subject these to your virus checking procedures prior to use.

Re: [Roundup-users] OperationalError: (1253, "COLLATION 'utf8mb3_bin' is not valid for CHARACTER SET 'utf8mb4'")

[Nagy G. <gab...@gm...>]

Hi John,

Thanks for detailed response. Today I had time for some testing.
It is important to note that I am not very familiar in MySQL, I just
report some observations. (One of the good things in Roundup that I do
not have to deal with direct SQL commands. I was very happy - and
surprised - when I found Roundup, a very flexible bug tracker: That is
exactly what I looked for, I did not believe such an open source
project exists. It took a while until I understood how Roundup works,
but now I find it very handy.)

First of all, I have no clue how I managed to create utf8mb4 encoded
tables in my Roundup's sql database. Now I cannot create utf8mb4 table
with Roundup 2.1.0 or 2.2.0 (no sql patches applied) even if I set
mysql_charset = utf8mb4 in config.ini. This is mystery how my Roundup
created such tables
(I did not upgrade or configure mariadb after my utf8mb4 tables were
created.)

Now I think mysql_charset = utf8mb4 has no real effect in Roundup
2.1.0. The only effect I see in the mysqldump of the database is:

/*!40101 SET NAMES utf8mb4 */,

but the databases still looks like this:

CREATE TABLE `_status` (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci;


Based on your previous email, the following patch (of back_mysql.py of
v2.1.0) solves the problems, plus the mysql_charset = utf8mb4 setting
in config.ini:

diff --git a/back_mysql.py.bak b/back_mysql.py
index b0ec1c6..7b128a4 100644
--- a/back_mysql.py.bak
+++ b/back_mysql.py
@@ -92,9 +92,9 @@ def db_create(config):
     kwargs = connection_dict(config)
     conn = MySQLdb.connect(**kwargs)
     cursor = conn.cursor()
-    command = "CREATE DATABASE %s"%config.RDBMS_NAME
+    command = "CREATE DATABASE %s COLLATE utf8mb4_unicode_ci"%config.RDBMS_NAME
     if sys.version_info[0] > 2:
-        command += ' CHARACTER SET utf8'
+        command += ' CHARACTER SET utf8mb4'
     logging.info(command)
     cursor.execute(command)
     conn.commit()
@@ -625,7 +625,7 @@ class Database(rdbms_common.Database):
                 raise
 
 class MysqlClass:
-    case_sensitive_equal = 'COLLATE utf8_bin ='
+    case_sensitive_equal = 'COLLATE utf8mb4_bin ='
 
     # TODO: AFAIK its version dependent for MySQL
     supports_subselects = False

Discussion:
1) The change to case_sensitive_equal = 'COLLATE utf8mb4_bin ='
eliminates the Error in the subject of this email on utf8mb4 tables.
(Unfortunately, utf8mb4_bin does not seem to be backward compatible,
because if I do exact_match_spec search with this patch applied in an
utf8mb3 table, I get the same error with the numbers 3 and 4
interchanged. Great...)

2) The patches on db_create() has the effect that 'roundup-admin
initalise' finally creates utf8mb4 encoded tables. I also tested the
implicit table creation (when you add a new class to schema.py), those
tables are also created with utf8mb4.
(The "COLLATE utf8_general_ci" change of Roundup v2.2.0 is not enough to
achieve this on my system.)

3) MySQL 8.0 documentation says this in
https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html :
"utf8: An alias for utf8mb3. In MySQL 8.0, this alias is deprecated;
use utf8mb4 instead. utf8 is expected in a future release to become an
alias for utf8mb4."
So if we wait long enough, this problem might disappear automatically.
:)

+1) Of course, these hardcoded "utf8mb4 commands" in the source code are
ugly (those are proof of concept codes), this should be somehow
configurable, or the code should figure out (from mysql_charset) these
codeparts.

Regards,
Gábor

P.S. I also thank you for your quick response to my bug report for
the sorted() method of the class MultilinkHTMLProperty.

> Hi Gabor:
> 
> I'm sorry this is becoming such a chore.
> 
> In message <20230530144304.3e55d8e3@Dell>, Nagy Gabor writes:
> >1) Thanks for the detailed information. First of all, MySQL's partial
> >(3-bit) utf8 implementation is a shame. I am not happy...  
> 
> I agree.
> 
> >I have to convert my Roundup databases to utf8mb4 to support all utf8
> >characters, and it is not so evident as it sounds (thanks for the
> >links, btw):
> >https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-conversion.html
> >I have to check that the "maximum permitted length" limitation does
> >not break anything, so I have to understand Roundup's SQL database,
> >which I did not wanted. :)  
> 
> I took a quick scan of the code in rdbms_common.py and back_mysql.py
> in the backends directory.
> 
> From that it looks like VARCHAR(255) is used for single character
> (I.E. ANSI/ASCII character set) represented values. For example
> encoded passwords, intervals. One Time Keys used to be VARCHAR but
> were converted to TEXT. So the conversion shouldn't matter as the
> fields will be the same size before/after.
> 
> The String type in Roundup is represented as TEXT in mysql. While
> strings can be large, files and message bodies are stored outside of
> the database. So in general, you shouldn't have BLOBs or
> multi-megabyte TEXT fields.
> 
> >2) After converting my test database to utf8mb4, and setting
> >mysql_charset = utf8mb4 in config.ini, the bug still persists, I got
> >precisely the same error message. Where does this utf8mb3_bin is come
> >from?... I even reinitalised the database with roundup-admin, with
> >no success.
> >
> >After doing a roundup-admin initialise with mysql_charset = utf8mb4,
> >mysql_dump showed lots of utf8mb3 databases, so its seems
> >mysql_charset had no effect. Strange.  
> 
> Indeed. I would expect that to have solved the problem.
> You are testing with 2.1.0 right? The changelog for 2.2.0 has:
> 
> - issue2551216 - create new mysql databases using COLLATE
>   utf8_general_ci to prevent crashes in test suite. (John Rouillard)
> 
> which was a no-op as far as I could tell with Python 3, but I may not
> have had the correct data in the db to trigger the issue. The patch
> for that change is:
> 
> diff -r eccb2f53566d -r 38e0fc1c7f11 roundup/backends/back_mysql.py
> --- a/roundup/backends/back_mysql.py    Tue Jul 05 19:45:12 2022 -0400
> +++ b/roundup/backends/back_mysql.py    Sun Jul 10 15:49:39 2022 -0400
> @@ -92,7 +92,7 @@
>      kwargs = connection_dict(config)
>      conn = MySQLdb.connect(**kwargs)
>      cursor = conn.cursor()
> -    command = "CREATE DATABASE %s"%config.RDBMS_NAME
> +    command = "CREATE DATABASE %s COLLATE
> utf8_general_ci"%config.RDBMS_NAME if sys.version_info[0] > 2:
>          command += ' CHARACTER SET utf8'
>      logging.info(command)
> 
> Maybe try making this change and re-test? Also I note that the
> hardcoded 'CHARACTER SET utf8' you mention below is seen in this diff.
> 
> >First I thought there are some problems in my
> >mariadb .cnf files, but I found some strange hardcoded lines in
> >backends/back_mysql.py:
> >
> >In db_create():
> >    if sys.version_info[0] > 2:
> >        command += ' CHARACTER SET utf8'
> >
> >In class MysqlClass:
> >    case_sensitive_equal = 'COLLATE utf8_bin ='  
> 
> Try changing the character set to use utf8mb4 and see if that
> helps. Running 'python3 -m pytest test/test_mysql.py' should turn up
> any major issues I hope.
> 
> My best guess is that you are onto something with
> 
>     case_sensitive_equal = 'COLLATE utf8_bin ='
> 
> The file rdbms_common.py sets
> 
>     case_sensitive_equal = '='
> 
> and uses that property in queries in _filter_sql for exact match if I
> understand the code correctly. This matches your observations about it
> only happening with exact match.
> 
> Ralf wrote that code so he would know if I am misinterpreting it
> incorrectly. There is a "utf8mb4_bin" collation. Maybe try changing
> 'utf8_bin' for 'utf8mb4_bin' see if it gets rid of the utf8mb3
> failure.
> 
> Maybe:
> 
>    https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci
> 
> will help? It seems to indicate that utf8mb4_unicode_ci or
> utf8mb4_general_ci are possible replacements for utf8_general_ci.
> 
>   https://forums.mysql.com/read.php?103,187048,188748#msg-188748
> 
> and the stackoverflow answer seem to indicate that my utf8_general_ci
> should be replaced by utf8_unicode_ci and I assume the same would be
> true for utf8mb4.
> 
> We don't target/recommend a specific version of MySQL/MariaDB. However
> MySQL 8.0 (with utf8mb4* collations IIUC) has been out since 2016, so
> more than the 5 years of support given my most LTS Linux releases. If
> the utf8mb4 variants work for you, we can consider requiring mysql >
> 8.0 for the 2.4.0 release next year.
> 
> The test suite could use some help from somebody who understands
> unicode better than I do and can create a test that sorts differently
> between utf8[mb4]_general_ci and utf8[mb4]_unicode_ci.
> 
> 
> --
> 				-- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] sorted() method of MultilinkHTMLProperty is broken?

[John P. R. <ro...@cs...>]

Hi Gabor:

In message <20230609120100.0780e9bf@Dell>, Nagy Gabor writes:
>I am using Roundup 2.1.0, and I cannot get the sorted() method of
>MultilinkHTMLProperty to work in the web interface.
>
>In schema.py, the class "issue" has the following definition:
>[...]
>When I try to list the documents of an issue, according to the Roundup
>documentation, I am trying to do this in issue.item.html:
>
><tr tal:repeat="i python:context.documents.sorted('d_feltoltes', reverse=True)">
>
>And this leads to "KeyError: 'd_feltoltes'", induced by the command 
>
>prop = self._db.getclass(self._classname).getprops()[property]
>
>in cgi/templating.py/MultilinkHTMLProperty/sorted().
>
>After replacing self._classname to self._prop.classname in that codeline,
>it seems to work as expected.
>
>Am I completely oversee something, or this sorted() method is broken,
>but nobody uses it?

I think the method is broken. That code (along with some other
templating code) is not covered by the test suite, so it is quite
possible this bug has existed for the past 2 years.

Your fix looks correct as far as I can tell by walking through the code
in the debugger.

I have committed your fix in:  changeset:   7478:d267b0454500

Thanks for the report and the fix.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Roundup-users] sorted() method of MultilinkHTMLProperty is broken?

[Nagy G. <gab...@gm...>]

Dear Users,

I am using Roundup 2.1.0, and I cannot get the sorted() method of
MultilinkHTMLProperty to work in the web interface.

In schema.py, the class "issue" has the following definition:

issue = IssueClass(db, "issue",
...
                   documents=Multilink("doc", rev_multilink="owner"),
...
)

The multilinked "doc" class has the following definition:

doc = FileClass(db, "doc",
...
                d_feltoltes=Date(),    # this is d_upload in Hungarian
...
)

When I try to list the documents of an issue, according to the Roundup
documentation, I am trying to do this in issue.item.html:

<tr tal:repeat="i python:context.documents.sorted('d_feltoltes', reverse=True)">

And this leads to "KeyError: 'd_feltoltes'", induced by the command 

prop = self._db.getclass(self._classname).getprops()[property]

in cgi/templating.py/MultilinkHTMLProperty/sorted().

After replacing self._classname to self._prop.classname in that codeline,
it seems to work as expected.

Am I completely oversee something, or this sorted() method is broken,
but nobody uses it?

As I see, this is not patched in Roundup 2.2.0 either.

Regards,
Gábor

PS. I have not fixed the previous mySQL issue yet, I will report the result here.

[Roundup-users] Roundup 2.3.0b2 released

[John P. R. <ro...@cs...>]

Hello Everybody:

I released Roundup 2.3.0b2 a few minutes ago.

(Beta 1 was broken and was using the 2.2.0 announcement.)

You can test it with:

  pip install roundup==2.3.0b2

in your venv. Docs for it are at:

   https://www.roundup-tracker.org/dev_docs/

and are up to date except that docs/upgrading.html seems to be cached
on an older version. Adding ?nocache=1 to the url returns the right
version. I wish there was some way to flush sourceforge's cache.

Please test the release. The last release escaped with a few bugs that
I had hoped would be caught during the alpha/beta period. Also since
the docs have been partly overhauled, checking those would be welcome
as well.

I am still planning on a 2.3.0 release on 13 July.

The announcement message, change log etc. is:

[...]

Among the notable improvements from the 2.2.0 release are:

* Dockerfile demo mode implemented.

* SQLite backends can use WAL mode to reduce blocking between readers
  and writers.

* Redis can be used for session database with SQLite and dbm
  backends. Provides a major performance improvement.

* roundup-mailgw can use OAUTH authentication to SMTP
  server. (roundup-mailgw command line options changed as a result.)

* Postgres full text index can now be enabled.

* Modifications to in-reply-to threading when there are multiple
  matches.

* Many updates to documentation to make it scannable, useful and
  work on mobile.

* Admin documentation includes a section on setting up Content
  Security Policy (CSP)

* REST now allows rate limiting headers to be accessed by client
  javascript.

* Default number of rounds for PBKDF2 updated to 2M to account for
  improvements in password crackers and CPU power.

* Support PBKDF2 with SHA512 for password storage

* Deprecate SSHA password hash function.

* roundup-admin reindex can be done in batches to managle load
  incurred by reindexing.

* roundup-admin can list avaailable templates and their installed
  locations.

* Crash fixes in detector handling, configuration handling, 

The file CHANGES.txt has a detailed list of feature additions and
bug fixes (51) for each release. The most recent changes from
there are at the end of this announcement. Also see the
information in doc/upgrading.txt.

If you find bugs, please report them to issues AT roundup-tracker.org
or create an account at https://issues.roundup-tracker.org and open a
new ticket. If you have patches to fix the issues they can be attached
to the email or uploaded to the tracker.

[...]

Recent Changes
==============

>From 2.2.0 to 2.3.0b2

Fixed:
------

- Updated directions for verifying Roundup distribution using pgp.
- Dockerfile healthcheck fixed so it works when trackers are
  specified on command line. Also cleanup of unneeded
  packages. (John Rouillard)
- issue2551224 - Replace dbm db for sessions and otks when using
  sqlite. New databases are created for session data (db-session)
  and one time key data (db-otk). The data is ephemeral so no
  need to migrate. (John Rouillard)
- issue2551223 - Timestamps are truncated in mysql and postgresql
  for session and otk database tables. Modify db schema to use a
  numeric type that preserves more significant figures. See
  upgrading.txt for required steps. (John Rouillard)
- added more testing of BasicDatabase to support use of SQLite
  for that purpose. Had to fix memory, rdbms and dbm edge cases
  due to new tests. (John Rouillard)
- issue2551138 - roundup-server with ssl under python2 throws
  traceback on socket close. Not sure how this got fixed,
  but after fixing issue2551137 it was not an issue anymore.
- issue2551137 - roundup-server won't run with ssl under python3
  Fixed by using SocketIO and manually adding buffering io and
  catching SSL.ZeroReturnError indicating SSL has been shut down.
- add caching header for text/javascript in addition to depricated
  application/javascript. (John Rouillard)
- Enable postgres-fts: fix indexer-common::get_indexer so it returns a
  postgresql-fts Test code paths in get_indexer. (John Rouillard)
- Fix Postgres native-fts, implement a two phase initialization of the
  indexer. The native-fts one gets assigned after the database
  connection is open. (John Rouillard)
- fix crash if postgresql native-fts backend is asked to index content
  with null bytes. (John Rouillard)
- issue2551232 - modify in-reply-to threading when multiple matches
  Change how in-reply-to threading works in the mailgw. If there is
  more than one issue with a matching parent message, fall back to
  subject matching. See upgrading.txt for details. (John Rouillard)
- issue2551195 - port scripts from optparse to argparse (Ralf Schlatterbeck)
- issue2551246 - mitigation, document how -u doesn't work for
  roundup-admin. (John Rouillard)
- Document better that files in the template or static_files
  directories accessed via @@file are available to any user with the
  url. (John Rouillard)
- Fix final exception handler in roundup-server to send proper
  Content-Length header to the client. (John Rouillard)
- Fix traceback if Origin header is missing. (John Rouillard)
- issue2551250: Fix sorting of detectors even if there are two with the
  same name and priority (can happen if they are created in two
  different files).
- Fix Traceback when a numeric order attribute is empty (Ralf
  Schlatterbeck)
- Update some template schema files to assign Register permissions for the
  Anonymous user. Replaces the old Create permission. (John Rouillard)
- Allow '*' and explicit origins in allowed_api_origins. Only return 
  'Access-Control-Allow-Credentials' when not matching '*'. Fixes
  security issue with rest when using '*'.  (John Rouillard)
- issue2551263: In REST response expose rate limiting, sunset, allow
  HTTP headers to calling javascript.  (John Rouillard)
- issue2551257: When downloading an attached (user supplied file),
  make sure that an 'X-Content-Type-Options: nosniff' header is sent.
  (John Rouillard)
- issue2551252 - default number of rounds for PKDF2 password increased
  to 2,000,000.  (John Rouillard)
- issue2551251 - migrate/re-encrypt PBKDF2 password if stored
  password used a smaller number of rounds than set in
  password_pbkdf2_default_rounds.  (John Rouillard)
- upgrade from jquery-3.5.1 to jquery-3.6.3. Update user.help.html
  to new version.  (John Rouillard)
- Dockerfile scanned with hadolint. Fixed multiple issues. (John Rouillard)
- fix crash due to invalid initialization/reset of configuration.py
  option_validators. Crashed roundup-admin on second command if an
  option_validator was added by a detector or extension. (John Rouillard)
- Dockerfile uses dumb-init to properly wait for child/zombie
  processes. Defense against child process starting from detector
  and becoming a zombie when its roundup-server instance exits.
  (John Rouillard)
- Move installed frontend/Zope back to frontend/ZRoundup
  directory. This better identifies the directory when copied into
  the Zope framework. It also matches existing
  documentation. (John Rouilard)
- Multiple fixes/updates for installation documentation.
  Including docker shell/admin/demo mdoes. (John Rouillard)
- Invalid item identifiers passed to REST endpoint return a 404
  rather than a 400 error. E.G. /rest/data/issue/issue4 (rather
  than .../issue/4). (John Rouillard)

Features:
---------

- Add warning about limited Python 2 support lifetime to install and
  upgrading docs. (John Rouillard)
- Dockerfile supports demo mode for instant gratification
  8-). Also supports shell and admin mode (John Rouillard)
- Dockerfile build allows adding additional python packages via
  pip, setting UID tracker is run under. (John Rouillard)
- issue2551140 - Added redis as a session and otk database for use
  with anydbm and sqlite primary databases. (John Rouillard)
- issue2550559 - Pretty printing / formatting for Number types.
  Added pretty(format='%0.3f') method to NumberHTMLProperty to
  print numeric values. If value is None, return empty string
  otherwise str() of value. (John Rouillard)
- sqlite native-fts backend now uses the stopwords list in config.ini
  to filter words from queries. (Stopwords are still indexed so that
  phrase/proximity searches still work.) (John Rouillard)
- sqlite databases use WAL mode when *created* to improve read
  concurrency. Existing sqlite database still use rollback journal
  mode. See upgrading.txt for details. (John Rouillard)
- issue2551233 - create new roundup-admin command "templates" list all
  template names, location and descriptions. Should help find where
  /usr/share/roundup/templates is buried during some install
  mechanisms. Does not need a tracker home to run. (John Rouillard)
- Add OAuth authentication to the mailgw script. Now IMAPS can be used
  with OAuth as required by several large cloud providers. Move command
  line processing of the mailgw script to ``argparse``. Note that the
  command line options of the mailgw have changed, see upgrading.txt for
  details. (Ralf Schlatterbeck)
- issue2551243: schema-dump.py enhanced with anti-CSRF headers. Flake8
  cleanup and python2 support. (John Rouillard)
- issue2551253 - new password hash PBDKF2-SHA512 added. Not
  available by default. Follow directions in upgrading document
  to use.  (John Rouillard)
- roundup-admin migrate command reports the schema version.
- issue2551262 - the mail gateway subject prefix now allows spaces
  before/after prefix. Also allow spaces between classname and id
  number in prefix designator. So "[ issue 23   ] subject" is parsed
  like "[issue23] subject". (John Rouillard)
- [doc]: add section on implementing CSP for Roundup to admin
  doc. (John Rouillard)
- issue2551265 - deprecate SSHA password hash method. Users using SSHA
  passwords will have their passwords transprently upgraded to PBKDF2
  derived hash on next login. (John Rouillard)
- issue2551253 - Modify password PBKDF2 method to use SHA512. New
  hash function using PBKDF2-SHA512 available. Will be default in
  future. Directions for upgrading security by using it now is
  included in upgrading.txt. (John Rouillard)
- issue2551275 - Allow configuring max_children in roundup-server.
  When using roundup-server in fork mode, allow raising number of
  forked children above the default of 40. (Original patch by Joseph
  Myers, config settings and docs by John Rouillard.)
- roundup-admin genconfig does not need a tracker home to run. (John
  Rouillard)
- issue2551190 - Allow roundup-admin reindex to work in
  batches. Running roundup-admin -i ... reindex issue:1-1000 will
  reindex the first 1000 issues while reporting any missing issues
  in the range. Also completion progress is reported when indexing a
  specific class.


--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Roundup-users] OperationalError: (1253, "COLLATION 'utf8mb3_bin' is not valid for CHARACTER SET 'utf8mb4'")

[John P. R. <ro...@cs...>]

Hi Gabor:

I'm sorry this is becoming such a chore.

In message <20230530144304.3e55d8e3@Dell>, Nagy Gabor writes:
>1) Thanks for the detailed information. First of all, MySQL's partial
>(3-bit) utf8 implementation is a shame. I am not happy...

I agree.

>I have to convert my Roundup databases to utf8mb4 to support all utf8
>characters, and it is not so evident as it sounds (thanks for the
>links, btw):
>https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-conversion.html
>I have to check that the "maximum permitted length" limitation does
>not break anything, so I have to understand Roundup's SQL database,
>which I did not wanted. :)

I took a quick scan of the code in rdbms_common.py and back_mysql.py
in the backends directory.

From that it looks like VARCHAR(255) is used for single character
(I.E. ANSI/ASCII character set) represented values. For example
encoded passwords, intervals. One Time Keys used to be VARCHAR but
were converted to TEXT. So the conversion shouldn't matter as the
fields will be the same size before/after.

The String type in Roundup is represented as TEXT in mysql. While
strings can be large, files and message bodies are stored outside of
the database. So in general, you shouldn't have BLOBs or
multi-megabyte TEXT fields.

>2) After converting my test database to utf8mb4, and setting
>mysql_charset = utf8mb4 in config.ini, the bug still persists, I got
>precisely the same error message. Where does this utf8mb3_bin is come
>from?... I even reinitalised the database with roundup-admin, with no success.
>
>After doing a roundup-admin initialise with mysql_charset = utf8mb4,
>mysql_dump showed lots of utf8mb3 databases, so its seems mysql_charset
>had no effect. Strange.

Indeed. I would expect that to have solved the problem.
You are testing with 2.1.0 right? The changelog for 2.2.0 has:

- issue2551216 - create new mysql databases using COLLATE
  utf8_general_ci to prevent crashes in test suite. (John Rouillard)

which was a no-op as far as I could tell with Python 3, but I may not
have had the correct data in the db to trigger the issue. The patch for
that change is:

diff -r eccb2f53566d -r 38e0fc1c7f11 roundup/backends/back_mysql.py
--- a/roundup/backends/back_mysql.py    Tue Jul 05 19:45:12 2022 -0400
+++ b/roundup/backends/back_mysql.py    Sun Jul 10 15:49:39 2022 -0400
@@ -92,7 +92,7 @@
     kwargs = connection_dict(config)
     conn = MySQLdb.connect(**kwargs)
     cursor = conn.cursor()
-    command = "CREATE DATABASE %s"%config.RDBMS_NAME
+    command = "CREATE DATABASE %s COLLATE utf8_general_ci"%config.RDBMS_NAME
     if sys.version_info[0] > 2:
         command += ' CHARACTER SET utf8'
     logging.info(command)

Maybe try making this change and re-test? Also I note that the
hardcoded 'CHARACTER SET utf8' you mention below is seen in this diff.

>First I thought there are some problems in my
>mariadb .cnf files, but I found some strange hardcoded lines in
>backends/back_mysql.py:
>
>In db_create():
>    if sys.version_info[0] > 2:
>        command += ' CHARACTER SET utf8'
>
>In class MysqlClass:
>    case_sensitive_equal = 'COLLATE utf8_bin ='

Try changing the character set to use utf8mb4 and see if that
helps. Running 'python3 -m pytest test/test_mysql.py' should turn up
any major issues I hope.

My best guess is that you are onto something with

    case_sensitive_equal = 'COLLATE utf8_bin ='

The file rdbms_common.py sets

    case_sensitive_equal = '='

and uses that property in queries in _filter_sql for exact match if I
understand the code correctly. This matches your observations about it
only happening with exact match.

Ralf wrote that code so he would know if I am misinterpreting it
incorrectly. There is a "utf8mb4_bin" collation. Maybe try changing
'utf8_bin' for 'utf8mb4_bin' see if it gets rid of the utf8mb3
failure.

Maybe:

   https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci

will help? It seems to indicate that utf8mb4_unicode_ci or
utf8mb4_general_ci are possible replacements for utf8_general_ci.

  https://forums.mysql.com/read.php?103,187048,188748#msg-188748

and the stackoverflow answer seem to indicate that my utf8_general_ci
should be replaced by utf8_unicode_ci and I assume the same would be
true for utf8mb4.

We don't target/recommend a specific version of MySQL/MariaDB. However
MySQL 8.0 (with utf8mb4* collations IIUC) has been out since 2016, so
more than the 5 years of support given my most LTS Linux releases. If
the utf8mb4 variants work for you, we can consider requiring mysql >
8.0 for the 2.4.0 release next year.

The test suite could use some help from somebody who understands
unicode better than I do and can create a test that sorts differently
between utf8[mb4]_general_ci and utf8[mb4]_unicode_ci.


--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.