Re: [TCLCORE] TCT/TIP reform

[Steve L. <st...@di...>]

Thanks for your thoughts Gustaf.

I’m keen for us to focus on the initial question of expanding the TCT and adding more policy and procedure to the TIP process, as outlined in my initial email.  The question of the tools can be handled in time.

Having said that, I will note that the discussion of GIt vs Fossil vs whatever is not a black-and-white decision and that there are pros and cons.  While Fossil may be a hurdle for those more familiar with Git (or rather, Github) the reverse is also true. Let’s kick that can down the road and deal with it as an implementation issue.  Personally I would be comfortable using whatever the majority of core developers prefer.  In the meantime you might want to read https://fossil-scm.org/forum/forumpost/84949ed7ec19dc03cc32108d8ccfec47e4de2b73fc4e326e219850c9e8d4fa7f.

And (as Rolf reminded me yesterday) we still have to get 9.0 out the door.

-- Steve
On 16 May 2023 at 5:10 AM +0800, Gustaf Neumann <ne...@wu...>, wrote:
> Dear all,
> A few cents from my point of view. I was exposed to the Tcl source code for some longer time (XOTcl, NSF, ...) and act currently as the maintainer of NaviServer (multi-threaded server, C code) and partly of OpenACS. Several companies based their products on Tcl + NaviServer, having with significant usage numbers, which are often reluctant to changes. I see myself mostly a user of Tcl ecosystem, since I do not have currently the time to follow in detail the tcl-core discussions, so please apologize for some off-topic comments. I am also a contributor to some other larger C based software packages with quite different contribution rules. I do understand the voices asking for some kind of TCT/TIP reform. From my experiences, the existing mechanisms served us well so far, the core team members react quickly to serious problems, being constructive and helpful. But certainly, we have some aging problems (as other projects as well). With some shudders, I remember a conference of a language community smaller than the Tcl community, where most participants looked like 70+ (like a retirees' club excursion). From my point of view, we should not make life more complicated than necessary for potential contributors, but instead, reduce some of the hurdles to increase the number of potential contributions. It would be great if contributing would become easier and more transparent, also for "part-time" contributors, which should not have to have write permissions on the central repository. In particular, we should find ways to accept pull requests which are the standard for contributing in many projects, opening contributions especially for non-core members. A PR serves as a means for communication and discussion, has typically transparent decision processes (a certain number of reviewers), can be linked with automated testing and conflict resolution. It can be probably setup in a way, that different kinds of reviewers (C, Tcl, large users) can serve in an acceptance process. Although fossil has many nice features, for me, it is more of a hurdle than a relief. Maybe there is a way to sync fossil with git bidirectional, such that PR can be accepted as well.
> all the best -g
> On 12.05.23 09:25, Arjen Markus wrote:
> > Hi everyone,
> >
> > I find Steve's proposal regarding TCT/TIP quite refreshing. I have not felt qualified to comment on the C side of things, a matter of involvement in the code, no expertise in the topics that brought on much of the discussion, limited time for digging into it and, more importantly perhaps, mostly working with the scripting side of things. But I do have an opinion on some aspects :), having experimented with various forms of using Tcl in combination with another language.
> >
> > Just wanted to say this. I am eager to see what this results in.
> >
> > Regards,
> >
> > Arjen
> >
> > From: Steve Landers <st...@di...>
> > Sent: donderdag 11 mei 2023 02:16
> > To: tcl...@li...
> > Subject: [TCLCORE] TCT/TIP reform
> >
> >
> > Caution: This message was sent from outside of Deltares. Please do not click links or open attachments unless you recognize the source of this email and know the content is safe. Please report all suspicious emails to "Ser...@de..." as an attachment.
> >
> > During the last few months there has been a lot of friction in the Tcl community resulting from concern about the direction of some changes being made to the core. Although these now look to have been resolved there was a lot of time, effort and emotional energy expended to get there. Not to mention the wasted effort on the TIPs that were replaced.  And there are a number of other issues simmering with respect to implementation and coding style as evidenced by ongoing chatter on the Tclers Chat and mailing list.
> >
> > I have been reflecting on the situation and wondering how we might reduce the friction and turn activity into achievement,  working from the assumption that all involved have the best interests of Tcl/Tk at heart.   In the process I have leaned on several Tclers whose wisdom and opinions I greatly value:  Ashok Nadkarni, Brian Griffin, Scott Goodwin and Kevin Walzer.  This document records my thoughts in the hope of stimulating a discussion and concrete proposals that lead to positive change.
> >
> > My conclusion is that the TCT/TIP process mostly works but needs improving. Let me explain why ...
> >
> > TIP #0 is the foundation of the TCT. Although it was written for a different time and different circumstances the objective remains the same: to stimulate progress within reasonable constraints and avoid doing harm.  It mostly works but the TYANNOTT policy (Two Yesses and No Nos or Two Thirds) breaks down at a time when there is a flood of TIPs and not enough active TCT members for adequate review. Also, TIP #0 talks about three stages of a proposed change - approval, implementation and integration - but only the first has been formalised.  And finally there is a broader issue,  the Tcl/Tk release cadence is glacially slow when compared with other Open Source projects, which means useful features may not get wide adoption for years, if not decades. That results in pressure to get proposed changes into a release when perhaps they are not ready.
> >
> > So this proposal addresses three broad areas in TIP #0 that need refining:
> >
> > • the need for broader participation in reviewing and voting on TIPs
> > • the need for a more formal implementation and integration process once a TIP is approved
> > • the need for an improved release cadence
> >
> > Reviewing and Voting on TIPs
> >
> > There are currently 13 members of the TCT of which three have not been active recently, which  leaves a small group of people available for reviewing proposals. Based on recent voting it is apparent that many of active members have struggled to keep up with the load of reviewing TIPs, myself included.  The result is that many recent TIPs have passed with just a few YES votes and (arguably) have not had adequate scrutiny.   This is worse if a particular developer is running ahead of the pack and identifying problems that need addressing.
> >
> > So what's the solution?  I've come to the conclusion that we need a lot more people on the TCT, to spread the load and get more eyes viewing proposals.
> >
> > But what sort of people?
> >
> > Tcl is more than the core Tcl language , it includes the APIs to access the core, the documentation and the distributions.  And currently we may have more core maintainers outside of the TCT than within, a situation that was not foreseen in TIP #0.   So (in my opinion) the TCT should not just include core developers but also need people with an interest in the stability, performance and direction of the core. That would include people who depend on the APIs such as extension maintainers, people who have experience in deploying applications based on Tcl, people who integrate Tcl (e.g. distribution maintainers), and people who maintain interfaces to Tcl and Tk from other languages. Complementary skills are needed: some people will be better at core coding, some at code review, others at API design, others at documentation and communication.
> >
> > And who should the people be?
> >
> > I suggest any long term active community member who is well respected and who is willing to serve should be considered.  Especially people who have to interface with Tcl (and live with the consequences of bad API design) and those who have deployed systems based on Tcl (so their knowledge is experiential and not just theoretical).  The more the merrier, in the hope that at least a few more people with different skills and interests will be voting on each TIP. Put differently (and perhaps ungraciously) this is an attempt to dilute the influence of any one individual and to harness the wisdom of the crowd.
> >
> > Sure some people can be "prickly" (can't we all at times?) but most long term members of the Tcl community have the best interest of Tcl at heart and have a critical mindset that will lead to potential problems being identified earlier and addressed in a way that is widely acceptable.  Hopefully the various skills and weaknesses will balance and we'll have more active involvement.
> >
> > One other consideration is we might choose to have Tcl and Tk subgroups, and inviting people from other language communities to get involved in the Tk language direction.
> >
> > I further suggest that TCT members should be expected to be involved in a vote or review (more on that later) at least once per year, otherwise they are marked as "inactive".  Not that there would any consequence to being inactive, since the TCT voting is 2/3 of those present not of active members.
> >
> > TIP Implementations
> >
> > The question of implementation needs clarification. Should an implementation be required before voting on a TIP, given that it commits the proposer to work without any guarantee of the work being used? Over the years it has been implied that this should be the case, but there is nothing in TIP #0 that says this - arguably the opposite.   This needs to be clarified.
> >
> > I can see both sides of the argument: on one hand why should someone invest time and energy with no indication of whether it will be used,  on the other hand why should a proposal be evaluated if there is no guarantee an implementation will be forthcoming.
> >
> > We could explicitly to allow TIPs to be voted on without an implementation (as happens now) but there is a risk this will be seen as a “Feature Request” and not a commitment to implement if approved.
> >
> > Another alternative is to require an implementation before a vote, but to add extra states to the TIP workflow to explicitly require a discussion phase before implementation.  This will give some indication to the TIP proposers as to whether a TIP has a good chance to succeed before they expend effort on an implementation.  Currently we have Draft, Voting, Accepted and Final (plus Rejected or Withdrawn or Obsolete).   What if we refined the TIP system to support Draft, Discussion, Implementation, Voting, Accepted and Final (i.e. integrated) plus Rejected/Withdrawn/Obsolete?  In other words require an implementation before a vote but only after the developer has “tested the waters”.  A larger TCT and more engagement from the community will make this more practical.
> >
> > Feature integration
> >
> > Integration is desperately in need of a more formal process:
> >
> > • there is no formal mechanism to ensure that code merged to trunk actually reflects the TIP or meets acceptable code quality standards
> > • there have been recent instances of TIPs and implementations being changed during a discussion, making it difficult to assess the merits of a proposal
> > • core devs are branching off different locations: some branch off 8.7 and merge changes forward, others branch off 9.0 and merge backwards - making it more difficult to support parallel development and for reviewing changes
> > • some branches contain multiple, intertwined changes so it is difficult to review individual changes
> > • there have been instances of people re-writing other people’s code in amongst functional changes:   some were egregious - e.g. changing spacing - while others were more subtle, but the net effect is to make it difficult to identify and review functional changes
> >
> > My suggestion is that we formalise the integration process, along the lines of:
> >
> > • there should be one branch per TIP containing only changes pertaining to that TIP (for ease of review)
> > • every feature branch must be based on the latest development branch (i.e. Tcl 9.0 at the moment) for consistency and ease of review
> > • there should be a review and formal sign-off before by a TCT member before integration into the trunk
> > • the branch must pass the test suite on key platforms before being submitted for review (it may be possible to automate this through the existing Continuous Integration facility)
> >
> > Looking at the process for approving a merge, it needn't be onerous. Even a single reviewer signing-off on a merge would be better than the current situation.   That means a single positive code review is sufficient but if someone objects then a 2/3 vote is needed to integrate.  Does this add complexity?  Definitely.  Would it avoid the "commit wars"?  Hopefully. Would it lead to a better core? Surely.
> >
> > Release procedures
> >
> > With more people involved and a commitment to more regular releases we could start addressing the load on the release manager.  This might include documentation of the steps involved in creating a release with a view to enlisting the help of others,  automation where possible and checklists to monitor progress.
> >
> > Conclusion
> >
> > Change is needed but it can be evolutionary rather than revolutionary.  With that change will come inevitable pressures to change policy and procedures but those can be dealt with in time.  However the current situation is wasting time and energy that could be better directed to improving Tcl, Tk and associated technologies.
> >
> > I intend to turn this proposal into a series of TIPs after a suitable period for feedback and discussion.
> >
> > Steve Landers
> > May 2023
> > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core