srvx-devel — srvx development discussion

[srvx-devel] next generation database approach

[Entrope <en...@us...>]

There was some discussion today about what kind of database srvx
should use in its reincarnation.  There are three choices that I see:
whether to use multiple databases (versus just one), the kind of
database to use, and how to give other processes access to the data.

The question of multiple databases is mostly an issue if we have
in-memory, write-all-at-once databases (which we do in srvx-1.x).
Reading or writing all the databases for GamesNET at once can take a
few seconds, even on an Athlon 1 GHz.  If separate databases are used,
it is easier for them to be out of sync.  So this is mostly dependent
on the next issue.

I've seen three suggestions for the kind of database to use:

- The current recdb code (or a faster version of it, such as saxdb in
the CVS HEAD).  This is a hierarchical, string-keyed human-readable
database with several types of data.  The limitations we have seen
with it are: long read and write times; all the data being in-memory
at once (which will not be much of a problem until we grow a lot); and
lack of any way to do IPC operations on the data.  (Any other process
would have to talk to it through IRC, which is ugly and unreliable.)

- An out-of-process relational database, such as SQL.  This eliminates
the read and write time issues, moves the problem of what data to keep
in memory to another process, and easily allows others to access the
data.  However, it has drawbacks: any read or write becomes much more
expensive, due to the context switch (or even network access); we need
to add locking logic in many places; and our database schema is
constrained to what is in the database (or how we talk to it).  To be
more precise, updating the database schema is hard.

- A flat database with a relational information and serialization
layer on top.  This is approximately how my proposed "oodb" is
structured; Berkeley DB (or something similar) could provide the flat
database operations.  It solves the long read and write times (by only
writing some dirty data at once, it amortizes storing data over time)
and allows us to control how much data we keep in memory.  Direct
access to the database from other processes is probably harder than
with SQL.

The third issue is how to give other processes access to the data.  If
we use a standard format (or protocol) to store the data, we can
directly access it from other processes.  If we use a proprietary
format (or protocol), we would have to provide some sort of IPC
bridge.  This might be through IRC (eww), or might be through some
other direct TCP connection.

The drawbacks with giving other processes direct access to the data
are that we need to have rigid (and correct) rules about how to keep
the database consistent (this probably requires transactions) and
about how to lock parts of the database (or the whole thing) while you
work on it, and it is hard to extend the format.  If we provide an IPC
bridge, we can provide atomic operations and an extensible format
through that interface.  (SOAPy RPC over XML over HTTP!  Zoom
zooooom!!!</winer>)

If we look at what's practical, I see these choices:

1) Multi-file saxdb with IPC bridge
   - Status quo
2) One-file saxdb with IPC bridge
   - Probably too slow
3) SQL server with direct access
   - Buzzword compliant!
4) SQL server with IPC bridge
   - Almost no point in having the DB out of process
5) oodb on Berkeley DB with IPC bridge
   - Possible code reuse advantages; freedom of storage format
6) oodb on Berkeley DB with direct access
   - Possible code reuse advantages
7) oodb with proprietary DB with IPC bridge
   - Best control over caching and writing back data, but much code

Right now we use (1), but I think we want to get rid of it, and we
probably don't want (2) either.  Did I leave anything out?

-- Entrope

[srvx-devel] srvx-1.1 and srvx-2.0

[Entrope <en...@us...>]

(I have no idea if AfterNET is still interested in srvx or not.  If
not, somebody tell me, and I'll stop sending to the sr...@af...
list. :)

The current CVS HEAD for srvx (what we call -unstable) has been a
testing ground for a lot of next-generation features that would go
into srvx-2.0.  However, a lot of what we want is very awkward to do
in C, so we're probably going to end up rewriting it in C++ or C#.
And that will take time.

In the mean time, we plan on making a new fork from the current
srvx-1.0 tree (the srvx-1_0 tag) to work on a srvx-1.1, and release it
soonish.  Here's a list of what will be in 1.1 and what will only be
in 2.0 (we'll of course aim for 2.0 to have a strict superset of 1.1's
capabilities); the rough criteria were how much code it changes, and
whether it would be helped a lot by virtual functions:

IN SRVX-1.1
-----------
More restrictions on when/how "god mode" can be used
HelpServ (an automatic question queueing service)
Per-user flags in ChanServ
Additional notes for ChanServ (e.g. web page, staff notes on history;
  probably some will only be visible to staff)
Most of the stuff on the SourceForge feature request tracker
A few additional ChanServ commands (such as !giveownership)
Expanded proxy testing support (in particular, being able to match
  arbitrary length strings, so HTTP CONNECT proxies can be tested
  interactively)

SRVX-2.0 ONLY
-------------
Multiple protocol support
Multiple connection core (for DCC, IPC, etc.)
Dynamically loaded modules
"Pluggable" services (can assign commands to arbitrary service bots)
I18N (or at least translation of various strings)
Freeform aliasing
.. and most other wide-ranging changes

-- Entrope

[srvx-devel] Re: [srvx-commits] CVS: services ABOUT-NLS,NONE,1.1 Makefile.am,1.12,1.13 aclocal.m4,1.12,1.13 autogen.sh,1.4,1.5 configure.ac,1.7,1.8

[Entrope <en...@us...>]

Hm, I hadn't really thought about localizing them on a per-user basis.
But that would be easy enough to do -- gettext(), which is the
(standard) interface I plan to use, has a function that lets you
specify a particular language to use.  The I18N library backend will
then figure out the closest available match and use it:

  char *xlated;
  xlated = dgettext(handle->pref_lang, orig_text);

This could be dropped in somewhere at the equivalent of helpfile.c.

The reason I used CSMSG_FOO_BAR in the first place was to try to make
it easier to do I18N.  It turns out that the GNU way of doing it, by
putting _() or N_() around translatable strings, is easier and more
flexible -- at least if the string is only used once.

-- Entrope

Adrian Dewhurst <dew...@ed...> writes:

> You know, I was thinking about I18N a week ago... thanks for the commits
> to remind me
> 
> With all the CSMSG_*, etc defines, think it would be feasible to make the
> language choosable by an authserv handle setting? Just a thought. Here's
> one method I thought might do it:
> 
> The defines convert to numeric values. There are arrays of messages for
> each language, and it uses lang_english[CSMSG_FOO_BAR]... the command that
> does the sending of messages takes the authserv info struct from the user
> it's to be sent to and decides the language to use from there... if there
> is no handle, it uses the default (via srvx.conf, naturally).
> 
> I actually can see large problems in using an array, but it'd be the
> fastest, and easiest to explain... in practice, another method would
> probably be required, though it gets the idea across. I do think I18N
> support in srvx would require it to be per-user and not
> per-network/process in order to be very useful. Myself, I could assist
> with French translations, though they would be far from perfect. We'd also
> need separate help files.
> 
> I was too lazy to look up the srvx devel list address before I go to sleep
> tonight, so if you don't mind forwarding it to the list address, then feel
> free to do so ;-)

Re: [srvx-devel] srvx development directions meeting

[Entrope <en...@us...>]

Minutes for the meeting:

Next stable version tentatively called 1.1
We definitely want modules for 1.1; dependency support needs to be improved
Need to flag possible abuse better (e.g. !voice self, !addop self in god mode -- or disallow and make another staff do it)
#1, #2, #3 in 1.1, using Zoot's rule ACL stuff
DCC/bot support in 1.1
HelpServ is extremely positively received; it goes in, too
Renaming ChanServ access levels is delayed
Fisich notes that having OpServ and ChanServ semi-separate is important; running two instances of srvx is an acceptable solution
Extra channel information (notes, problem histories, etc.): new idea; put just a simple list of notes in 1.1, more stuff can wait
Per-<user,channel> flags for ChanServ: just for plain !togop
Better help files: Not really defined well, but should include things like task-based help entries
Entrope and/or SailorFrag will keep list of help file entries that need to be changed
Database journalling: defer (too hard)
Rule based auth: can also use it with ?trace, etc
Rules:
  being in channel N
  user mode matches +N-M
  opserv level at least N
  channel mode +M-P in channel N
  access level at least N in channel M
  handle flags +N-M
  hostmask M!N@P
  unauthed for duration N
  unvisited channel N for duration M
  handle name matches M
  true
  false

-- Entrope

[srvx-devel] Re: srvx development directions meeting [resend]

[Entrope <en...@us...>]

It appears that my first attempt at sending this failed (I got two
corrupted copies, someone else said they didn't get it at all).
Sorry if anyone gets it twice.

--

Due to outside constraints, I cannot make the meeting tonight.  def
cannot either.  If people still want to attend, I hope Zoot can
organize it or something (he has the list of things I wanted to
discuss); otherwise, I plan to do it next Friday instead.

I apologize for any inconvenience this causes.

-- Entrope

Re: [srvx-devel] srvx development directions meeting

[Entrope <en...@us...>]

Due to outside constraints, I cannot make the meeting tonight.  def
cannot either.  If people still want to attend, I hope Zoot can
organize it or something (he has the list of things I wanted to
discuss); otherwise, I plan to do it next Friday instead.

I apologize for any inconvenience this causes.

-- Entrope

[srvx-devel] srvx development directions meeting

[Entrope <en...@us...>]

I'd like to have an informal meeting sometime soon about development
directions for the next version(s) of srvx.  Since 1.0 is almost
stable enough for release, I'd like to get ideas about what everyone
thinks is most important for the 1.2 release.

Changes that are already on the cvs HEAD:
- Port to other IRC server dialects (bahamut so far)
- Dynamically loadable modules (only lightly tested)
- Some general cleanup

Things that have been suggested (or non-trivial TODO items):
- Freeform command aliasing ("?hostkill foo" -> "?trace kill host foo")
- Merging services (or otherwise exposing commands in different services)
- More rules about who can execute commands (for example, on GamesNET,
  helpers should have to be in #support and/or #ecsi to use their
  "god mode")

(I'm sure I've forgotten a bunch of things on both those lists, but
those are the biggest ones that come to mind.)

Anyway, the point of the meeting would be to put priorities on things
and figure out what we want in the 1.2 branch and what we want to put
in 2.0.  Zoot has an architecture that covers the three TODO items I
listed above, but is probably too wide-ranging to put in 1.2; but I'd
like to see 1.2 more usable out-of-the-box by Afternet.

So, I'd like to have the meeting this Friday (August 31) at 8:30 PM
EDT (00:30 UTC) in #coders on GamesNET.  Anyone is welcome to attend.

-- Entrope

[srvx-devel] bugs to fix for 1.0

[Entrope <en...@us...>]

Now that Jedi killed most of the bugs in proxy checking, we've only
got a few bugs left to fix before I'd feel good about releasing a
final 1.0 release:

 * vsend_message() sometimes does "$b" wrong at the end of a line.  I
   have a comment about it at helpfile.c:127, but didn't figure out
   exactly why it sometimes makes 'b' visible at the start of the
   second line.

 * timeq bug; sometimes seems to forget events if they happen long
   enough in the future.  No idea why this happens.

 * Ideally, we could make ProxyCheck support a '.*' matcher to go
   with its '..' matcher, and it could search at different offsets
   forward.  This will complicate the code, but let us do better
   checks for things like HTTP proxies and Code Red.

 * Seldon reports that sometimes "/msg authserv handleinfo whoever"
   will report that "whoever is on vacation." even if they've authed
   since going on vacation and logging off.

 * IP hostmatching bugs - easy enough to fix; we can kill parse_ipmask
   and replace that bit of code in user_matches_glob() with an sprintf
   of their IP into a buffer and then a compare

 * All NickServ searches return no hits.  I think I know why.

I'll work on the last three -- I expect they'll be easy to fix
(tonight).  The first two need more debugging.  Any takers to work on
them?  (The third we might be able to put off until 1.1.  Is it worth
taking chances in 1.0 for?)

-- Entrope