Thread: [SQLObject] Re: problem: implementing BoolCol
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: Nick <ni...@dd...> - 2003-04-30 17:41:46
|
[ earlier discussion about implementing different SQL types and converting them to Python types ] I've been looking at the code for Constraints and Col, and it looks to me like there needs to be a distiction between real constraints (MaxLength, Unique, Not Null, for example) and type checking/casting. It's a little mixed right now. In terms of type checking and conversion, I would propose: 1) make validators part of Col instead of just random functions in Constraints that get referenced in Col. 2) at the rate it's going, there are going to be a lot of parallel _mysql, _postgres, _sqlite, etc. functions for every operation under Col. There needs to be a way to bundle a bunch of functions for the particular db api, probably related to whatever _connection type you're using. 3) for each db api, there needs to be convert-from-result-to-python and convert-from-python-to-sql-value functions as part of the Col class, encapsulated by the mechanism described in 2. I've got some ideas how to do this, but I don't want to go ripping through the code if you're going somewhere else with this. Nick |
From: Ian B. <ia...@co...> - 2003-05-05 17:51:49
|
On Wed, 2003-04-30 at 12:41, Nick wrote: > [ earlier discussion about implementing different SQL types and > converting them to Python types ] > > I've been looking at the code for Constraints and Col, and it looks to > me like there needs to be a distiction between real constraints > (MaxLength, Unique, Not Null, for example) and type checking/casting. > It's a little mixed right now. In terms of type checking and > conversion, I would propose: > > 1) make validators part of Col instead of just random functions in > Constraints that get referenced in Col. Yes, that's possible, i.e., StringCol checks for strings (or maybe calls str() on Python objects, for instance, or checks the length). And, I suppose, you could add new types for specific requirements, like numbers in a certain range (e.g. IntCol(min=0)) By making column types and validators separate it should be possible to use richer checks on objects. > 2) at the rate it's going, there are going to be a lot of parallel > _mysql, _postgres, _sqlite, etc. functions for every operation under > Col. There needs to be a way to bundle a bunch of functions for the > particular db api, probably related to whatever _connection type you're > using. It's challenging, because there's more than one way things can be built out -- types may be added over time, or databases may be added. > 3) for each db api, there needs to be convert-from-result-to-python and > convert-from-python-to-sql-value functions as part of the Col class, > encapsulated by the mechanism described in 2. > > I've got some ideas how to do this, but I don't want to go ripping > through the code if you're going somewhere else with this. I've been working on a validation/conversion system for a form processor. I'm fairly happy with what I've got, at least for forms, and it could be applied to SQLObject -- though I wonder if it's too heavy, since it would be invoked for every get or set of a column. It makes composition and definition of validators easier, but maybe that's not as important for something like SQLObject, where there's a lot more work in defining an object than just its validators. Anyway, I'd be interested on some feedback on it. You can see the system here: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/webware-sandbox/Sandbox/ianbicking/FormEncode/ Validator.py more specifically has the validation code. Doc strings may not be up to date (it's based on previous system, and the method names have been changed -- attemptToPython and attemptFromPython are public methods) Ian |
From: Nick <ni...@dd...> - 2003-05-05 18:50:13
|
On Mon, 2003-05-05 at 12:52, Ian Bicking wrote: > On Wed, 2003-04-30 at 12:41, Nick wrote: > > 3) for each db api, there needs to be convert-from-result-to-python and > > convert-from-python-to-sql-value functions as part of the Col class, > > encapsulated by the mechanism described in 2. > > > I've been working on a validation/conversion system for a form > processor. I'm fairly happy with what I've got, at least for forms, and > it could be applied to SQLObject -- though I wonder if it's too heavy, > since it would be invoked for every get or set of a column. It makes > composition and definition of validators easier, but maybe that's not as > important for something like SQLObject, where there's a lot more work in > defining an object than just its validators. I haven't seen the form stuff you're talking about (cvs.sf.net doesn't respond at the moment), but you only really need to do the conversion when updating the db, which is another case for pythonic transaction handling. Validation in most cases is lightweight, even if you're only going to match against a re, but necessary if you want to catch problems before they happen. If you didn't want validation, should should have used Col. Either that, or you can subclass XxxCol and override the validator to pass, assuming you move the validator into the class. Nick |
From: Ian B. <ia...@co...> - 2003-05-05 19:38:27
|
On Mon, 2003-05-05 at 13:49, Nick wrote: > I haven't seen the form stuff you're talking about (cvs.sf.net doesn't > respond at the moment), but you only really need to do the conversion > when updating the db, which is another case for pythonic transaction > handling. Validation in most cases is lightweight, even if you're only > going to match against a re, but necessary if you want to catch problems > before they happen. If you didn't want validation, should should have > used Col. Either that, or you can subclass XxxCol and override the > validator to pass, assuming you move the validator into the class. It should go both ways, so you can define the class of the result -- maybe unpickling a string, or creating a Point class, etc. Essentially it came down to a Validator object with two defined methods -- fromPython and toPython. But there were a bunch of other options that may not apply to SQLObject. Validation may be important when fetching from the database, if SQLObject doesn't have total control of the database, and the constraints aren't as restrictive as the validation. That might be an obscure case, though. Ian |
From: Nick <ni...@dd...> - 2003-05-05 21:58:10
|
On Mon, 2003-05-05 at 14:39, Ian Bicking wrote: > Validation may be important when fetching from the database, if > SQLObject doesn't have total control of the database, and the > constraints aren't as restrictive as the validation. That might be an > obscure case, though. Conversion is important, especially for DBM and SQLite, which only store strings, coming out of the database. Hoever, *validating* data coming back out of the database can be devastating to an application if it can't control the data going into the database from another source. You'd never be able to retrieve a row, even to fix the data, if you got an exception because the data already in there is bad. Nick |
From: Ian B. <ia...@co...> - 2003-05-06 07:12:59
|
On Mon, 2003-05-05 at 16:57, Nick wrote: > On Mon, 2003-05-05 at 14:39, Ian Bicking wrote: > > Validation may be important when fetching from the database, if > > SQLObject doesn't have total control of the database, and the > > constraints aren't as restrictive as the validation. That might be an > > obscure case, though. > > Conversion is important, especially for DBM and SQLite, which only store > strings, coming out of the database. Hoever, *validating* data coming > back out of the database can be devastating to an application if it > can't control the data going into the database from another source. > You'd never be able to retrieve a row, even to fix the data, if you got > an exception because the data already in there is bad. DBMConnection actually is more shelve-like, pickling all its data, so you get full types in and out. But if you want to move to another backend, validation to restrict that flexibility would serve you well, so it's not beyond all of this. At first I thought: yes, validating input from the backend isn't very useful, since there's no good way to recover in the case of invalid input. But on second thought, one design decision that I feel is good in Validator (and its predicessors) is that validation and conversion are a single process, because they are whether you want them to be or not. The easiest example would be for an integer (from a string). The converter would be int() itself (convenient :). But then there's an implicit validation too, because int("abc") isn't going to work. You will get an exception. So you need a way to handle these exceptions, whether or not you want to. How you do that isn't clear to me... in some cases they may be an example of corrupt data in the database (an all too common occurance). Throwing an exception is a pain, but really the only thing you can do at that point is to provide an alternate mechanism so a person can fix that. Doing direct queries is one possibility right now. In another instance, your validation is too tight. It's a bug, and should be fixed in your code. You should always be able to do obj.x = obj.x, and I think it deserves an error if something gets in the way of doing that (and if the validation is really odd or eclectic, you should be overriding _get_x and _set_x so you can be as eclectic as you want). That error will show up eventually -- better sooner than later. But, in both cases it's a real pain to deal with it automatically or programmatically. But it's not clear to me how it should be dealt with. Using Validator you can actually be explicit about it, like: Validator.Int(ifInvalid=None) So "x" becomes None when you validate it. Or you could do: Validator.Any(Validator.Int(), Validator.Validator()) Which if an input can't be turned into an integer, it returns it as is (Validator.Validator incidentally being the identity function of validators). That said, I'm still open to more ideas, but handling exceptional behavior here is a pain in the but, most definitely. Maybe Validator could distinguish between validating input and validating output (which it doesn't currently -- it just distinguishes conversion). Then you could at least choose. I think with some more long-winded programming, even with its features Validator should be fairly efficient (i.e., multiple implementations of attemptToPython and attemptFromPython, so you only pay for the features you use). Of course, I don't have any performance data at all, so maybe that's premature optimization... Ian |