Krang

Hey Sam --=20

> * Analysis
>=20
> This problem occurs because Krang relies solely on URLs to match
> stories during import.  Since the URL has changes Krang has no way to
> know that the stories match between beta and production.  This problem
> occurs in all cases where Krang allows users to change the key it uses
> to match objects during import - users, groups, media, templates,
> categories and sites all have this problem.
>=20
> * Solution
>=20
> I propose to add a universally-unique identifier (UUID) to Krang
> objects affected by this problem.  For example, the story table would
> gain a story_uuid column containing a UUID which is guaranteed to
> never change.  These UUID columns would be added to the export XML
> format.  On import Krang would prefer matches on UUIDs, falling back
> to the current behavior when no match is found.


I definitely see the problem you describe.  It's even more pronounced
with Krang list items, which have no URL.  (And Contributors, too, I
imagine.)

You'll have to be careful about this approach.  Right now the logic for
understanding under what circumstance a KDS will update or add an object
is fairly straight-forward.  A "UUID" might end up just being another
breakable identifier.  You might not want to have a fall-back, at all.
Maybe you simply want UUIDs to be the only way items are identified from
now on?

How will you prevent URL collisions with this scheme, if URLs are no
longer the unique identifier?

What are you going to base your UUID on?  How will you assure that UUIDs
are never used more than once, even across multiple servers?  (I.e., You
can't use a "auto-increment" INT.)  MD5?  Is that unique enough?


-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

On Wed, 25 Oct 2006, Jesse Erlbaum wrote:

> You'll have to be careful about this approach.  Right now the logic for
> understanding under what circumstance a KDS will update or add an object
> is fairly straight-forward.  A "UUID" might end up just being another
> breakable identifier.  You might not want to have a fall-back, at all.
> Maybe you simply want UUIDs to be the only way items are identified from
> now on?

I'm not sure that will work for all use-cases.  For example:

   - In production Joe creates foo.com/news/awesome

   - In beta I start work on a new and improved foo.com/news/awesome

   - I finish work and try to move the new story in production

If there's no fall-back to URL-matching then import goes boom since it
can't have two stories with the same URL.

> How will you prevent URL collisions with this scheme, if URLs are no
> longer the unique identifier?

I'll prevent it by making sure it won't happen, I guess.  Regardless
of a UUID match Krang can't allow two of anything to have matching
URLs.  If the import code can't resolve the problem then it'll just
have to fail.

> What are you going to base your UUID on?

Data::UUID.  I almost picked MySQL's UUID() but it doesn't work with
replication, which isn't used much for Krang but definitely shouldn't
be broken for something as small as this.

> How will you assure that UUIDs are never used more than once, even
> across multiple servers?  (I.e., You can't use a "auto-increment"
> INT.)  MD5?  Is that unique enough?

Data::UUID promises this will never happen.  I don't know whether to
trust it, but a look at the code makes me think it's at least very
very unlikely.

-sam

Hey Sam --

I see the advantage of a unique key which is pure metadata (i.e.,
without any other meaning) and consistent across servers.  People have
an expectation that the story IDs work like this and are constantly
baffled why a story (or other object) has a different ID on a different
server.  UUID would certainly give them what they want.  (FWIW, I've
been thinking about some sort of "Inter-Krang Protocol" for a while
which could use this, but that's another subject.)  My immediate concern
is the complexity of multiple IDs in the system.  Should we drop the
database IDs in favor of this UUID?


> I'm not sure that will work for all use-cases.  For example:
>=20
>    - In production Joe creates foo.com/news/awesome
>=20
>    - In beta I start work on a new and improved foo.com/news/awesome
>=20
>    - I finish work and try to move the new story in production
>=20
> If there's no fall-back to URL-matching then import goes boom since it
> can't have two stories with the same URL.

In the situation you describe, what should be the behavior?  Overwrite
Joe's story?  What if Joe doesn't want his story to be overwritten?
Isn't that the same problem you started with before the UUID?

It seems that a UUID would support the following use-cases:

Case A:
  1. You create a story (UUID: "123") on server A
  2. Joe creates story "456" on server B, at same URL as story "123"
  3. You import story 123 from server A, which results in duplicate URL
exception

Case B:
  1. You create a story (UUID: "123") on server A
  2. Story "123" is copied to server B
  3. You edit story "123" on server A
  4. Joe changes the category of story "123" on server B
  5. Re-import of story "123" from A to B causes (?????) story 123 to be
moved back to original category, and content to be updated.
  6. If another story has been created in the old URL of story 123 on
server B, an exception will be thrown.

In this situation, what does keying on URL do that isn't done by UUID?
Is there any need to have that "fall back" anymore?


> I'll prevent it by making sure it won't happen, I guess.  Regardless
> of a UUID match Krang can't allow two of anything to have matching
> URLs.  If the import code can't resolve the problem then it'll just
> have to fail.

I guess you'd need a first pass to check for dups?  Also, maybe there
should be an option to skip colliding objects?  Maybe an option to link
UUIDs which is default off?  (I guess this might be the "fall back"
behavior you describe.)


> Data::UUID promises this will never happen.  I don't know whether to
> trust it, but a look at the code makes me think it's at least very
> very unlikely.

Seems that's based on time plus some machine-uniqueness data.  Works for
me.


TTYL,

-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

> Case B:
>   1. You create a story (UUID: "123") on server A
>   2. Story "123" is copied to server B
>   3. You edit story "123" on server A
>   4. Joe changes the category of story "123" on server B
>   5. Re-import of story "123" from A to B causes (?????)=20
> story 123 to be
> moved back to original category, and content to be updated.


It occurs to me that in this case you might want the story ("123") to
remain in the new category, but to be updated with the new content.
This begs for a "diff"-style solution.

-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

On Wed, 25 Oct 2006, Jesse Erlbaum wrote:

> I see the advantage of a unique key which is pure metadata (i.e.,
> without any other meaning) and consistent across servers.  People have
> an expectation that the story IDs work like this and are constantly
> baffled why a story (or other object) has a different ID on a different
> server.  UUID would certainly give them what they want.  (FWIW, I've
> been thinking about some sort of "Inter-Krang Protocol" for a while
> which could use this, but that's another subject.)  My immediate concern
> is the complexity of multiple IDs in the system.

I agree - there's danger in the complexity.  I guess I'm willing to
deal with it because I absolutely have to solve this problem and I
can't think of a simpler way to do it.

> Should we drop the database IDs in favor of this UUID?

I don't think so.  They're huge, for one thing - 128-bits to be exact.
We'd pretty much have to stop showing them in the UI - even if we
could fit them on the screen they're so long that user's would no
longer be able to remember them.  They also won't index nearly as well
as the current IDs, so Krang would be slower.

>>    - In production Joe creates foo.com/news/awesome
>>
>>    - In beta I start work on a new and improved foo.com/news/awesome
>>
>>    - I finish work and try to move the new story in production
>>
>> If there's no fall-back to URL-matching then import goes boom since it
>> can't have two stories with the same URL.
>
> In the situation you describe, what should be the behavior?  Overwrite
> Joe's story?

Yes, I think so.  That's the behavior now and I think it's pretty
reasonable.

> What if Joe doesn't want his story to be overwritten?

Then he can revert it, I guess.

> Isn't that the same problem you started with before the UUID?

It could be - depending on the user's expectation.  In my example I
intended to convey that the intent of the person working in beta was
to produce a new version of the story, starting from scratch.

> It seems that a UUID would support the following use-cases:
>
> Case A:
>  1. You create a story (UUID: "123") on server A
>  2. Joe creates story "456" on server B, at same URL as story "123"
>  3. You import story 123 from server A, which results in duplicate URL
> exception

You'd rather have an exception in this case, rather than an update?
That seems like a pretty big change for Krang.  Maybe this isn't too
likely for stories but I think this kind of parallel development is
happens for templates and media all the time.  Media is often loaded
onto multiple systems via FTP, for example, which doesn't offer any
metadata.

> Case B:
>  1. You create a story (UUID: "123") on server A
>  2. Story "123" is copied to server B
>  3. You edit story "123" on server A
>  4. Joe changes the category of story "123" on server B
>  5. Re-import of story "123" from A to B causes (?????) story 123 to be
> moved back to original category, and content to be updated.
>  6. If another story has been created in the old URL of story 123 on
> server B, an exception will be thrown.
>
> In this situation, what does keying on URL do that isn't done by UUID?
> Is there any need to have that "fall back" anymore?

Perhaps not, depending on the use-case.  One problem will be existing
beta-production pairs.  When they upgrade to this release all their
objects will get fresh UUIDs - without some other kind of matching any
future imports will be a real pain.

>> I'll prevent it by making sure it won't happen, I guess.  Regardless
>> of a UUID match Krang can't allow two of anything to have matching
>> URLs.  If the import code can't resolve the problem then it'll just
>> have to fail.
>
> I guess you'd need a first pass to check for dups?

Probably all I need to do is catch the exception from the DB - URLs
should all have UNIQUE indexes I think.

> Also, maybe there should be an option to skip colliding objects?
> Maybe an option to link UUIDs which is default off?  (I guess this
> might be the "fall back" behavior you describe.)

I can make it fully configurable.  I think I'd default to having both
behaviors - first try UUID and fallback to the old method if no match
is found.  I can make it so that either can be turned off, through an
option to krang_import.  Sound good?

-sam

On Wed, 25 Oct 2006, Jesse Erlbaum wrote:

> It occurs to me that in this case you might want the story ("123") to
> remain in the new category, but to be updated with the new content.

I can't think of any easy way to tell the difference between cases
where you would want to change the URL during update and cases where
you wouldn't.  I suppose it could be optional.

> This begs for a "diff"-style solution.

Diffing hierarchical data is hard!  I did a diff and patch for
tempates once - it's been moderately useful.

-sam

> I agree - there's danger in the complexity.  I guess I'm willing to
> deal with it because I absolutely have to solve this problem and I
> can't think of a simpler way to do it.

I understand.  Your plan sounds good for the simple case you present
(being able to change the URL of a story as a part of an import).

-Jesse-

Jesse Erlbaum wrote:
> Any suggestions for names?  "Preview Server" replacing "Site Server"?
> "Front-end Server"?  "Live Server"?

I can't any reason you'd want both a new one and the existing one.  I'd 
just keep the name and make the Site Server better.

- Perrin

> I can't any reason you'd want both a new one and the existing=20
> one.  I'd=20
> just keep the name and make the Site Server better.


I could, except that Site Server is currently all about preview -- not
publish.  You think it's better to extend it to published content?

Also, there was some thought that it might not be desirable to have the
"preview" be on the same server as the "publish".  Thoughts?

-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

On Wed, 2006-10-25 at 12:29 -0400, Jesse Erlbaum wrote:
> I could, except that Site Server is currently all about preview -- not
> publish.  You think it's better to extend it to published content?

My thinking is that in order for preview to work, your preview and live
servers need identical configurations.  That means the current Site
Server option is only useful if your live server looks exactly like the
Site Server.  If you plan to change one, you probably need to change
both.

> Also, there was some thought that it might not be desirable to have the
> "preview" be on the same server as the "publish".  Thoughts?

I don't see any harm in it.  We run all of our servers that way, with
separate docroots for each of course.

- Perrin

> My thinking is that in order for preview to work, your=20
> preview and live
> servers need identical configurations.  That means the current Site
> Server option is only useful if your live server looks=20
> exactly like the
> Site Server.  If you plan to change one, you probably need to change
> both.

In development we use Site Server as an approximation of the live
server.  I see your point, and agree.  I just wanted you to have another
use case in mind.


> I don't see any harm in it.  We run all of our servers that way, with
> separate docroots for each of course.

What about separate IP addresses?  Is an Apache NameVirtualHost
sufficient, particularly if you're running many sites?  What about
multiple instances?  Should their be a separate IP address for each
instance?  (Currently it's at the root level of krang.conf.)


-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

On Wed, 2006-10-25 at 14:38 -0400, Jesse Erlbaum wrote:
> What about separate IP addresses?  Is an Apache NameVirtualHost
> sufficient, particularly if you're running many sites?  What about
> multiple instances?  Should their be a separate IP address for each
> instance?  (Currently it's at the root level of krang.conf.)

Anyone doing something complicated will have their own setup for this.
I'm assuming we're still just talking about a bare-bones static site.
In that case, name-based virtualhosts should be fine.

- Perrin

> Anyone doing something complicated will have their own setup for this.

That is what I was on the fence about (vis-=E0-vis separate IP =
addresses).  You're probably right about that for now.  (Although, I see =
that changing in the future.)


> I'm assuming we're still just talking about a bare-bones static site.
> In that case, name-based virtualhosts should be fine.

I'm not thinking about static sites.  I'm thinking about using this for =
interactive apps, too.  Specifically, I have a dream of being able to =
distribute the CMS + apps for a site via a single Krang add-on.


-Jesse-


--
=20
Jesse Erlbaum
The Erlbaum Group
jesse@...
Phone: 212-684-6161
Fax: 212-684-6226
=20

Michael Peters wrote:
> I'm planning to add some new security minded features to krang in the next
> couple of days. 

The next feature has to do with creating and enforcing a password policy. Since
this will vary foreach organization I think it should be as easy as possible to
override with an addon. My thought was to create a PasswordHandler (is there a
better name?) which is used to perform the check against a username/password
combo. I'll move the current checks into this and then it's just a matter of
creating a PasswordHandler and updating krang_addon.conf for a particular orgs
needs.

The 2nd part of this is to create an optional "Change password every X days"
feature (which will be turned off by default or if that particular config var is
0). This will require some changes to Krang::Handler and Krang::CGI::Login.

Any thoughts or objections to these as well?

-- 
Michael Peters
Developer
Plus Three, LP

Michael Peters wrote:

> The 2nd part of this is to create an optional "Change password every X days"
> feature (which will be turned off by default or if that particular config var is
> 0). This will require some changes to Krang::Handler and Krang::CGI::Login.

I've implemented this using 2 config vars PasswordChangeTime (number of days
between changes) and PasswordChangeCount (number of old passwords to remember
and not let you use again).

-- 
Michael Peters
Developer
Plus Three, LP

Looks ancient:

  # ll /var/lock/cvs/krang/templates/MyPref
  total 0
  -rw-rw-r--  1 adancygier adancygier 0 Aug  9 12:17
#cvs.rfl.krang-cvs.thepirtgroup.com.32644


I've deleted it.

=20

> -----Original Message-----
> From: krang-devel-bounces@...
> [mailto:krang-devel-bounces@...] On Behalf=20
> Of Michael Peters
> Sent: Wednesday, October 25, 2006 11:40 AM
> To: krang-devel
> Subject: [Krang-devel] cvs lock in templates/MyPref
>=20
> I just tried committing a minor update to=20
> templates/MyPref/edit.tmpl and was
> informed that it was currently locked by adancygier. Aaron,=20
> are you currently
> doing something with these or is this an old lock?
>=20
> --=20
> Michael Peters
> Developer
> Plus Three, LP
>=20
>=20
> --------------------------------------------------------------
> -----------
> Using Tomcat but need to do more? Need to support web=20
> services, security?
> Get stuff done quickly with pre-integrated technology to make=20
> your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on=20
> Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&;
> dat=3D121642
> _______________________________________________
> Krang-devel mailing list
> Krang-devel@...
> https://lists.sourceforge.net/lists/listinfo/krang-devel
>=20

Michael Peters wrote:
> I'm trying to make an automated build/test setup for one of our big projects
> that use krang and I thought I'd make it work for krang too. We use smolder
> (http://sourceforge.net/projects/smolder) to monitor the test runs and I was
> going to setup the krang results in a public space.

If anyone wants to see a Krang test on smolder:
http://70.42.42.115:8080/app/public_projects/smoke_reports/3

and then a details HTML report can be seen here:

http://70.42.42.115:8080/app/public_projects/report_details/2/html

Try clicking on the "Details?" links and the individual tests' boxes.

-- 
Michael Peters
Developer
Plus Three, LP

On Wed, 11 Oct 2006, Jesse Erlbaum wrote:

> I intend to start by committing the latest version, and then I'm going
> to write a little script to untar the old source releases and make svn
> tags for them all.  We won't have every individual revision, but we'll
> have all the major ones.

Sounds great!  I think there are CVS-to-SVN migration tools that can
migrate all the CVS revision history into SVN.  If this was possible I
think it would desirable - I find the revision history to be very
helpful in jogging my terrible memory.

-sam