Re: [GD-General] Scripting

[brian h. <bri...@py...>]

> I think data description (let alone "rich" data description) is reason
> enough to use something like Lua (which started its life as a data
> description language).

I agree, and in fact that's what I use Lua for right now.  It stores my 
configuration variables, and I also use it to execute extremely 
crude "level scripts" for some of our 2D games.  But I've struggled a 
bit to figure out how to go past that basic functionality, which is 
part of the reason that I'm so down on scripting recently =)

> I think you're taking on a couple separate issues here, 1) the
> language itself, and 2) the development environment (i.e. the
> mechanics of how you get your code into the running game).

Sure, I kinda tossed them in together, but at the same time, they're 
both valid reasons why scripting is a pain in the ass for non-
developers to use.  Lack of a debugger alone can be an absolute killer, 
especially if you have a huge numbers of scripts and you have some 
small corner case that only manifests itself under certain conditions.

> I can say that about Python (and Lua) with a straight face.  Although
> "intrinsically easier" is pretty subjective, in the absence of user
> studies.

I hate to be so curmudgeonly, but I think you're too experienced to 
make that call =)  Even a simple one line bit of Python looks like 
gibberish to a non-programmer.  I mean, this is something pretty simple:

data = filter(lambda x: x>=' ' and x<='z', data)

Yet you show that to an artist, and they're not going to have any idea 
what it does.  Something as basic as parens for a function call -- or a 
function call, period -- can be daunting, and this is something most 
languages will have.

Same with this chunk of Lua:

function List:reverse()
  local t,n={},getn(self)
  for i=1,n do t[i]=self[n-i+1] end -- reverse
  for i=1,n do self[i]=t[i] end -- copy back
end

Now, I know that these aren't that obtuse, but they're obtuse enough to 
a non-programmer that I just think the difference between good C code 
and the above isn't going to be that huge either way.

> I generally agree with your point that the "language" part
> of "scripting language" is *not* the killer feature that makes it
> worth doing.

Right, that's a good summary of the way I feel.
 
> IMO, in order to get designer friendliness and faster prototyping, the
> key thing to focus on is the "development environment".  (And I don't
> mean "IDE" per se; I mean whatever it is specifically that a designer
> has to do to see her changes in the game, and all the little
> mechanical details of how feedback comes out, and how you debug, etc.)

Agreed.  For example, a level designer/mapper should access scripting 
facilities from inside their level editor.  Assuming they even need to 
see the scripting code.
 
> I've never worked on a commerical game whose average edit-compile-run
> turnaround was less than a minute (and usually it has been at least 2x
> worse than that).

One minute, okay.  Five minutes, even that's fine.

For what it's worth, when I worked on Quake 2 and Quake 3, complete 
build times were on the order of a minute IIRC, and compile-edit-run 
was measured in seconds.  I don't think anything magical was done for 
that either -- it was straight ANSI C with the subsystems broken apart 
into DLLs to avoid interdependencies.

But these days, I'm hearing stories about build times measured in the 
tens of minutes or, in some cases, hours.  HOURS.  I just find this 
unimaginably bad, and in many cases it's because of heavy use of C++ 
templates or STL or other things like that.  The rationale is that 
stuff is bigger, huger, and more complex these days, but at what cost?

> The script-whatever-run cycle has the *potential*
> to be very quick, although it also tends to get bogged down by
> suboptimal process, in my experience. 

I would argue that it gets bogged down more often than not.  When you 
talk to people that have written scripting languages or used them, 
inevitably it's trading off one set of evils for another set that are 
often just as bad and sometimes even worse.

> Like, scripting engines where
> it's theoretically trivial to inject updated scripts into the running
> game, but nobody exposed the functionality in a way that the designers
> could use.  A simple "load(script)" command that you can trigger from
> the console in the running game goes a long way.  In general, I think
> using a scripting language for a console language is a handy thing.

I think this is what Bioware did with Baldur's Gate et. al.

> Well, you can do either of the above, or both at the same time, with a
> scripting language.  (And, as I said, I think there are advantages to
> using a scripting language's parser to digest your data.)

The main reason I started using Lua for my cvars is that I found I was 
writing Yet Another Crappy Parser, and figured I'd just Lua handle it.

The thing with scripting languages though is that finding the right 
area to bifurcate is extremely hard.  I really think this is glossed 
over too much.  Using our door example, you have a situation where you 
have to neatly define "What parts of being a door does the script 
decide; and what parts of being a door does the engine decide; and how 
do they reconcile these aspects?"

And no matter where you draw that line, you incur some kind of cost in 
development time, run-time cost, robustness, features, etc.  At one 
extreme, the "script door" is nothing but data defs read by the engine, 
at which point you have very little flexibility since any new 
capabilities implemented by the engine have to be exported repeatedly 
to the script designers.  "Oh, I can now make doors out of different 
materials, must support this capability".

At the other end of the spectrum, you have ALL properties of the game 
done in the script, and the engine is solely responsible for low level 
system tasks like rendering, marshalling input, low level audio, etc.  
At that point you're also in a special form of hell, because you've 
effectively exported huge chunks of your engine to the script so that 
the script can do meaningful work, AND you've put yourself in a 
position where your primary development pipeline is being gated by 
scripting tools which have a high likelihood of sucking.

And, honestly, if you make any major changes to your engine, you're 
probably breaking the scripts anyway.

> finalDoor = {
>   type = door,
>   key = function(opener)
>     if opener == evil_wizard then
>       nil
>     else
>       redKey
>     end
>   end
> }

That's a slippery slope you walk upon =)

> Anyway, I also struggle with the issues you raise.  I do think it's
> very helpful to know about all the things that are wrong with
> scripting languages before deciding to use one, and especially after
> you're committed to using one.

In this particular case, I went from using Lua to looking at everything 
else to deciding to roll my own to back to using Lua, all in the span 
of about a week.

It was extremely helpful to me trying to write my own, because it made 
me acutely aware of all the different implementation and design trade 
offs that a language has to make, and by extension I learned why 
different languages are good at different things and bad at others.

And that's why I've reached this philosophy now, because once I gave 
myself the "freedom" to write my own language, I realized that the 
_language_ wasn't the problem.  The language and its implementations 
can eventually become a problem, e.g. Lua's garbage collector or 
Python's footprint, but long before you reach that point, the 
integration and development process issues will become a much bigger 
concern.

I'm quite amazed at the number of developers I run into that use 
scripting languages but deep down inside still aren't comfortable with 
how and where they've integrated them into their games, which is why 
I'm so leery of scripting at this point.  You hear a lot about backdoor 
hacks to get pieces of the engine and scripts to communicate something 
that isn't cleanly defined by the pre-determined APIs.

For many developers, it makes a lot of sense, I just fear for the 
developers (and games) that feel a certain obligation to be script 
driven because everyone else.

For example, console games -- obviously the mod community isn't going 
to be targeting console games, so I'm curious to hear why/when/how 
someone would fully script AI, combat, quests, levels, etc. for a 
console game.  I'm sure there are good reasons, but I'm real curious if 
the actual benefits derived were worth the hassle (and if they couldn't 
have been reached some other way).

This is a separate subject, but I really do feel that in many ways, 
games are being overengineered.  Tempting technologies like C++, STL, 
scripts, garbage collection/ref-counting/smart pointers, keep rearing 
their heads, and they make such promises of how everything will be 
better, but when you get right down to brass tacks, rarely have things 
improved.

I have a C++ application framework code base, and I have some open 
source stuff I'm developing on the side that is ANSI C.  The latter 
stuff, without exception, is better architected than the former, 
because C++ hierarchies have a tendency of "hardening".  Code acquires 
a sense of mass and inertia that makes it difficult to modify, and when 
you have huge amounts of interdependent code, like a C++ class 
hierarchy, then trivial changes become non-trivial.

STL is the same thing -- it's so great to have something like std::set 
or std::list available all the time, but holy crap, is writing a linked 
list REALLY that stressful?  Because the last couple times I used STL, 
the 15 second list vs. the 5 minute self-rolled list time advantage 
evaporated the first time I had to try to read STL code in the debugger 
and look at opaque pointers with no idea what died and where.  But I 
was convinced that STL was a "better" way of doing things.

Then you start hearing horror stories about build times with STL; rapid 
rise in binary data size; inability to debug code deep inside STL; etc. 
and suddenly you find that your project is suffering in many different 
areas because you wanted to take the easy way out and reuse a very 
popular and successful library.

This isn't about NIH vs. non-NIH, it's about theoretical vs. actual 
benefits of things that are just touted as "better" without any real 
objective data to back it up.  It's about purity of design vs. 
pragmatism in development.  It's 80s software engineering dogma vs. old 
fashioned "just getting it done".

Are STL and C++ worthwhile if you can no longer have an incremental 
build cycle?  Are scripts worthwhile if you can no longer debug them, 
or if you spend just as much time dealing with script<->engine 
communication as you would have just writing the scripted stuff 
directly into the engine?  Is SourceSafe's GUI better than using cvs at 
the command line if you can no longer check out stuff remotely from 
home?  Is garbage collection really that much better if you now have to 
start searching for cycles and be real careful about how you allocate 
objects during run-time, and when you gc, and when/how you delete root 
objects that can cause cascading gc?

Okay, I'll be quiet now, I feel better =)

-Brian