ICU data packaging suggestions/help wanted!

[<sr...@jt...>]

ICU for C/C++ users,
   For 2.6 we will be overhauling pkgdata and the makefiles which build 
data.  Please reply here to the list (ic...@os..., not 
icu4c-support) with your favorite, or least favorite, data packaging 
issues.  I'll throw out some items which have become hot issues 
recently.  Note, we can't commit to doing everything, of course.  But 
we want to hear your feedback, especially if you would like to help 
out! I would like to get some proposals and prototypes out soon.

* Limit on number of files.   Currently, ICU uses makefiles to generate 
a list file (on both msvc and unix). This frequently runs out of 
command space.  Instead of using makefiles (ucmlocal.mk, ucmfiles.mk, 
reslocal.mk, etc), current thinking is to use flat text files with a 
simplified format.
    Impact on users:  removal of limits on number of converters, etc.  
Faster processing.    Breaking change to existing ucmlocal/reslocal 
user files, or ICU customizations.

* Have pkgdata perform actions directly without using intermediate 
makefiles.  If you've ever had to debug data packaging issues, you know 
that an intermediate makefile is generated, and then gnumake or nmake 
is executed on that file.  Instead, pkgdata will be changed to directly 
execute the appropriate commands.
     Impact on users:  Faster processing.  Fewer bugs.  Should not have 
any usability impact.

* Build .o files directly from .dat files, i.e., without invoking the 
compiler ("generating C source for ..") .  Impact: incredible speed 
improvements - on selected platforms.  Windows version already works 
this way.

Something I'd like to see, but don't have time for:
* have pkgdata be able to run genrb or makeconv directly- i.e. manage 
part of the building and not just packaging.

Regards,
  Steven

On Wednesday, March 12, 2003, at 03:10 PM, Markus Scherer wrote:
> an...@id... wrote:
>> Can I suggest modifying that link step to change into the directory 
>> with
>> the source *.o's, first?  Stripping out the path to each .o  (eg., 
>> from
>> ../data/out/build/icudata_dat.o  to  icudata_dat.o)  trims the total
>> length of the command from over 20,000 bytes to under 12,000.
>
> Steven and George will work on this for 2.6 and solicit suggestions 
> and feedback from this list. The actual discussion will be on the icu 
> list, not this support list. (Separate list subscription.)
>
> <soapbox>
>
> I would like to take this opportunity to invite contributions for the 
> data library building. Specifically, it would be very useful if we 
> could build the ICU data DLL/.so/library directly from a .dat file by 
> slapping on some header and footer information to make a .o and then 
> just link it into a runtime library executable. This would get us 
> around some of the messy and slow build process via data .c and .o 
> files for each data item.
>
> We already do this for Windows (and always did, see 
> source/tools/gencmn/gencmn.c), but could not get it to work anywhere 
> else yet for lack of time.
>
> Please!
>
> As a side effect, a user building custom data would then not need a C 
> compiler any more, just a linker. On Windows, this is the difference 
> between using the free Platform SDK (which includes the linker) and a 
> paid-for compiler.
>
> Aside from the fact that building the data library from pieces would 
> be under 5 seconds...
>
> </soapbox>
>
> Seriously. Thanks!
>
> markus