lcms-user — For users of the little cms library

Re: [Lcms-user] lcms2-2.9 release candidate 1 available for testing

[Christian S. <rea...@mo...>]

> Am 27.09.2017 um 09:16 schrieb Martí Maria <mar...@li...>:
> 
> Hi,
> 
> I've uploaded a tarball with lcms2-2.9 release candidate:

Building here gives errors on linking on macOS 32-bit:



/bin/sh ../../libtool  --tag=CC   --mode=link clang  -O2 -mmacosx-version-min=10.4 -DTARGET_MACOS=1 -arch i386 -I.. -Dunix=1  -fvisibility=hidden -D_THREAD_SAFE    -o transicc transicc.o ../common/xgetopt.o ../common/vprf.o ../../src/liblcms2.la  
libtool: link: clang -O2 -mmacosx-version-min=10.4 -DTARGET_MACOS=1 -arch i386 -I.. -Dunix=1 -fvisibility=hidden -D_THREAD_SAFE -o .libs/transicc transicc.o ../common/xgetopt.o ../common/vprf.o  ../../src/.libs/liblcms2.dylib -lm -lpthread
Undefined symbols for architecture i386:
 "_cmsAllocNamedColorList", referenced from:
     _ComponentNames in transicc.o
 "_cmsAppendNamedColor", referenced from:
     _ComponentNames in transicc.o
 "_cmsBuildGamma", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsChannelsOf", referenced from:
     _main in transicc.o
     _ComponentNames in transicc.o
 "_cmsCloseProfile", referenced from:
     _main in transicc.o
 "_cmsCreateGrayProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreateLab2ProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreateLab4Profile", referenced from:
     _main in transicc.o
 "_cmsCreateLab4ProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreateLinearizationDeviceLink", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreateNULLProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreateProofingTransform", referenced from:
     _main in transicc.o
 "_cmsCreateTransform", referenced from:
     _main in transicc.o
 "_cmsCreateXYZProfile", referenced from:
     _main in transicc.o
 "_cmsCreateXYZProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsCreate_sRGBProfileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsD50_xyY", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsDeleteTransform", referenced from:
     _CloseTransforms in transicc.o
 "_cmsDoTransform", referenced from:
     _main in transicc.o
 "_cmsDupNamedColorList", referenced from:
     _main in transicc.o
 "_cmsFormatterForColorspaceOfProfile", referenced from:
     _main in transicc.o
 "_cmsFormatterForPCSOfProfile", referenced from:
     _main in transicc.o
 "_cmsFreeNamedColorList", referenced from:
     _CloseTransforms in transicc.o
 "_cmsFreeToneCurve", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsGetColorSpace", referenced from:
     _main in transicc.o
 "_cmsGetDeviceClass", referenced from:
     _main in transicc.o
 "_cmsGetNamedColorList", referenced from:
     _main in transicc.o
 "_cmsGetPCS", referenced from:
     _main in transicc.o
 "_cmsGetProfileInfoASCII", referenced from:
     _PrintInfo in vprf.o
 "_cmsGetSupportedIntents", referenced from:
     _PrintRenderingIntents in vprf.o
 "_cmsIT8Alloc", referenced from:
     _main in transicc.o
 "_cmsIT8DefineDblFormat", referenced from:
     _main in transicc.o
 "_cmsIT8Free", referenced from:
     _main in transicc.o
 "_cmsIT8GetData", referenced from:
     _main in transicc.o
 "_cmsIT8GetDataDbl", referenced from:
     _main in transicc.o
 "_cmsIT8GetPatchName", referenced from:
     _main in transicc.o
 "_cmsIT8GetPropertyDbl", referenced from:
     _main in transicc.o
 "_cmsIT8LoadFromFile", referenced from:
     _main in transicc.o
 "_cmsIT8SaveToFile", referenced from:
     _main in transicc.o
 "_cmsIT8SetComment", referenced from:
     _main in transicc.o
 "_cmsIT8SetData", referenced from:
     _main in transicc.o
 "_cmsIT8SetDataDbl", referenced from:
     _main in transicc.o
 "_cmsIT8SetDataFormat", referenced from:
     _main in transicc.o
 "_cmsIT8SetPropertyDbl", referenced from:
     _main in transicc.o
 "_cmsIT8SetPropertyStr", referenced from:
     _main in transicc.o
 "_cmsIsTag", referenced from:
     _main in transicc.o
     _PrintColorantTable in vprf.o
 "_cmsNamedColorCount", referenced from:
     _main in transicc.o
     _PrintColorantTable in vprf.o
 "_cmsNamedColorIndex", referenced from:
     _main in transicc.o
 "_cmsNamedColorInfo", referenced from:
     _main in transicc.o
     _PrintColorantTable in vprf.o
 "_cmsOpenProfileFromFileTHR", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsReadTag", referenced from:
     _main in transicc.o
     _PrintColorantTable in vprf.o
 "_cmsSetAdaptationState", referenced from:
     _main in transicc.o
 "_cmsSetAlarmCodes", referenced from:
     _main in transicc.o
 "_cmsSetLogErrorHandler", referenced from:
     _InitUtils in vprf.o
 "_cmsWhitePointFromTemp", referenced from:
     _OpenStockProfile in vprf.o
 "_cmsstrcasecmp", referenced from:
     _OpenStockProfile in vprf.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)


nm src/.libs/liblcms2.2.dylib 

Shows the entries:

00035b10 t _cmsIT8LoadFromFile

But small t, so not public.


So I changed hidden in "-fvisibility=hidden" in configure file to xxx, and the 

checking whether C compiler accepts -fvisibility=xxx… no

Happens and the visibility flag is not used.

Than everything works fine and builds.
Can you make sure the flag is not used when building shared library?

Sincerely
Christian

-- 
Read our blog about news on our plugins:

http://www.mbsplugins.de/

Re: [Lcms-user] lcms2-2.9 release candidate 1 available for testing

[Richard H. <hug...@gm...>]

On 27 September 2017 at 08:16, Martí Maria <mar...@li...> wrote:
> I've uploaded a tarball with lcms2-2.9 release candidate:

Seems to pass all the colord tests locally, I'll build and tests on
all weird architectures tomorrow.

Richard.

[Lcms-user] lcms2-2.9 release candidate 1 available for testing

[Martí M. <mar...@li...>]

Hi,

I've uploaded a tarball with lcms2-2.9 release candidate:

http://www.littlecms.com/lcms2-2.9rc1.tar.gz

It is basically a maintenance release.

Best regards
Marti Maria
The LittleCMS project
http://www.littlecms.com/

Re: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation and lcms context

[Marco F. <Mar...@en...>]

Hi Noel,

thanks. Your "GetColorTransform" sounds exactly like what I'm doing :)



Von: Noel Carboni [mailto:NCa...@Pr...]
Gesendet: Freitag, 25. August 2017 07:46
An: Marco Freudenberger; 'Martí Maria'; lcm...@li...
Betreff: RE: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation and lcms context

Hi Marco,

If it helps build confidence, we have for years in our DLL been using a transform caching scheme and multi-threading the actual transforms.  We don't manage Little CMS contexts at all (i.e., we use the library functions that don't have the ContextID parameter).

Our key function is called GetColorTransform, which grabs a mutex to ensure only one thread can traverse the function at any one time, then A. searches for a cached transform that already exists for the given input arguments and returns its handle immediately if one exists, or B. generates and caches the transform handle, then returns it.

We multithread the transforms like crazy.  As an example, a 12 core circa 2012 Xeon system transforms all the pixels in an 80 megapixel 16 bit RGBA image in an eighth of a second...

08:36:41.208 Thread 0x1A34: Line 2317:  Category 'TraceColorManagement': Level 2: ConvertColorOf() - Dividing the image, height 9610, into 24 parts of 400.4 rows each for multi-threaded color conversion.

08:36:41.333 Thread 0x1A34: Line 2345:  Performance Message: Total Time = 121.544 milliseconds, ConvertColorOf() - Completed color conversion using 24 simultaneous threads.  Image:  8398 x 9610 pixels x 16 bits per color.

That's something like 660 megapixels per second.  :-)

Given our application, relatively few transforms actually get created and so the caching is incredibly effective at keeping performance up.  Oh, and we don't replace the default memory manager.

-Noel

From: Marco Freudenberger [mailto:Mar...@en...]
Sent: Fri, August 25, 2017 6:27 AM
To: Martí Maria; lcm...@li...<mailto:lcm...@li...>
Subject: Re: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation and lcms context

Hi Marti!

Thanks a lot for your answer. I was under the impression by reading the documentation and previous post here, that besides the plug-in and memory-management configuration (I'm not using any of them right now), the context had something to do with concurrent access to global "internal data structures" of littlecms when working with multiple threads.
If not, that's actually great and I just can use my transformation across multiple threads without recreating it.

Sounds like it's no problem if I just cache the transformation itself.

Thanks a lot for your time answering this question, appreciate it!
Marco

Von: Martí Maria [mailto:mar...@li...]
Gesendet: Freitag, 25. August 2017 05:13
An: Marco Freudenberger; lcm...@li...<mailto:lcm...@li...>
Betreff: [EXTERNAL]Re: [Lcms-user] Multithreading - Transformation and lcms context




Hi Marco,

Keep in mind cmsContext are only a way to store the plug-in configuration you are using. So, if you provide your own memory management routines, you may want to create a context at the very begin of your program to let lcms use your memory manager instead of the default malloc functions. contexts also stores some global settings. But that's all, contexts are not critical to threads. Deleting the context would just unregister your memory manager, or other plug-ins, if any.

On the other hand, cmsDoTransform and specially cmsDoTransformLineStride are re-entrant and can be used on different threads. You can create the transform in the base thread and apply the transform in many other different threads. The only requirement is memory should be accessible. If you provide your memory manager, the context is the way you channel the malloc and free calls to your memory manager.

You could just create the transform and keep it open while needed. That's what I do in the color translator and works fine. There is no need to convert to a devicelink.

Now for your questions:

> 1. Does that mean I could actually just cache the transformation (cmsHTRANSFORM) and re-use it, although the lcms context originally used to load the profiles the transformation was deleted? Can I delete the transformation from a different thread it was created from?

If you provide you own memory management functions, then creates the transform, then unregister you memory manager, further cmsDeleteTranform() would call default free(), is likely something will fail. I would just create the context at the begin of program and keep it open all the time to make sure your memory manager is being consistently called. Again, this is just to keep track of the plug-ins you are using. And yes, you can create the transform and reuse it. Even with different buffer formats in some cases.
> 2. Alternatively, if that doesn't work, could I just cache the transformation together with the lcmsContext it was created in and use it from different threads concurrently on different images? And if so, could multiple DIFFERENT transformations, which might be applied concurrently, share the same context (assuming they are not CREATED concurrently) or would I need a separate context per transformation?

Of course yes, many different transformations can share the same context. That is how it is supposed to work: you create one and only one context, put there all your plug--ins and operates in this context all the time. The plug-in API documentation would give you more details on contexts and plugins.

Hope that helps

Marti


On 8/25/2017 10:34 AM, Marco Freudenberger wrote:
All,

I've got a multithreading / lcms context related question on lcms2. I'm currently using lcms 2.7, but certainly can upgrade to latest release, if that makes any difference.
I tried to find the answer to this somewhere in the documentation, but couldn't get a definite answer.

For a project, I need to do color space conversion (from RGB -> CMYK, not that it matters) in a dll.
The CMYK profiles are mostly large table based device profiles, so creating the transformation takes a while (depending on the image size, most often longer than applying it). The code might be called thousands of times in an hour. As all the transformations are likely from the same one or two RGB profile(s) to the same (or a little number of different) CMYK profile(s), I would like to cache transformations.

The problem is, that the dll calls might be called from different threads and might be called concurrently.

For each of the calls, in the original implementation I was:

-          creating a lcms context

-          load the profiles

-          create the transformation

-          close the profiles

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation

-          delete the lcms context.

As I said, I would like to cache a small number of transformations.

As I wasn't sure about the multithreading (transformation potentially used in different threads) and concurrency (code might be called multiple times) aspects, my first implementation is trying to play save going through a device link:

-          create a lcms context

-          check if transformation from IN->OUT is in chache. if not: load profiles, create transformation, close profiles, CREATE A DEVICELINK FROM TRANSFORMATION, SAVE DEVICELINK IN MEMORY (CACHE)

-          if it is in cache: load transformation from devicelink in cache

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation (not the devicelink).

-          delete the lcms context.

-          if cache item needs to be cleared, delete the devicelink buffer.

That process as two downsides:
(a) loading a transformation from devicelink profile is much faster than creating it from the in and out profile, but still takes quite some time.
(b) the results are slightly different (not as accurate?) when loading the transformation from devicelink. I'm getting some CMYK value differences up to ~2 or 3 values per channel on 8 bit images.

What I'm wondering is: Once I've got a handle to a transformation, I can apply it to one image in multiple threads (although right now I'm doing that while the lcmsContext still exists). Also, to apply it, I don't need to pass in a lcms context to cmsDoTransform  (at least not explicitly - maybe a handle is stored in the transformation).


So - my questions:
1. Does that mean I could actually just cache the transformation (cmsHTRANSFORM) and re-use it, although the lcms context originally used to load the profiles the transformation was deleted? Can I delete the transformation from a different thread it was created from?

2. Alternatively, if that doesn't work, could I just cache the transformation together with the lcmsContext it was created in and use it from different threads concurrently on different images? And if so, could multiple DIFFERENT transformations, which might be applied concurrently, share the same context (assuming they are not CREATED concurrently) or would I need a separate context per transformation?

Thanks for any useful input...
Marco



------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most

engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________

Lcms-user mailing list

Lcm...@li...<mailto:Lcm...@li...>

https://lists.sourceforge.net/lists/listinfo/lcms-user

Re: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation and lcms context

[Noel C. <NCa...@Pr...>]

Hi Marco,

 

If it helps build confidence, we have for years in our DLL been using a
transform caching scheme and multi-threading the actual transforms.  We
don't manage Little CMS contexts at all (i.e., we use the library
functions that don't have the ContextID parameter).

 

Our key function is called GetColorTransform, which grabs a mutex to
ensure only one thread can traverse the function at any one time, then
A. searches for a cached transform that already exists for the given
input arguments and returns its handle immediately if one exists, or B.
generates and caches the transform handle, then returns it.

 

We multithread the transforms like crazy.  As an example, a 12 core
circa 2012 Xeon system transforms all the pixels in an 80 megapixel 16
bit RGBA image in an eighth of a second...

 

08:36:41.208 Thread 0x1A34: Line 2317:  Category 'TraceColorManagement':
Level 2: ConvertColorOf() - Dividing the image, height 9610, into 24
parts of 400.4 rows each for multi-threaded color conversion.

 

08:36:41.333 Thread 0x1A34: Line 2345:  Performance Message: Total Time
= 121.544 milliseconds, ConvertColorOf() - Completed color conversion
using 24 simultaneous threads.  Image:  8398 x 9610 pixels x 16 bits per
color.

 

That's something like 660 megapixels per second.  :-)

 

Given our application, relatively few transforms actually get created
and so the caching is incredibly effective at keeping performance up.
Oh, and we don't replace the default memory manager.

 

-Noel

 

From: Marco Freudenberger
[mailto:Mar...@en...] 
Sent: Fri, August 25, 2017 6:27 AM
To: Martí Maria; lcm...@li...
Subject: Re: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation
and lcms context

 

Hi Marti!

 

Thanks a lot for your answer. I was under the impression by reading the
documentation and previous post here, that besides the plug-in and
memory-management configuration (I’m not using any of them right now),
the context had something to do with concurrent access to global
“internal data structures” of littlecms when working with multiple
threads.

If not, that’s actually great and I just can use my transformation
across multiple threads without recreating it. 

 

Sounds like it’s no problem if I just cache the transformation itself. 

 

Thanks a lot for your time answering this question, appreciate it!

Marco

 

Von: Martí Maria [mailto:mar...@li...] 
Gesendet: Freitag, 25. August 2017 05:13
An: Marco Freudenberger; lcm...@li...
Betreff: [EXTERNAL]Re: [Lcms-user] Multithreading - Transformation and
lcms context

 

 

Hi Marco,

Keep in mind cmsContext are only a way to store the plug-in
configuration you are using. So, if you provide your own memory
management routines, you may want to create a context at the very begin
of your program to let lcms use your memory manager instead of the
default malloc functions. contexts also stores some global settings. But
that's all, contexts are not critical to threads. Deleting the context
would just unregister your memory manager, or other plug-ins, if any.

On the other hand, cmsDoTransform and specially cmsDoTransformLineStride
are re-entrant and can be used on different threads. You can create the
transform in the base thread and apply the transform in many other
different threads. The only requirement is memory should be accessible.
If you provide your memory manager, the context is the way you channel
the malloc and free calls to your memory manager.

You could just create the transform and keep it open while needed.
That's what I do in the color translator and works fine. There is no
need to convert to a devicelink.

Now for your questions:

> 1. Does that mean I could actually just cache the transformation
(cmsHTRANSFORM) and re-use it, although the lcms context originally used
to load the profiles the transformation was deleted? Can I delete the
transformation from a different thread it was created from?

If you provide you own memory management functions, then creates the
transform, then unregister you memory manager, further
cmsDeleteTranform() would call default free(), is likely something will
fail. I would just create the context at the begin of program and keep
it open all the time to make sure your memory manager is being
consistently called. Again, this is just to keep track of the plug-ins
you are using. And yes, you can create the transform and reuse it. Even
with different buffer formats in some cases.

> 2. Alternatively, if that doesn’t work, could I just cache the
transformation together with the lcmsContext it was created in and use
it from different threads concurrently on different images? And if so,
could multiple DIFFERENT transformations, which might be applied
concurrently, share the same context (assuming they are not CREATED
concurrently) or would I need a separate context per transformation?

Of course yes, many different transformations can share the same
context. That is how it is supposed to work: you create one and only one
context, put there all your plug--ins and operates in this context all
the time. The plug-in API documentation would give you more details on
contexts and plugins.

Hope that helps

Marti

 

On 8/25/2017 10:34 AM, Marco Freudenberger wrote:

All,

 

I’ve got a multithreading / lcms context related question on lcms2. I’m
currently using lcms 2.7, but certainly can upgrade to latest release,
if that makes any difference.

I tried to find the answer to this somewhere in the documentation, but
couldn’t get a definite answer.

 

For a project, I need to do color space conversion (from RGB -> CMYK,
not that it matters) in a dll.

The CMYK profiles are mostly large table based device profiles, so
creating the transformation takes a while (depending on the image size,
most often longer than applying it). The code might be called thousands
of times in an hour. As all the transformations are likely from the same
one or two RGB profile(s) to the same (or a little number of different)
CMYK profile(s), I would like to cache transformations.

 

The problem is, that the dll calls might be called from different
threads and might be called concurrently.

 

For each of the calls, in the original implementation I was:

-          creating a lcms context

-          load the profiles

-          create the transformation

-          close the profiles

-          apply the transformation to the image (potentially in
multiple threads)

-          delete the transformation

-          delete the lcms context. 

 

As I said, I would like to cache a small number of transformations.

 

As I wasn’t sure about the multithreading (transformation potentially
used in different threads) and concurrency (code might be called
multiple times) aspects, my first implementation is trying to play save
going through a device link:

-          create a lcms context

-          check if transformation from IN->OUT is in chache. if not:
load profiles, create transformation, close profiles, CREATE A
DEVICELINK FROM TRANSFORMATION, SAVE DEVICELINK IN MEMORY (CACHE)

-          if it is in cache: load transformation from devicelink in
cache

-          apply the transformation to the image (potentially in
multiple threads)

-          delete the transformation (not the devicelink).

-          delete the lcms context. 

-          if cache item needs to be cleared, delete the devicelink
buffer.

 

That process as two downsides: 

(a) loading a transformation from devicelink profile is much faster than
creating it from the in and out profile, but still takes quite some
time. 

(b) the results are slightly different (not as accurate?) when loading
the transformation from devicelink. I’m getting some CMYK value
differences up to ~2 or 3 values per channel on 8 bit images. 

 

What I’m wondering is: Once I’ve got a handle to a transformation, I can
apply it to one image in multiple threads (although right now I’m doing
that while the lcmsContext still exists). Also, to apply it, I don’t
need to pass in a lcms context to cmsDoTransform  (at least not
explicitly – maybe a handle is stored in the transformation).

 

 

So – my questions:

1. Does that mean I could actually just cache the transformation
(cmsHTRANSFORM) and re-use it, although the lcms context originally used
to load the profiles the transformation was deleted? Can I delete the
transformation from a different thread it was created from?

 

2. Alternatively, if that doesn’t work, could I just cache the
transformation together with the lcmsContext it was created in and use
it from different threads concurrently on different images? And if so,
could multiple DIFFERENT transformations, which might be applied
concurrently, share the same context (assuming they are not CREATED
concurrently) or would I need a separate context per transformation?

 

Thanks for any useful input…

Marco

 





------------------------------------------------------------------------
------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot





_______________________________________________
Lcms-user mailing list
Lcm...@li...
https://lists.sourceforge.net/lists/listinfo/lcms-user

 

Re: [Lcms-user] [EXTERNAL]Re: Multithreading - Transformation and lcms context

[Marco F. <Mar...@en...>]

Hi Marti!

Thanks a lot for your answer. I was under the impression by reading the documentation and previous post here, that besides the plug-in and memory-management configuration (I'm not using any of them right now), the context had something to do with concurrent access to global "internal data structures" of littlecms when working with multiple threads.
If not, that's actually great and I just can use my transformation across multiple threads without recreating it.

Sounds like it's no problem if I just cache the transformation itself.

Thanks a lot for your time answering this question, appreciate it!
Marco

Von: Martí Maria [mailto:mar...@li...]
Gesendet: Freitag, 25. August 2017 05:13
An: Marco Freudenberger; lcm...@li...
Betreff: [EXTERNAL]Re: [Lcms-user] Multithreading - Transformation and lcms context




Hi Marco,

Keep in mind cmsContext are only a way to store the plug-in configuration you are using. So, if you provide your own memory management routines, you may want to create a context at the very begin of your program to let lcms use your memory manager instead of the default malloc functions. contexts also stores some global settings. But that's all, contexts are not critical to threads. Deleting the context would just unregister your memory manager, or other plug-ins, if any.

On the other hand, cmsDoTransform and specially cmsDoTransformLineStride are re-entrant and can be used on different threads. You can create the transform in the base thread and apply the transform in many other different threads. The only requirement is memory should be accessible. If you provide your memory manager, the context is the way you channel the malloc and free calls to your memory manager.

You could just create the transform and keep it open while needed. That's what I do in the color translator and works fine. There is no need to convert to a devicelink.

Now for your questions:

> 1. Does that mean I could actually just cache the transformation (cmsHTRANSFORM) and re-use it, although the lcms context originally used to load the profiles the transformation was deleted? Can I delete the transformation from a different thread it was created from?

If you provide you own memory management functions, then creates the transform, then unregister you memory manager, further cmsDeleteTranform() would call default free(), is likely something will fail. I would just create the context at the begin of program and keep it open all the time to make sure your memory manager is being consistently called. Again, this is just to keep track of the plug-ins you are using. And yes, you can create the transform and reuse it. Even with different buffer formats in some cases.
> 2. Alternatively, if that doesn't work, could I just cache the transformation together with the lcmsContext it was created in and use it from different threads concurrently on different images? And if so, could multiple DIFFERENT transformations, which might be applied concurrently, share the same context (assuming they are not CREATED concurrently) or would I need a separate context per transformation?

Of course yes, many different transformations can share the same context. That is how it is supposed to work: you create one and only one context, put there all your plug--ins and operates in this context all the time. The plug-in API documentation would give you more details on contexts and plugins.

Hope that helps

Marti


On 8/25/2017 10:34 AM, Marco Freudenberger wrote:
All,

I've got a multithreading / lcms context related question on lcms2. I'm currently using lcms 2.7, but certainly can upgrade to latest release, if that makes any difference.
I tried to find the answer to this somewhere in the documentation, but couldn't get a definite answer.

For a project, I need to do color space conversion (from RGB -> CMYK, not that it matters) in a dll.
The CMYK profiles are mostly large table based device profiles, so creating the transformation takes a while (depending on the image size, most often longer than applying it). The code might be called thousands of times in an hour. As all the transformations are likely from the same one or two RGB profile(s) to the same (or a little number of different) CMYK profile(s), I would like to cache transformations.

The problem is, that the dll calls might be called from different threads and might be called concurrently.

For each of the calls, in the original implementation I was:

-          creating a lcms context

-          load the profiles

-          create the transformation

-          close the profiles

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation

-          delete the lcms context.

As I said, I would like to cache a small number of transformations.

As I wasn't sure about the multithreading (transformation potentially used in different threads) and concurrency (code might be called multiple times) aspects, my first implementation is trying to play save going through a device link:

-          create a lcms context

-          check if transformation from IN->OUT is in chache. if not: load profiles, create transformation, close profiles, CREATE A DEVICELINK FROM TRANSFORMATION, SAVE DEVICELINK IN MEMORY (CACHE)

-          if it is in cache: load transformation from devicelink in cache

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation (not the devicelink).

-          delete the lcms context.

-          if cache item needs to be cleared, delete the devicelink buffer.

That process as two downsides:
(a) loading a transformation from devicelink profile is much faster than creating it from the in and out profile, but still takes quite some time.
(b) the results are slightly different (not as accurate?) when loading the transformation from devicelink. I'm getting some CMYK value differences up to ~2 or 3 values per channel on 8 bit images.

What I'm wondering is: Once I've got a handle to a transformation, I can apply it to one image in multiple threads (although right now I'm doing that while the lcmsContext still exists). Also, to apply it, I don't need to pass in a lcms context to cmsDoTransform  (at least not explicitly - maybe a handle is stored in the transformation).


So - my questions:
1. Does that mean I could actually just cache the transformation (cmsHTRANSFORM) and re-use it, although the lcms context originally used to load the profiles the transformation was deleted? Can I delete the transformation from a different thread it was created from?

2. Alternatively, if that doesn't work, could I just cache the transformation together with the lcmsContext it was created in and use it from different threads concurrently on different images? And if so, could multiple DIFFERENT transformations, which might be applied concurrently, share the same context (assuming they are not CREATED concurrently) or would I need a separate context per transformation?

Thanks for any useful input...
Marco





------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most

engaging tech sites, Slashdot.org! http://sdm.link/slashdot




_______________________________________________

Lcms-user mailing list

Lcm...@li...<mailto:Lcm...@li...>

https://lists.sourceforge.net/lists/listinfo/lcms-user

Re: [Lcms-user] Multithreading - Transformation and lcms context

[Martí M. <mar...@li...>]

Hi Marco,

Keep in mind cmsContext are only a way to store the plug-in 
configuration you are using. So, if you provide your own memory 
management routines, you may want to create a context at the very begin 
of your program to let lcms use your memory manager instead of the 
default malloc functions. contexts also stores some global settings. But 
that's all, contexts are not critical to threads. Deleting the context 
would just unregister your memory manager, or other plug-ins, if any.

On the other hand, cmsDoTransform and specially cmsDoTransformLineStride 
are re-entrant and can be used on different threads. You can create the 
transform in the base thread and apply the transform in many other 
different threads. The only requirement is memory should be accessible. 
If you provide your memory manager, the context is the way you channel 
the malloc and free calls to your memory manager.

You could just create the transform and keep it open while needed. 
That's what I do in the color translator and works fine. There is no 
need to convert to a devicelink.

Now for your questions:

> 1. Does that mean I could actually just  cache the transformation (cmsHTRANSFORM) and re-use it, although the 
lcms context originally used to load the profiles the transformation was 
deleted? Can I delete the transformation from a different thread it was 
created from?

If you provide you own memory management functions, then creates the 
transform, then unregister you memory manager, further 
cmsDeleteTranform() would call default free(), is likely something will 
fail. I would just create the context at the begin of program and keep 
it open all the time to make sure your memory manager is being 
consistently called. Again, this is just to keep track of the plug-ins 
you are using. And yes, you can create the transform and reuse it. Even 
with different buffer formats in some cases.

> 2. Alternatively, if  that doesn’t work, could I just cache the transformation together with 
the lcmsContext it was created in and use it from different threads 
concurrently on different images? And if so, could multiple DIFFERENT 
transformations, which might be applied concurrently, share the same 
context (assuming they are not CREATED concurrently) or would I need a 
separate context per transformation?

Of course yes, many different transformations can share the same 
context. That is how it is supposed to work: you create one and only one 
context, put there all your plug--ins and operates in this context all 
the time. The plug-in API documentation would give you more details on 
contexts and plugins.

Hope that helps

Marti


On 8/25/2017 10:34 AM, Marco Freudenberger wrote:
>
> All,
>
> I’ve got a multithreading / lcms context related question on lcms2. 
> I’m currently using lcms 2.7, but certainly can upgrade to latest 
> release, if that makes any difference.
>
> I tried to find the answer to this somewhere in the documentation, but 
> couldn’t get a definite answer.
>
> For a project, I need to do color space conversion (from RGB -> CMYK, 
> not that it matters) in a dll.
>
> The CMYK profiles are mostly large table based device profiles, so 
> creating the transformation takes a while (depending on the image 
> size, most often longer than applying it). The code might be called 
> thousands of times in an hour. As all the transformations are likely 
> from the same one or two RGB profile(s) to the same (or a little 
> number of different) CMYK profile(s), I would like to cache 
> transformations.
>
> The problem is, that the dll calls might be called from different 
> threads and might be called concurrently.
>
> For each of the calls, in the original implementation I was:
>
> -creating a lcms context
>
> -load the profiles
>
> -create the transformation
>
> -close the profiles
>
> -apply the transformation to the image (potentially in multiple threads)
>
> -delete the transformation
>
> -delete the lcms context.
>
> As I said, I would like to cache a small number of transformations.
>
> As I wasn’t sure about the multithreading (transformation potentially 
> used in different threads) and concurrency (code might be called 
> multiple times) aspects, my first implementation is trying to play 
> save going through a device link:
>
> -create a lcms context
>
> -check if transformation from IN->OUT is in chache. if not: load 
> profiles, create transformation, close profiles, CREATE A DEVICELINK 
> FROM TRANSFORMATION, SAVE DEVICELINK IN MEMORY (CACHE)
>
> -if it is in cache: load transformation from devicelink in cache
>
> -apply the transformation to the image (potentially in multiple threads)
>
> -delete the transformation (not the devicelink).
>
> -delete the lcms context.
>
> -if cache item needs to be cleared, delete the devicelink buffer.
>
> That process as two downsides:
>
> (a) loading a transformation from devicelink profile is much faster 
> than creating it from the in and out profile, but still takes quite 
> some time.
>
> (b) the results are slightly different (not as accurate?) when loading 
> the transformation from devicelink. I’m getting some CMYK value 
> differences up to ~2 or 3 values per channel on 8 bit images.
>
> What I’m wondering is: Once I’ve got a handle to a transformation, I 
> can apply it to one image in multiple threads (although right now I’m 
> doing that while the lcmsContext still exists). Also, to apply it, I 
> don’t need to pass in a lcms context to cmsDoTransform  (at least not 
> explicitly – maybe a handle is stored in the transformation).
>
> So – my questions:
>
> 1. Does that mean I could actually just cache the transformation 
> (cmsHTRANSFORM) and re-use it, although the lcms context originally 
> used to load the profiles the transformation was deleted? Can I delete 
> the transformation from a different thread it was created from?
>
> 2. Alternatively, if that doesn’t work, could I just cache the 
> transformation together with the lcmsContext it was created in and use 
> it from different threads concurrently on different images? And if so, 
> could multiple DIFFERENT transformations, which might be applied 
> concurrently, share the same context (assuming they are not CREATED 
> concurrently) or would I need a separate context per transformation?
>
> Thanks for any useful input…
>
> Marco
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> Lcms-user mailing list
> Lcm...@li...
> https://lists.sourceforge.net/lists/listinfo/lcms-user

[Lcms-user] Multithreading - Transformation and lcms context

[Marco F. <Mar...@en...>]

All,

I've got a multithreading / lcms context related question on lcms2. I'm currently using lcms 2.7, but certainly can upgrade to latest release, if that makes any difference.
I tried to find the answer to this somewhere in the documentation, but couldn't get a definite answer.

For a project, I need to do color space conversion (from RGB -> CMYK, not that it matters) in a dll.
The CMYK profiles are mostly large table based device profiles, so creating the transformation takes a while (depending on the image size, most often longer than applying it). The code might be called thousands of times in an hour. As all the transformations are likely from the same one or two RGB profile(s) to the same (or a little number of different) CMYK profile(s), I would like to cache transformations.

The problem is, that the dll calls might be called from different threads and might be called concurrently.

For each of the calls, in the original implementation I was:

-          creating a lcms context

-          load the profiles

-          create the transformation

-          close the profiles

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation

-          delete the lcms context.

As I said, I would like to cache a small number of transformations.

As I wasn't sure about the multithreading (transformation potentially used in different threads) and concurrency (code might be called multiple times) aspects, my first implementation is trying to play save going through a device link:

-          create a lcms context

-          check if transformation from IN->OUT is in chache. if not: load profiles, create transformation, close profiles, CREATE A DEVICELINK FROM TRANSFORMATION, SAVE DEVICELINK IN MEMORY (CACHE)

-          if it is in cache: load transformation from devicelink in cache

-          apply the transformation to the image (potentially in multiple threads)

-          delete the transformation (not the devicelink).

-          delete the lcms context.

-          if cache item needs to be cleared, delete the devicelink buffer.

That process as two downsides:
(a) loading a transformation from devicelink profile is much faster than creating it from the in and out profile, but still takes quite some time.
(b) the results are slightly different (not as accurate?) when loading the transformation from devicelink. I'm getting some CMYK value differences up to ~2 or 3 values per channel on 8 bit images.

What I'm wondering is: Once I've got a handle to a transformation, I can apply it to one image in multiple threads (although right now I'm doing that while the lcmsContext still exists). Also, to apply it, I don't need to pass in a lcms context to cmsDoTransform  (at least not explicitly - maybe a handle is stored in the transformation).


So - my questions:
1. Does that mean I could actually just cache the transformation (cmsHTRANSFORM) and re-use it, although the lcms context originally used to load the profiles the transformation was deleted? Can I delete the transformation from a different thread it was created from?

2. Alternatively, if that doesn't work, could I just cache the transformation together with the lcmsContext it was created in and use it from different threads concurrently on different images? And if so, could multiple DIFFERENT transformations, which might be applied concurrently, share the same context (assuming they are not CREATED concurrently) or would I need a separate context per transformation?

Thanks for any useful input...
Marco

Re: [Lcms-user] One More Possible Important Compiler Warning

[Phil R. <phi...@or...>]

Since there has been a fair amount of changes recently which might
be compiler sensitive, I pulled the trunk just now (well 2 hours ago) 
and tried out
building OpenJDK 7, 8 and 9 with those bits using the JDK "offical"
compilers used to build each of those releases on Windows, Mac,
Linux (various versions), and Solaris. I had just one new build warning
that I noticed (but we do suppress a few) related to a usage in the
change in the return type of cmsReadRawTag from signed int to unsigned int :

jdk/src/java.desktop/share/native/liblcms/LCMS.c: In function 
'_writeCookedTag':

jdk/src/java.desktop/share/native/liblcms/LCMS.c:853:29: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
                  if (tagSize ==  cmsReadRawTag(pfTarget, s, buf, tagSize)) {


So no problem compiling the LCMS code itself. Just the one small
change needed in our code that calls into it.

We don't (yet) use VS2017 for compiling JDK on Windows, but it
seems that is being verified already by Noel, so far, so good.

-phil.

On 08/24/2017 09:24 AM, Martí Maria wrote:
>
>
> Thanks Noel, I have incorporated your changes in the main trunk. But 
> don't worry too much, this function is not used by the rest of the 
> library as it is kept here for compatibility with old apps.
>
> Thanks again
>
> Marti
>
>
> On 8/9/2017 8:34 PM, Noel Carboni wrote:
>>
>> Hi Marti,
>>
>> By the way, there is one more possibly important warning you should 
>> think about addressing for release 2.9, this time issued from the VS 
>> 2017 Code Analyzer:
>>
>> *c:\dev\prodigital\trunk\libraries\littlecms\src\cmsgamma.c(1153): 
>> warning C6262: Function uses '49196' bytes of stack: exceeds 
>> /analyze:stacksize '16384'.  Consider moving some data to heap.*
>>
>> This is because of the following code, in cmsgamma.c:
>>
>> // Maxim number of nodes
>>
>> #define MAX_NODES_IN_CURVE   4097
>>
>> ...
>>
>> // Smooths a curve sampled at regular intervals.
>>
>> cmsBool CMSEXPORT cmsSmoothToneCurve(cmsToneCurve* Tab, 
>> cmsFloat64Number lambda)
>>
>> {
>>
>> cmsFloat32Number w[MAX_NODES_IN_CURVE], y[MAX_NODES_IN_CURVE], 
>> z[MAX_NODES_IN_CURVE];
>>
>> The w, y, and z arrays are a bit on the large size for stack 
>> variables.  I did some reading on Windows stack allocations and the 
>> stack may be quite limited under some conditions.  It's possible the 
>> same applies to other architectures too.
>>
>> I recommend considering changing the code to malloc several temporary 
>> buffers instead of allocating blocks of stack space.  Here's a 
>> suggested implementation using _cmsCalloc / _cmsFree, and which is 
>> more structured and now properly reports all errors.  It passes all 
>> tests and performance is good, per the testbed, though I recommend a 
>> very careful review.  Both the current Github code and proposed new 
>> code blocks are in this file.  Compile the new code for comparative 
>> testing by defining the symbol PRODIGITAL_OPTIMIZATIONS, which can 
>> also be used to find the changes in the source.
>>
>> http://Noel.ProDigitalSoftware.com/temp/cmsgamma_proposed_changes.zip
>>
>> Please let me know if you have questions.
>>
>> -Noel
>>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> Lcms-user mailing list
> Lcm...@li...
> https://lists.sourceforge.net/lists/listinfo/lcms-user

Re: [Lcms-user] One More Possible Important Compiler Warning

[Martí M. <mar...@li...>]

Thanks Noel, I have incorporated your changes in the main trunk. But 
don't worry too much, this function is not used by the rest of the 
library as it is kept here for compatibility with old apps.

Thanks again

Marti


On 8/9/2017 8:34 PM, Noel Carboni wrote:
>
> Hi Marti,
>
> By the way, there is one more possibly important warning you should 
> think about addressing for release 2.9, this time issued from the VS 
> 2017 Code Analyzer:
>
> *c:\dev\prodigital\trunk\libraries\littlecms\src\cmsgamma.c(1153): 
> warning C6262: Function uses '49196' bytes of stack: exceeds 
> /analyze:stacksize '16384'.  Consider moving some data to heap.*
>
> This is because of the following code, in cmsgamma.c:
>
> // Maxim number of nodes
>
> #define MAX_NODES_IN_CURVE   4097
>
> ...
>
> // Smooths a curve sampled at regular intervals.
>
> cmsBool CMSEXPORT cmsSmoothToneCurve(cmsToneCurve* Tab, 
> cmsFloat64Number lambda)
>
> {
>
> cmsFloat32Number w[MAX_NODES_IN_CURVE], y[MAX_NODES_IN_CURVE], 
> z[MAX_NODES_IN_CURVE];
>
> The w, y, and z arrays are a bit on the large size for stack 
> variables.  I did some reading on Windows stack allocations and the 
> stack may be quite limited under some conditions.  It's possible the 
> same applies to other architectures too.
>
> I recommend considering changing the code to malloc several temporary 
> buffers instead of allocating blocks of stack space.  Here's a 
> suggested implementation using _cmsCalloc / _cmsFree, and which is 
> more structured and now properly reports all errors.  It passes all 
> tests and performance is good, per the testbed, though I recommend a 
> very careful review.  Both the current Github code and proposed new 
> code blocks are in this file.  Compile the new code for comparative 
> testing by defining the symbol PRODIGITAL_OPTIMIZATIONS, which can 
> also be used to find the changes in the source.
>
> http://Noel.ProDigitalSoftware.com/temp/cmsgamma_proposed_changes.zip
>
> Please let me know if you have questions.
>
> -Noel
>

Re: [Lcms-user] Revision 485 "Constify some big arrays to allow to compiler put them in read only section." breaks VS 2017 builds

[Noel C. <NCa...@Pr...>]

Looks like you've got the trunk code compiling again Marti.  Thank you.

I presume, given that the tables are linked lists, that those tables are
not designed to be expanded (i.e., by linking additional entries to
them).  That they are qualified as const now precludes that.

 

I'm sending these messages about development code to the lcms-user
list...  Is there an lcms-developer list I should rather be using
instead?

 

-Noel

 

From: Noel Carboni [mailto:NCa...@Pr...] 
Sent: Tue, August 22, 2017 12:09 PM
To: 'lcm...@li...'
Subject: Revision 485 "Constify some big arrays to allow to compiler put
them in read only section." breaks VS 2017 builds

 

Hi folks,

 

The current Little CMS trunk does not build at all with the VS 2017 C++
compiler.

 

    ..\..\..\src\cmspack.c(3018): error C2440: 'initializing': cannot
convert from 'const cmsFormatters16 *' to 'cmsFormatters16 *'

    ..\..\..\src\cmspack.c(3018): note: Conversion loses qualifiers

    ..\..\..\src\cmspack.c(3030): error C2440: 'initializing': cannot
convert from 'const cmsFormattersFloat *' to 'cmsFormattersFloat *'

    ..\..\..\src\cmspack.c(3030): note: Conversion loses qualifiers

    ..\..\..\src\cmspack.c(3175): error C2440: 'initializing': cannot
convert from 'const cmsFormatters16 *' to 'cmsFormatters16 *'

    ..\..\..\src\cmspack.c(3175): note: Conversion loses qualifiers

 

    ..\..\..\src\cmstypes.c(5329): error C2440: 'initializing': cannot
convert from 'const _cmsTagTypeLinkedList *' to
'_cmsTagTypeLinkedList_st *'

    ..\..\..\src\cmstypes.c(5329): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5419): error C2664: 'cmsTagTypeHandler
*GetHandler(cmsTagTypeSignature,_cmsTagTypeLinkedList
*,_cmsTagTypeLinkedList *)': cannot convert argument 3 from 'const
_cmsTagTypeLinkedList [31]' to '_cmsTagTypeLinkedList *'

    ..\..\..\src\cmstypes.c(5419): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5524): error C2440: 'initializing': cannot
convert from 'const _cmsTagLinkedList *' to '_cmsTagLinkedList_st *'

    ..\..\..\src\cmstypes.c(5524): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5624): error C2440: '=': cannot convert from
'const _cmsTagLinkedList [64]' to '_cmsTagLinkedList *'

    ..\..\..\src\cmstypes.c(5624): note: Conversion loses qualifiers

 

The change is a good idea but the whole job has to be done, so that all
the types match everywhere.

 

-Noel

[Lcms-user] Revision 485 "Constify some big arrays to allow to compiler put them in read only section." breaks VS 2017 builds

[Noel C. <NCa...@Pr...>]

Hi folks,

 

The current Little CMS trunk does not build at all with the VS 2017 C++
compiler.

 

    ..\..\..\src\cmspack.c(3018): error C2440: 'initializing': cannot
convert from 'const cmsFormatters16 *' to 'cmsFormatters16 *'

    ..\..\..\src\cmspack.c(3018): note: Conversion loses qualifiers

    ..\..\..\src\cmspack.c(3030): error C2440: 'initializing': cannot
convert from 'const cmsFormattersFloat *' to 'cmsFormattersFloat *'

    ..\..\..\src\cmspack.c(3030): note: Conversion loses qualifiers

    ..\..\..\src\cmspack.c(3175): error C2440: 'initializing': cannot
convert from 'const cmsFormatters16 *' to 'cmsFormatters16 *'

    ..\..\..\src\cmspack.c(3175): note: Conversion loses qualifiers

 

    ..\..\..\src\cmstypes.c(5329): error C2440: 'initializing': cannot
convert from 'const _cmsTagTypeLinkedList *' to
'_cmsTagTypeLinkedList_st *'

    ..\..\..\src\cmstypes.c(5329): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5419): error C2664: 'cmsTagTypeHandler
*GetHandler(cmsTagTypeSignature,_cmsTagTypeLinkedList
*,_cmsTagTypeLinkedList *)': cannot convert argument 3 from 'const
_cmsTagTypeLinkedList [31]' to '_cmsTagTypeLinkedList *'

    ..\..\..\src\cmstypes.c(5419): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5524): error C2440: 'initializing': cannot
convert from 'const _cmsTagLinkedList *' to '_cmsTagLinkedList_st *'

    ..\..\..\src\cmstypes.c(5524): note: Conversion loses qualifiers

    ..\..\..\src\cmstypes.c(5624): error C2440: '=': cannot convert from
'const _cmsTagLinkedList [64]' to '_cmsTagLinkedList *'

    ..\..\..\src\cmstypes.c(5624): note: Conversion loses qualifiers

 

The change is a good idea but the whole job has to be done, so that all
the types match everywhere.

 

-Noel

Re: [Lcms-user] Build warnings on OSX / clang

[Martí M. <mar...@li...>]

Fixed by 
https://github.com/mm2/Little-CMS/commit/7e32c6fcb930754406f1174591dee85627eb0030

Thank you

Marti



On 8/9/2017 3:17 PM, Philip Race wrote:
> [RESEND]
>
> I submitted the current git repo state into the JDK build system and 
> GCC 4.82 (if I believe our logging)
> did not like these lines for .. the reason it states ..
> /sjdk/src/java.desktop/share/native/liblcms/cmsopt.c:1550:48: error: 
> signed and unsigned type in conditional expression [-Werror=sign-compare]
>      ri = (l1 < 0) ? 0 : ((l1 > 16384) ? 16384U : l1);
>                                                 ^
> /s/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:1551:48: error: 
> signed and unsigned type in conditional expression [-Werror=sign-compare]
>      gi = (l2 < 0) ? 0 : ((l2 > 16384) ? 16384U : l2);
>                                                 ^
> /s/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:1552:48: error: 
> signed and unsigned type in conditional expression [-Werror=sign-compare]
>      bi = (l3 < 0) ? 0 : ((l3 > 16384) ? 16384U : l3);
>
> -phil.
>
>

[Lcms-user] One More Possible Important Compiler Warning

[Noel C. <NCa...@Pr...>]

Hi Marti,

 

By the way, there is one more possibly important warning you should
think about addressing for release 2.9, this time issued from the VS
2017 Code Analyzer:

 

c:\dev\prodigital\trunk\libraries\littlecms\src\cmsgamma.c(1153):
warning C6262: Function uses '49196' bytes of stack:  exceeds
/analyze:stacksize '16384'.  Consider moving some data to heap.

 

This is because of the following code, in cmsgamma.c:

 

 

// Maxim number of nodes

#define MAX_NODES_IN_CURVE   4097

 

...

 

// Smooths a curve sampled at regular intervals.

cmsBool  CMSEXPORT cmsSmoothToneCurve(cmsToneCurve* Tab,
cmsFloat64Number lambda)

{

    cmsFloat32Number w[MAX_NODES_IN_CURVE], y[MAX_NODES_IN_CURVE],
z[MAX_NODES_IN_CURVE];

 

 

The w, y, and z arrays are a bit on the large size for stack variables.
I did some reading on Windows stack allocations and the stack may be
quite limited under some conditions.  It's possible the same applies to
other architectures too.

 

I recommend considering changing the code to malloc several temporary
buffers instead of allocating blocks of stack space.  Here's a suggested
implementation using _cmsCalloc / _cmsFree, and which is more structured
and now properly reports all errors.  It passes all tests and
performance is good, per the testbed, though I recommend a very careful
review.  Both the current Github code and proposed new code blocks are
in this file.  Compile the new code for comparative testing by defining
the symbol PRODIGITAL_OPTIMIZATIONS, which can also be used to find the
changes in the source.

 

http://Noel.ProDigitalSoftware.com/temp/cmsgamma_proposed_changes.zip

 

Please let me know if you have questions.

 

-Noel

 

Re: [Lcms-user] Build warnings on OSX / clang

[Phil R. <phi...@or...>]

I've experienced oddities (in other code) where version N of a compiler 
issued a quite valid
looking warning and rev N+m decided it was no longer a problem ...

So yes, testing with various compiler versions is good and I don't think 
they
always monotonically get better.

-phil.


On 08/09/2017 09:08 AM, Noel Carboni wrote:
>
> An anecdote:  On the advice of Bob Friesenhahn who a few days ago 
> suggested building things with multiple compilers to check code, I've 
> been doing that with our in-house products lately...
>
> We hadn't thought to take the time to do that before.  Specifically, 
> we're rebuilding our Visual Studio C++ projects with clang and the 
> -Wall option, and sure enough it's uncovered several bona fide bugs 
> that had escaped the notice of Visual Studio's compiler alone.  I 
> guess they all have some blind spots!
>

Re: [Lcms-user] Build warnings on OSX / clang

[Noel C. <NCa...@Pr...>]

For what it's worth I saw both your recent messages, Philip.  It's kind
of up to Marti whether he wants to tweak that code to make it not emit a
warning.

 

Indeed those instructions are mixing unsigned and signed values (e.g.,
under some runtime conditions the signed cmsS1Fixed14Number values in
l1, l2, l3 could be assigned to the unsigned cmsUInt32Number variables
ri, gi, bi.  Interestingly, Visual Studio's C++ compiler does not flag
that as a warning...  It's possible it has figured out that the code
does not logically allow a signed negative value to be copied to an
unsigned lvalue.

 

In any case, for other compilers a change that's valid everywhere using
casting to work around this warning would be to code the following:

 

    // Now we have to clip to 0..1.0 range

    ri = (l1 < 0) ? 0 : ((l1 > 16384) ? 16384U : (cmsUInt32Number) l1);

    gi = (l2 < 0) ? 0 : ((l2 > 16384) ? 16384U : (cmsUInt32Number) l2);

    bi = (l3 < 0) ? 0 : ((l3 > 16384) ? 16384U : (cmsUInt32Number) l3);

 

An anecdote:  On the advice of Bob Friesenhahn who a few days ago
suggested building things with multiple compilers to check code, I've
been doing that with our in-house products lately...  

 

We hadn't thought to take the time to do that before.  Specifically,
we're rebuilding our Visual Studio C++ projects with clang and the -Wall
option, and sure enough it's uncovered several bona fide bugs that had
escaped the notice of Visual Studio's compiler alone.  I guess they all
have some blind spots!

 

Many thanks for the idea, Bob!

 

-Noel

Re: [Lcms-user] Build warnings on OSX / clang

[Philip R. <phi...@or...>]

[RESEND]

I submitted the current git repo state into the JDK build system and GCC 
4.82 (if I believe our logging)
did not like these lines for .. the reason it states ..
/sjdk/src/java.desktop/share/native/liblcms/cmsopt.c:1550:48: error: 
signed and unsigned type in conditional expression [-Werror=sign-compare]
      ri = (l1 < 0) ? 0 : ((l1 > 16384) ? 16384U : l1);
                                                 ^
/s/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:1551:48: error: 
signed and unsigned type in conditional expression [-Werror=sign-compare]
      gi = (l2 < 0) ? 0 : ((l2 > 16384) ? 16384U : l2);
                                                 ^
/s/jdk/src/java.desktop/share/native/liblcms/cmsopt.c:1552:48: error: 
signed and unsigned type in conditional expression [-Werror=sign-compare]
      bi = (l3 < 0) ? 0 : ((l3 > 16384) ? 16384U : l3);

-phil.

[Lcms-user] War on warnings

[Aaron B. <bo...@gm...>]

Thanks to everyone for their help with the warnings - my OSX build is now
completely
warning free! It is a pleasure to see that green bar with 0 warnings.

Cheers,
Aaron

Re: [Lcms-user] OSX Warnings

[Aaron B. <bo...@gm...>]

Thanks, guys.  In this particular case, I think the problem is just
using a signed int to store something that always be positive:

fl->nFunctions

and

c->ParameterCount[Pos]



On Fri, Aug 4, 2017 at 8:51 AM, Bob Friesenhahn <
bfr...@si...> wrote:

> On Fri, 4 Aug 2017, Martí Maria wrote:
>
>
>> Hi Bob,
>>
>> Thanks for pointing out this. Yes, x64 is supported and tested. I have
>> done a search of "size_t" and the core library does not use it, nor
>> unsigned long. The places where integer overflow can happen are known, as
>> this may represent a security risk.
>>
>
> Size_t is implicitly used by memcpy(), malloc(), and other OS functions
> which require a size type.  Doing size computations using the size_t type
> is not a bad thing, especially if it feeds an interface already using the
> size_t type.
>
> On the flip side, Windows provides only a 32-bit stdio API (even in a
> 64-bit build), which results in more opportunity for problems.
>
>
> Bob
> --
> Bob Friesenhahn
> bfr...@si..., http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Lcms-user mailing list
> Lcm...@li...
> https://lists.sourceforge.net/lists/listinfo/lcms-user
>
>

Re: [Lcms-user] OSX Warnings

[Bob F. <bfr...@si...>]

On Fri, 4 Aug 2017, Martí Maria wrote:

>
> Hi Bob,
>
> Thanks for pointing out this. Yes, x64 is supported and tested. I have done a 
> search of "size_t" and the core library does not use it, nor unsigned long. 
> The places where integer overflow can happen are known, as this may represent 
> a security risk.

Size_t is implicitly used by memcpy(), malloc(), and other OS 
functions which require a size type.  Doing size computations using 
the size_t type is not a bad thing, especially if it feeds an 
interface already using the size_t type.

On the flip side, Windows provides only a 32-bit stdio API (even in a 
64-bit build), which results in more opportunity for problems.

Bob
-- 
Bob Friesenhahn
bfr...@si..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Re: [Lcms-user] OSX Warnings

[Noel C. <NCa...@Pr...>]

>Have test compiles also been done for 64-bit Windows builds?

Yes, that's what I have been doing.

Not to worry, the 64 bit Visual Studio compiler for Windows lets you
know if you have coded implicit conversions that could lose data.  I'll
confirm what Marti has said - the Little CMS library compiles clean
w/regard to 32 vs. 64 bit variables (not to mention runs properly) on
x64 Windows.

>> http://my.cdash.org/buildSummary.php?buildid=1262991

>For 64-bit Windows builds, 'unsigned long' is only a 32-bit type but
'size_t' is a 64-bit type.  It seems likely that
>there are many more problems when computing size_t values given
apparent assumptions that 'unsigned long' will be >large enough.
Integer overflow may occur.
>There will also be more prominence of sign-extension problems, which
can create security risks.

Note that the warnings Aaron showed are specifically about sign
conversion, not about data size, because it is possible data could be
lost (e.g., negative numbers turned into positive numbers) no matter
what size the variables are.

It is always possible to implicitly promote a smaller unsigned integer
into a larger one, thus you would not see a warning if a 32 bit unsigned
int value were provided in the 3rd argument of a memmove call.

Although IMHO it's often better form to use size_t when you are
expressing the size of something in code, even if you know it will
always fit in a smaller unsigned int or unsigned long variable.

That being said, I have seen both better and worse generated code by
using a 64 bit size_t data type for e.g., an index variable.  If there
are sufficient registers for the compiler to optimize the coded routine
well, the generated code is usually better, since the machine
instructions needed to convert 32 bit values to 64 bit values are
eliminated.  But it isn't always the case, especially if there's
register starvation (e.g., a function is a bit more complex than what
"fits" well with the processor architecture).  In those cases, when data
has to be pushed onto / popped off of the stack, the additional overhead
of moving 64 bits to/from RAM where 32 would suffice (because the number
is small, something like 0x0000000000000004) can actually slow things
down.

-Noel

Re: [Lcms-user] OSX Warnings

[Martí M. <mar...@li...>]

Hi Bob,

Thanks for pointing out this. Yes, x64 is supported and tested. I have 
done a search of "size_t" and the core library does not use it, nor 
unsigned long. The places where integer overflow can happen are known, 
as this may represent a security risk.

Best regards

Marti


On 8/3/2017 7:29 PM, Bob Friesenhahn wrote:
> For 64-bit Windows builds, 'unsigned long' is only a 32-bit type but 
> 'size_t' is a 64-bit type.  It seems likely that there are many more 
> problems when computing size_t values given apparent assumptions that 
> 'unsigned long' will be large enough. Integer overflow may occur. 
> There will also be more prominence of sign-extension problems, which 
> can create security risks.
>
> Have test compiles also been done for 64-bit Windows builds?
>
> Bob

Re: [Lcms-user] OSX Warnings

[Bob F. <bfr...@si...>]

On Wed, 2 Aug 2017, Aaron Boxer wrote:

> Thanks to Noel and Marti for these improvements !
>
> On OSX, my warning count has gone from 20 to 2.
>
> Here are the two remaining warnings:
>
> http://my.cdash.org/buildSummary.php?buildid=1262991

For 64-bit Windows builds, 'unsigned long' is only a 32-bit type but 
'size_t' is a 64-bit type.  It seems likely that there are many more 
problems when computing size_t values given apparent assumptions that 
'unsigned long' will be large enough.  Integer overflow may occur. 
There will also be more prominence of sign-extension problems, which 
can create security risks.

Have test compiles also been done for 64-bit Windows builds?

Bob
-- 
Bob Friesenhahn
bfr...@si..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

[Lcms-user] OSX Warnings

[Aaron B. <bo...@gm...>]

Thanks to Noel and Marti for these improvements !

On OSX, my warning count has gone from 20 to 2.

Here are the two remaining warnings:

http://my.cdash.org/buildSummary.php?buildid=1262991

Aaron

Re: [Lcms-user] Build warnings on OSX / clang

[Martí M. <mar...@li...>]

Hello,


Following Noel Carboni's suggestion and proof of concept, I have 
submitted a big commit that  gets rid of signed/unsigned mixing warnings 
when compiling as C++ with -Wall -Wpedantic

I've checked it with several compilers, clang, Visual Studio 15 and gcc 
3, 4 and 5, Borland C 5.5 and Embarcadero C++ 10.1. All passes the 
testbed and all my internal test beds on plug-ins.

Some thoughts:

- There are changes in the API. Most are identifying what 'int' means 
really. I regard this as bug fixes, since size of int may change. Ditto 
for some parameters that had to be declared as unsigned and were not. 
Overall, no client app should be broken in any case.
- The interpolation routines are untouched, rather than putting tons of 
casts I used #pragma to prevent warnings. The default interpolation is 
fine-tuned to match Photoshop, this is very valuable for some users and 
changing it is extremely risky. For things like throughput gains, please 
see below.
- In some places there are now ugly casts that bring nothing but noise, 
but I understand in many other places a strict type check is very 
desirable, so here it is.

Now regarding throughput optimization.

 From time to time people keeps sending me improvements for increasing 
throughput. Many times that involves changing the default interpolation 
code or adding SSE or VEC assembly in the base lcms2. I appreciate those 
collaborations, but the default interpolation is matching quite well 
photoshop and there is a clean, documented ways to increase throughput 
without sacrificing compatibility. That is the plug-in system. There are 
several plug-in types devoted to throughput for different usages. The 
right function to call when using those plug-in is 
cmsDoTransformLineStride(), which prepares buffers in the proper way.

Just as an example, I have plug-ins for fast processing that delivers:

P E R F O R M A N C E   T E S T S   8  B I T S
==============================================

                                   MPixel/sec. MByte/sec.
8 bits on CLUT profiles         : 28.42        85.26
8 bits on Matrix-Shaper profiles: 85.56        256.68
8 bits on same Matrix-Shaper    : 432.43       1297.30
8 bits on curves                : 253.97       761.90

P E R F O R M A N C E   T E S T S  F L O A T  (P L U G I N)
===========================================================

                                   MPixel/sec. MByte/sec.
Floating point on CLUT profiles  : 20.13 241.51       (x 12.3)
Floating point on Matrix-Shaper  : 19.05 228.57       (x 7.8)
Floating point on same MatrixSh  : 106.67 1280.00      (x 53.9)
Floating point on curves         : 44.82 537.82       (x 20.5)

And others for other types with similar results. Sorry, those plugins 
are not released as open source and I want not to spam you with 
commercial ads. It is just and example that what can be done without 
touching the default routines.


You can reach latest changes at 
https://github.com/mm2/Little-CMS/commit/55df275dc58fc817a59bf52c1b2bb564433c3999 
or just by downloading the github lcms2 trunk

I will be packing lcms2.9 in few days. This version is supposed to add 
stability and will remain for a while as because personal matters I have 
actually little time to dedicate to lcms.

Thanks again, Noel for all your hard work in this issue.
Best regards
Marti


On 7/26/2017 6:26 AM, Noel Carboni wrote:
>
> Hi guys,
>
> I've had a pass through the code.  I made several hundred changes to 
> 20 or so source files, and now have it building here with warnings set 
> to max in Visual Studio 2017.  Mostly the changes were to make 
> unsigned variables out of signed variables, and I didn't have to add 
> much casting.
>
> I have tested it in my own application (which of course only exercises 
> part of the library), and it seems to work great. I'm not presently 
> set up to run the battery of tests included with the software, so I 
> hope someone can help with that.  It would be also interesting to know 
> if other compilers (e.g., clang) find fault where Visual Studio does not.
>
> Here's the source code:
>
> http://Noel.ProDigitalSoftware.com/temp/LittleCMS_2.8NC.zip
>
> I've made a number of changes to the release 2.8 sources with the 
> following philosophies in mind:
>
> 1.  Don't alter the external interface (lcms2.h)
>
> 2.  Minimize the changes Marti will have to review and do them as 
> safely as possible.
>
> 3.  Make the use of signed and unsigned types consistent to reduce the 
> hundreds of warnings emitted when -Wall is used.
>
> 4.  Use as few casts as possible.
>
> 4.  Maintain efficiency.
>
> With this set of files you can now enable -Wall in Visual Studio 2017 
> builds for ongoing LittleCMS development work, to gain the benefits of 
> tighter type checking, but with the following caveats:
>
> A.  You'll most likely want to specifically disable several of the 
> -Wall warnings by putting the following additional options on the 
> C/C++ compiler command line:
>
>          /wd4711 /wd4820 /wd4061 /wd4774 /wd4710 /wd4668
>
> These specifically quiet the following Visual Studio 2017 C++ 
> warnings, which may not be helpful to you:
>
> warning C4711: function 'xxxxxxxxxxxxxxxxx' selected for automatic 
> inline expansion
>
> warning C4820: 'xxxxxxxx': 'n' bytes padding added after data member
>
>          warning C4061: enumerator 'xxxxxxxxxxxxx' in switch of enum 
> 'yyyyyyyyyyyyy' is not explicitly handled by a case label
>
> warning C4774: '_snprintf' : format string expected in argument 3 is 
> not a string literal
>
> warning C4710: 'int _snprintf(char *const ,const ::size_t,const char 
> *const ,...)': function not inlined
>
> warning C4668: '_M_HYBRID_X86_ARM64' is not defined as a preprocessor 
> macro, replacing with '0' for '#if/#elif'
>
> B.  There is one warning, in cmsxform.c, that may indicate a real 
> problem, especially in light of compilation of this library for 
> different systems with different compilers.  Marti, you may want to 
> review the code and decide whether to change it or suppress the warning:
>
> ..\..\..\src\cmsxform.c(799): warning C4191: 'type cast': unsafe 
> conversion from '_cmsTransform2Fn' to '_cmsTransformFn'
>
> The above text, to help Marti see what I've done, is also in a comment 
> block in lcms2_internal.h.
>
> I suggest careful review and testing of these changes should be done.  
> I don't think I have corrected any serious logic errors, and I hope I 
> haven't caused any new ones.  The compiler's additional scrutiny can 
> be brought to bear going forward to help Marti et. al. with ongoing 
> development.
>
> -Noel
>
> *From:*Aaron Boxer [mailto:bo...@gm...]
> *Sent:* Mon, July 24, 2017 7:03 PM
> *To:* Marti Maria
> *Cc:* Noel Carboni; lcm...@li...
> *Subject:* Re: [Lcms-user] Build warnings on OSX / clang
>
> Thanks to Noel for offering fixes, and to Marti for graciously 
> accepting :)
>
> I've gone through similar issues with my JPEG 2000 codec, much of the 
> code inherited from developers
> who didn't worry about these issues.  I spent a lot of time converting 
> signed to unsigned for all quantities
> that must always be positive : number of image components, image 
> dimensions, etc.
>
> Also, removed as many casts as possible. The code is now more 
> resilient to overflow, and compiles with no
>
> warnings at maximum -Wall compiler settings. I think it is worth the 
> effort.
>
> Regards,
>
> Aaron
>
>
>
>
> On Mon, Jul 24, 2017 at 6:31 PM, Marti Maria 
> <mar...@li... <mailto:mar...@li...>> wrote:
>
> Sounds great. All improvements are more than welcome.
>
> Regards
>
> Marti
>
> On 25 Jul 2017 00:27, Noel Carboni <NCa...@Pr... 
> <mailto:NCa...@Pr...>> wrote:
>
> Hi Marti,
>
> As C/C++ programmers we have to face the fact that the language is 
> growing ever more tightly typed.
>
> It's not quite correct even to promote implicitly from signed to 
> unsigned when you "just know" that the values will be small, since 
> under some conditions negative numbers could have meaning.  Or worse, 
> be given special meaning in the future.  It's tempting to return -1 as 
> a special error or "not found" case, but that can ultimately cause 
> conflict with code that expects only signed values.
>
> It's only really properly done - and thus "safe" from future 
> maintenance problems - if you use consistent types across the board, 
> or add casts occasionally when you really do want to express that 
> you're fitting a small positive unsigned value into a signed field.
>
> Looking your released code over just now I think that it can be 
> substantially cleaned up without a lot of casts added.  There are a 
> lot of cases where you have a habit of using "int" when what you 
> really want is "unsigned int" (e.g., for number of channels, where 
> negative values don't have meaning).  :)
>
> Tell you what; I'll go through all the code and see what I can come up 
> with to get it down to zero warnings at max warning level.  I don't 
> think it'll take me more than a few man-hours.  I'll send the result 
> to you and you can compare the files and see whether you feel the 
> changes are safe enough.  I'll make it my mission not to change the 
> logic or your intent with the style.
>
> -Noel
>
> *From:*Marti Maria [mailto:mar...@li... 
> <mailto:mar...@li...>]
> *Sent:* Mon, July 24, 2017 6:09 PM
> *To:* Noel Carboni
> *Cc:* lcm...@li... 
> <mailto:lcm...@li...>
> *Subject:* Re: [Lcms-user] Build warnings on OSX / clang
>
> Hi,
>
> Sometimes I use  signed to unsigned promotion without cast. This is 
> safe  if values are small and positive. Mostly to avoid casts in 
> memset, memmove, calloc, etc. which looks really ugly.
>
> If you can found other cases, like that one in cmsgamma.c, please let 
> me know. But this latter was fixed after the original report.
>
> Regards
>
> Marti
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Lcms-user mailing list
> Lcm...@li... <mailto:Lcm...@li...>
> https://lists.sourceforge.net/lists/listinfo/lcms-user
>