dev-cpp-users

[Dev-C++] Optimized memory allocation

[Shane B. <ast...@te...>]

Hello,

    I have recently been attempting to find ways to speed up the process =
of loading 48k records from a MySQL SELECT statement, which has a 32 =
char primary key.  One of the more intensive processes involved is =
calling "new" to create a new object 48k times.  In standard C, I would =
malloc(RowCount * sizeof(class_type)), and using pointer math, simply =
iterate whenever I needed a new structure.  With C++, a call to new is =
required for construction purposes.  Is there an optimized method =
wherein I can allocate a dynamic number to be preallocated more =
efficiently all at once in one call?
    I have considered using "new class_type[RowCount]()" to allocate an =
array (haven't tried to see if that would actually work), but I was =
wondering if there was a more efficient means to preallocate objects.

Thanks,
    -Shane

RE: [Dev-C++] Optimized memory allocation

[Ioannis V. <iv...@em...>]

new whatever[SIZE] is equivalent to malloc(SIZE*sizeof(whatever)), that =
is
it should boe slower.
=20
But better use an std::vector:
=20
=20
#include <vector>
=20
// ...
=20
using std::vector;
=20
// ...
=20
vector<class_type> array(48);
=20
// ...
array[0]=3D ...
=20
=20
=20
=20
=20
Ioannis Vranos

* Programming pages:  <http://www.noicys.freeurl.com/>
http://www.noicys.freeurl.com
* Alternative URL 1:  <http://run.to/noicys> http://run.to/noicys
* Alternative URL 2:  <http://www.noicys.cjb.net/> =
http://www.noicys.cjb.net

=20
=20
=20

-----Original Message-----
From: dev...@li...
[mailto:dev...@li...] On Behalf Of Shane =
Bryldt
Sent: Friday, June 20, 2003 3:39 AM
To: C++
Subject: [Dev-C++] Optimized memory allocation


Hello,
=20
    I have recently been attempting to find ways to speed up the process =
of
loading 48k records from a MySQL SELECT statement, which has a 32 char
primary key.  One of the more intensive processes involved is calling =
"new"
to create a new object 48k times.  In standard C, I would =
malloc(RowCount *
sizeof(class_type)), and using pointer math, simply iterate whenever I
needed a new structure.  With C++, a call to new is required for
construction purposes.  Is there an optimized method wherein I can =
allocate
a dynamic number to be preallocated more efficiently all at once in one
call?
    I have considered using "new class_type[RowCount]()" to allocate an
array (haven't tried to see if that would actually work), but I was
wondering if there was a more efficient means to preallocate objects.
=20
Thanks,
    -Shane

RE: [Dev-C++] Optimized memory allocation

[Ioannis V. <iv...@em...>]

Bug fix:
=20

-----Original Message-----
From: dev...@li...
[mailto:dev...@li...] On Behalf Of Ioannis
Vranos
Sent: Friday, June 20, 2003 5:48 AM
To: 'C++'
Subject: RE: [Dev-C++] Optimized memory allocation


 =3D=3D>  new whatever[SIZE]  should be equivalent to
malloc(SIZE*sizeof(whatever))  in terms of speed, where whatever is a =
POD
type.
=20
 But better use an std::vector:
=20
=20
#include <vector>
=20
// ...
=20
using std::vector;
=20
// ...
=20
vector<class_type> array(48);
=20
// ...
array[0]=3D ...
=20
=20
=20
=20
=20
Ioannis Vranos

* Programming pages:  <http://www.noicys.freeurl.com/>
http://www.noicys.freeurl.com
* Alternative URL 1:  <http://run.to/noicys> http://run.to/noicys
* Alternative URL 2:  <http://www.noicys.cjb.net/> =
http://www.noicys.cjb.net

=20
=20
=20

-----Original Message-----
From: dev...@li...
[mailto:dev...@li...] On Behalf Of Shane =
Bryldt
Sent: Friday, June 20, 2003 3:39 AM
To: C++
Subject: [Dev-C++] Optimized memory allocation


Hello,
=20
    I have recently been attempting to find ways to speed up the process =
of
loading 48k records from a MySQL SELECT statement, which has a 32 char
primary key.  One of the more intensive processes involved is calling =
"new"
to create a new object 48k times.  In standard C, I would =
malloc(RowCount *
sizeof(class_type)), and using pointer math, simply iterate whenever I
needed a new structure.  With C++, a call to new is required for
construction purposes.  Is there an optimized method wherein I can =
allocate
a dynamic number to be preallocated more efficiently all at once in one
call?
    I have considered using "new class_type[RowCount]()" to allocate an
array (haven't tried to see if that would actually work), but I was
wondering if there was a more efficient means to preallocate objects.
=20
Thanks,
    -Shane

Re: [Dev-C++] Optimized memory allocation

[Shane B. <ast...@te...>]

MessageYour idea suggest a significant performance boost, if I was =
referencing the memory with accessing optimization.  The issue here, is =
accessing, it's allocation performance.
As I believe you meant,
"new class_type[count]"
is equivilent to,
"malloc(count * sizeof(class_type))"
This is the fastest method I can determine for allocation, however the =
issue is that I want to allocate this memory to be used and delete'd not =
as an array, but perhaps individually.  What I mean is, currently the =
memory is allocated with many calls to "new", whereas one call to new, =
allocating all the memory at once would be faster.  However, I cannot =
take a pointer to a specific element of the array and only delete that =
memory when it's no longer needed, I would require a management system =
of the array.

Herein lies the problem, the array must be allocated and deleted =
entirely.  Multiple calls to "new" is required to ensure each memory is =
allocated and internally tracked separately.  I do not want one large =
chunk of memory accessed as an array, but rather simply to preallocate =
the required memory all at once for a specific number of objects.

The actual situation, involves as I said before, loading 48k records =
from a database, and for each, creating an object.  I do not want this =
in an array, each object is later stored in an STL map and indexed =
according to a unique value.  I just want to preallocate, calling =
constructors correctly, 48k objects, and then use one at a time into the =
map, and may later be individually deleted out of the map by expiration =
of individual records.

Your suggestion to use a vector does not imply allocating with new, as =
is needed for each to be allocated correctly and constructed properly.  =
There may not be a better solution than calling new 48k times, =
unfortunately.

I thought perhaps at a lower level you could preallocate a whole chunk =
of memory, then with another special call, construct an object in =
preallocated memory.  This would speed up as I would just iterate =
through the construction of each object after allocating for 48k =
objects, instead of calling new 48k times separately allocating each =
time.

With using any sort of memory manager to manage an array of preallocated =
memory would probably negate the speed increase.

    -Shane
p.s. I am really nitpicking here, the entire process takes about 30 =
seconds on average, running on the same server as the database, on low =
end home hardware, 350mhz, 256megs of ram, and a standard 5400rpm IDE =
drive.  Nothing special, if there are no lower level calls as I =
described above, then I can live with the loading time.
  ----- Original Message -----=20
  From: Ioannis Vranos=20
  To: 'C++'=20
  Sent: Thursday, June 19, 2003 8:47 PM
  Subject: RE: [Dev-C++] Optimized memory allocation


  new whatever[SIZE] is equivalent to malloc(SIZE*sizeof(whatever)), =
that is it should boe slower.

  But better use an std::vector:


  #include <vector>

  // ...

  using std::vector;

  // ...

  vector<class_type> array(48);

  // ...
  array[0]=3D ...





  Ioannis Vranos

  * Programming pages: http://www.noicys.freeurl.com
  * Alternative URL 1: http://run.to/noicys
  * Alternative URL 2: http://www.noicys.cjb.net=20



    -----Original Message-----
    From: dev...@li... =
[mailto:dev...@li...] On Behalf Of Shane =
Bryldt
    Sent: Friday, June 20, 2003 3:39 AM
    To: C++
    Subject: [Dev-C++] Optimized memory allocation


    Hello,

        I have recently been attempting to find ways to speed up the =
process of loading 48k records from a MySQL SELECT statement, which has =
a 32 char primary key.  One of the more intensive processes involved is =
calling "new" to create a new object 48k times.  In standard C, I would =
malloc(RowCount * sizeof(class_type)), and using pointer math, simply =
iterate whenever I needed a new structure.  With C++, a call to new is =
required for construction purposes.  Is there an optimized method =
wherein I can allocate a dynamic number to be preallocated more =
efficiently all at once in one call?
        I have considered using "new class_type[RowCount]()" to allocate =
an array (haven't tried to see if that would actually work), but I was =
wondering if there was a more efficient means to preallocate objects.

    Thanks,
        -Shane

RE: [Dev-C++] Optimized memory allocation

[Ioannis V. <iv...@em...>]

#include <vector>
=20
=20
// ...
=20
using std::vector;
=20
// ...
=20
vector<my_class>array;
=20
=20
=20
// Reserves space for 100 "my_class"es, does not create any objects.
array.reserve(100);
=20
=20
// Creates a my_class at the end copying the value using the assignment
operator.
// At push_back i passed a temporary here.
array.push_back(my_class(7) );
=20
=20
// Equivalent to realloc(). array[0] remains my_class(7)
// and 9 more objects get created with the default constructor.
=20
array.resize(10);
=20
=20
array.erase(array.begin()+6);
=20
=20
// Vector (random access) iterators behave like pointers
vector<my_class>::iterator p=3Darray.begin();
=20
=20
//array.end() points one after the end
=20
array.erase(p+3);
=20
=20
// array.size() returns the number of objects
//array.capacity() the number of reserved object space
=20
=20
=20
It is a long story, you have to read vectors. :) I suggest "The C++
Programming Language" 3rd Edition or Special Edition by Bjarne =
Stroustrup
(the creator of C++). Keep in mind that you must have finished some
introductory book first. Check this:
=20
http://www.research.att.com/~bs/3rd.html
=20
=20
=20
=20
Ioannis Vranos

* Programming pages: http://www.noicys.freeurl.com
<http://www.noicys.freeurl.com/>=20
* Alternative URL 1: http://run.to/noicys
* Alternative URL 2: http://www.noicys.cjb.net =
<http://www.noicys.cjb.net/>


=20
=20
=20
=20

-----Original Message-----
From: Shane Bryldt [mailto:ast...@te...]=20
Sent: Friday, June 20, 2003 11:04 AM
To: Ioannis Vranos
Cc: C++
Subject: Re: [Dev-C++] Optimized memory allocation


Your idea suggest a significant performance boost, if I was referencing =
the
memory with accessing optimization.  The issue here, is accessing, it's
allocation performance.
As I believe you meant,
"new class_type[count]"
is equivilent to,
"malloc(count * sizeof(class_type))"
This is the fastest method I can determine for allocation, however the =
issue
is that I want to allocate this memory to be used and delete'd not as an
array, but perhaps individually.  What I mean is, currently the memory =
is
allocated with many calls to "new", whereas one call to new, allocating =
all
the memory at once would be faster.  However, I cannot take a pointer to =
a
specific element of the array and only delete that memory when it's no
longer needed, I would require a management system of the array.
=20
Herein lies the problem, the array must be allocated and deleted =
entirely.
Multiple calls to "new" is required to ensure each memory is allocated =
and
internally tracked separately.  I do not want one large chunk of memory
accessed as an array, but rather simply to preallocate the required =
memory
all at once for a specific number of objects.
=20
The actual situation, involves as I said before, loading 48k records =
from a
database, and for each, creating an object.  I do not want this in an =
array,
each object is later stored in an STL map and indexed according to a =
unique
value.  I just want to preallocate, calling constructors correctly, 48k
objects, and then use one at a time into the map, and may later be
individually deleted out of the map by expiration of individual records.
=20
Your suggestion to use a vector does not imply allocating with new, as =
is
needed for each to be allocated correctly and constructed properly.  =
There
may not be a better solution than calling new 48k times, unfortunately.
=20
I thought perhaps at a lower level you could preallocate a whole chunk =
of
memory, then with another special call, construct an object in =
preallocated
memory.  This would speed up as I would just iterate through the
construction of each object after allocating for 48k objects, instead of
calling new 48k times separately allocating each time.
=20
With using any sort of memory manager to manage an array of preallocated
memory would probably negate the speed increase.
=20
    -Shane
p.s. I am really nitpicking here, the entire process takes about 30 =
seconds
on average, running on the same server as the database, on low end home
hardware, 350mhz, 256megs of ram, and a standard 5400rpm IDE drive.  =
Nothing
special, if there are no lower level calls as I described above, then I =
can
live with the loading time.

----- Original Message -----=20
From: Ioannis Vranos <mailto:iv...@em...> =20
To: 'C++' <mailto:dev...@li...> =20
Sent: Thursday, June 19, 2003 8:47 PM
Subject: RE: [Dev-C++] Optimized memory allocation


new whatever[SIZE] is equivalent to malloc(SIZE*sizeof(whatever)), that =
is
it should boe slower.
=20
But better use an std::vector:
=20
=20
#include <vector>
=20
// ...
=20
using std::vector;
=20
// ...
=20
vector<class_type> array(48);
=20
// ...
array[0]=3D ...
=20
=20
=20
=20
=20
Ioannis Vranos

* Programming pages:  <http://www.noicys.freeurl.com>
http://www.noicys.freeurl.com
* Alternative URL 1:  <http://run.to/noicys> http://run.to/noicys
* Alternative URL 2:  <http://www.noicys.cjb.net/> =
http://www.noicys.cjb.net

=20
=20
=20

-----Original Message-----
From: dev...@li...
[mailto:dev...@li...] On Behalf Of Shane =
Bryldt
Sent: Friday, June 20, 2003 3:39 AM
To: C++
Subject: [Dev-C++] Optimized memory allocation


Hello,
=20
    I have recently been attempting to find ways to speed up the process =
of
loading 48k records from a MySQL SELECT statement, which has a 32 char
primary key.  One of the more intensive processes involved is calling =
"new"
to create a new object 48k times.  In standard C, I would =
malloc(RowCount *
sizeof(class_type)), and using pointer math, simply iterate whenever I
needed a new structure.  With C++, a call to new is required for
construction purposes.  Is there an optimized method wherein I can =
allocate
a dynamic number to be preallocated more efficiently all at once in one
call?
    I have considered using "new class_type[RowCount]()" to allocate an
array (haven't tried to see if that would actually work), but I was
wondering if there was a more efficient means to preallocate objects.
=20
Thanks,
    -Shane

RE: [Dev-C++] Optimized memory allocation

[Louis G. <lg...@ya...>]

Ioannis,
What's the difference betweem the 3rd edition and the special edition ?
Which one do you recommend for someone who knows the basics of C++?

--- Ioannis Vranos <iv...@em...> wrote:
> #include <vector>
>  
>  
> // ...
>  
> using std::vector;
>  
> // ...
>  
> vector<my_class>array;
>  
>  
>  
> // Reserves space for 100 "my_class"es, does not create any objects.
> array.reserve(100);
>  
>  
> // Creates a my_class at the end copying the value using the assignment
> operator.
> // At push_back i passed a temporary here.
> array.push_back(my_class(7) );
>  
>  
> // Equivalent to realloc(). array[0] remains my_class(7)
> // and 9 more objects get created with the default constructor.
>  
> array.resize(10);
>  
>  
> array.erase(array.begin()+6);
>  
>  
> // Vector (random access) iterators behave like pointers
> vector<my_class>::iterator p=array.begin();
>  
>  
> //array.end() points one after the end
>  
> array.erase(p+3);
>  
>  
> // array.size() returns the number of objects
> //array.capacity() the number of reserved object space
>  
>  
>  
> It is a long story, you have to read vectors. :) I suggest "The C++
> Programming Language" 3rd Edition or Special Edition by Bjarne
> Stroustrup
> (the creator of C++). Keep in mind that you must have finished some
> introductory book first. Check this:
>  
> http://www.research.att.com/~bs/3rd.html
>  
>  
>  
>  
> Ioannis Vranos
> 
> * Programming pages: http://www.noicys.freeurl.com
> <http://www.noicys.freeurl.com/> 
> * Alternative URL 1: http://run.to/noicys
> * Alternative URL 2: http://www.noicys.cjb.net
> <http://www.noicys.cjb.net/>
> 
> 
>  
>  
>  
>  
> 
> -----Original Message-----
> From: Shane Bryldt [mailto:ast...@te...] 
> Sent: Friday, June 20, 2003 11:04 AM
> To: Ioannis Vranos
> Cc: C++
> Subject: Re: [Dev-C++] Optimized memory allocation
> 
> 
> Your idea suggest a significant performance boost, if I was referencing
> the
> memory with accessing optimization.  The issue here, is accessing, it's
> allocation performance.
> As I believe you meant,
> "new class_type[count]"
> is equivilent to,
> "malloc(count * sizeof(class_type))"
> This is the fastest method I can determine for allocation, however the
> issue
> is that I want to allocate this memory to be used and delete'd not as an
> array, but perhaps individually.  What I mean is, currently the memory
> is
> allocated with many calls to "new", whereas one call to new, allocating
> all
> the memory at once would be faster.  However, I cannot take a pointer to
> a
> specific element of the array and only delete that memory when it's no
> longer needed, I would require a management system of the array.
>  
> Herein lies the problem, the array must be allocated and deleted
> entirely.
> Multiple calls to "new" is required to ensure each memory is allocated
> and
> internally tracked separately.  I do not want one large chunk of memory
> accessed as an array, but rather simply to preallocate the required
> memory
> all at once for a specific number of objects.
>  
> The actual situation, involves as I said before, loading 48k records
> from a
> database, and for each, creating an object.  I do not want this in an
> array,
> each object is later stored in an STL map and indexed according to a
> unique
> value.  I just want to preallocate, calling constructors correctly, 48k
> objects, and then use one at a time into the map, and may later be
> individually deleted out of the map by expiration of individual records.
>  
> Your suggestion to use a vector does not imply allocating with new, as
> is
> needed for each to be allocated correctly and constructed properly. 
> There
> may not be a better solution than calling new 48k times, unfortunately.
>  
> I thought perhaps at a lower level you could preallocate a whole chunk
> of
> memory, then with another special call, construct an object in
> preallocated
> memory.  This would speed up as I would just iterate through the
> construction of each object after allocating for 48k objects, instead of
> calling new 48k times separately allocating each time.
>  
> With using any sort of memory manager to manage an array of preallocated
> memory would probably negate the speed increase.
>  
>     -Shane
> p.s. I am really nitpicking here, the entire process takes about 30
> seconds
> on average, running on the same server as the database, on low end home
> hardware, 350mhz, 256megs of ram, and a standard 5400rpm IDE drive. 
> Nothing
> special, if there are no lower level calls as I described above, then I
> can
> live with the loading time.
> 
> ----- Original Message ----- 
> From: Ioannis Vranos <mailto:iv...@em...>  
> To: 'C++' <mailto:dev...@li...>  
> Sent: Thursday, June 19, 2003 8:47 PM
> Subject: RE: [Dev-C++] Optimized memory allocation
> 
> 
> new whatever[SIZE] is equivalent to malloc(SIZE*sizeof(whatever)), that
> is
> it should boe slower.
>  
> But better use an std::vector:
>  
>  
> #include <vector>
>  
> // ...
>  
> using std::vector;
>  
> // ...
>  
> vector<class_type> array(48);
>  
> // ...
> array[0]= ...
>  
>  
>  
>  
>  
> Ioannis Vranos
> 
> * Programming pages:  <http://www.noicys.freeurl.com>
> http://www.noicys.freeurl.com
> * Alternative URL 1:  <http://run.to/noicys> http://run.to/noicys
> * Alternative URL 2:  <http://www.noicys.cjb.net/>
> http://www.noicys.cjb.net
> 
>  
>  
>  
> 
> -----Original Message-----
> From: dev...@li...
> [mailto:dev...@li...] On Behalf Of Shane
> Bryldt
> Sent: Friday, June 20, 2003 3:39 AM
> To: C++
> Subject: [Dev-C++] Optimized memory allocation
> 
> 
> Hello,
>  
>     I have recently been attempting to find ways to speed up the process
> of
> 
=== message truncated ===


__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com

RE: [Dev-C++] Optimized memory allocation

[Ioannis V. <iv...@em...>]

 -----Original Message-----
> From: dev...@li...=20
> [mailto:dev...@li...] On Behalf=20
> Of Louis Godio
> Sent: Saturday, June 21, 2003 3:54 PM
> To: Ioannis Vranos; Dev-C++ Mailing List
> Subject: RE: [Dev-C++] Optimized memory allocation
>=20
>=20
> Ioannis,
> What's the difference betweem the 3rd edition and the special=20
> edition ?
> Which one do you recommend for someone who knows the basics of C++?



For recent printings only the hard cover of the special edition. For =
first
printings of the 3rd edition, special edition has sone new appendices =
and
more than 1000 corrections. I suggest special edition because the hard =
cover
is really good it withstands heavy use (for example I have caught the =
book
on the air while it was falling from the hardcover only, noumerous times =
and
it has not been sustained serious damages. We are talking about a heavy =
book
here, so the hard cover is really very strong and of high quality). If =
you
get the 3rd edition with the simple paper cover, I am sure it will
eventually fall apart if you try to read it cover to cover.





Ioannis Vranos
=20
* Programming pages: http://www.noicys.freeurl.com
* Alternative URL 1: http://run.to/noicys
* Alternative URL 2: http://www.noicys.cjb.net