postgres-xc-general

[Postgres-xc-general] "invalid memory alloc request size ..." error

[Hitoshi H. <hem...@la...>]

Hi,

We encountered "invalid memory alloc request size ..." error of
Postgres-XC, when preparing DBT-1 benchmark, and need
someones help.
(First of all, what does the error message mean?)

------------------
Version of XC: v1.1.beta

[process to cause an error]
- Data generation
cd <DBT-1 dir>/datagen/
./datagen -i 10000 -u 15000 -p <data dir>

- Prepare XC cluster
We are using following HA configuration.
We are not sure this configuration is essential for the error,
but we have never encountered the error without XC slave nodes.

ServerA: GTM (master)
ServerB: GTM (slave)
ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
Coordinator2 (slave), Datanode2 (slave)
ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
Coordinator1 (slave), Datanode1 (slave)

- Data loading
$ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
$ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
$ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"

[the error]
ERROR: invalid memory alloc request size 1073741824
CONTEXT: COPY address, line 33191219:
"33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj

[Other information]
It seems small sized data can be copied:

- DBT-1 tables with size (item 10000 ,user 15000)
address 4.7G ERROR
author 894K OK
cc_xacts 4.4G ERROR
customer 20G ERROR
item 5.0M OK
order_line 11G ERROR
orders 4.1G ERROR
stock 78K OK
--------------------

Best regards,

-hemmi

-- 
Hitoshi HEMMI
NTT Open Source Software Center
hem...@la...
Tel:(03)5860-5115
Fax:(03)5463-5490

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Nikhil S. <ni...@st...>]

Hi Hitoshi-san,

I am working on a similar issue coincidentally. Basically what's happening
is that the COPY command is creating tuples which will get freed up only at
the end when the command will finish, so the RAM memory ends up being
consumed and can lead to the error that you have provided.

See the discussion around RemoteQueryNext on another thread.

Regards,
Nikhils

On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
<hem...@la...>wrote:

> Hi,
>
> We encountered "invalid memory alloc request size ..." error of
> Postgres-XC, when preparing DBT-1 benchmark, and need
> someones help.
> (First of all, what does the error message mean?)
>
> ------------------
> Version of XC: v1.1.beta
>
> [process to cause an error]
> - Data generation
> cd <DBT-1 dir>/datagen/
> ./datagen -i 10000 -u 15000 -p <data dir>
>
> - Prepare XC cluster
> We are using following HA configuration.
> We are not sure this configuration is essential for the error,
> but we have never encountered the error without XC slave nodes.
>
> ServerA: GTM (master)
> ServerB: GTM (slave)
> ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
> Coordinator2 (slave), Datanode2 (slave)
> ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
> Coordinator1 (slave), Datanode1 (slave)
>
> - Data loading
> $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
> $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
>
> [the error]
> ERROR: invalid memory alloc request size 1073741824
> CONTEXT: COPY address, line 33191219:
> "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>
> [Other information]
> It seems small sized data can be copied:
>
> - DBT-1 tables with size (item 10000 ,user 15000)
> address 4.7G ERROR
> author 894K OK
> cc_xacts 4.4G ERROR
> customer 20G ERROR
> item 5.0M OK
> order_line 11G ERROR
> orders 4.1G ERROR
> stock 78K OK
> --------------------
>
> Best regards,
>
> -hemmi
>
> --
> Hitoshi HEMMI
> NTT Open Source Software Center
> hem...@la...
> Tel:(03)5860-5115
> Fax:(03)5463-5490
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Hitoshi H. <hem...@la...>]

Thank you, Nikhil.
We will wait until a patch is available.

Best,
-hemmi


Nikhil Sontakke さんは書きました:
> Hi Hitoshi-san,
>
> I am working on a similar issue coincidentally. Basically what's 
> happening is that the COPY command is creating tuples which will get 
> freed up only at the end when the command will finish, so the RAM 
> memory ends up being consumed and can lead to the error that you have 
> provided.
>
> See the discussion around RemoteQueryNext on another thread.
>
> Regards,
> Nikhils
>
> On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI 
> <hem...@la... <mailto:hem...@la...>> wrote:
>
>     Hi,
>
>     We encountered "invalid memory alloc request size ..." error of
>     Postgres-XC, when preparing DBT-1 benchmark, and need
>     someones help.
>     (First of all, what does the error message mean?)
>
>     ------------------
>     Version of XC: v1.1.beta
>
>     [process to cause an error]
>     - Data generation
>     cd <DBT-1 dir>/datagen/
>     ./datagen -i 10000 -u 15000 -p <data dir>
>
>     - Prepare XC cluster
>     We are using following HA configuration.
>     We are not sure this configuration is essential for the error,
>     but we have never encountered the error without XC slave nodes.
>
>     ServerA: GTM (master)
>     ServerB: GTM (slave)
>     ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>     Coordinator2 (slave), Datanode2 (slave)
>     ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>     Coordinator1 (slave), Datanode1 (slave)
>
>     - Data loading
>     $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>     $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>     $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter
>     '>';"
>
>     [the error]
>     ERROR: invalid memory alloc request size 1073741824
>     CONTEXT: COPY address, line 33191219:
>     "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>
>     [Other information]
>     It seems small sized data can be copied:
>
>     - DBT-1 tables with size (item 10000 ,user 15000)
>     address 4.7G ERROR
>     author 894K OK
>     cc_xacts 4.4G ERROR
>     customer 20G ERROR
>     item 5.0M OK
>     order_line 11G ERROR
>     orders 4.1G ERROR
>     stock 78K OK
>     --------------------
>
>     Best regards,
>
>     -hemmi
>
>     --
>     Hitoshi HEMMI
>     NTT Open Source Software Center
>     hem...@la... <mailto:hem...@la...>
>     Tel:(03)5860-5115
>     Fax:(03)5463-5490
>
>
>     ------------------------------------------------------------------------------
>     Get 100% visibility into Java/.NET code with AppDynamics Lite!
>     It's a free troubleshooting tool designed for production.
>     Get down to code-level detail for bottlenecks, with <2% overhead.
>     Download for free and get started troubleshooting in minutes.
>     http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>
>
> -- 
> StormDB - http://www.stormdb.com
> The Database Cloud 


-- 
Hitoshi HEMMI
NTT Open Source Software Center
hem...@la...
Tel:(03)5860-5115
Fax:(03)5463-5490

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Hitoshi H. <hem...@la...>]

Hi Nikhil,

Thank you for the patch.
I will be off from tomorrow, but I ask our team to test it
and post the result.

Best,
-hemmi


Nikhil Sontakke さんは書きました:
> Hi Hitoshi-san,
>
> PFA, patch which should help to use less memory while running COPY
> commands. This is against REL 1.1. Please do let me know if this helps
> in your case.
>
> Regards,
> Nikhils
>
>
> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
> <hem...@la... <mailto:hem...@la...>> wrote:
>
>     Thank you, Nikhil.
>     We will wait until a patch is available.
>
>     Best,
>     -hemmi
>
>
>     Nikhil Sontakke さんは書きました:
>     > Hi Hitoshi-san,
>     >
>     > I am working on a similar issue coincidentally. Basically what's
>     > happening is that the COPY command is creating tuples which will get
>     > freed up only at the end when the command will finish, so the RAM
>     > memory ends up being consumed and can lead to the error that you
>     have
>     > provided.
>     >
>     > See the discussion around RemoteQueryNext on another thread.
>     >
>     > Regards,
>     > Nikhils
>     >
>     > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>     > <hem...@la...
>     <mailto:hem...@la...>
>     <mailto:hem...@la...
>     <mailto:hem...@la...>>> wrote:
>     >
>     > Hi,
>     >
>     > We encountered "invalid memory alloc request size ..." error of
>     > Postgres-XC, when preparing DBT-1 benchmark, and need
>     > someones help.
>     > (First of all, what does the error message mean?)
>     >
>     > ------------------
>     > Version of XC: v1.1.beta
>     >
>     > [process to cause an error]
>     > - Data generation
>     > cd <DBT-1 dir>/datagen/
>     > ./datagen -i 10000 -u 15000 -p <data dir>
>     >
>     > - Prepare XC cluster
>     > We are using following HA configuration.
>     > We are not sure this configuration is essential for the error,
>     > but we have never encountered the error without XC slave nodes.
>     >
>     > ServerA: GTM (master)
>     > ServerB: GTM (slave)
>     > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>     > Coordinator2 (slave), Datanode2 (slave)
>     > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>     > Coordinator1 (slave), Datanode1 (slave)
>     >
>     > - Data loading
>     > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>     > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>     > $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter
>     > '>';"
>     >
>     > [the error]
>     > ERROR: invalid memory alloc request size 1073741824
>     > CONTEXT: COPY address, line 33191219:
>     > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>     >
>     > [Other information]
>     > It seems small sized data can be copied:
>     >
>     > - DBT-1 tables with size (item 10000 ,user 15000)
>     > address 4.7G ERROR
>     > author 894K OK
>     > cc_xacts 4.4G ERROR
>     > customer 20G ERROR
>     > item 5.0M OK
>     > order_line 11G ERROR
>     > orders 4.1G ERROR
>     > stock 78K OK
>     > --------------------
>     >
>     > Best regards,
>     >
>     > -hemmi
>     >
>     > --
>     > Hitoshi HEMMI
>     > NTT Open Source Software Center
>     > hem...@la... <mailto:hem...@la...>
>     <mailto:hem...@la...
>     <mailto:hem...@la...>>
>     > Tel:(03)5860-5115
>     > Fax:(03)5463-5490
>     >
>     >
>     >
>     ------------------------------------------------------------------------------
>     > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>     > It's a free troubleshooting tool designed for production.
>     > Get down to code-level detail for bottlenecks, with <2% overhead.
>     > Download for free and get started troubleshooting in minutes.
>     >
>     http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>     >
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>>
>     > _______________________________________________
>     > Postgres-xc-general mailing list
>     > Pos...@li...
>     <mailto:Pos...@li...>
>     > <mailto:Pos...@li...
>     <mailto:Pos...@li...>>
>     > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>     >
>     >
>     >
>     >
>     > --
>     > StormDB - http://www.stormdb.com
>     > The Database Cloud
>
>
>     --
>     Hitoshi HEMMI
>     NTT Open Source Software Center
>     hem...@la... <mailto:hem...@la...>
>     Tel:(03)5860-5115
>     Fax:(03)5463-5490
>
>
>     ------------------------------------------------------------------------------
>     Get 100% visibility into Java/.NET code with AppDynamics Lite!
>     It's a free troubleshooting tool designed for production.
>     Get down to code-level detail for bottlenecks, with <2% overhead.
>     Download for free and get started troubleshooting in minutes.
>     http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>     <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>     _______________________________________________
>     Postgres-xc-general mailing list
>     Pos...@li...
>     <mailto:Pos...@li...>
>     https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>
>
>
>
> -- 
> StormDB - http://www.stormdb.com
> The Database Cloud 


-- 
Hitoshi HEMMI
NTT Open Source Software Center
hem...@la...
Tel:(03)5860-5115
Fax:(03)5463-5490

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Eiji U. <uwa...@as...>]

Hi Nikhil,

At the request of Hemmi-san,
By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
However, the results did not change unfortunately.

[results]
$ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
ERROR:  invalid memory alloc request size 1073741824
CONTEXT:  COPY address, line 30305382: "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."

Regards,
uwaizumi


(2013/08/15 18:06), Hitoshi HEMMI wrote:
> Hi Nikhil,
> 
> Thank you for the patch.
> I will be off from tomorrow, but I ask our team to test it
> and post the result.
> 
> Best,
> -hemmi
> 
> 
> Nikhil Sontakke さんは書きました:
>> Hi Hitoshi-san,
>>
>> PFA, patch which should help to use less memory while running COPY
>> commands. This is against REL 1.1. Please do let me know if this helps
>> in your case.
>>
>> Regards,
>> Nikhils
>>
>>
>> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
>> <hem...@la... <mailto:hem...@la...>> wrote:
>>
>>      Thank you, Nikhil.
>>      We will wait until a patch is available.
>>
>>      Best,
>>      -hemmi
>>
>>
>>      Nikhil Sontakke さんは書きました:
>>      > Hi Hitoshi-san,
>>      >
>>      > I am working on a similar issue coincidentally. Basically what's
>>      > happening is that the COPY command is creating tuples which will get
>>      > freed up only at the end when the command will finish, so the RAM
>>      > memory ends up being consumed and can lead to the error that you
>>      have
>>      > provided.
>>      >
>>      > See the discussion around RemoteQueryNext on another thread.
>>      >
>>      > Regards,
>>      > Nikhils
>>      >
>>      > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>>      > <hem...@la...
>>      <mailto:hem...@la...>
>>      <mailto:hem...@la...
>>      <mailto:hem...@la...>>> wrote:
>>      >
>>      > Hi,
>>      >
>>      > We encountered "invalid memory alloc request size ..." error of
>>      > Postgres-XC, when preparing DBT-1 benchmark, and need
>>      > someones help.
>>      > (First of all, what does the error message mean?)
>>      >
>>      > ------------------
>>      > Version of XC: v1.1.beta
>>      >
>>      > [process to cause an error]
>>      > - Data generation
>>      > cd <DBT-1 dir>/datagen/
>>      > ./datagen -i 10000 -u 15000 -p <data dir>
>>      >
>>      > - Prepare XC cluster
>>      > We are using following HA configuration.
>>      > We are not sure this configuration is essential for the error,
>>      > but we have never encountered the error without XC slave nodes.
>>      >
>>      > ServerA: GTM (master)
>>      > ServerB: GTM (slave)
>>      > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>>      > Coordinator2 (slave), Datanode2 (slave)
>>      > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>>      > Coordinator1 (slave), Datanode1 (slave)
>>      >
>>      > - Data loading
>>      > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>>      > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>>      > $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter
>>      > '>';"
>>      >
>>      > [the error]
>>      > ERROR: invalid memory alloc request size 1073741824
>>      > CONTEXT: COPY address, line 33191219:
>>      > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>>      >
>>      > [Other information]
>>      > It seems small sized data can be copied:
>>      >
>>      > - DBT-1 tables with size (item 10000 ,user 15000)
>>      > address 4.7G ERROR
>>      > author 894K OK
>>      > cc_xacts 4.4G ERROR
>>      > customer 20G ERROR
>>      > item 5.0M OK
>>      > order_line 11G ERROR
>>      > orders 4.1G ERROR
>>      > stock 78K OK
>>      > --------------------
>>      >
>>      > Best regards,
>>      >
>>      > -hemmi
>>      >
>>      > --
>>      > Hitoshi HEMMI
>>      > NTT Open Source Software Center
>>      > hem...@la... <mailto:hem...@la...>
>>      <mailto:hem...@la...
>>      <mailto:hem...@la...>>
>>      > Tel:(03)5860-5115
>>      > Fax:(03)5463-5490
>>      >
>>      >
>>      >
>>      ------------------------------------------------------------------------------
>>      > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>      > It's a free troubleshooting tool designed for production.
>>      > Get down to code-level detail for bottlenecks, with <2% overhead.
>>      > Download for free and get started troubleshooting in minutes.
>>      >
>>      http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>      <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>>      >
>>      <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>      <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>>
>>      > _______________________________________________
>>      > Postgres-xc-general mailing list
>>      > Pos...@li...
>>      <mailto:Pos...@li...>
>>      > <mailto:Pos...@li...
>>      <mailto:Pos...@li...>>
>>      > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>      >
>>      >
>>      >
>>      >
>>      > --
>>      > StormDB - http://www.stormdb.com
>>      > The Database Cloud
>>
>>
>>      --
>>      Hitoshi HEMMI
>>      NTT Open Source Software Center
>>      hem...@la... <mailto:hem...@la...>
>>      Tel:(03)5860-5115
>>      Fax:(03)5463-5490
>>
>>
>>      ------------------------------------------------------------------------------
>>      Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>      It's a free troubleshooting tool designed for production.
>>      Get down to code-level detail for bottlenecks, with <2% overhead.
>>      Download for free and get started troubleshooting in minutes.
>>      http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>      <http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk>
>>      _______________________________________________
>>      Postgres-xc-general mailing list
>>      Pos...@li...
>>      <mailto:Pos...@li...>
>>      https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>>
>>
>>
>> -- 
>> StormDB - http://www.stormdb.com
>> The Database Cloud
> 
> 


-- 
-----------------------------------
  上泉 英二
  uwa...@as...
-----------------------------------

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Nikhil S. <ni...@st...>]

Thanks Eiji san,



>
> At the request of Hemmi-san,
> By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
> However, the results did not change unfortunately.
>
> [results]
> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
> ERROR:  invalid memory alloc request size 1073741824
> CONTEXT:  COPY address, line 30305382:
> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
>
>
Yeah, as I slept over it I realized that what I am trying to solve is a
"COPY TO" issue and what you guys must be facing is a "COPY FROM" error. I
will try to look at this after when I can make more progress on the "COPY
TO" issue.

Regards,
Nikhils

> Regards,
> uwaizumi
>
>
> (2013/08/15 18:06), Hitoshi HEMMI wrote:
> > Hi Nikhil,
> >
> > Thank you for the patch.
> > I will be off from tomorrow, but I ask our team to test it
> > and post the result.
> >
> > Best,
> > -hemmi
> >
> >
> > Nikhil Sontakke さんは書きました:
> >> Hi Hitoshi-san,
> >>
> >> PFA, patch which should help to use less memory while running COPY
> >> commands. This is against REL 1.1. Please do let me know if this helps
> >> in your case.
> >>
> >> Regards,
> >> Nikhils
> >>
> >>
> >> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
> >> <hem...@la... <mailto:hem...@la...>>
> wrote:
> >>
> >>      Thank you, Nikhil.
> >>      We will wait until a patch is available.
> >>
> >>      Best,
> >>      -hemmi
> >>
> >>
> >>      Nikhil Sontakke さんは書きました:
> >>      > Hi Hitoshi-san,
> >>      >
> >>      > I am working on a similar issue coincidentally. Basically what's
> >>      > happening is that the COPY command is creating tuples which will
> get
> >>      > freed up only at the end when the command will finish, so the RAM
> >>      > memory ends up being consumed and can lead to the error that you
> >>      have
> >>      > provided.
> >>      >
> >>      > See the discussion around RemoteQueryNext on another thread.
> >>      >
> >>      > Regards,
> >>      > Nikhils
> >>      >
> >>      > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
> >>      > <hem...@la...
> >>      <mailto:hem...@la...>
> >>      <mailto:hem...@la...
> >>      <mailto:hem...@la...>>> wrote:
> >>      >
> >>      > Hi,
> >>      >
> >>      > We encountered "invalid memory alloc request size ..." error of
> >>      > Postgres-XC, when preparing DBT-1 benchmark, and need
> >>      > someones help.
> >>      > (First of all, what does the error message mean?)
> >>      >
> >>      > ------------------
> >>      > Version of XC: v1.1.beta
> >>      >
> >>      > [process to cause an error]
> >>      > - Data generation
> >>      > cd <DBT-1 dir>/datagen/
> >>      > ./datagen -i 10000 -u 15000 -p <data dir>
> >>      >
> >>      > - Prepare XC cluster
> >>      > We are using following HA configuration.
> >>      > We are not sure this configuration is essential for the error,
> >>      > but we have never encountered the error without XC slave nodes.
> >>      >
> >>      > ServerA: GTM (master)
> >>      > ServerB: GTM (slave)
> >>      > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
> >>      > Coordinator2 (slave), Datanode2 (slave)
> >>      > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
> >>      > Coordinator1 (slave), Datanode1 (slave)
> >>      >
> >>      > - Data loading
> >>      > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
> >>      > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
> >>      > $ psql -p 5422 -c "copy address from '/tmp/address.data'
> delimiter
> >>      > '>';"
> >>      >
> >>      > [the error]
> >>      > ERROR: invalid memory alloc request size 1073741824
> >>      > CONTEXT: COPY address, line 33191219:
> >>      > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
> >>      >
> >>      > [Other information]
> >>      > It seems small sized data can be copied:
> >>      >
> >>      > - DBT-1 tables with size (item 10000 ,user 15000)
> >>      > address 4.7G ERROR
> >>      > author 894K OK
> >>      > cc_xacts 4.4G ERROR
> >>      > customer 20G ERROR
> >>      > item 5.0M OK
> >>      > order_line 11G ERROR
> >>      > orders 4.1G ERROR
> >>      > stock 78K OK
> >>      > --------------------
> >>      >
> >>      > Best regards,
> >>      >
> >>      > -hemmi
> >>      >
> >>      > --
> >>      > Hitoshi HEMMI
> >>      > NTT Open Source Software Center
> >>      > hem...@la... <mailto:hem...@la...>
> >>      <mailto:hem...@la...
> >>      <mailto:hem...@la...>>
> >>      > Tel:(03)5860-5115
> >>      > Fax:(03)5463-5490
> >>      >
> >>      >
> >>      >
> >>
>  ------------------------------------------------------------------------------
> >>      > Get 100% visibility into Java/.NET code with AppDynamics Lite!
> >>      > It's a free troubleshooting tool designed for production.
> >>      > Get down to code-level detail for bottlenecks, with <2% overhead.
> >>      > Download for free and get started troubleshooting in minutes.
> >>      >
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>      <
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >
> >>      >
> >>      <
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>      <
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>
> >>      > _______________________________________________
> >>      > Postgres-xc-general mailing list
> >>      > Pos...@li...
> >>      <mailto:Pos...@li...>
> >>      > <mailto:Pos...@li...
> >>      <mailto:Pos...@li...>>
> >>      > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>      >
> >>      >
> >>      >
> >>      >
> >>      > --
> >>      > StormDB - http://www.stormdb.com
> >>      > The Database Cloud
> >>
> >>
> >>      --
> >>      Hitoshi HEMMI
> >>      NTT Open Source Software Center
> >>      hem...@la... <mailto:hem...@la...>
> >>      Tel:(03)5860-5115
> >>      Fax:(03)5463-5490
> >>
> >>
> >>
>  ------------------------------------------------------------------------------
> >>      Get 100% visibility into Java/.NET code with AppDynamics Lite!
> >>      It's a free troubleshooting tool designed for production.
> >>      Get down to code-level detail for bottlenecks, with <2% overhead.
> >>      Download for free and get started troubleshooting in minutes.
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>      <
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >
> >>      _______________________________________________
> >>      Postgres-xc-general mailing list
> >>      Pos...@li...
> >>      <mailto:Pos...@li...>
> >>      https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>
> >>
> >>
> >>
> >> --
> >> StormDB - http://www.stormdb.com
> >> The Database Cloud
> >
> >
>
>
> --
> -----------------------------------
>   上泉 英二
>   uwa...@as...
> -----------------------------------
>
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Nikhil S. <ni...@st...>]

What I think is happening here is that the COPY process is trying to send
data to the datanodes, but they are not consuming it at the same rate.
Hence the send_some call sends only a portion of the data and the rest of
the data has to remain buffered. This builds up eventually to more than
half GB and the subsequent repalloc call to double the buffer fails because
1GB is the max limit. This is evident from the below COPY error message as
well.

What we should do is maybe if the data outbuffer size crosses a pre-defined
limit like 16MB or so, then we should use pgxc_node_flush to ensure that
all of that has reached the datanode. After some point even the socket will
not be handle direct large send_some calls and we will increase the buffer
size unnecessarily.

Regards,
Nikhils


On Fri, Aug 16, 2013 at 8:37 AM, Nikhil Sontakke <ni...@st...>wrote:

> Thanks Eiji san,
>
>
>
>>
>> At the request of Hemmi-san,
>> By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
>> However, the results did not change unfortunately.
>>
>> [results]
>> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
>> ERROR:  invalid memory alloc request size 1073741824
>> CONTEXT:  COPY address, line 30305382:
>> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
>>
>>
> Yeah, as I slept over it I realized that what I am trying to solve is a
> "COPY TO" issue and what you guys must be facing is a "COPY FROM" error. I
> will try to look at this after when I can make more progress on the "COPY
> TO" issue.
>
> Regards,
> Nikhils
>
>> Regards,
>> uwaizumi
>>
>>
>> (2013/08/15 18:06), Hitoshi HEMMI wrote:
>> > Hi Nikhil,
>> >
>> > Thank you for the patch.
>> > I will be off from tomorrow, but I ask our team to test it
>> > and post the result.
>> >
>> > Best,
>> > -hemmi
>> >
>> >
>> > Nikhil Sontakke さんは書きました:
>> >> Hi Hitoshi-san,
>> >>
>> >> PFA, patch which should help to use less memory while running COPY
>> >> commands. This is against REL 1.1. Please do let me know if this helps
>> >> in your case.
>> >>
>> >> Regards,
>> >> Nikhils
>> >>
>> >>
>> >> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
>> >> <hem...@la... <mailto:hem...@la...>>
>> wrote:
>> >>
>> >>      Thank you, Nikhil.
>> >>      We will wait until a patch is available.
>> >>
>> >>      Best,
>> >>      -hemmi
>> >>
>> >>
>> >>      Nikhil Sontakke さんは書きました:
>> >>      > Hi Hitoshi-san,
>> >>      >
>> >>      > I am working on a similar issue coincidentally. Basically what's
>> >>      > happening is that the COPY command is creating tuples which
>> will get
>> >>      > freed up only at the end when the command will finish, so the
>> RAM
>> >>      > memory ends up being consumed and can lead to the error that you
>> >>      have
>> >>      > provided.
>> >>      >
>> >>      > See the discussion around RemoteQueryNext on another thread.
>> >>      >
>> >>      > Regards,
>> >>      > Nikhils
>> >>      >
>> >>      > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>> >>      > <hem...@la...
>> >>      <mailto:hem...@la...>
>> >>      <mailto:hem...@la...
>> >>      <mailto:hem...@la...>>> wrote:
>> >>      >
>> >>      > Hi,
>> >>      >
>> >>      > We encountered "invalid memory alloc request size ..." error of
>> >>      > Postgres-XC, when preparing DBT-1 benchmark, and need
>> >>      > someones help.
>> >>      > (First of all, what does the error message mean?)
>> >>      >
>> >>      > ------------------
>> >>      > Version of XC: v1.1.beta
>> >>      >
>> >>      > [process to cause an error]
>> >>      > - Data generation
>> >>      > cd <DBT-1 dir>/datagen/
>> >>      > ./datagen -i 10000 -u 15000 -p <data dir>
>> >>      >
>> >>      > - Prepare XC cluster
>> >>      > We are using following HA configuration.
>> >>      > We are not sure this configuration is essential for the error,
>> >>      > but we have never encountered the error without XC slave nodes.
>> >>      >
>> >>      > ServerA: GTM (master)
>> >>      > ServerB: GTM (slave)
>> >>      > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>> >>      > Coordinator2 (slave), Datanode2 (slave)
>> >>      > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>> >>      > Coordinator1 (slave), Datanode1 (slave)
>> >>      >
>> >>      > - Data loading
>> >>      > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>> >>      > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>> >>      > $ psql -p 5422 -c "copy address from '/tmp/address.data'
>> delimiter
>> >>      > '>';"
>> >>      >
>> >>      > [the error]
>> >>      > ERROR: invalid memory alloc request size 1073741824
>> >>      > CONTEXT: COPY address, line 33191219:
>> >>      > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>> >>      >
>> >>      > [Other information]
>> >>      > It seems small sized data can be copied:
>> >>      >
>> >>      > - DBT-1 tables with size (item 10000 ,user 15000)
>> >>      > address 4.7G ERROR
>> >>      > author 894K OK
>> >>      > cc_xacts 4.4G ERROR
>> >>      > customer 20G ERROR
>> >>      > item 5.0M OK
>> >>      > order_line 11G ERROR
>> >>      > orders 4.1G ERROR
>> >>      > stock 78K OK
>> >>      > --------------------
>> >>      >
>> >>      > Best regards,
>> >>      >
>> >>      > -hemmi
>> >>      >
>> >>      > --
>> >>      > Hitoshi HEMMI
>> >>      > NTT Open Source Software Center
>> >>      > hem...@la... <mailto:hem...@la...
>> >
>> >>      <mailto:hem...@la...
>> >>      <mailto:hem...@la...>>
>> >>      > Tel:(03)5860-5115
>> >>      > Fax:(03)5463-5490
>> >>      >
>> >>      >
>> >>      >
>> >>
>>  ------------------------------------------------------------------------------
>> >>      > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> >>      > It's a free troubleshooting tool designed for production.
>> >>      > Get down to code-level detail for bottlenecks, with <2%
>> overhead.
>> >>      > Download for free and get started troubleshooting in minutes.
>> >>      >
>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >
>> >>      >
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>
>> >>      > _______________________________________________
>> >>      > Postgres-xc-general mailing list
>> >>      > Pos...@li...
>> >>      <mailto:Pos...@li...>
>> >>      > <mailto:Pos...@li...
>> >>      <mailto:Pos...@li...>>
>> >>      >
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>      >
>> >>      >
>> >>      >
>> >>      >
>> >>      > --
>> >>      > StormDB - http://www.stormdb.com
>> >>      > The Database Cloud
>> >>
>> >>
>> >>      --
>> >>      Hitoshi HEMMI
>> >>      NTT Open Source Software Center
>> >>      hem...@la... <mailto:hem...@la...>
>> >>      Tel:(03)5860-5115
>> >>      Fax:(03)5463-5490
>> >>
>> >>
>> >>
>>  ------------------------------------------------------------------------------
>> >>      Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> >>      It's a free troubleshooting tool designed for production.
>> >>      Get down to code-level detail for bottlenecks, with <2% overhead.
>> >>      Download for free and get started troubleshooting in minutes.
>> >>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >>      <
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> >
>> >>      _______________________________________________
>> >>      Postgres-xc-general mailing list
>> >>      Pos...@li...
>> >>      <mailto:Pos...@li...>
>> >>      https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> StormDB - http://www.stormdb.com
>> >> The Database Cloud
>> >
>> >
>>
>>
>> --
>> -----------------------------------
>>   上泉 英二
>>   uwa...@as...
>> -----------------------------------
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>> It's a free troubleshooting tool designed for production.
>> Get down to code-level detail for bottlenecks, with <2% overhead.
>> Download for free and get started troubleshooting in minutes.
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Postgres-xc-general mailing list
>> Pos...@li...
>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>
>
>
>
> --
> StormDB - http://www.stormdb.com
> The Database Cloud
>



-- 
StormDB - http://www.stormdb.com
The Database Cloud

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Eiji U. <uwa...@as...>]

I think I'll try to output a detailed log of this matter.
I will output a detailed log with the following settings.

Postgres-XC
log_min_messages = debug5

Server
iostat and vmstat

Some time it may take, but I will report the results.

Regards,
uwaizumi

(2013/08/16 14:28), Nikhil Sontakke wrote:
> What I think is happening here is that the COPY process is trying to send
> data to the datanodes, but they are not consuming it at the same rate.
> Hence the send_some call sends only a portion of the data and the rest of
> the data has to remain buffered. This builds up eventually to more than
> half GB and the subsequent repalloc call to double the buffer fails because
> 1GB is the max limit. This is evident from the below COPY error message as
> well.
> 
> What we should do is maybe if the data outbuffer size crosses a pre-defined
> limit like 16MB or so, then we should use pgxc_node_flush to ensure that
> all of that has reached the datanode. After some point even the socket will
> not be handle direct large send_some calls and we will increase the buffer
> size unnecessarily.
> 
> Regards,
> Nikhils
> 
> 
> On Fri, Aug 16, 2013 at 8:37 AM, Nikhil Sontakke <ni...@st...>wrote:
> 
>> Thanks Eiji san,
>>
>>
>>
>>>
>>> At the request of Hemmi-san,
>>> By applying the patch to the Postgres-XC v1.1, I have conducted a DBT-1.
>>> However, the results did not change unfortunately.
>>>
>>> [results]
>>> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter '>';"
>>> ERROR:  invalid memory alloc request size 1073741824
>>> CONTEXT:  COPY address, line 30305382:
>>> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
>>>
>>>
>> Yeah, as I slept over it I realized that what I am trying to solve is a
>> "COPY TO" issue and what you guys must be facing is a "COPY FROM" error. I
>> will try to look at this after when I can make more progress on the "COPY
>> TO" issue.
>>
>> Regards,
>> Nikhils
>>
>>> Regards,
>>> uwaizumi
>>>
>>>
>>> (2013/08/15 18:06), Hitoshi HEMMI wrote:
>>>> Hi Nikhil,
>>>>
>>>> Thank you for the patch.
>>>> I will be off from tomorrow, but I ask our team to test it
>>>> and post the result.
>>>>
>>>> Best,
>>>> -hemmi
>>>>
>>>>
>>>> Nikhil Sontakke さんは書きました:
>>>>> Hi Hitoshi-san,
>>>>>
>>>>> PFA, patch which should help to use less memory while running COPY
>>>>> commands. This is against REL 1.1. Please do let me know if this helps
>>>>> in your case.
>>>>>
>>>>> Regards,
>>>>> Nikhils
>>>>>
>>>>>
>>>>> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
>>>>> <hem...@la... <mailto:hem...@la...>>
>>> wrote:
>>>>>
>>>>>       Thank you, Nikhil.
>>>>>       We will wait until a patch is available.
>>>>>
>>>>>       Best,
>>>>>       -hemmi
>>>>>
>>>>>
>>>>>       Nikhil Sontakke さんは書きました:
>>>>>       > Hi Hitoshi-san,
>>>>>       >
>>>>>       > I am working on a similar issue coincidentally. Basically what's
>>>>>       > happening is that the COPY command is creating tuples which
>>> will get
>>>>>       > freed up only at the end when the command will finish, so the
>>> RAM
>>>>>       > memory ends up being consumed and can lead to the error that you
>>>>>       have
>>>>>       > provided.
>>>>>       >
>>>>>       > See the discussion around RemoteQueryNext on another thread.
>>>>>       >
>>>>>       > Regards,
>>>>>       > Nikhils
>>>>>       >
>>>>>       > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
>>>>>       > <hem...@la...
>>>>>       <mailto:hem...@la...>
>>>>>       <mailto:hem...@la...
>>>>>       <mailto:hem...@la...>>> wrote:
>>>>>       >
>>>>>       > Hi,
>>>>>       >
>>>>>       > We encountered "invalid memory alloc request size ..." error of
>>>>>       > Postgres-XC, when preparing DBT-1 benchmark, and need
>>>>>       > someones help.
>>>>>       > (First of all, what does the error message mean?)
>>>>>       >
>>>>>       > ------------------
>>>>>       > Version of XC: v1.1.beta
>>>>>       >
>>>>>       > [process to cause an error]
>>>>>       > - Data generation
>>>>>       > cd <DBT-1 dir>/datagen/
>>>>>       > ./datagen -i 10000 -u 15000 -p <data dir>
>>>>>       >
>>>>>       > - Prepare XC cluster
>>>>>       > We are using following HA configuration.
>>>>>       > We are not sure this configuration is essential for the error,
>>>>>       > but we have never encountered the error without XC slave nodes.
>>>>>       >
>>>>>       > ServerA: GTM (master)
>>>>>       > ServerB: GTM (slave)
>>>>>       > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1 (master),
>>>>>       > Coordinator2 (slave), Datanode2 (slave)
>>>>>       > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2 (master),
>>>>>       > Coordinator1 (slave), Datanode1 (slave)
>>>>>       >
>>>>>       > - Data loading
>>>>>       > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
>>>>>       > $ psql -a -p 5422 -f <DBT-1 dir>/scripts/pgsql/create_tables.sql
>>>>>       > $ psql -p 5422 -c "copy address from '/tmp/address.data'
>>> delimiter
>>>>>       > '>';"
>>>>>       >
>>>>>       > [the error]
>>>>>       > ERROR: invalid memory alloc request size 1073741824
>>>>>       > CONTEXT: COPY address, line 33191219:
>>>>>       > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
>>>>>       >
>>>>>       > [Other information]
>>>>>       > It seems small sized data can be copied:
>>>>>       >
>>>>>       > - DBT-1 tables with size (item 10000 ,user 15000)
>>>>>       > address 4.7G ERROR
>>>>>       > author 894K OK
>>>>>       > cc_xacts 4.4G ERROR
>>>>>       > customer 20G ERROR
>>>>>       > item 5.0M OK
>>>>>       > order_line 11G ERROR
>>>>>       > orders 4.1G ERROR
>>>>>       > stock 78K OK
>>>>>       > --------------------
>>>>>       >
>>>>>       > Best regards,
>>>>>       >
>>>>>       > -hemmi
>>>>>       >
>>>>>       > --
>>>>>       > Hitoshi HEMMI
>>>>>       > NTT Open Source Software Center
>>>>>       > hem...@la... <mailto:hem...@la...
>>>>
>>>>>       <mailto:hem...@la...
>>>>>       <mailto:hem...@la...>>
>>>>>       > Tel:(03)5860-5115
>>>>>       > Fax:(03)5463-5490
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>
>>>   ------------------------------------------------------------------------------
>>>>>       > Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>>>       > It's a free troubleshooting tool designed for production.
>>>>>       > Get down to code-level detail for bottlenecks, with <2%
>>> overhead.
>>>>>       > Download for free and get started troubleshooting in minutes.
>>>>>       >
>>>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>
>>>>>       >
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>
>>>>>       > _______________________________________________
>>>>>       > Postgres-xc-general mailing list
>>>>>       > Pos...@li...
>>>>>       <mailto:Pos...@li...>
>>>>>       > <mailto:Pos...@li...
>>>>>       <mailto:Pos...@li...>>
>>>>>       >
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>       >
>>>>>       > --
>>>>>       > StormDB - http://www.stormdb.com
>>>>>       > The Database Cloud
>>>>>
>>>>>
>>>>>       --
>>>>>       Hitoshi HEMMI
>>>>>       NTT Open Source Software Center
>>>>>       hem...@la... <mailto:hem...@la...>
>>>>>       Tel:(03)5860-5115
>>>>>       Fax:(03)5463-5490
>>>>>
>>>>>
>>>>>
>>>   ------------------------------------------------------------------------------
>>>>>       Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>>>>       It's a free troubleshooting tool designed for production.
>>>>>       Get down to code-level detail for bottlenecks, with <2% overhead.
>>>>>       Download for free and get started troubleshooting in minutes.
>>>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>>       <
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>>>
>>>>>       _______________________________________________
>>>>>       Postgres-xc-general mailing list
>>>>>       Pos...@li...
>>>>>       <mailto:Pos...@li...>
>>>>>       https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> StormDB - http://www.stormdb.com
>>>>> The Database Cloud
>>>>
>>>>
>>>
>>>
>>> --
>>> -----------------------------------
>>>    上泉 英二
>>>    uwa...@as...
>>> -----------------------------------
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
>>> It's a free troubleshooting tool designed for production.
>>> Get down to code-level detail for bottlenecks, with <2% overhead.
>>> Download for free and get started troubleshooting in minutes.
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Postgres-xc-general mailing list
>>> Pos...@li...
>>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>>
>>
>>
>>
>> --
>> StormDB - http://www.stormdb.com
>> The Database Cloud
>>
> 
> 
> 

Re: [Postgres-xc-general] "invalid memory alloc request size ..." error

[Koichi S. <koi...@gm...>]

Thanks Eiji for the effort.    Looking forward to your outcome.

Regards;
---
Koichi Suzuki


2013/8/21 Eiji UWAIZUMI <uwa...@as...>

> I think I'll try to output a detailed log of this matter.
> I will output a detailed log with the following settings.
>
> Postgres-XC
> log_min_messages = debug5
>
> Server
> iostat and vmstat
>
> Some time it may take, but I will report the results.
>
> Regards,
> uwaizumi
>
> (2013/08/16 14:28), Nikhil Sontakke wrote:
> > What I think is happening here is that the COPY process is trying to send
> > data to the datanodes, but they are not consuming it at the same rate.
> > Hence the send_some call sends only a portion of the data and the rest of
> > the data has to remain buffered. This builds up eventually to more than
> > half GB and the subsequent repalloc call to double the buffer fails
> because
> > 1GB is the max limit. This is evident from the below COPY error message
> as
> > well.
> >
> > What we should do is maybe if the data outbuffer size crosses a
> pre-defined
> > limit like 16MB or so, then we should use pgxc_node_flush to ensure that
> > all of that has reached the datanode. After some point even the socket
> will
> > not be handle direct large send_some calls and we will increase the
> buffer
> > size unnecessarily.
> >
> > Regards,
> > Nikhils
> >
> >
> > On Fri, Aug 16, 2013 at 8:37 AM, Nikhil Sontakke <ni...@st...
> >wrote:
> >
> >> Thanks Eiji san,
> >>
> >>
> >>
> >>>
> >>> At the request of Hemmi-san,
> >>> By applying the patch to the Postgres-XC v1.1, I have conducted a
> DBT-1.
> >>> However, the results did not change unfortunately.
> >>>
> >>> [results]
> >>> $ psql -p 5422 -c "copy address from '/tmp/address.data' delimiter
> '>';"
> >>> ERROR:  invalid memory alloc request size 1073741824
> >>> CONTEXT:  COPY address, line 30305382:
> >>>
> "30305382>TXnEREEwqOLV11zEFGDXV4AXEDEeqLn9JDDwGxHw>TCyFEgXDNVOCihBdmdExEBAAvnXdvEEBDEE>TGEgxF5RdiuAEz..."
> >>>
> >>>
> >> Yeah, as I slept over it I realized that what I am trying to solve is a
> >> "COPY TO" issue and what you guys must be facing is a "COPY FROM"
> error. I
> >> will try to look at this after when I can make more progress on the
> "COPY
> >> TO" issue.
> >>
> >> Regards,
> >> Nikhils
> >>
> >>> Regards,
> >>> uwaizumi
> >>>
> >>>
> >>> (2013/08/15 18:06), Hitoshi HEMMI wrote:
> >>>> Hi Nikhil,
> >>>>
> >>>> Thank you for the patch.
> >>>> I will be off from tomorrow, but I ask our team to test it
> >>>> and post the result.
> >>>>
> >>>> Best,
> >>>> -hemmi
> >>>>
> >>>>
> >>>> Nikhil Sontakke さんは書きました:
> >>>>> Hi Hitoshi-san,
> >>>>>
> >>>>> PFA, patch which should help to use less memory while running COPY
> >>>>> commands. This is against REL 1.1. Please do let me know if this
> helps
> >>>>> in your case.
> >>>>>
> >>>>> Regards,
> >>>>> Nikhils
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 14, 2013 at 3:38 PM, Hitoshi HEMMI
> >>>>> <hem...@la... <mailto:hem...@la...>>
> >>> wrote:
> >>>>>
> >>>>>       Thank you, Nikhil.
> >>>>>       We will wait until a patch is available.
> >>>>>
> >>>>>       Best,
> >>>>>       -hemmi
> >>>>>
> >>>>>
> >>>>>       Nikhil Sontakke さんは書きました:
> >>>>>       > Hi Hitoshi-san,
> >>>>>       >
> >>>>>       > I am working on a similar issue coincidentally. Basically
> what's
> >>>>>       > happening is that the COPY command is creating tuples which
> >>> will get
> >>>>>       > freed up only at the end when the command will finish, so the
> >>> RAM
> >>>>>       > memory ends up being consumed and can lead to the error that
> you
> >>>>>       have
> >>>>>       > provided.
> >>>>>       >
> >>>>>       > See the discussion around RemoteQueryNext on another thread.
> >>>>>       >
> >>>>>       > Regards,
> >>>>>       > Nikhils
> >>>>>       >
> >>>>>       > On Wed, Aug 14, 2013 at 1:56 PM, Hitoshi HEMMI
> >>>>>       > <hem...@la...
> >>>>>       <mailto:hem...@la...>
> >>>>>       <mailto:hem...@la...
> >>>>>       <mailto:hem...@la...>>> wrote:
> >>>>>       >
> >>>>>       > Hi,
> >>>>>       >
> >>>>>       > We encountered "invalid memory alloc request size ..." error
> of
> >>>>>       > Postgres-XC, when preparing DBT-1 benchmark, and need
> >>>>>       > someones help.
> >>>>>       > (First of all, what does the error message mean?)
> >>>>>       >
> >>>>>       > ------------------
> >>>>>       > Version of XC: v1.1.beta
> >>>>>       >
> >>>>>       > [process to cause an error]
> >>>>>       > - Data generation
> >>>>>       > cd <DBT-1 dir>/datagen/
> >>>>>       > ./datagen -i 10000 -u 15000 -p <data dir>
> >>>>>       >
> >>>>>       > - Prepare XC cluster
> >>>>>       > We are using following HA configuration.
> >>>>>       > We are not sure this configuration is essential for the
> error,
> >>>>>       > but we have never encountered the error without XC slave
> nodes.
> >>>>>       >
> >>>>>       > ServerA: GTM (master)
> >>>>>       > ServerB: GTM (slave)
> >>>>>       > ServerC: GTM-Proxy, Coordinator1 (master), Datanode1
> (master),
> >>>>>       > Coordinator2 (slave), Datanode2 (slave)
> >>>>>       > ServerD: GTM-Proxy, Coordinator2 (master), Datanode2
> (master),
> >>>>>       > Coordinator1 (slave), Datanode1 (slave)
> >>>>>       >
> >>>>>       > - Data loading
> >>>>>       > $ createdb -E SQL_ASCII -T template0 -p 5422 DBT1
> >>>>>       > $ psql -a -p 5422 -f <DBT-1
> dir>/scripts/pgsql/create_tables.sql
> >>>>>       > $ psql -p 5422 -c "copy address from '/tmp/address.data'
> >>> delimiter
> >>>>>       > '>';"
> >>>>>       >
> >>>>>       > [the error]
> >>>>>       > ERROR: invalid memory alloc request size 1073741824
> >>>>>       > CONTEXT: COPY address, line 33191219:
> >>>>>       > "33191219>EDpcFnEEAe954E5Tg6XbkiBVGLAcuHBhFAEElEE7>zxPzRj
> >>>>>       >
> >>>>>       > [Other information]
> >>>>>       > It seems small sized data can be copied:
> >>>>>       >
> >>>>>       > - DBT-1 tables with size (item 10000 ,user 15000)
> >>>>>       > address 4.7G ERROR
> >>>>>       > author 894K OK
> >>>>>       > cc_xacts 4.4G ERROR
> >>>>>       > customer 20G ERROR
> >>>>>       > item 5.0M OK
> >>>>>       > order_line 11G ERROR
> >>>>>       > orders 4.1G ERROR
> >>>>>       > stock 78K OK
> >>>>>       > --------------------
> >>>>>       >
> >>>>>       > Best regards,
> >>>>>       >
> >>>>>       > -hemmi
> >>>>>       >
> >>>>>       > --
> >>>>>       > Hitoshi HEMMI
> >>>>>       > NTT Open Source Software Center
> >>>>>       > hem...@la... <mailto:
> hem...@la...
> >>>>
> >>>>>       <mailto:hem...@la...
> >>>>>       <mailto:hem...@la...>>
> >>>>>       > Tel:(03)5860-5115
> >>>>>       > Fax:(03)5463-5490
> >>>>>       >
> >>>>>       >
> >>>>>       >
> >>>>>
> >>>
> ------------------------------------------------------------------------------
> >>>>>       > Get 100% visibility into Java/.NET code with AppDynamics
> Lite!
> >>>>>       > It's a free troubleshooting tool designed for production.
> >>>>>       > Get down to code-level detail for bottlenecks, with <2%
> >>> overhead.
> >>>>>       > Download for free and get started troubleshooting in minutes.
> >>>>>       >
> >>>>>
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>>       <
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>
> >>>>>       >
> >>>>>       <
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>>       <
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>>
> >>>>>       > _______________________________________________
> >>>>>       > Postgres-xc-general mailing list
> >>>>>       > Pos...@li...
> >>>>>       <mailto:Pos...@li...>
> >>>>>       > <mailto:Pos...@li...
> >>>>>       <mailto:Pos...@li...>>
> >>>>>       >
> >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>>>       >
> >>>>>       >
> >>>>>       >
> >>>>>       >
> >>>>>       > --
> >>>>>       > StormDB - http://www.stormdb.com
> >>>>>       > The Database Cloud
> >>>>>
> >>>>>
> >>>>>       --
> >>>>>       Hitoshi HEMMI
> >>>>>       NTT Open Source Software Center
> >>>>>       hem...@la... <mailto:
> hem...@la...>
> >>>>>       Tel:(03)5860-5115
> >>>>>       Fax:(03)5463-5490
> >>>>>
> >>>>>
> >>>>>
> >>>
> ------------------------------------------------------------------------------
> >>>>>       Get 100% visibility into Java/.NET code with AppDynamics Lite!
> >>>>>       It's a free troubleshooting tool designed for production.
> >>>>>       Get down to code-level detail for bottlenecks, with <2%
> overhead.
> >>>>>       Download for free and get started troubleshooting in minutes.
> >>>>>
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>>       <
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>>>
> >>>>>       _______________________________________________
> >>>>>       Postgres-xc-general mailing list
> >>>>>       Pos...@li...
> >>>>>       <mailto:Pos...@li...>
> >>>>>
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> StormDB - http://www.stormdb.com
> >>>>> The Database Cloud
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------
> >>>    上泉 英二
> >>>    uwa...@as...
> >>> -----------------------------------
> >>>
> >>>
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> >>> It's a free troubleshooting tool designed for production.
> >>> Get down to code-level detail for bottlenecks, with <2% overhead.
> >>> Download for free and get started troubleshooting in minutes.
> >>>
> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> >>> _______________________________________________
> >>> Postgres-xc-general mailing list
> >>> Pos...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >>>
> >>
> >>
> >>
> >> --
> >> StormDB - http://www.stormdb.com
> >> The Database Cloud
> >>
> >
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Introducing Performance Central, a new site from SourceForge and
> AppDynamics. Performance Central is your source for news, insights,
> analysis and resources for efficient Application Performance Management.
> Visit us today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>