From: Ad S. v. S. <ad....@gm...> - 2013-02-13 08:29:12
|
Hi, Does anyone have experience with csv import or export of let's say 20.000 lines? Piece of cake for the query engine or quite a challenge? The export is about addresses for a mailing. I have a question of a customer who is in the mailing services and many of his customers have problems keeping their address list in shape and do mailings in an organized way. So his question to me is if a wiki could be of help. Thanks! Ad |
From: Leonard W. <leo...@ho...> - 2013-02-13 10:24:52
|
Are you talking about http://www.mediawiki.org/wiki/Extension:External_Data or http://www.mediawiki.org/wiki/Extension:Data_Transfer ? I have imported rather large CSV files with Data Transfer without any troubles (makes for a long job queue, though). When reading CSV tables with External Data, however, I've had trouble with the server timing out. I guess it depends on what your server can handle, but I've typically had to break my CSV's down to chunks of < 5 000 lines. /Leo _______________Leonard Wal...@ho...@leo_wallentin+46 (0) 735 - 933 543 > From: ad....@gm... > Date: Wed, 13 Feb 2013 09:29:02 +0100 > To: Sem...@li... > Subject: [Semediawiki-user] Large csv import/export > > Hi, > > Does anyone have experience with csv import or export of let's say 20.000 lines? Piece of cake for the query engine or quite a challenge? > The export is about addresses for a mailing. > I have a question of a customer who is in the mailing services and many of his customers have problems keeping their address list in shape and do mailings in an organized way. So his question to me is if a wiki could be of help. > > Thanks! Ad > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Semediawiki-user mailing list > Sem...@li... > https://lists.sourceforge.net/lists/listinfo/semediawiki-user |
From: Ad S. v. S. <ad....@gm...> - 2013-02-13 11:06:19
|
My experience is limited to http://www.mediawiki.org/wiki/Extension:Data_Transfer which is very suitable for this use case. Do you know how many lines per job are processed? On 13 februari 2013, at 11:24, Leonard Wallentin <leo...@ho...> wrote: > Are you talking about http://www.mediawiki.org/wiki/Extension:External_Data or http://www.mediawiki.org/wiki/Extension:Data_Transfer ? > > I have imported rather large CSV files with Data Transfer without any troubles (makes for a long job queue, though). When reading CSV tables with External Data, however, I've had trouble with the server timing out. I guess it depends on what your server can handle, but I've typically had to break my CSV's down to chunks of < 5 000 lines. > > /Leo > _______________ > Leonard Wallentin > leo...@ho... > @leo_wallentin > +46 (0) 735 - 933 543 > > > > From: ad....@gm... > > Date: Wed, 13 Feb 2013 09:29:02 +0100 > > To: Sem...@li... > > Subject: [Semediawiki-user] Large csv import/export > > > > Hi, > > > > Does anyone have experience with csv import or export of let's say 20.000 lines? Piece of cake for the query engine or quite a challenge? > > The export is about addresses for a mailing. > > I have a question of a customer who is in the mailing services and many of his customers have problems keeping their address list in shape and do mailings in an organized way. So his question to me is if a wiki could be of help. > > > > Thanks! Ad > > ------------------------------------------------------------------------------ > > Free Next-Gen Firewall Hardware Offer > > Buy your Sophos next-gen firewall before the end March 2013 > > and get the hardware for free! Learn more. > > http://p.sf.net/sfu/sophos-d2d-feb > > _______________________________________________ > > Semediawiki-user mailing list > > Sem...@li... > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user |
From: Yaron K. <ya...@wi...> - 2013-02-13 14:58:29
|
Hi Ad, It's one job per line/page. -Yaron On Wed, Feb 13, 2013 at 6:06 AM, Ad Strack van Schijndel < ad....@gm...> wrote: > My experience is limited to > http://www.mediawiki.org/wiki/Extension:Data_Transfer which is very > suitable for this use case. > > Do you know how many lines per job are processed? > > > On 13 februari 2013, at 11:24, Leonard Wallentin < > leo...@ho...> wrote: > > > Are you talking about > http://www.mediawiki.org/wiki/Extension:External_Data or > http://www.mediawiki.org/wiki/Extension:Data_Transfer ? > > > > I have imported rather large CSV files with Data Transfer without any > troubles (makes for a long job queue, though). When reading CSV tables with > External Data, however, I've had trouble with the server timing out. I > guess it depends on what your server can handle, but I've typically had to > break my CSV's down to chunks of < 5 000 lines. > > > > /Leo > > _______________ > > Leonard Wallentin > > leo...@ho... > > @leo_wallentin > > +46 (0) 735 - 933 543 > > > > > > > From: ad....@gm... > > > Date: Wed, 13 Feb 2013 09:29:02 +0100 > > > To: Sem...@li... > > > Subject: [Semediawiki-user] Large csv import/export > > > > > > Hi, > > > > > > Does anyone have experience with csv import or export of let's say > 20.000 lines? Piece of cake for the query engine or quite a challenge? > > > The export is about addresses for a mailing. > > > I have a question of a customer who is in the mailing services and > many of his customers have problems keeping their address list in shape and > do mailings in an organized way. So his question to me is if a wiki could > be of help. > > > > > > Thanks! Ad > > > > ------------------------------------------------------------------------------ > > > Free Next-Gen Firewall Hardware Offer > > > Buy your Sophos next-gen firewall before the end March 2013 > > > and get the hardware for free! Learn more. > > > http://p.sf.net/sfu/sophos-d2d-feb > > > _______________________________________________ > > > Semediawiki-user mailing list > > > Sem...@li... > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Semediawiki-user mailing list > Sem...@li... > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > -- WikiWorks · MediaWiki Consulting · http://wikiworks.com |
From: Laurent A. <la...@al...> - 2013-02-13 23:11:54
|
I observed the same thing as Leonard. Server time outs forced me to chunk large uploads into 5000 increments. For larger jobs, it is better to look into bot frameworks. I can check and edit/create about 250 pages an hour with a script. After that, there have been presentations floating around about how to go directly to the database for very large jobs. (SNPedia maybe?) - Laurent On Wed, Feb 13, 2013 at 9:58 AM, Yaron Koren <ya...@wi...> wrote: > Hi Ad, > > It's one job per line/page. > > -Yaron > > On Wed, Feb 13, 2013 at 6:06 AM, Ad Strack van Schijndel < > ad....@gm...> wrote: > > > My experience is limited to > > http://www.mediawiki.org/wiki/Extension:Data_Transfer which is very > > suitable for this use case. > > > > Do you know how many lines per job are processed? > > > > > > On 13 februari 2013, at 11:24, Leonard Wallentin < > > leo...@ho...> wrote: > > > > > Are you talking about > > http://www.mediawiki.org/wiki/Extension:External_Data or > > http://www.mediawiki.org/wiki/Extension:Data_Transfer ? > > > > > > I have imported rather large CSV files with Data Transfer without any > > troubles (makes for a long job queue, though). When reading CSV tables > with > > External Data, however, I've had trouble with the server timing out. I > > guess it depends on what your server can handle, but I've typically had > to > > break my CSV's down to chunks of < 5 000 lines. > > > > > > /Leo > > > _______________ > > > Leonard Wallentin > > > leo...@ho... > > > @leo_wallentin > > > +46 (0) 735 - 933 543 > > > > > > > > > > From: ad....@gm... > > > > Date: Wed, 13 Feb 2013 09:29:02 +0100 > > > > To: Sem...@li... > > > > Subject: [Semediawiki-user] Large csv import/export > > > > > > > > Hi, > > > > > > > > Does anyone have experience with csv import or export of let's say > > 20.000 lines? Piece of cake for the query engine or quite a challenge? > > > > The export is about addresses for a mailing. > > > > I have a question of a customer who is in the mailing services and > > many of his customers have problems keeping their address list in shape > and > > do mailings in an organized way. So his question to me is if a wiki could > > be of help. > > > > > > > > Thanks! Ad > > > > > > > ------------------------------------------------------------------------------ > > > > Free Next-Gen Firewall Hardware Offer > > > > Buy your Sophos next-gen firewall before the end March 2013 > > > > and get the hardware for free! Learn more. > > > > http://p.sf.net/sfu/sophos-d2d-feb > > > > _______________________________________________ > > > > Semediawiki-user mailing list > > > > Sem...@li... > > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > > > > > > ------------------------------------------------------------------------------ > > Free Next-Gen Firewall Hardware Offer > > Buy your Sophos next-gen firewall before the end March 2013 > > and get the hardware for free! Learn more. > > http://p.sf.net/sfu/sophos-d2d-feb > > _______________________________________________ > > Semediawiki-user mailing list > > Sem...@li... > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > > > -- > WikiWorks · MediaWiki Consulting · http://wikiworks.com > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Semediawiki-user mailing list > Sem...@li... > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > -- - Laurent Alquier http://www.linfa.net |