numpy-discussion

[Numpy-discussion] reading large files

[<Sve...@ff...>]

I would like to read files >2Gbyte. From earlier posting I believed it
should be possible with python 2.5.

I am getting MemoryError when trying to read a file larger than 2Gb.

I am using python2.5 and numpy1.0rc1.

Any solution to the problem?

=20

SEH

Re: [Numpy-discussion] reading large files

[Bill B. <wb...@gm...>]

Do you also have a 64-bit processor?  Just checking since you didn't mention it.
--bb

On 9/22/06, Sve...@ff... <Sve...@ff...> wrote:
>
>
>
>
> I would like to read files >2Gbyte. From earlier posting I believed it
> should be possible with python 2.5.
>
> I am getting MemoryError when trying to read a file larger than 2Gb.
>
> I am using python2.5 and numpy1.0rc1.
>
> Any solution to the problem?
>
>
>
> SEH
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>

Re: [Numpy-discussion] reading large files

[Mike R. <mik...@al...>]

On 9/22/06, Bill Baxter <wb...@gm...> wrote:
>
> Do you also have a 64-bit processor?  Just checking since you didn't
> mention it.
> --bb
>
> On 9/22/06, Sve...@ff... <Sve...@ff...> wrote:
> > I would like to read files >2Gbyte. From earlier posting I believed it
> > should be possible with python 2.5.
> >
> > I am getting MemoryError when trying to read a file larger than 2Gb.
> >
> > I am using python2.5 and numpy1.0rc1.
>

Also, how much memory do you have? You will have to use memory mapping as
well if the file size is greater than your memory size.

But you should be hopeful. I have been working with > 10 GB files for some
time using beta versions of Python 2.5 and numpy-1.0 on a 64-bit AMD machine
with 4 GB of memory.

Mike

-- 
mik...@al...