Hi! I am a newbies in Ubuntu and nco .Lately ,I used ncks for downloading Opendap data from hycom website in Ubuntu 14.04 .But my output file could not be larger than 100MB .If it was larger than 100MB ,the result is segmentation fault (core dump),if it was smaller than 100MB I can output file successfully.Here is a simple code I use to determine the output file size:
ncks -O -D 4 http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.6 -d X,2549,3574 -d Y,1505,2400 -d MT,1,1 -v v ~/v2.nc
plus,I tried different methods to correct by unsuccesful :
(1)Ironically, It can be used in windows ,and can get an output file larger than 100MB,but I want to know what went wrong in Ubuntu shell.
(2)I tried ulimit -s to increase stake , but still did not work.
(3)I tried opendap by matlab ,it can download data sucessfully larger than 100MB.
(4)My ulimit -a :
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 512975
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 512000
cpu time (seconds, -t) unlimited
max user processes (-u) 512975
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
PS. my original shell code:
!/bin/sh
let lengthT=100
for (( i=1; i<=lengthT; i=i+1))
do
time1=$(date +%s -d '1900-12-31 ')
time2=$((606024*(39343+i)))
time1=$(($time1+$time2))
time1=$(date +%Y-%m-%d -d "1970-01-01 UTC $time1 seconds");
ncks -v v -d X,2549,3574 -d Y,1505,3000 -d MT,$i,$i http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.6 -O v_$time1.nc
done
end
Last edit: lorttyliu 2014-07-14
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello lorttyliu,
Welcome to NCO.
We hope you have a pleasant stay!
Limits like this indicate problems with OS limits, hardware, firewalls or
DAP/web server limits because NCO has no built-in filesize limits.
Many people regularly transfer much larger files via NCO/DAP.
You can verify whether this is an NCO problem by using nccopy (from Unidata) instead of ncks to try obtaining a file > 100 MB.
It is odd that it works on Windows not Linux.
This suggests the problem may be with the DAP client (your machine) rather than the DAP server (remote machine).
I'm not skilled at diagnosing DAP configuration issues.
I would welcome advice on diagnosis from others reading this thread.
If all else fails maybe you can alter your script to download 100 MB at a time.
Happy to fix any NCO problems that this uncovers.
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi! I am a newbies in Ubuntu and nco .Lately ,I used ncks for downloading Opendap data from hycom website in Ubuntu 14.04 .But my output file could not be larger than 100MB .If it was larger than 100MB ,the result is segmentation fault (core dump),if it was smaller than 100MB I can output file successfully.Here is a simple code I use to determine the output file size:
ncks -O -D 4 http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.6 -d X,2549,3574 -d Y,1505,2400 -d MT,1,1 -v v ~/v2.nc
plus,I tried different methods to correct by unsuccesful :
(1)Ironically, It can be used in windows ,and can get an output file larger than 100MB,but I want to know what went wrong in Ubuntu shell.
(2)I tried ulimit -s to increase stake , but still did not work.
(3)I tried opendap by matlab ,it can download data sucessfully larger than 100MB.
(4)My ulimit -a :
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 512975
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 512000
cpu time (seconds, -t) unlimited
max user processes (-u) 512975
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
PS. my original shell code:
!/bin/sh
let lengthT=100
for (( i=1; i<=lengthT; i=i+1))
do
time1=$(date +%s -d '1900-12-31 ')
time2=$((606024*(39343+i)))
time1=$(($time1+$time2))
time1=$(date +%Y-%m-%d -d "1970-01-01 UTC $time1 seconds");
ncks -v v -d X,2549,3574 -d Y,1505,3000 -d MT,$i,$i http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.6 -O v_$time1.nc
done
end
Last edit: lorttyliu 2014-07-14
Hello lorttyliu,
Welcome to NCO.
We hope you have a pleasant stay!
Limits like this indicate problems with OS limits, hardware, firewalls or
DAP/web server limits because NCO has no built-in filesize limits.
Many people regularly transfer much larger files via NCO/DAP.
You can verify whether this is an NCO problem by using nccopy (from Unidata) instead of ncks to try obtaining a file > 100 MB.
It is odd that it works on Windows not Linux.
This suggests the problem may be with the DAP client (your machine) rather than the DAP server (remote machine).
I'm not skilled at diagnosing DAP configuration issues.
I would welcome advice on diagnosis from others reading this thread.
If all else fails maybe you can alter your script to download 100 MB at a time.
Happy to fix any NCO problems that this uncovers.
cz