Menu
▾
▴
#49 Save bloc by bloc in unique scan file
Hi!
I'm trying to create an e57 file (with only 1 scan in it) using point cloud that would not fit in RAM.
e57 can save multiple scan in one file. My goal is to generate one and only one big scan in a file, with streaming datas.
For example :
I have 3 files, each contains points, but I don't know how much. I create an e57 file and then , open my 3 files one by one, and save the points in the created e57. I found the way to create 3 scans inside the e57, but not how to append points to an unique scan.
Since I have to create an index each time using eWriter.NewData3D(header), I can't modify header afterwards, so I can't change the number of points nor the number of rows. And if I try to add points to the scan, first points are overwrited.
For example, I tried to do something like that (and some other variants, but no one worked :()
Create a writer
Create a header
Fille the header with informations
(nRows = 1000, nColumns = 1, nPoints = 1000, X, Y, Z, Intensity, R, G, B)
//And then
size_t size_3 = 1000;
std::vector<int64_t> idElementValue_w;
std::vector<int64_t> startPointIndex_w;
vector<int64_t> pointCount_w;
std::default_random_engine generator;
std::uniform_real_distribution<float> distribution(0, 1000);
std::default_random_engine generator_color;
std::uniform_int_distribution<int> distribution_coolor(0, 255);
int scanIndex = eWriter.NewData3D(header);
for (int l = 0; l < 3; ++l) {
e57::CompressedVectorWriter dataWriter = eWriter.SetUpData3DPointsData(
scanIndex,
size_3,
xData_w.data(), yData_w.data(), zData_w.data(),
_isInvalidData_w.data(),
intData_w.data(),
NULL,
redData_w.data(), greenData_w.data(), blueData_w.data()
);
int startPoint = 0;
int count = 0;
header.nRows += 1000; // useless since we don't recreate an index for scan after that
for (size_t i = 0; i < size_3; i++)
{
//_isInvalidData[count] = isInvalidData[count];
size_t index = i;
xData_w[count] = distribution(generator);
yData_w[count] = l * 1000;
zData_w[count] = distribution(generator);
intData_w[count] = 0.5;
redData_w[count] = distribution_coolor(generator_color);
greenData_w[count] = distribution_coolor(generator_color);
blueData_w[count] = distribution_coolor(generator_color);
count++;
}
dataWriter.write(count);
idElementValue_w.push_back(l);
startPointIndex_w.push_back(startPoint);
pointCount_w.push_back(size_3);
startPoint += count;
dataWriter.close();
}
eWriter.WriteData3DGroupsData(scanIndex, 0,
&idElementValue_w[0], &startPointIndex_w[0], &pointCount_w[0]);
//In the end, only size_3 points are stored in the file, the size_3 points from the last loop
Is there a way to achieve what I want to do?
I'm not sure if it is understandable, and if not, I can try to rephrase it!
Thanks a lot!
Related
Hi Ralph,
The problem that I see is that NewData3D() needs to be inside the for loop
for (int l = 0; l < 3; ++l) {
e57::Data3D header;
// setup new header for each scan
scanIndex = eWriter.NewData3D(header);
e57::CompressedVectorWriter dataWriter = eWriter.SetUpData3DPointsData(
dataWriter.write(count);
dataWriter.close();
}
The close should update the header for you on the total number of points.
Stan Coleby
E57 Committee member
stan.coleby@gmail.com
801 209-0183
On Thu, Dec 6, 2018 at 7:18 AM Raph Schim kirbx@users.sourceforge.net
wrote:
Related
Bug Reports: #49
Hi Stan,
thanks for the answer!
I think I didn't explained well, sorry for that.
If I understood well the e57 format, within 1 file, I can have one or more
scans
(something like this (schematically) :
Header
Scan 1 :
... // Points for first scan
Scan 2:
... // Points for second scan
Scan n:
...// Points for the nth scan
)
What I would like is to have only 1 scan in my file. So my file look like
this :
Header
Scan :
... //List of alls points.
By creating the header and calling for NewData3D within the for loop,
several scans will be created, no? It won't update the unique scan I want
within my file but everytime create another scan with the points I'm
giving, isn't it?
I'll give it a shot anyways when I can, but I think I already tried this,
and if yes, the result is what I said.
Thanks a lot, again!
Best regards,
Raphaël S.
Le lun. 10 déc. 2018 à 23:16, Stan Coleby stancoleby@users.sourceforge.net
a écrit :
--
SCHIMCHOWITSCH Raphaël
Related
Bug Reports: #49
Hi Ralph,
If you want all points as one scan you need to move the
SetUpData3DPointsData() to the outside of the loop and only close it when
all points have been added. The total points added is adjusted by the
close() after all points have been written out.
The size required by the SetUpData3DPointsData() is just the buffer size
and can be any block size you want to work with. The count on the write()
is the number of points in the buffer and can be anything less than or
equal to the buffer size.
Hope this will help.
Best reqards,
Stan Coleby
On Tue, Dec 11, 2018 at 1:08 AM Raph Schim kirbx@users.sourceforge.net
wrote:
Related
Bug Reports: #49
Hi stan! Once again, thank you, that's it! But I already tried this, and
there is still a problem.
In fact, the problem is the number I put in
header.pointsSize
header.rowMaximum *
and
heaeder.pointGroupingSchemes.groupingByLine.pointCountSize.I don't know
before how many points there will be (I want to open several files, apply
filters on them, and when I'm done with filtering one file, I add points in
the scan, and repeat it, until I finish. So I don't know how many points
there will be), and if, for example, I put 1000 in these 3 values, when I
add points in my scan, I won't have more than 1000 points in my final file.
And I can't change those values afterwards (in the loop) since I have to
call for NewData3D before entering the loop.
Is there a way I can change the values contained in the header after
calling to NewData3D*? that would be the answer to my problem.
I'm sorry my question was maybe poorly expressed at the beggining, but
thank you again for your answer!
Best regards,
Raphaël S.
Le mar. 11 déc. 2018 à 18:45, Stan Coleby stancoleby@users.sourceforge.net
a écrit :
--
SCHIMCHOWITSCH Raphaël
Related
Bug Reports: #49
I found a way, but It's probably an ugly way.
Before everything in the function, I create a scanIndex (using NewData3D) for the new file I want to create. And in numPoints, I put the maximum I can have. (For example, if I have 3 scans with 1000 points each, in the header I put
header.indexBounds.rowMaximum = 3000and same for pointsSize and pointGroupingSchemes.groupingByLine.PointCountSize.Then I create a datawriter for this header.
Then I read my scans 1 by 1, do What i want on them (Filter for me), then write all remaining elements in the dataWriter I previously created.
It works but I see several problem.
First, since I can't modify an header afterwards, my file will have an header mentionning it have 3000 points, and 3000 rows, but it can have only 2400. Here it is not important, but when I have 2billions points, I will have to reserve 2billions points for each vector I will use instead of maybe the 1billion that are actually present in my file.
Second problem I see is that I have touse large enough sizeChunks for my datawriter, since It will crash if I want to write more point that there actually is. But more sizeChunks is big, the more the memory I have to use ... For example :
here, I will be able to write sizechunks point at once maximum. But what if I want to load several scan before writing in the huge unique scan. For example, I take 10 scans, and apply filter on all these point, maybe at the end I will have sizeChunks + 1 points.
Since I first wrote that the dataWriter will accept only sizeChunks points at once, I won't be able to copy the point and it will crash. It doesn't allow me to do what I want with the flexibility I would like to have, because I have to know beforehand how much point there will be, and I can't predict that...
is there no other method to write several scan into 1 unique big scan?