Don:
Thanks for your input.
I believe I have this working. I am attaching three source files: they are
changed versions of
ShpOutputSTream, ShxOutputSTream and DbfOutputSTream. I have called them
each ScalableShp.. Shx etc.
To write to shapefiles in an incremental fashion, the user would supply
two filenames, final name and a tmp name.
S/he would then call the older methods as s/he gets the data. Several
writeGeometry(), writeIndex(), writeModel() will be called.
Finally after all points have been supplied, the user would call
finalzeFile():
ScalableShpOutputStream mMetersShp shp = new
ScalableShpOutputStream("final.shp", "tmp.shp");
ScalableShxOutputStream mMetersShx shx = new .... ;
ScalableDbfOutputStream mMetersDbf = new ....;
while (!done) {
EsriLayer pointsLayer = getMorePoints();
if (pointsLayer == null)
done = true;
EsriGraphicList list = pointsLayer.getEsriGraphicList();
DbfTableModel model = pointsLayer.getModel();
int[][] imageData = shp.writeGeometry( list );
shx.writeIndex( imageData, list.getType(), list.getExtents() );
dbf.writeModel(model);
}
Shp.finalizeFile();
Shx.finalizeFile();
Dbf.finalizeFile(model);
If you want, please review the code and make it available to others in the
next version.
Nav
Don Dietrick <dietrick_at_bbn.com>
Sent by: owner-openmap-users_at_bbn.com
06/12/2007 06:31 PM
To
navaneet.kumar_at_us.elster.com
cc
openmap-users_at_bbn.com
Subject
Re: [OpenMap Users] generating very large shapefiles (scalability)
Hi Nav,
The code below looks like it writes all of the points out in one shot.
You could write out a shape file incrementally, but I think there is
an admin issue you need to take care of. I think all of the files
have the number of records stored in their headers, so that number
would have to be updated after you've written a batch of records to
the file. You've have to check out the writers code to figure out
which bytes represent the record count in the three files. You'd
probably have to modify the writer code a little bit to handle the
incremental record writing, too (the DbfOutputStream,
ShpOutputStream, ShxOutputStream). I don't think you can do it with
the classes as currently written.
Hope this helps,
- Don
On Jun 11, 2007, at 2:12 PM, navaneet.kumar_at_us.elster.com wrote:
>
>
>
> Guys:
>
> I have been using Openmap for a few years and it has worked
> perfectly. Thanks.
>
> I use openmap to create shapefiles. As the project has grown over
> time, these shapefiles
> are getting quite big. The dbf file especially gets big, on the
> order of 50 MB or so.
>
> As I am writing to the shapefiles, I do not want to consume too
> much memory. Since the shapefile
> may contain hundreds of thousands of points or lines, I would
> rather write to the shapefile incremently
> and not all points in one shot.
>
> How do I go about doing that? Does something like this works:
>
> String name = "someLayer";
> shxStr = new FileOutputStream( name + ".shx" );
> shpStr = new FileOutputStream( name + ".shp" );
> dbfStr = new FileOutputStream( name + ".dbf" );
>
> ShxOutputStream shxStream = new ShxOutputStream( shxStr);
> ShpOutputStream shpStream = new ShpOutputStream( shpStr);
> DbfOutputStream dbfStream = new DbfOutputStream(dbfStr);
>
>
> boolean done = false;
> int nameLength = 20;
> while(done == false) {
> Vector nodes = getMoreNodes();// get more points to write
> if (nodes == null) {
> done = true;
> continue;
> }
>
> EsriLayer pointsLayer = new EsriLayer("Points Layer",
> ShapeConstants.SHAPE_TYPE_POINT, 2);
> DbfTableModel pointsModel = pointsLayer.getModel();
> setModelColumn(pointsModel, 0, 0, nameLength, "name",
> DbfTableModel.TYPE_CHARACTER);
> setModelColumn(pointsModel, 1, 0, 6, "type",
> DbfTableModel.TYPE_CHARACTER);
>
> for (Enumeration e= nodes.elements(); e.hasMoreElements(); ) {
> Integer irn = (Integer) e.nextElement();
> ArrayList al = new ArrayList();
> MeterInfo info = getMeterInfo(irn);
> if (info != null) {
> EsriPoint p = new EsriPoint(info.mLat, info.mLon);
> al.add(0, info.mName);
> al.add(1, "orphan");
> pointsLayer.addRecord(p, al);
> }
> }
>
> EsriGraphicList list = layer.getEsriGraphicList();
> DbfTableModel model = layer.getModel();
> int[][] imageData = shpStream.writeGeometry( list );
> shxStream.writeIndex( imageData, list.getType(),
> list.getExtents() );
> dbfStream.writeModel(model);
> }
> shpStr.close();
> shxStr.close();
> dbfStr.close();
>
> Thanks
>
> Nav Kumar
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Don Dietrick, dietrick_at_bbn.com
BBN Technologies, Cambridge, MA
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
--
[To unsubscribe to this list send an email to "majdart_at_bbn.com"
with the following text in the BODY of the message "unsubscribe
openmap-users"]
______________________________________________________________________
This email has been spam and virus checked by Elster IT Services.
--
[To unsubscribe to this list send an email to "majdart_at_bbn.com"
with the following text in the BODY of the message "unsubscribe openmap-users"]
Received on Thu Jun 14 2007 - 13:07:14 EDT