Re: [OpenMap Users] generating very large shapefiles (scalability)

From: Don Dietrick <dietrick_at_bbn.com>
Date: Tue, 12 Jun 2007 18:30:18 -0400

Hi Nav,

The code below looks like it writes all of the points out in one shot.

You could write out a shape file incrementally, but I think there is
an admin issue you need to take care of. I think all of the files
have the number of records stored in their headers, so that number
would have to be updated after you've written a batch of records to
the file. You've have to check out the writers code to figure out
which bytes represent the record count in the three files. You'd
probably have to modify the writer code a little bit to handle the
incremental record writing, too (the DbfOutputStream,
ShpOutputStream, ShxOutputStream). I don't think you can do it with
the classes as currently written.

Hope this helps,

- Don




On Jun 11, 2007, at 2:12 PM, navaneet.kumar_at_us.elster.com wrote:

>
>
>
> Guys:
>
> I have been using Openmap for a few years and it has worked
> perfectly. Thanks.
>
> I use openmap to create shapefiles. As the project has grown over
> time, these shapefiles
> are getting quite big. The dbf file especially gets big, on the
> order of 50 MB or so.
>
> As I am writing to the shapefiles, I do not want to consume too
> much memory. Since the shapefile
> may contain hundreds of thousands of points or lines, I would
> rather write to the shapefile incremently
> and not all points in one shot.
>
> How do I go about doing that? Does something like this works:
>
> String name = "someLayer";
> shxStr = new FileOutputStream( name + ".shx" );
> shpStr = new FileOutputStream( name + ".shp" );
> dbfStr = new FileOutputStream( name + ".dbf" );
>
> ShxOutputStream shxStream = new ShxOutputStream( shxStr);
> ShpOutputStream shpStream = new ShpOutputStream( shpStr);
> DbfOutputStream dbfStream = new DbfOutputStream(dbfStr);
>
>
> boolean done = false;
> int nameLength = 20;
> while(done == false) {
> Vector nodes = getMoreNodes();// get more points to write
> if (nodes == null) {
> done = true;
> continue;
> }
>
> EsriLayer pointsLayer = new EsriLayer("Points Layer",
> ShapeConstants.SHAPE_TYPE_POINT, 2);
> DbfTableModel pointsModel = pointsLayer.getModel();
> setModelColumn(pointsModel, 0, 0, nameLength, "name",
> DbfTableModel.TYPE_CHARACTER);
> setModelColumn(pointsModel, 1, 0, 6, "type",
> DbfTableModel.TYPE_CHARACTER);
>
> for (Enumeration e= nodes.elements(); e.hasMoreElements(); ) {
> Integer irn = (Integer) e.nextElement();
> ArrayList al = new ArrayList();
> MeterInfo info = getMeterInfo(irn);
> if (info != null) {
> EsriPoint p = new EsriPoint(info.mLat, info.mLon);
> al.add(0, info.mName);
> al.add(1, "orphan");
> pointsLayer.addRecord(p, al);
> }
> }
>
> EsriGraphicList list = layer.getEsriGraphicList();
> DbfTableModel model = layer.getModel();
> int[][] imageData = shpStream.writeGeometry( list );
> shxStream.writeIndex( imageData, list.getType(),
> list.getExtents() );
> dbfStream.writeModel(model);
> }
> shpStr.close();
> shxStr.close();
> dbfStr.close();
>
> Thanks
>
> Nav Kumar



=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Don Dietrick, dietrick_at_bbn.com
BBN Technologies, Cambridge, MA
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


--
[To unsubscribe to this list send an email to "majdart_at_bbn.com"
with the following text in the BODY of the message "unsubscribe openmap-users"]
Received on Tue Jun 12 2007 - 18:30:39 EDT

This archive was generated by hypermail 2.3.0 : Tue Mar 28 2017 - 23:25:08 EDT