Re: [OpenMap Users] reading very large shapefiles/dbf files

From: Don Dietrick <dfdietrick_at_gmail.com>
Date: Mon, 4 Oct 2010 15:08:36 -0400

Hi Chad,

Whether the dbf/shape file streams or not depends on how the code is
accessing it. If the file is local, then the unused records will be
jumped and it works as you hope it does. If the files are remote (via
URL), I think the unused records still have to be streamed before
skipping - but they aren't stored in memory, so there is still some
benefit. BinaryFile hides the implementation for you, but that's what
is happening under the covers.

The SpatialIndex class creates bounding boxes for the shapes inside
the shape file - so it doesn't matter what their shape is. It's a
very simple spatial filtering mechanism that allows the code to skip
records that are clearly of no interest. If you need accurate spatial
filtering, I'd still do the bounding box check, and then do a more
detailed check using the com.bbn.openmap.geo package components
(GeoExtent and Intersection).

Hope this helps, let me know if you have questions.

Don

On Mon, Oct 4, 2010 at 2:45 PM, C S <usmsci_at_yahoo.com> wrote:
> Don,
>
>  thanks for the reply. I am glad that you mentioned the SpatialIndex because reading every shapefile index is exactly what i don't want to do. This will also alleviate having to read(i hope) or stream an entire dbf file. Am i correct in assuming that i can only siphon out dbf records that are of interest to me rather than streaming the whole dbf file?
>
> I am going to practice with some of these classes and might have some follow up questions for you, if you dont mind, if there are things that dont make sense. About the SpatialIndex though, you mentioned that i can only return shapefile records that lie within a bounding box. Does this also work for shapes that arent a perfect square/rectangle?(multiple points, irregular shapes), because this is a target area for me to support.
>
> thanks again for your time....
>
> Cws(Chad)
>
> --- On Mon, 10/4/10, Don Dietrick <dfdietrick_at_gmail.com> wrote:
>
>> From: Don Dietrick <dfdietrick_at_gmail.com>
>> Subject: Re: [OpenMap Users] reading very large shapefiles/dbf files
>> To: "C S" <usmsci_at_yahoo.com>
>> Date: Monday, October 4, 2010, 1:13 PM
>> Hi cws,
>>
>> You can do this, using the EsriGraphicFactory and the
>> DbfFile classes.
>>  The EsriGraphicFactory has methods that let you read from
>> a shp file
>> one record at a time.  You'll have to manage the byte
>> offsets, to
>> manage where the factory reads in the file for the next
>> record.  The
>> getEsriGraphics(...) method source code can show you how to
>> do that.
>> If you want to be able to sort through the shp file for
>> only shapes
>> that lie within a bounding box, check out the SpatialIndex
>> class.
>> It's what the ShapeLayer uses to control how the shape file
>> is read.
>> You can create a spatial index from the shape file using
>> that class.
>>
>> The DbfFile class lets you retrieve records from the dbf
>> file - so you
>> can either read that file, look at the attributes and then
>> determine
>> which shapes to get based on those attributes, or you can
>> use the
>> spatial index to do your filtering spatially before
>> bothering to read
>> the dbf file.
>>
>> Hope this helps,
>>
>> Don
>>
>>
>> On Mon, Oct 4, 2010 at 12:24 PM, C S <usmsci_at_yahoo.com>
>> wrote:
>> > Hi all,
>> >
>> >   i have some code currently, that reads in
>> shapefiles/dbf files using the openmap API. I have ran into
>> a bottleneck now reading very large shapefiles. The system
>> that is reading the shapefiles is using web services in java
>> and it doesnt have enough memory to load and stream in
>> multiple files at once and sometimes even one very very
>> large file.
>> >
>> > I am currently using the below code to read in the
>> files:
>> >
>> > ShapeFile shapeFile = null;
>> > DbfInputStream dbfFile = null;
>> > DbfTableModel dbfTable = null;
>> >
>> > .....
>> > .....
>> >
>> > shapeFile = new ShapeFile(shpInfile);
>> > dbfFile = new DbfInputStream(dbfInputStream);
>> > dbfTable = new DbfTableModel(dbfFile);
>> >
>> > is there a better way to read in the shapefiles and/or
>> dbf blocks at a time to preserve memory at run time? My
>> thought is, is that i can read a section in memory, do some
>> calculations, read the next block and write over the
>> previous block of memory with new data and so on.
>> >
>> > Any help please on this matter is appreciated.
>> thanks!
>> >
>> > -cws
>> >
>> >
>> >
>> >
>> > --
>> > [To unsubscribe to this list send an email to "majdart_at_bbn.com"
>> > with the following text in the BODY of the message
>> "unsubscribe openmap-users"]
>> >
>>
>
>
>
>

--
[To unsubscribe to this list send an email to "majdart_at_bbn.com"
with the following text in the BODY of the message "unsubscribe openmap-users"]
Received on Mon Oct 04 2010 - 15:09:32 EDT

This archive was generated by hypermail 2.3.0 : Tue Mar 28 2017 - 23:25:09 EDT