What're the chances that we could see something like SpatiaLite replace the shapefile soon?
There are some immediate advantages, I think, but I also suspect the reaction from the general user that's not so geeky is going to be that it's too complicated. SRID? Huh?
What about some of you folks that have been around since PC ARC/INFO? How'd the switch from coverages to shapefiles go down?
dylan
I knew I had read something about this just recently. Took me a minute to find it.
I have an existential map: It has 'you are here' written all over it.
I hope SpatiaLite replaces the shapefile. While I haven't tried SpatiaLite yet, what I've read has me excited about some of the possibilities it could offer. I certainly anticipate it will gain traction in the FOSS community, but I agree that there will be those who are very reluctant or who find it complicated.
It boggles my mind that in 2009 we're still using shapefiles, with their many limitations, as a sort of lingua franca for GIS data. ESRI seems to be in no hurry to open up the file geodatabase as well, although that may become a moot point if SpatiaLite is widely adopted.
The switch from coverages to shapefiles was facilitated by one major event: ESRI published the binary format of the shapefile. Within a couple of years, every major commercial GIS package as well as the emerging open-source ones (JShape and OpenMap spring to mind), supported it. With that kind of support, it became a de facto standard quickly. No more passing around E00 files.
If you were used to the coverage, the shapefile took some getting used to. The coverage stored multiple feature types (point, line, polygon, node, route, region, etc.) for a related data set. It also stored topological information. While that was nice, people seemed to shed those pretty quickly in favor of a simple, open format.
Interestingly, as ESRI has extended some of those features back into the geodatabase, they haven't been huge "gotta haves". Ultimately, I would say the switch was not only smooth but also overwhelming. The fact that the shapefile is still the primary format in use speaks to the fact that it hit some kind of sweet spot that no one (ESRI, open-source, other commercial vendors, etc.) has been able to duplicate with a subsequent format.
I think SpatiaLite is at least mature enough to serve as a replacement to the shapefile and I think it's a format ESRI should really look at.
Bill
Bill DollinsZekiah Technologies, Inc.
It's pretty neat. Just like the name says, lightweight and spatial. But still wholly adequate. I do see it continuing to gain traction in the FOSS community, like as with QGIS 1.1. I'd still be surprised to see the end-user care much for it in the ESRI world without at least the perception that SpatiaLite is as simple for them to use as a shapefile.
I think your point about the shapefile still being used is a pretty important point, Bill. What would it take to make the switch, though? Get ESRI behind it, I suppose? Maybe they'd support it so people would quit asking them to open the file gdb... But ultimately I think it has to be both more powerful/useful/robust than a bunch of files lumped together by name, but just as- if not more--simple to use than a shapefile.
This topic seems to be cropping up quite a bit lately.
I just "installed" spatialite yesterday and it looks like a good portable file with solid sql querying ability but you still have to convert to shapefile to view it. Until venders are supporting it, I don't see it spreading widely. Most people don't know all the in's and out's of shapefiles but when they open their favorite gis client there's a simple way to view/edit it.
I think shapefiles are still hanging on mainly because much of the community that uses them hasn't (yet) really outgrown them. There's also a 'devil I know' factor, but I think the main reason is that shapefiles can still get the job done.
For now. Chances are, though, the shapefile is living on borrowed time. In many ways, the 'database' attached to a given shapefile is really just a spreadsheet. The coverage was an attempt to address the issue, but it really just amounted to a collection of shapefiles and their accompanying spreadsheets. For most people, though, this is good enough. Need more data attached to that point? Just slap another column onto the spreadsheet. It's the result of 2-dimensional thinking, but it usually suffices.
But the demands of the profession are rapidly outgrowing 2-dimensional ideas. We need more dimensions, and shapefiles can't deliver them. The 3D vs. 2.5D situation is a good example. A shape in a shapefile cannot be truly represented in 3 dimensions. It's like a drawing of a cube. It looks like a cube, alright, but it's awfully flat. So when we want to look at an actual cube, we need to look somewhere other than a shapefile.
I'm inclined to think that the eventual shift away from shapefiles will be driven by the profession, not by the vendors. The mountains of data we're getting buried under are only getting larger, and we need smarter, faster and more efficient methods of dealing with them. 'Lightweight' is an excellent start. 'Lightweight and adequate' is almost there.
I guess what I'm saying, Dylan, is that you answered your own question. What would it take to make the switch? Just something "more powerful/useful/robust than a bunch of files lumped together by name, but just as- if not more--simple to use than a shapefile."
My question is: Can Spatialite be that 'something'?
Dylan, you hit the nail on the head with simplicity. Robust is nice but simplicity trumps it. I can point at a shapefile on my disk and it shows up on my map. Users want something close to that. The fact that SpatiaLite can store multiple data sets makes it ever so slightly more complex so that complexity would need to be hidden and it wouldn't be that hard to do. I could make a SQLite database with multiple data sets look for all the world like multiple individual files in ArcCatalog vary easily. The user would need be none the wiser.
The shapefile has been on "borrowed time" for ages now but I think it must be borrowing from the Federal Reserve as it never seems to run out. The demise of the shapefile has been predicted, wished for, and pushed along for years but it's still here. I won't be the one to declare anything to be the thing that kills it.
Incidentally, the coverage was not an attempt to address any issue related to the shapefile as it pre-dated the shapefile by better than a decade. The workstation version of the coverage was actually a decent format given the state of the technology at the time it was created. It stored multiple related feature types and maintained topological relationships among them. Within a workspace (a directory), you had multiple coverages (subdirectories). Those subdirectories contained the spatial data (Arc) while all in the workspace shared an Info database. Info was very much relational, it just didn't use SQL. The main knock on the coverage in the 90s was that it was closed and proprietary. The reaction to that fact shaped the GIS industry. OGC stood up to advocate for open data formats and we know where that led. In reaction to that initial agitation, ESRI published the shapefile format as a means to quell calls for publication of the coverage. And we know where that led.
As for addressing the issue of "mountains of data", SpatiaLite is not the answer to that either. It is a great way to store data locally in a manner similar to how we use shapefiles currently. That said, it is no more appropriate for enterprise data management than is a network file share full of shapefiles. It can, however, be part of the answer to address the needs of mobile or disconnected users as well as a fantastic workspace for analysts doing their day-to-day work. That is one part of an overall organizational approach that has to include workflows, training, automation and the like. The bigger it gets, the more it's going to take management from and organizational perspective.
The fact that QGIS will support SpatiaLite is a great first step but it needs to be followed by uDig, ArcGIS, MapInfo, Manifold, GeoServer....you get my point. When users turn around at every corner and hear SpatiaLite as a data option, they'll start to use it. Just like the shapefile, it boils down to acceptance.
Interesting, thanks for the replies. Bill, you've mentioned both that the end user will need to see SpatiaLite at every corner and that ESRI should take a look at it. Won't the former require the latter? ArcMap is obviously the most popular desktop app and without users seeing SpatiaLite (or whatever new format we're hoping for) in ArcMap they won't be seeing it at every corner, even with all those open source options you mentioned.
What would it take for ESRI to support it? Why would they adopt SpatiaLite (or something similar)? Especially when they're sitting on the file geodatabase, which, my impression has been they must be hoping/expecting to become the next storage unit.
Well, I think ESRI should look at it but I'd be surprised if they do anytime soon. I say that because it is relatively new and, short of critical mass, I'd be surprised if they reallocate resources from a product team at this point for it. However, they did add support for PostgreSQL/PostGIS at 9.3 so anything is possible.
That said, the beauty of ArcGIS is that is can be customized and extended so I'm going to start taking a look at that. There's no real reason to wait for ESRI to support SpatiaLite when extending ArcGIS to support it wouldn't be a particularly difficult thing. Perhaps if some third-party tools stimulate demand, then maybe they will look more closely at it for a future release.
Cool, keep us posted! That could go a long way toward increasing the usage of SpatiaLite and who knows, maybe it will get some attention at ESRI.
OK...the problem I am seeing in my current work environment is that the customer would rather pay for an ESRI solution than have a robust working application for free in a matter of minutes. Nevermind the fact that we spent a couple weeks trying to figure out who the procurement person was so that we could license all of our ESRI software and actually get work done! They simply wanted ESRI because that is what someone told them they needed. Oracle too! Oracle Spatial is good stuff but it's really expensive and honestly, why do you need Oracle 11G and ArcSDE? I dunno...
SpatiaLite is freaking revolutionary! It's a fully functional relational and spatial database and there is no need to install anything or configure anything in order to use it. When was the last time you tried to "configure" an Oracle database? Ughhhh......
MY POINT IS that even though SpatiaLite is an absolutely revolutionary and powerful spatial database, there are certain organizations that may or may not deem it important enough to put it on "the list". I have written about this before and my opinion has not changed one little bit! Free software is grrreat but our stake holders are always going to want a phone number to call when something goes south of cheese! I am probably one of SpatiaLite's biggest fans because I see it as being the right path this industry needs to go down but my struggle is getting others to listen. Honestly, I don't think the shapefile will ever go away and it certainly won't be replaced but we can improve upon data interoperability and dissemination by making it better. What if your ESRI example data could be delivered in a single file as a fully functional relational and spatial database that your GIS app could interact with natively? This would be leaps and bounds over the 5 files that usually accompany a single shapefile! That is dangerous thinking I know but wouldn't it be cool!!!
I encourage anyone who is interested to download SpatiaLite and play with it! I foresee this taking off in the same way KML did! Do you know that KML is NOT as bad-ass as everyone makes it out to be? Limitations galore, right? The bottom line is that it is an open format that a large company adopted and evangelized as their own and then offered a free application that their users could play with. I look forward to seeing what may or may not become of SQLite and it's spatial extension, SpatiaLite.
Adam Estrada - The GIS Forum
Glad to see you weigh in, Adam. I saw the SpatiaLight Google group a while back. I think it certainly represents a direction- if not the direction--we should be pursuing, but how to we get there? Why aren't people listening to you evanglize it? What don't they like about it?
I guess, though, until it's as easy to use as a shapefile in ArcMap it doesn't matter how much better it is at spatial data.
PLEASE SEE THE THE LICENSE FROM THE ORIGINAL SOURCE FOR ALL POSTS PULLED INTO THE COMMUNITY MASHUP BLOG SECTION