GBIF data heat maps - Heat maps over Google Maps for Flash

Maps like everything else seems to be trendy. And nowadays the sexy thing in mapping is the creation of Heat Maps. The best way to understand what they are is to see them:

You can also take a look at this post from one of my favorite blogs on what is and what is not a heat map.

Well for long time I wanted to give it a try and yesterday I had the time to experiment a bit. The idea was to display GBIF available data as a Heat Map over Google Maps. Here you have an screenshot for Quercus ilex:

And if you want to try for yourself here it is (some usability issue, the search box is on the bottom right corner):

Launch GBIF Heat Maps

So how does it work? It was actually easier than I expected:

1) Get the data: I am using the so called "Density tables" from GBIF. You can access them through GBIF web services API at http://es.mirror.gbif.org/ws/rest/density . For example in a query like this one for Quercus ilex (of course you need to get the taxonconceptkey from a previous request to the services): 

http://es.mirror.gbif.org/ws/rest/density/list?taxonconceptkey=14261031

This works fine but has some problems. The first one is that GBIF goes down almost every evening. Tim can maybe explain why. Thats why I am using the spanish mirror (look at the url) and I recommend you to do the same.

Second problem is the verbosity of the XML schema being used. For downloading the Animalia, well thats the biggest concept you can get probably, the result is 14.1 MB of XML. And thats just to get a list of cellIds (if anybody is interested we can post details about CellIds) with counts on them, exactly an array of 34,871 numbers. Even worst is handling them on a web client like this one, parsing such a huge xml output kills the browser. The GBIF webservices API deserve its own blog post I would say together with Tim.

But what is new is that I have supercow powers on GBIF :D I am working for GBIF right now and have access to a test database. In a testing environment I developed a little server app that publish the same density service but using the AMF protocol. I used AMFPHP for this if anybody is interested. There are two good things about using AMF: The output now is around 150 KB for the same thing and AMF is natively supported by Flash so there is no need to be parsed it goes straight into memory as AS3 Objects.

2) Create a Het Map from the data: Once the data is on the client I make use of a Class from Jordi Boggiano called HeatMap.as that creates Sprites as the result. In my case I decided to create a Spring, think like an Image, of 1 pixel per cellId creating a 360x180 pixel image (cellId is equivalent to a 1 degree box).

3) Overlay the image on Google Maps: When you have the Sprite, or even earlier but thats too many details, what you do is overlaying in Google Maps for Flash using a GroundOverlay object that takes care of the reprojection and adapting it to the map. The GroundOverlay is explained in the doc as a way to overlay images but it accepts actually any Sprite.

Done! (almost)

4) Ok, there are some problems: Yes, it is not perfect, these are the pending issues:

• The GroundOverlay seems to not be reprojecting correctly the Sprite I generate and in the very north and south everything is not correctly overlayed.
• The resolution of the Heat Map is a little bit poor, bu actually represent the quality of the data we have. Some interpolation could be done to make it look nicer.
• The colours of the Heat Map do not fit well with the actual Google Maps layers. When there is small data then you can not see it almost.

I still dont feel confident with the code to release it yet. I hope I can work a little bit more on it so that i can be proud, but if you desperately need it let me know.

Just another notice. Yesterday Universal Mind released a preview of a new product: Spatial Key. I am always impressed with what this people does and follow the blogs from their developers (like this one and this one). They are kind of my RIA and web GIS heroes. The new product they have released actually look very much like what I wanted to do in Biodiversity Atlas for data anlysis. It lets people explore geographically and temporally huge datasets. Tim suggested me to contact them and I will do. Nevertheless it is great to have such a great tool available to get ideas on interaction design. Good job Universal Mind, you really rock.

We want to see your comments!

Update: 

Some people asked for different quality settings on the heat map. I have modified the application so that you get now a set of controls to define different quality and drawing options. By default the app tries to figure out depending on the number of occurrences, but maybe thats not the best, depends on how the data is dsitributed. In a final product I think I would NOT provide this functionality to the user, too much for my taste. You know, less is more.

Update 2:

There is a following post with correctly reprojected data and source here.